2024-03-30T03:30:06.705 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T03:30:06.705 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} --archive /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435 --name rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi --verbose --owner scheduled_rishabh@teuthology -- /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/orig.config.yaml 2024-03-30T03:30:06.765 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T03:30:06.862 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435 branch: wip-rishabh-testing-20240327.051042-reef description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7629435' kernel: client: branch: testing flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-rishabh-testing-20240327.051042-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240327.051042-reef sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 8052 sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f sleep_before_teardown: 0 subset: 497/512 suite: fs suite_branch: wip-rishabh-testing-20240327.051042-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLp9uxysJxnBuYZiO+DnHycmfUebGiDiHI5DiVTfPUaHsatE07i0IWHz4qRqrXOR9S1dbVVWs5V2a9AMGb6ZZwM= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChxT9Qy5LIUldjLYonVCLcq3DDsp8y4977m4nZE51QPNBRejvuoHOOw61rNqkGQUZAnDpZ+ruC3pQ9o0+Wb7eg= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKF3oidbcYiEISL3SZW28ovg1Xn7gOPBmKsLkOW9aGGueB2vGeKAal9lIMX3o+rKbbDxDit/fb3xYx0fe8jFfSw= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_18:05:24 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T03:30:06.862 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa; will attempt to use it 2024-03-30T03:30:06.863 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks 2024-03-30T03:30:06.863 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T03:30:06.864 INFO:teuthology.task.internal:Checking packages... 2024-03-30T03:30:06.886 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '470100a4ccbcf7769052bf09a5413881ccf1b97f' 2024-03-30T03:30:06.886 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T03:30:06.886 INFO:teuthology.packaging:ref: None 2024-03-30T03:30:06.886 INFO:teuthology.packaging:tag: None 2024-03-30T03:30:06.886 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240327.051042-reef 2024-03-30T03:30:06.886 INFO:teuthology.packaging:sha1: 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:30:06.886 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rishabh-testing-20240327.051042-reef 2024-03-30T03:30:07.028 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1018.g470100a4 2024-03-30T03:30:07.030 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T03:30:07.036 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T03:30:07.037 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T03:30:07.085 INFO:teuthology.task.internal:Saving configuration 2024-03-30T03:30:07.107 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T03:30:07.113 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T03:30:07.139 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.845700', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLp9uxysJxnBuYZiO+DnHycmfUebGiDiHI5DiVTfPUaHsatE07i0IWHz4qRqrXOR9S1dbVVWs5V2a9AMGb6ZZwM='} 2024-03-30T03:30:07.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.846856', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChxT9Qy5LIUldjLYonVCLcq3DDsp8y4977m4nZE51QPNBRejvuoHOOw61rNqkGQUZAnDpZ+ruC3pQ9o0+Wb7eg='} 2024-03-30T03:30:07.184 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.844227', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKF3oidbcYiEISL3SZW28ovg1Xn7gOPBmKsLkOW9aGGueB2vGeKAal9lIMX3o+rKbbDxDit/fb3xYx0fe8jFfSw='} 2024-03-30T03:30:07.185 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T03:30:07.191 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-30T03:30:07.191 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-30T03:30:07.191 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-30T03:30:07.191 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T03:30:07.295 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbf3cf32160>, signals=[15]) 2024-03-30T03:30:07.295 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T03:30:07.301 INFO:teuthology.task.internal:Opening connections... 2024-03-30T03:30:07.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T03:30:07.303 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:30:07.384 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-03-30T03:30:07.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:30:07.462 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T03:30:07.462 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:30:07.547 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T03:30:07.554 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-30T03:30:07.575 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-30T03:30:07.575 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-30T03:30:07.636 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T03:30:07.636 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T03:30:07.636 INFO:teuthology.orchestra.run.smithi023.stdout:ID="rhel" 2024-03-30T03:30:07.636 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="fedora" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="8.6" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el8" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T03:30:07.637 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T03:30:07.638 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:30:07.638 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T03:30:07.638 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T03:30:07.638 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T03:30:07.638 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T03:30:07.639 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-30T03:30:07.662 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-03-30T03:30:07.683 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-03-30T03:30:07.683 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-03-30T03:30:07.743 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T03:30:07.743 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:ID="rhel" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="fedora" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="8.6" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el8" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T03:30:07.744 INFO:teuthology.orchestra.run.smithi154.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T03:30:07.745 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T03:30:07.746 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-03-30T03:30:07.770 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-30T03:30:07.792 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-30T03:30:07.792 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-30T03:30:07.853 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Red Hat Enterprise Linux" 2024-03-30T03:30:07.853 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8.6 (Ootpa)" 2024-03-30T03:30:07.853 INFO:teuthology.orchestra.run.smithi193.stdout:ID="rhel" 2024-03-30T03:30:07.853 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="fedora" 2024-03-30T03:30:07.853 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8.6" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.redhat.com/" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-30T03:30:07.854 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:30:07.855 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2024-03-30T03:30:07.855 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2024-03-30T03:30:07.855 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2024-03-30T03:30:07.855 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2024-03-30T03:30:07.855 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-30T03:30:07.878 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T03:30:07.889 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T03:30:07.894 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T03:30:07.894 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-30T03:30:07.897 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-03-30T03:30:07.899 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-30T03:30:07.919 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T03:30:07.925 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T03:30:07.925 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-30T03:30:07.958 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-03-30T03:30:07.960 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-30T03:30:07.996 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T03:30:08.073 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T03:30:08.087 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-30T03:30:08.087 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-30T03:30:08.087 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-30T03:30:08.088 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T03:30:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:30:08.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&ref=testing 2024-03-30T03:30:08.432 DEBUG:teuthology.task.kernel:sha1 for client.0 is 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:30:08.432 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "1df0c7e8a2354c42d738e5454ba83ea7332deac2"... 2024-03-30T03:30:08.432 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-30T03:30:08.453 INFO:teuthology.orchestra.run.smithi023.stdout:4.18.0-513.9.1.el8_9.x86_64 2024-03-30T03:30:08.453 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-513.9.1.el8_9.x86_64 vs 1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:30:08.453 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-30T03:30:08.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum upgrade -y linux-firmware 2024-03-30T03:30:10.054 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2024-03-30T03:30:10.054 INFO:teuthology.orchestra.run.smithi023.stdout:Unable to read consumer identity 2024-03-30T03:30:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:30:10.302 INFO:teuthology.orchestra.run.smithi023.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2024-03-30T03:30:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:30:12.323 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-03-30T03:30:13.876 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-03-30T03:30:17.709 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 433 kB/s | 24 kB 00:00 2024-03-30T03:30:18.571 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-30T03:30:18.572 INFO:teuthology.orchestra.run.smithi023.stdout:Nothing to do. 2024-03-30T03:30:18.572 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-30T03:30:18.695 INFO:teuthology.task.kernel:Downloading kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-30T03:30:18.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=1df0c7e8a2354c42d738e5454ba83ea7332deac2 2024-03-30T03:30:18.841 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ 2024-03-30T03:30:18.841 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/ --input-file=- 2024-03-30T03:30:18.844 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi023.front.sepia.ceph.com to complete... 2024-03-30T03:30:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30 03:30:51 URL:https://1.chacra.ceph.com/r/kernel/testing/1df0c7e8a2354c42d738e5454ba83ea7332deac2/centos/8/flavors/default/x86_64/kernel-6.8.0_g1df0c7e8a235-1.x86_64.rpm [401601944/401601944] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-30T03:30:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:FINISHED --2024-03-30 03:30:51-- 2024-03-30T03:30:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:Total wall clock time: 33s 2024-03-30T03:30:51.512 INFO:teuthology.orchestra.run.smithi023.stderr:Downloaded: 1 files, 383M in 32s (11.8 MB/s) 2024-03-30T03:30:51.516 INFO:teuthology.task.kernel:Installing kernel 1df0c7e8a2354c42d738e5454ba83ea7332deac2 on client.0... 2024-03-30T03:30:51.516 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-30T03:30:53.461 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying... ######################################## 2024-03-30T03:30:54.403 INFO:teuthology.orchestra.run.smithi023.stdout:Preparing... ######################################## 2024-03-30T03:30:54.993 INFO:teuthology.orchestra.run.smithi023.stdout:Updating / installing... 2024-03-30T03:31:38.382 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-6.8.0_g1df0c7e8a235-1 ######################################## 2024-03-30T03:31:39.905 INFO:teuthology.orchestra.run.smithi023.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T03:32:01.826 INFO:teuthology.orchestra.run.smithi023.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-30T03:34:16.813 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-30T03:34:16.814 DEBUG:teuthology.orchestra.run.smithi023:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-30T03:34:16.871 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/System.map 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/config 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-30T03:34:16.872 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-30T03:34:16.873 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/arch/x86/kvm/kvm.ko 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto 2024-03-30T03:34:16.874 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ansi_cprng.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/anubis.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/arc4.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_memcpy.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_pq.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_tx.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/async_xor.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/async_tx/raid6test.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authenc.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/authencesn.ko 2024-03-30T03:34:16.875 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blake2b_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_common.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/blowfish_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/camellia_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast5_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast6_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cast_common.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/ccm.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cmac.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crc32_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_engine.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/crypto_user.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/cts.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/des_generic.ko 2024-03-30T03:34:16.876 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/dh_generic.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/echainiv.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/essiv.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/fcrypt.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/khazad.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/lz4hc.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/md4.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/michael_mic.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcbc.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/pcrypt.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/rmd160.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/seed.ko 2024-03-30T03:34:16.877 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/serpent_generic.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tcrypt.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/tea.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_common.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/twofish_generic.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/vmac.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/wp512.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xcbc.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xor.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/crypto/xxhash_generic.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/acpi_pad.ko 2024-03-30T03:34:16.878 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/custom_method.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/ec_sys.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/platform_profile.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbs.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/sbshc.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/acpi/video.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/acard-ahci.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ahci_platform.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/ata_generic.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/libahci_platform.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_acpi.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ali.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_amd.ko 2024-03-30T03:34:16.879 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_artop.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atiixp.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_atp867x.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd640.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cmd64x.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_cypress.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_efar.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt366.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt37x.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it8213.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_it821x.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_jmicron.ko 2024-03-30T03:34:16.880 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_marvell.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_mpiix.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_netcell.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ninja32.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87410.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_ns87415.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_oldpiix.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_opti.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_optidma.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pcmcia.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_piccolo.ko 2024-03-30T03:34:16.881 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_rdc.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sch.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_serverworks.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sil680.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sis.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_sl82c105.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_triflex.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pata_via.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/pdc_adma.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_inic162x.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_mv.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_nv.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_promise.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_qstor.ko 2024-03-30T03:34:16.882 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sil24.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sis.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_svw.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_sx4.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_uli.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_via.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ata/sata_vsc.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/auxdisplay/ks0108.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-30T03:34:16.883 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/bcma/bcma.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/aoe/aoe.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/brd.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/drbd/drbd.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/floppy.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/loop.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-30T03:34:16.884 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/nbd.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/null_blk/null_blk.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/pktcdvd.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/rbd.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/virtio_blk.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/xen-blkfront.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/block/zram/zram.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hangcheck-timer.ko 2024-03-30T03:34:16.885 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/via-rng.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-30T03:34:16.886 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/lp.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/mwave/mwave.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/ppdev.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tlclk.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-30T03:34:16.887 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/uv_mmtimer.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/char/virtio_console.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp 2024-03-30T03:34:16.888 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/ccp/ccp.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-aes.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/padlock-sha.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca 2024-03-30T03:34:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dca/dca.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/amd64_edac.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/e752x_edac.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/edac_mce_amd.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3000_edac.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i3200_edac.ko 2024-03-30T03:34:16.890 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5100_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i5400_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7300_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i7core_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/i82975x_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/ie31200_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/sb_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/edac/x38_edac.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/edd.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-ich.ko 2024-03-30T03:34:16.891 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-appleir.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-aureal.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-axff.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-dr.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elecom.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-elo.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-emsff.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gaff.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gt683r.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-gyration.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-30T03:34:16.892 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-holtekff.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-hyperv.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-icade.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-keytouch.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-kye.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lcpower.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-led.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-lenovo.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-multitouch.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-ortek.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-penmount.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-petalynx.ko 2024-03-30T03:34:16.893 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-picolcd.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-pl.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-plantronics.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-primax.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-rmi.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-common.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-30T03:34:16.894 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-roccat.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-saitek.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-samsung.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sjoy.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sony.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-speedlink.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-steelseries.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-sunplus.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tivo.ko 2024-03-30T03:34:16.895 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-tmff.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-topseed.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-twinhan.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-uclogic.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-waltop.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-wiimote.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-xinmo.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zpff.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/hid-zydacron.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/uhid.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hid/wacom.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_balloon.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_utils.ko 2024-03-30T03:34:16.896 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hv/hv_vmbus.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/abituguru3.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7414.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ad7418.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adc128d818.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1025.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1026.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1029.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm1031.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adm9240.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ads7828.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7410.ko 2024-03-30T03:34:16.897 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7411.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7462.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7470.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7475.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/adt7x10.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/amc6821.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/applesmc.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asb100.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asc7621.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/atxp1.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/coretemp.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/dme1737.ko 2024-03-30T03:34:16.898 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds1621.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ds620.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc1403.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/emc6w201.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71805f.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f71882fg.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/f75375s.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fam15h_power.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/fschmd.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g760a.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/g762.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl518sm.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/gl520sm.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-30T03:34:16.899 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5500_temp.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/i5k_amb.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmaem.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ibmpex.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina209.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ina2xx.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/it87.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k10temp.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/k8temp.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lineage-pem.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm63.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm73.ko 2024-03-30T03:34:16.900 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm75.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm77.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm78.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm80.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm83.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm85.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm87.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm90.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm92.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm93.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95234.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95241.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/lm95245.ko 2024-03-30T03:34:16.901 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc2945.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4151.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4215.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4222.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4245.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4260.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/ltc4261.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max16065.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1619.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max1668.ko 2024-03-30T03:34:16.902 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max197.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6639.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6650.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/max6697.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/mcp3021.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6683.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775-core.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct6775.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/nct7802.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87360.ko 2024-03-30T03:34:16.906 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pc87427.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pcf8591.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-30T03:34:16.907 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/powr1220.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5627.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch5636.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht15.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sht21.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/shtc1.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/sis5595.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47b397.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m1.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/smsc47m192.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/thmc50.ko 2024-03-30T03:34:16.908 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp102.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp103.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp401.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/tmp421.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via-cputemp.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/via686a.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt1211.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/vt8231.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627ehf.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83627hf.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83781d.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83791d.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83792d.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83793.ko 2024-03-30T03:34:16.909 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83795.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l785ts.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/hwmon/w83l786ng.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-30T03:34:16.910 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-30T03:34:16.911 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-dev.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-mux.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-smbus.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/i2c/i2c-stub.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_core.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-30T03:34:16.912 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp 2024-03-30T03:34:16.913 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-30T03:34:16.914 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/gameport/gameport.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/joydev.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/appletouch.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/bcm5974.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/cyapatp.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/sermouse.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-30T03:34:16.915 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/altera_ps2.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/arc_ps2.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/serio/serio_raw.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/input/sparse-keymap.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-blinkm.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lm3530.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-lp3944.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/leds-ss4200.ko 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger 2024-03-30T03:34:16.916 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/bcache/bcache.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-bio-prison.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache-smq.ko 2024-03-30T03:34:16.917 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-cache.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-crypt.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-delay.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-flakey.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-log-userspace.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-multipath.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-queue-length.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-raid.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-round-robin.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-service-time.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-switch.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-thin-pool.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/dm-verity.ko 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data 2024-03-30T03:34:16.918 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid0.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid1.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid10.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/md/raid456.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptbase.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptctl.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptfc.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptlan.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptsas.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptscsih.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/message/fusion/mptspi.ko 2024-03-30T03:34:16.919 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_ich.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/lpc_sch.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/mfd-core.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/sm501.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/viperboard.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/vx855.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/mfd/wl1273-core.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds9802als.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/apds990x.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/bh1770glc.ko 2024-03-30T03:34:16.920 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/cb710/cb710.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/at24.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/eeprom/max6875.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/enclosure.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/hpilo.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29003.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/isl29020.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei 2024-03-30T03:34:16.921 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-me.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei-txe.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/mei/mei.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_7xx1.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tifm_core.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/tsl2550.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_balloon.ko 2024-03-30T03:34:16.922 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/bonding/bonding.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/dummy.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/eql.ko 2024-03-30T03:34:16.923 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-30T03:34:16.924 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-30T03:34:16.925 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-30T03:34:16.926 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-30T03:34:16.927 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-30T03:34:16.928 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/dnet.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ethoc.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/fealnx.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e 2024-03-30T03:34:16.929 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-30T03:34:16.930 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/jme.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-30T03:34:16.931 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-30T03:34:16.932 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-30T03:34:16.933 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-30T03:34:16.934 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro 2024-03-30T03:34:16.935 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via 2024-03-30T03:34:16.936 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/geneve.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-30T03:34:16.937 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ifb.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvlan.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/macvtap.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/mii.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/net_failover.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/netconsole.ko 2024-03-30T03:34:16.938 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/nlmon.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/amd.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/bcm87xx.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/broadcom.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/cicada.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/davicom.ko 2024-03-30T03:34:16.939 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/dp83640.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/et1011c.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/icplus.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/lxt.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/marvell.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/micrel.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/national.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/phylink.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/qsemi.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/realtek.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/smsc.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/ste10Xp.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/phy/vitesse.ko 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp 2024-03-30T03:34:16.940 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_async.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppoe.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pppox.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/ppp/pptp.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slhc.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/slip/slip.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/sungem_phy.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tap.ko 2024-03-30T03:34:16.941 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_random.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/tun.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/veth.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/virtio_net.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan 2024-03-30T03:34:16.942 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/vxlan/vxlan.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/net/xen-netfront.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ntb/ntb.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme-core.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/nvme/host/nvme.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_cs.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_pc.ko 2024-03-30T03:34:16.943 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/parport/parport_serial.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pci/xen-pcifront.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/i82092.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/pd6729.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel 2024-03-30T03:34:16.944 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/acerhdf.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-30T03:34:16.945 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/hdaps.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/intel_ips.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-30T03:34:16.946 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/platform/x86/wmi.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/clients/pps_parport.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/pps/pps_core.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ptp/ptp_kvm.ko 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc 2024-03-30T03:34:16.947 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-em3027.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-30T03:34:16.948 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-max6900.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-30T03:34:16.949 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/rtc/rtc-x1205.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-9xxx.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-sas.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/3w-xxxx.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/BusLogic.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/a100u2w.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/advansys.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-30T03:34:16.950 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/am53c974.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/atp870u.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bfa/bfa.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ch.ko 2024-03-30T03:34:16.951 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dc395x.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/dmx3191d.ko 2024-03-30T03:34:16.952 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/esp_scsi.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/fnic/fnic.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hpsa.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hptiop.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/hv_storvsc.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/initio.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ipr.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ips.ko 2024-03-30T03:34:16.953 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/isci/isci.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libfc/libfc.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/libsas/libsas.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-30T03:34:16.954 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/mvumi.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/pmcraid.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla1280.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-30T03:34:16.955 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/raid_class.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_debug.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/ses.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/st.ko 2024-03-30T03:34:16.956 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/stex.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/virtio_scsi.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/scsi/wd719x.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ssb/ssb.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_file.ko 2024-03-30T03:34:16.957 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_iblock.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_mod.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_pscsi.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/target_core_user.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-30T03:34:16.958 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-30T03:34:16.959 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_gsm.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/n_hdlc.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/nozomi.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/arc_uart.ko 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm 2024-03-30T03:34:16.960 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_aec.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_cif.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_pci_generic.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/uio/uio_sercos3.ko 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class 2024-03-30T03:34:16.961 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-acm.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usblp.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/class/usbtmc.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/mdc800.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/image/microtek.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/adutux.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/appledisplay.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi26.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/emi62.ko 2024-03-30T03:34:16.962 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ezusb.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/idmouse.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/iowarrior.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/ldusb.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/legousbtower.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usb3503.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usblcd.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/uss720.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/misc/yurex.ko 2024-03-30T03:34:16.963 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/phy/phy-generic.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/aircable.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ark3116.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ch341.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cp210x.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cyberjack.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/empeg.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-30T03:34:16.964 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/io_ti.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipaq.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ipw.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ir-usb.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mct_u232.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7720.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/mos7840.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/navman.ko 2024-03-30T03:34:16.965 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/omninet.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/opticon.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/option.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/oti6858.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/pl2303.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcaux.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/qcserial.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/quatech2.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/safe_serial.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/sierra.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ssu100.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/symbolserial.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-30T03:34:16.966 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_debug.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/visor.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/whiteheat.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/uas.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-30T03:34:16.967 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-karma.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/storage/usb-storage.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-30T03:34:16.968 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_net.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/vhost/vhost_scsi.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/apple_bl.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/lcd.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio 2024-03-30T03:34:16.969 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_balloon.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_mmio.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/virtio/virtio_ring.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2482.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/masters/ds2490.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-30T03:34:16.970 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/w1/wire.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-30T03:34:16.971 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/hpwdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/i6300esb.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ib700wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ibmasr.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/it87_wdt.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/machzwd.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/nv_tco.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-30T03:34:16.972 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/softdog.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/via_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/wdt_pci.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/watchdog/xen_wdt.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-evtchn.ko 2024-03-30T03:34:16.973 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntalloc.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-gntdev.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xen-privcmd.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/9p/9p.ko 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs 2024-03-30T03:34:16.974 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/affs/affs.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/befs/befs.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/binfmt_misc.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/btrfs/btrfs.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cachefiles/cachefiles.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ceph/ceph.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/coda/coda.ko 2024-03-30T03:34:16.975 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/cramfs/cramfs.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/dlm/dlm.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/f2fs/f2fs.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/fat.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/msdos.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fat/vfat.ko 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse 2024-03-30T03:34:16.976 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/cuse.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/fuse/fuse.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/gfs2/gfs2.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfs/hfs.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/hfsplus/hfsplus.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/isofs/isofs.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/jfs/jfs.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/lockd/lockd.ko 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix 2024-03-30T03:34:16.977 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/minix/minix.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/netfs/netfs.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfs.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv3.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs/nfsv4.ko 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common 2024-03-30T03:34:16.978 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/grace.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfs_common/nfs_acl.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nfsd/nfsd.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nilfs2/nilfs2.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-celtic.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-centeuro.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-croatian.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-cyrillic.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-gaelic.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-greek.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-iceland.ko 2024-03-30T03:34:16.979 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-inuit.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-roman.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-romanian.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/mac-turkish.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1250.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1251.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp1255.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp737.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp775.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp850.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp852.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp855.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp857.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp860.ko 2024-03-30T03:34:16.980 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp861.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp862.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp863.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp864.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp865.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp866.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp869.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp874.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp932.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp936.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp949.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_cp950.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_euc-jp.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-1.ko 2024-03-30T03:34:16.981 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-13.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-14.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-15.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-2.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-3.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-4.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-5.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-6.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-7.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_iso8859-9.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-r.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-ru.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_koi8-u.ko 2024-03-30T03:34:16.982 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/nls/nls_utf8.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-30T03:34:16.983 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/overlayfs/overlay.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/pstore/ramoops.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/reiserfs/reiserfs.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/romfs/romfs.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/client/cifs.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_arc4.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/smb/common/cifs_md4.ko 2024-03-30T03:34:16.984 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/squashfs/squashfs.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/sysv/sysv.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/udf/udf.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/ufs/ufs.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/fs/xfs/xfs.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/rcu/rcutorture.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/torture.ko 2024-03-30T03:34:16.985 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/asn1_encoder.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc-itu-t.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crc8.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libarc4.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/crypto/libdes.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/libcrc32c.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lru_cache.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4_compress.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/lz4/lz4hc_compress.ko 2024-03-30T03:34:16.986 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/math/cordic.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/raid6/raid6_pq.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_bm.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_fsm.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/ts_kmp.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/lib/zstd/zstd_compress.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/mm/hwpoison-inject.ko 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net 2024-03-30T03:34:16.987 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/6lowpan.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_dest.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_fragment.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_hop.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_mobility.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_routing.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/6lowpan/nhc_udp.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/garp.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/mrp.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/p8022.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/psnap.ko 2024-03-30T03:34:16.988 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/802/stp.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/8021q/8021q.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_fd.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_rdma.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/9p/9pnet_virtio.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/batman-adv/batman-adv.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/br_netfilter.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/bridge.ko 2024-03-30T03:34:16.989 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-30T03:34:16.990 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/ebtables.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph 2024-03-30T03:34:16.991 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ceph/libceph.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/failover.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/core/pktgen.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_diag.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv4.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dccp/dccp_ipv6.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dns_resolver/dns_resolver.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/dsa_core.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_brcm.ko 2024-03-30T03:34:16.992 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_dsa.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_none.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/dsa/tag_trailer.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ah4.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/esp4.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/fou.ko 2024-03-30T03:34:16.993 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/gre.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/inet_diag.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_gre.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_tunnel.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ip_vti.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipcomp.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/ipip.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-30T03:34:16.994 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-30T03:34:16.995 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_bic.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_dctcp.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_diag.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_highspeed.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_htcp.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_hybla.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_illinois.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_lp.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_scalable.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_vegas.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_veno.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_westwood.ko 2024-03-30T03:34:16.996 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tcp_yeah.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/tunnel4.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_diag.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/udp_tunnel.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ah6.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/esp6.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/fou6.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_tunnel.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ip6_vti.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/ipcomp6.ko 2024-03-30T03:34:16.997 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-30T03:34:16.998 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/sit.ko 2024-03-30T03:34:16.999 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/tunnel6.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/key/af_key.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_core.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_eth.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ip6.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_netlink.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/l2tp/l2tp_ppp.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/llc/llc.ko 2024-03-30T03:34:17.000 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mac802154/mac802154.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/mpls/mpls_gso.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-30T03:34:17.001 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-30T03:34:17.002 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conncount.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack.ko 2024-03-30T03:34:17.003 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-30T03:34:17.004 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_log_syslog.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_irc.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_sip.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-30T03:34:17.005 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nf_tables.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_log.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_chain_nat.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_compat.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_ct.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fib.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-30T03:34:17.006 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_hash.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_limit.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_log.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_masq.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_nat.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_numgen.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_queue.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_quota.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_redir.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_reject_inet.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_socket.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/nft_tunnel.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_AUDIT.ko 2024-03-30T03:34:17.007 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_CT.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_DSCP.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HL.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_HMARK.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LED.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_LOG.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NETMAP.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFLOG.ko 2024-03-30T03:34:17.008 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_RATEEST.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_SECMARK.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TEE.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TPROXY.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_TRACE.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_addrtype.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_bpf.ko 2024-03-30T03:34:17.009 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cgroup.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cluster.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_comment.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connbytes.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlabel.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connlimit.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_connmark.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_conntrack.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_cpu.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dccp.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_devgroup.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_dscp.ko 2024-03-30T03:34:17.010 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ecn.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_esp.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hashlimit.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_helper.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_hl.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipcomp.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_iprange.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_ipvs.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_l2tp.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_length.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_limit.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mac.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_mark.ko 2024-03-30T03:34:17.011 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_multiport.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nat.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_nfacct.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_osf.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_owner.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_physdev.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_pkttype.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_policy.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_quota.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_rateest.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_realm.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_recent.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_sctp.ko 2024-03-30T03:34:17.012 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_set.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_socket.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_state.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_statistic.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_string.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_tcpmss.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_time.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netfilter/xt_u32.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/netlink/netlink_diag.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/nsh/nsh.ko 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch 2024-03-30T03:34:17.013 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/openvswitch.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-geneve.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-gre.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/openvswitch/vport-vxlan.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/packet/af_packet_diag.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_rdma.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/rds/rds_tcp.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_csum.ko 2024-03-30T03:34:17.014 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_gact.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_mirred.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_nat.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_pedit.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_police.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_simple.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_skbedit.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/act_vlan.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_basic.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_bpf.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_flow.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_fw.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_route.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/cls_u32.ko 2024-03-30T03:34:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_cmp.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_ipset.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_meta.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_nbyte.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_text.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/em_u32.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_choke.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_codel.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_drr.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_fq.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_gred.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hfsc.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_hhf.ko 2024-03-30T03:34:17.016 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_htb.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_ingress.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_mqprio_lib.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_multiq.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_netem.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_pie.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_plug.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_prio.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_qfq.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_red.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfb.ko 2024-03-30T03:34:17.017 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_sfq.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_tbf.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sched/sch_teql.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sctp/sctp_diag.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/sunrpc.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-30T03:34:17.018 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/diag.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/tipc/tipc.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/unix/unix_diag.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys 2024-03-30T03:34:17.019 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/security/keys/trusted-keys/trusted.ko 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/kernel/virt/lib/irqbypass.ko 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.alias.bin 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.bin 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.builtin.modinfo 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.dep.bin 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.devname 2024-03-30T03:34:17.020 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.order 2024-03-30T03:34:17.021 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.softdep 2024-03-30T03:34:17.021 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols 2024-03-30T03:34:17.021 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/modules.symbols.bin 2024-03-30T03:34:17.021 INFO:teuthology.orchestra.run.smithi023.stdout:/lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T03:34:17.025 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g1df0c7e8a235 2024-03-30T03:34:17.025 DEBUG:teuthology.orchestra.run.smithi023:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout:postinstall scriptlet (using /bin/sh): 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout: /usr/bin/kernel-install add 6.8.0-g1df0c7e8a235 /lib/modules/6.8.0-g1df0c7e8a235/vmlinuz 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout:for file in vmlinuz System.map config; do 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout: if ! cmp --silent "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235"; then 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout: cp "/lib/modules/6.8.0-g1df0c7e8a235/${file}" "/boot/${file}-6.8.0-g1df0c7e8a235" 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout: fi 2024-03-30T03:34:17.050 INFO:teuthology.orchestra.run.smithi023.stdout:done 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:new-kernel-pkg --remove 6.8.0-g1df0c7e8a235 --rminitrd --initrdfile=/boot/initramfs-6.8.0-g1df0c7e8a235.img 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-install remove 6.8.0-g1df0c7e8a235 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/update-bootloader --remove 6.8.0-g1df0c7e8a235 2024-03-30T03:34:17.051 INFO:teuthology.orchestra.run.smithi023.stdout:fi 2024-03-30T03:34:17.052 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-30T03:34:17.127 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-30T03:34:17.127 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-30T03:34:17.127 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.02 2024-03-30T03:34:17.127 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 123.el8 2024-03-30T03:34:17.127 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Thu 14 Dec 2023 12:39:42 AM UTC 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Group : System Environment/Base 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 9265450 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Thu 31 Mar 2022 07:24:35 PM UTC, Key ID 199e2f91fd431d51 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.02-123.el8.src.rpm 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Mon 28 Mar 2022 02:12:09 PM UTC 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-vm-09.build.eng.bos.redhat.com 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Relocations : (not relocatable) 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : Red Hat, Inc. 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : Red Hat, Inc. 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-30T03:34:17.128 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:34:17.129 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-30T03:34:17.130 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-30T03:34:17.130 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 6.8.0-g1df0c7e8a235 2024-03-30T03:34:17.130 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-30T03:34:17.915 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-30T03:34:20.271 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-30T03:34:20.273 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-30T03:34:20.330 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.z7cmFuB2cj 2024-03-30T03:34:20.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.z7cmFuB2cj 2024-03-30T03:34:20.419 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.z7cmFuB2cj 2024-03-30T03:34:20.535 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.z7cmFuB2cj is 6KB 2024-03-30T03:34:20.585 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.z7cmFuB2cj 2024-03-30T03:34:20.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-30T03:34:20.667 INFO:teuthology.orchestra.run.smithi023.stdout:2db68f91c8f90721a6596efe657a5324-0-rescue.conf 2024-03-30T03:34:20.667 INFO:teuthology.orchestra.run.smithi023.stdout:2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235.conf 2024-03-30T03:34:20.667 INFO:teuthology.orchestra.run.smithi023.stdout:963f8b6c99184de696879b3f7da9ecce-0-rescue.conf 2024-03-30T03:34:20.667 INFO:teuthology.orchestra.run.smithi023.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-372.9.1.el8.x86_64.conf 2024-03-30T03:34:20.667 INFO:teuthology.orchestra.run.smithi023.stdout:963f8b6c99184de696879b3f7da9ecce-4.18.0-513.9.1.el8_9.x86_64.conf 2024-03-30T03:34:20.668 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 2db68f91c8f90721a6596efe657a5324-6.8.0-g1df0c7e8a235 2024-03-30T03:34:20.975 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-30T03:34:50.978 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-30T03:34:50.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:34:50.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:35:25.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-30T03:35:34.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:35:34.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:35:37.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-30T03:35:49.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:35:49.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:35:52.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-30T03:36:07.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:36:07.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:36:39.626 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-30T03:36:40.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:36:40.086 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g1df0c7e8a235"... 2024-03-30T03:36:40.086 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-30T03:36:40.126 INFO:teuthology.orchestra.run.smithi023.stdout:6.8.0-g1df0c7e8a235 2024-03-30T03:36:40.126 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g1df0c7e8a235 vs 6.8.0-g1df0c7e8a235 2024-03-30T03:36:40.126 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T03:36:40.126 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2024-03-30T03:36:41.127 DEBUG:teuthology.parallel:result is None 2024-03-30T03:36:41.127 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T03:36:41.134 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T03:36:41.135 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T03:36:41.139 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T03:36:41.142 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T03:36:41.177 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T03:36:41.186 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T03:36:41.197 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T03:36:41.197 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T03:36:41.222 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T03:36:41.224 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T03:36:41.278 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T03:36:41.285 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T03:36:41.286 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T03:36:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:36:41.313 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T03:36:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:36:41.333 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T03:36:41.336 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T03:36:41.356 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T03:36:41.406 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.430 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.440 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.487 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.492 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.527 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T03:36:41.531 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T03:36:41.538 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T03:36:41.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T03:36:41.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T03:36:41.545 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T03:36:41.607 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T03:36:41.617 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T03:36:41.617 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T03:36:41.621 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T03:36:41.623 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T03:36:41.658 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:41.773 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:41.864 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:41.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:42.019 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:36:42.020 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T03:36:42.085 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:42.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:42.187 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:42.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:42.318 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:36:42.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T03:36:42.387 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:42.435 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T03:36:42.492 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:42.571 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T03:36:42.626 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:36:42.626 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T03:36:42.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-30T03:36:42.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-03-30T03:36:42.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-30T03:36:42.759 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T03:36:42.787 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T03:36:42.800 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T03:36:43.258 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T03:36:43.267 INFO:teuthology.task.internal:Starting timer... 2024-03-30T03:36:43.267 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T03:36:43.276 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T03:36:43.286 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-30T03:36:43.287 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-30T03:36:43.397 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-30T03:36:43.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-03-30T03:36:43.471 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-03-30T03:36:43.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-30T03:36:43.538 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-30T03:36:43.540 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T03:36:43.541 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-30T03:36:43.587 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-30T03:36:43.588 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-03-30T03:36:43.612 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-03-30T03:36:43.613 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-30T03:36:43.639 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-30T03:36:43.639 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-30T03:36:43.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T03:36:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:36:43.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T03:36:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:36:43.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T03:36:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T03:36:43.762 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T03:36:43.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-30T03:36:43.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-03-30T03:36:43.885 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-30T03:36:43.913 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T03:36:43.927 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-30T03:36:43.933 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-30T03:36:44.059 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T03:36:44.082 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T03:36:44.083 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi154.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T03:48:19.308 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-30T03:48:19.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:48:19.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:48:19.400 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-30T03:48:19.510 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-30T03:48:19.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-30T03:48:19.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:48:19.592 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-03-30T03:48:19.662 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-03-30T03:48:19.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T03:48:19.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T03:48:19.754 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-30T03:48:19.780 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-30T03:48:19.780 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T03:48:19.791 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T03:48:19.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T03:48:19.792 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T03:48:19.796 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T03:48:19.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T03:48:19.798 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T03:48:19.798 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T03:48:19.833 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T03:48:19.850 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T03:48:19.853 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T03:48:19.865 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-30T03:48:19.870 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T03:48:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-30T03:48:19.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-03-30T03:48:19.945 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-03-30T03:48:19.951 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-30T03:48:19.961 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T03:48:19.963 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-30T03:48:19.976 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T03:48:19.977 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T03:48:19.992 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T03:48:19.992 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-30T03:48:20.020 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-30T03:48:20.041 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.045 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.052 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-30T03:48:20.067 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.071 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.085 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-30T03:48:20.205 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2024-03-30T03:48:20.212 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T03:48:20.212 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-30T03:48:20.212 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.212 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.212 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.213 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T03:48:20.214 INFO:teuthology.run_tasks:Running task pexec... 2024-03-30T03:48:20.223 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-30T03:48:20.224 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T03:48:20.224 DEBUG:teuthology.orchestra.run.smithi023:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T03:48:20.224 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2024-03-30T03:48:20.224 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T03:48:20.225 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T03:48:20.225 DEBUG:teuthology.orchestra.run.smithi193:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-30T03:48:20.600 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:20.601 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:20.744 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:22.469 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 34 kB/s | 4.1 kB 00:00 2024-03-30T03:48:22.483 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 4.5 kB 00:00 2024-03-30T03:48:22.815 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 24 kB/s | 4.1 kB 00:00 2024-03-30T03:48:22.837 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-03-30T03:48:23.191 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-03-30T03:48:23.207 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-03-30T03:48:25.952 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout:Resetting modules: 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-03-30T03:48:25.953 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:48:25.994 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-30T03:48:25.995 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T03:48:25.995 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout:Resetting modules: 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-30T03:48:25.996 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:48:26.599 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-30T03:48:26.600 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout:Resetting modules: 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-30T03:48:26.601 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:48:39.509 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-30T03:48:40.028 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:41.116 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-30T03:48:41.637 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:41.903 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 kB/s | 4.5 kB 00:00 2024-03-30T03:48:42.649 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 20 kB/s | 4.1 kB 00:00 2024-03-30T03:48:42.966 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-30T03:48:43.610 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2024-03-30T03:48:43.633 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 kB/s | 4.1 kB 00:00 2024-03-30T03:48:43.971 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 37 kB/s | 4.5 kB 00:00 2024-03-30T03:48:45.636 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 36 kB/s | 4.1 kB 00:00 2024-03-30T03:48:45.639 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-30T03:48:45.802 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================================== 2024-03-30T03:48:45.802 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================================== 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout:Installing group/module packages: 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T03:48:45.803 INFO:teuthology.orchestra.run.smithi193.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout:Installing module profiles: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools/common 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout:Enabling module streams: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: container-tools rhel8 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:48:45.804 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================================== 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 26 M 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 87 M 2024-03-30T03:48:45.805 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-30T03:48:45.951 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 646 kB/s | 94 kB 00:00 2024-03-30T03:48:45.976 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 145 kB/s | 25 kB 00:00 2024-03-30T03:48:45.989 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 kB/s | 4.5 kB 00:00 2024-03-30T03:48:46.018 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): yajl-2.1.0-12.el8.x86_64.rpm 612 kB/s | 41 kB 00:00 2024-03-30T03:48:46.068 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): criu-libs-3.18-4.module+el8.9.0+20326+3 150 kB/s | 39 kB 00:00 2024-03-30T03:48:46.157 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 5.6 MB/s | 1.0 MB 00:00 2024-03-30T03:48:46.244 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 4.1 MB/s | 739 kB 00:00 2024-03-30T03:48:46.269 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 2.1 MB/s | 239 kB 00:00 2024-03-30T03:48:46.394 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 23 MB/s | 8.8 MB 00:00 2024-03-30T03:48:46.411 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 1.1 MB/s | 154 kB 00:00 2024-03-30T03:48:46.528 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 13 MB/s | 3.7 MB 00:00 2024-03-30T03:48:46.595 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 14 MB/s | 2.5 MB 00:00 2024-03-30T03:48:46.720 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 25 MB/s | 8.1 MB 00:00 2024-03-30T03:48:46.745 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): udica-0.2.6-20.module+el8.9.0+21243+a5 228 kB/s | 49 kB 00:00 2024-03-30T03:48:46.746 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:48:46.746 INFO:teuthology.orchestra.run.smithi193.stdout:Total 27 MB/s | 26 MB 00:00 2024-03-30T03:48:47.035 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-30T03:48:47.051 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-30T03:48:47.055 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-30T03:48:47.055 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================== 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================== 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout:Installing group/module packages: 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T03:48:47.219 INFO:teuthology.orchestra.run.smithi154.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout:Installing module profiles: 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools/common 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling module streams: 2024-03-30T03:48:47.220 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools rhel8 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout:========================================================================================================== 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:48:47.221 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 26 M 2024-03-30T03:48:47.222 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 87 M 2024-03-30T03:48:47.222 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-30T03:48:47.283 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-30T03:48:47.292 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-30T03:48:47.450 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): criu-libs-3.18-4.module+el8.9.0+20326+3 173 kB/s | 39 kB 00:00 2024-03-30T03:48:47.508 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-pyxdg-0.25-16.el8.noarch.rpm 328 kB/s | 94 kB 00:00 2024-03-30T03:48:47.542 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): yajl-2.1.0-12.el8.x86_64.rpm 446 kB/s | 41 kB 00:00 2024-03-30T03:48:47.609 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-pytoml-0.1.14-5.git7dea353.el8. 64 kB/s | 25 kB 00:00 2024-03-30T03:48:47.651 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 7.1 MB/s | 1.0 MB 00:00 2024-03-30T03:48:47.743 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 5.4 MB/s | 739 kB 00:00 2024-03-30T03:48:47.819 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.4 MB/s | 239 kB 00:00 2024-03-30T03:48:47.944 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 22 MB/s | 8.8 MB 00:00 2024-03-30T03:48:47.961 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 1.1 MB/s | 154 kB 00:00 2024-03-30T03:48:47.990 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-30T03:48:48.060 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 12 MB/s | 3.7 MB 00:00 2024-03-30T03:48:48.203 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 10 MB/s | 2.5 MB 00:00 2024-03-30T03:48:48.220 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 309 kB/s | 49 kB 00:00 2024-03-30T03:48:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 22 MB/s | 8.1 MB 00:00 2024-03-30T03:48:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:48:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:Total 23 MB/s | 26 MB 00:01 2024-03-30T03:48:48.516 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T03:48:48.596 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-30T03:48:48.617 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-30T03:48:48.617 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-30T03:48:48.749 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T03:48:48.847 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-30T03:48:48.855 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-30T03:48:48.898 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T03:48:49.091 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T03:48:49.294 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T03:48:49.393 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-30T03:48:49.504 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T03:48:49.547 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-30T03:48:49.582 INFO:teuthology.orchestra.run.smithi023.stdout:========================================================================================================== 2024-03-30T03:48:49.582 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-30T03:48:49.582 INFO:teuthology.orchestra.run.smithi023.stdout:========================================================================================================== 2024-03-30T03:48:49.582 INFO:teuthology.orchestra.run.smithi023.stdout:Installing group/module packages: 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: aardvark-dns x86_64 2:1.7.0-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 1.0 M 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: buildah x86_64 1:1.31.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.8 M 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman noarch 75-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 739 k 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: crun x86_64 1.8.7-1.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 239 k 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: netavark x86_64 2:1.7.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman noarch 4.6.0-2.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 154 k 2024-03-30T03:48:49.583 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo x86_64 2:1.13.3-3.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 8.1 M 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox x86_64 0.0.99.4-5.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 2.5 M 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout: udica noarch 0.2.6-20.module+el8.9.0+21243+a586538b rhel-8-for-x86_64-appstream-rpms 49 k 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout: yajl x86_64 2.1.0-12.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2024-03-30T03:48:49.584 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout: criu-libs x86_64 3.18-4.module+el8.9.0+20326+387084d0 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout:Installing module profiles: 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools/common 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout: container-tools rhel8 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:48:49.585 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-30T03:48:49.586 INFO:teuthology.orchestra.run.smithi023.stdout:========================================================================================================== 2024-03-30T03:48:49.586 INFO:teuthology.orchestra.run.smithi023.stdout:Install 13 Packages 2024-03-30T03:48:49.586 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:48:49.587 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 26 M 2024-03-30T03:48:49.587 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 87 M 2024-03-30T03:48:49.587 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-30T03:48:49.756 INFO:teuthology.orchestra.run.smithi023.stdout:(1/13): python3-pyxdg-0.25-16.el8.noarch.rpm 550 kB/s | 94 kB 00:00 2024-03-30T03:48:49.764 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T03:48:49.790 INFO:teuthology.orchestra.run.smithi023.stdout:(2/13): python3-pytoml-0.1.14-5.git7dea353.el8. 121 kB/s | 25 kB 00:00 2024-03-30T03:48:49.807 INFO:teuthology.orchestra.run.smithi023.stdout:(3/13): criu-libs-3.18-4.module+el8.9.0+20326+3 178 kB/s | 39 kB 00:00 2024-03-30T03:48:49.930 INFO:teuthology.orchestra.run.smithi023.stdout:(4/13): yajl-2.1.0-12.el8.x86_64.rpm 237 kB/s | 41 kB 00:00 2024-03-30T03:48:50.033 INFO:teuthology.orchestra.run.smithi023.stdout:(5/13): aardvark-dns-1.7.0-1.module+el8.9.0+212 4.2 MB/s | 1.0 MB 00:00 2024-03-30T03:48:50.052 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T03:48:50.150 INFO:teuthology.orchestra.run.smithi023.stdout:(6/13): cockpit-podman-75-1.module+el8.9.0+2124 3.3 MB/s | 739 kB 00:00 2024-03-30T03:48:50.183 INFO:teuthology.orchestra.run.smithi023.stdout:(7/13): crun-1.8.7-1.module+el8.9.0+21243+a5865 1.6 MB/s | 239 kB 00:00 2024-03-30T03:48:50.223 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T03:48:50.368 INFO:teuthology.orchestra.run.smithi023.stdout:(8/13): buildah-1.31.3-3.module+el8.9.0+21243+a 16 MB/s | 8.8 MB 00:00 2024-03-30T03:48:50.410 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T03:48:50.412 INFO:teuthology.orchestra.run.smithi023.stdout:(9/13): python3-podman-4.6.0-2.module+el8.9.0+2 737 kB/s | 154 kB 00:00 2024-03-30T03:48:50.509 INFO:teuthology.orchestra.run.smithi023.stdout:(10/13): netavark-1.7.0-2.module+el8.9.0+21243+ 10 MB/s | 3.7 MB 00:00 2024-03-30T03:48:50.575 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T03:48:50.610 INFO:teuthology.orchestra.run.smithi023.stdout:(11/13): toolbox-0.0.99.4-5.module+el8.9.0+2124 12 MB/s | 2.5 MB 00:00 2024-03-30T03:48:50.634 INFO:teuthology.orchestra.run.smithi023.stdout:(12/13): udica-0.2.6-20.module+el8.9.0+21243+a5 396 kB/s | 49 kB 00:00 2024-03-30T03:48:50.694 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:50.721 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T03:48:50.802 INFO:teuthology.orchestra.run.smithi023.stdout:(13/13): skopeo-1.13.3-3.module+el8.9.0+21243+a 19 MB/s | 8.1 MB 00:00 2024-03-30T03:48:50.802 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:48:50.802 INFO:teuthology.orchestra.run.smithi023.stdout:Total 21 MB/s | 26 MB 00:01 2024-03-30T03:48:50.921 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T03:48:50.962 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:51.174 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T03:48:51.283 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-30T03:48:51.310 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-30T03:48:51.310 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-30T03:48:51.339 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T03:48:51.831 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-30T03:48:51.841 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-30T03:48:52.076 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:52.264 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T03:48:52.286 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:52.518 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T03:48:52.674 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-30T03:48:52.697 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T03:48:52.701 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T03:48:53.159 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T03:48:53.160 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:53.160 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:53.160 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T03:48:53.160 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T03:48:53.160 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T03:48:53.197 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 1/13 2024-03-30T03:48:53.428 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 2/13 2024-03-30T03:48:53.615 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : yajl-2.1.0-12.el8.x86_64 3/13 2024-03-30T03:48:53.776 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T03:48:53.818 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 4/13 2024-03-30T03:48:53.993 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 5/13 2024-03-30T03:48:54.028 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T03:48:54.210 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:54.283 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 6/13 2024-03-30T03:48:54.479 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 7/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T03:48:54.631 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T03:48:54.632 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:54.632 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:54.632 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T03:48:54.632 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T03:48:54.632 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T03:48:55.506 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:55.766 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 10/13 2024-03-30T03:48:57.132 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 11/13 2024-03-30T03:48:57.379 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 12/13 2024-03-30T03:48:57.539 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 13/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86 3/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 4/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a58653 5/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x 6/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cockpit-podman-75-1.module+el8.9.0+21243+a586538b. 7/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 8/13 2024-03-30T03:48:58.223 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x 9/13 2024-03-30T03:48:58.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-podman-4.6.0-2.module+el8.9.0+21243+a58653 10/13 2024-03-30T03:48:58.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x8 11/13 2024-03-30T03:48:58.224 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x 12/13 2024-03-30T03:49:07.233 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T03:49:07.233 INFO:teuthology.orchestra.run.smithi193.stdout:Installed products updated. 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T03:49:07.800 INFO:teuthology.orchestra.run.smithi193.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:49:07.801 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-30T03:49:07.956 DEBUG:teuthology.parallel:result is None 2024-03-30T03:49:08.890 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T03:49:08.891 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.446 INFO:teuthology.orchestra.run.smithi154.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:49:09.447 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-30T03:49:09.592 DEBUG:teuthology.parallel:result is None 2024-03-30T03:49:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarc 13/13 2024-03-30T03:49:10.813 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: aardvark-dns-2:1.7.0-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: buildah-1:1.31.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: cockpit-podman-75-1.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: criu-libs-3.18-4.module+el8.9.0+20326+387084d0.x86_64 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: crun-1.8.7-1.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: netavark-2:1.7.0-2.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: python3-podman-4.6.0-2.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:11.484 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: skopeo-2:1.13.3-3.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: toolbox-0.0.99.4-5.module+el8.9.0+21243+a586538b.x86_64 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: udica-0.2.6-20.module+el8.9.0+21243+a586538b.noarch 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: yajl-2.1.0-12.el8.x86_64 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:49:11.485 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-30T03:49:11.724 DEBUG:teuthology.parallel:result is None 2024-03-30T03:49:11.724 INFO:teuthology.run_tasks:Running task install... 2024-03-30T03:49:11.734 DEBUG:teuthology.task.install:project ceph 2024-03-30T03:49:11.734 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'}} 2024-03-30T03:49:11.734 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f'} 2024-03-30T03:49:11.734 INFO:teuthology.task.install:Using flavor: default 2024-03-30T03:49:11.740 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T03:49:11.740 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-30T03:49:11.741 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T03:49:11.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:49:11.744 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T03:49:11.744 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:49:11.745 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'tag': None, 'wait_for_package': False} 2024-03-30T03:49:11.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:49:11.902 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T03:49:11.902 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T03:49:11.906 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T03:49:11.906 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T03:49:11.974 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/ 2024-03-30T03:49:11.974 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1018.g470100a4 2024-03-30T03:49:12.130 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T03:49:12.130 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:49:12.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T03:49:12.163 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T03:49:12.164 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T03:49:12.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T03:49:12.247 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T03:49:12.247 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:49:12.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T03:49:12.315 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T03:49:12.315 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T03:49:12.316 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T03:49:12.346 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-30T03:49:12.346 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:49:12.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-30T03:49:12.379 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-30T03:49:12.379 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/470100a4ccbcf7769052bf09a5413881ccf1b97f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-30T03:49:12.385 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-03-30T03:49:12.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-03-30T03:49:12.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T03:49:12.452 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-30T03:49:12.514 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T03:49:12.532 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-30T03:49:12.578 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-30T03:49:12.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-30T03:49:12.600 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-30T03:49:12.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-30T03:49:12.765 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:12.979 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:13.061 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:14.079 INFO:teuthology.orchestra.run.smithi154.stdout:61 files removed 2024-03-30T03:49:14.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T03:49:14.251 INFO:teuthology.orchestra.run.smithi193.stdout:61 files removed 2024-03-30T03:49:14.303 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T03:49:14.471 INFO:teuthology.orchestra.run.smithi154.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:14.545 INFO:teuthology.orchestra.run.smithi023.stdout:61 files removed 2024-03-30T03:49:14.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-30T03:49:14.638 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:15.082 INFO:teuthology.orchestra.run.smithi023.stdout:Updating Subscription Management repositories. 2024-03-30T03:49:16.351 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 264 kB/s | 80 kB 00:00 2024-03-30T03:49:16.488 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 220 kB/s | 80 kB 00:00 2024-03-30T03:49:16.643 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 39 kB/s | 9.9 kB 00:00 2024-03-30T03:49:16.816 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 34 kB/s | 9.9 kB 00:00 2024-03-30T03:49:16.943 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 5.4 kB/s | 1.5 kB 00:00 2024-03-30T03:49:17.102 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2024-03-30T03:49:17.158 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 4.7 kB/s | 1.5 kB 00:00 2024-03-30T03:49:17.192 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 189 kB/s | 80 kB 00:00 2024-03-30T03:49:17.325 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-03-30T03:49:17.643 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 24 kB/s | 9.9 kB 00:00 2024-03-30T03:49:17.977 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 4.9 kB/s | 1.5 kB 00:00 2024-03-30T03:49:18.152 INFO:teuthology.orchestra.run.smithi023.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-03-30T03:49:18.618 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-03-30T03:49:19.447 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2024-03-30T03:49:20.436 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 16 MB 00:03 2024-03-30T03:49:22.177 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 492 kB/s | 24 kB 00:00 2024-03-30T03:49:23.612 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 66 MB 00:01 2024-03-30T03:49:23.972 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 498 kB/s | 24 kB 00:00 2024-03-30T03:49:24.071 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 370 kB/s | 24 kB 00:00 2024-03-30T03:49:24.832 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 11 MB/s | 8.8 MB 00:00 2024-03-30T03:49:25.454 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 46 MB/s | 60 MB 00:01 2024-03-30T03:49:28.775 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 31 MB/s | 66 MB 00:02 2024-03-30T03:49:34.262 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 60 MB 00:01 2024-03-30T03:49:36.123 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 13 MB/s | 8.8 MB 00:00 2024-03-30T03:49:38.999 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 66 MB 00:01 2024-03-30T03:49:41.460 INFO:teuthology.orchestra.run.smithi023.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 39 MB/s | 60 MB 00:01 2024-03-30T03:49:45.112 INFO:teuthology.orchestra.run.smithi154.stdout:Red Hat CodeReady Linux Builder for RHEL 8 x86_ 11 MB/s | 8.8 MB 00:00 2024-03-30T03:49:52.234 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:52.235 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:52.253 INFO:teuthology.orchestra.run.smithi154.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:52.255 INFO:teuthology.orchestra.run.smithi154.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:52.256 INFO:teuthology.orchestra.run.smithi154.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T03:49:52.257 INFO:teuthology.orchestra.run.smithi154.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T03:49:52.260 INFO:teuthology.orchestra.run.smithi154.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T03:49:52.264 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T03:49:52.265 INFO:teuthology.orchestra.run.smithi154.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T03:49:52.480 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T03:49:52.495 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T03:49:52.496 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T03:49:52.497 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T03:49:52.498 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T03:49:52.499 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T03:49:52.500 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T03:49:52.501 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T03:49:52.502 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T03:49:52.503 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T03:49:52.504 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T03:49:52.505 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T03:49:52.506 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T03:49:52.507 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T03:49:52.508 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling module streams: 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql 10 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout:=================================================================================================================================== 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout:Install 130 Packages 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 16 Packages 2024-03-30T03:49:52.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:49:52.510 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 213 M 2024-03-30T03:49:52.510 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-03-30T03:49:52.738 INFO:teuthology.orchestra.run.smithi154.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 35 kB/s | 6.4 kB 00:00 2024-03-30T03:49:53.155 INFO:teuthology.orchestra.run.smithi154.stdout:(2/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 9.2 MB/s | 5.5 MB 00:00 2024-03-30T03:49:53.221 INFO:teuthology.orchestra.run.smithi154.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 2.4 MB/s | 166 kB 00:00 2024-03-30T03:49:53.455 INFO:teuthology.orchestra.run.smithi154.stdout:(4/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 9.8 MB/s | 2.3 MB 00:00 2024-03-30T03:49:53.480 INFO:teuthology.orchestra.run.smithi154.stdout:(5/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 1.1 MB/s | 854 kB 00:00 2024-03-30T03:49:53.606 INFO:teuthology.orchestra.run.smithi154.stdout:(6/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 10 MB/s | 1.6 MB 00:00 2024-03-30T03:49:53.888 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:53.889 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:53.906 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:53.908 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:53.909 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T03:49:53.910 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T03:49:53.913 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T03:49:53.917 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T03:49:53.918 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T03:49:54.135 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout:=================================================================================================================================== 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout:=================================================================================================================================== 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T03:49:54.149 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T03:49:54.150 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T03:49:54.151 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T03:49:54.152 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T03:49:54.153 INFO:teuthology.orchestra.run.smithi193.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T03:49:54.154 INFO:teuthology.orchestra.run.smithi193.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T03:49:54.155 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T03:49:54.156 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T03:49:54.157 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T03:49:54.158 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout:Enabling module streams: 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql 10 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout:=================================================================================================================================== 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout:Install 130 Packages 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 16 Packages 2024-03-30T03:49:54.159 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:49:54.160 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 213 M 2024-03-30T03:49:54.160 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-30T03:49:54.382 INFO:teuthology.orchestra.run.smithi193.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 41 kB/s | 6.4 kB 00:00 2024-03-30T03:49:54.431 INFO:teuthology.orchestra.run.smithi154.stdout:(7/146): ceph-common-18.2.2-1018.g470100a4.el8. 12 MB/s | 22 MB 00:01 2024-03-30T03:49:55.024 INFO:teuthology.orchestra.run.smithi193.stdout:(2/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 1.3 MB/s | 854 kB 00:00 2024-03-30T03:49:55.124 INFO:teuthology.orchestra.run.smithi193.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 1.6 MB/s | 166 kB 00:00 2024-03-30T03:49:55.198 INFO:teuthology.orchestra.run.smithi154.stdout:(8/146): ceph-osd-18.2.2-1018.g470100a4.el8.x86 12 MB/s | 18 MB 00:01 2024-03-30T03:49:55.348 INFO:teuthology.orchestra.run.smithi154.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 163 kB/s | 24 kB 00:00 2024-03-30T03:49:56.048 INFO:teuthology.orchestra.run.smithi154.stdout:(10/146): ceph-radosgw-18.2.2-1018.g470100a4.el 8.1 MB/s | 13 MB 00:01 2024-03-30T03:49:56.173 INFO:teuthology.orchestra.run.smithi154.stdout:(11/146): cephfs-mirror-18.2.2-1018.g470100a4.e 2.0 MB/s | 261 kB 00:00 2024-03-30T03:49:56.240 INFO:teuthology.orchestra.run.smithi154.stdout:(12/146): libcephfs-devel-18.2.2-1018.g470100a4 467 kB/s | 31 kB 00:00 2024-03-30T03:49:56.283 INFO:teuthology.orchestra.run.smithi193.stdout:(4/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 2.0 MB/s | 2.3 MB 00:01 2024-03-30T03:49:56.399 INFO:teuthology.orchestra.run.smithi154.stdout:(13/146): libcephfs2-18.2.2-1018.g470100a4.el8. 4.8 MB/s | 775 kB 00:00 2024-03-30T03:49:56.483 INFO:teuthology.orchestra.run.smithi154.stdout:(14/146): libcephsqlite-18.2.2-1018.g470100a4.e 1.7 MB/s | 142 kB 00:00 2024-03-30T03:49:56.566 INFO:teuthology.orchestra.run.smithi154.stdout:(15/146): librados-devel-18.2.2-1018.g470100a4. 1.6 MB/s | 139 kB 00:00 2024-03-30T03:49:56.567 INFO:teuthology.orchestra.run.smithi193.stdout:(5/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 2.4 MB/s | 5.5 MB 00:02 2024-03-30T03:49:56.658 INFO:teuthology.orchestra.run.smithi154.stdout:(16/146): libradosstriper1-18.2.2-1018.g470100a 5.1 MB/s | 475 kB 00:00 2024-03-30T03:49:56.892 INFO:teuthology.orchestra.run.smithi193.stdout:(6/146): ceph-common-18.2.2-1018.g470100a4.el8. 8.1 MB/s | 22 MB 00:02 2024-03-30T03:49:57.458 INFO:teuthology.orchestra.run.smithi154.stdout:(17/146): librgw2-18.2.2-1018.g470100a4.el8.x86 8.2 MB/s | 6.5 MB 00:00 2024-03-30T03:49:57.484 INFO:teuthology.orchestra.run.smithi193.stdout:(7/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 1.3 MB/s | 1.6 MB 00:01 2024-03-30T03:49:57.500 INFO:teuthology.orchestra.run.smithi154.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 1.1 MB/s | 45 kB 00:00 2024-03-30T03:49:57.626 INFO:teuthology.orchestra.run.smithi154.stdout:(19/146): python3-ceph-common-18.2.2-1018.g4701 1.0 MB/s | 129 kB 00:00 2024-03-30T03:49:57.701 INFO:teuthology.orchestra.run.smithi154.stdout:(20/146): python3-cephfs-18.2.2-1018.g470100a4. 2.6 MB/s | 196 kB 00:00 2024-03-30T03:49:57.717 INFO:teuthology.orchestra.run.smithi193.stdout:(8/146): ceph-mon-18.2.2-1018.g470100a4.el8.x86 4.2 MB/s | 4.8 MB 00:01 2024-03-30T03:49:57.760 INFO:teuthology.orchestra.run.smithi154.stdout:(21/146): python3-rados-18.2.2-1018.g470100a4.e 6.2 MB/s | 371 kB 00:00 2024-03-30T03:49:57.809 INFO:teuthology.orchestra.run.smithi193.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 266 kB/s | 24 kB 00:00 2024-03-30T03:49:57.818 INFO:teuthology.orchestra.run.smithi154.stdout:(22/146): ceph-mon-18.2.2-1018.g470100a4.el8.x8 1.1 MB/s | 4.8 MB 00:04 2024-03-30T03:49:57.843 INFO:teuthology.orchestra.run.smithi154.stdout:(23/146): python3-rbd-18.2.2-1018.g470100a4.el8 4.3 MB/s | 366 kB 00:00 2024-03-30T03:49:57.919 INFO:teuthology.orchestra.run.smithi154.stdout:(24/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 1.2 MB/s | 95 kB 00:00 2024-03-30T03:49:57.986 INFO:teuthology.orchestra.run.smithi154.stdout:(25/146): python3-rgw-18.2.2-1018.g470100a4.el8 641 kB/s | 107 kB 00:00 2024-03-30T03:49:58.253 INFO:teuthology.orchestra.run.smithi154.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 680 kB/s | 181 kB 00:00 2024-03-30T03:49:58.361 INFO:teuthology.orchestra.run.smithi154.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 215 kB/s | 23 kB 00:00 2024-03-30T03:49:58.486 INFO:teuthology.orchestra.run.smithi154.stdout:(28/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 6.1 MB/s | 3.5 MB 00:00 2024-03-30T03:49:58.562 INFO:teuthology.orchestra.run.smithi154.stdout:(29/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 690 kB/s | 138 kB 00:00 2024-03-30T03:49:58.787 INFO:teuthology.orchestra.run.smithi154.stdout:(30/146): ceph-mgr-dashboard-18.2.2-1018.g47010 6.0 MB/s | 1.8 MB 00:00 2024-03-30T03:49:58.871 INFO:teuthology.orchestra.run.smithi154.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 2.9 MB/s | 245 kB 00:00 2024-03-30T03:49:58.946 INFO:teuthology.orchestra.run.smithi154.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 658 kB/s | 49 kB 00:00 2024-03-30T03:49:59.021 INFO:teuthology.orchestra.run.smithi154.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 196 kB/s | 15 kB 00:00 2024-03-30T03:49:59.096 INFO:teuthology.orchestra.run.smithi154.stdout:(34/146): ceph-volume-18.2.2-1018.g470100a4.el8 3.4 MB/s | 263 kB 00:00 2024-03-30T03:49:59.188 INFO:teuthology.orchestra.run.smithi154.stdout:(35/146): cephadm-18.2.2-1018.g470100a4.el8.noa 1.8 MB/s | 173 kB 00:00 2024-03-30T03:49:59.230 INFO:teuthology.orchestra.run.smithi154.stdout:(36/146): cephfs-top-18.2.2-1018.g470100a4.el8. 436 kB/s | 18 kB 00:00 2024-03-30T03:49:59.243 INFO:teuthology.orchestra.run.smithi193.stdout:(10/146): ceph-osd-18.2.2-1018.g470100a4.el8.x8 7.8 MB/s | 18 MB 00:02 2024-03-30T03:49:59.343 INFO:teuthology.orchestra.run.smithi193.stdout:(11/146): cephfs-mirror-18.2.2-1018.g470100a4.e 2.6 MB/s | 261 kB 00:00 2024-03-30T03:49:59.393 INFO:teuthology.orchestra.run.smithi193.stdout:(12/146): libcephfs-devel-18.2.2-1018.g470100a4 621 kB/s | 31 kB 00:00 2024-03-30T03:49:59.481 INFO:teuthology.orchestra.run.smithi154.stdout:(37/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2024-03-30T03:49:59.510 INFO:teuthology.orchestra.run.smithi193.stdout:(13/146): libcephfs2-18.2.2-1018.g470100a4.el8. 6.5 MB/s | 775 kB 00:00 2024-03-30T03:49:59.539 INFO:teuthology.orchestra.run.smithi154.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-03-30T03:49:59.569 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:59.569 INFO:teuthology.orchestra.run.smithi193.stdout:(14/146): libcephsqlite-18.2.2-1018.g470100a4.e 2.4 MB/s | 142 kB 00:00 2024-03-30T03:49:59.570 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2024-03-30T03:49:59.590 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:59.592 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2024-03-30T03:49:59.594 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2024-03-30T03:49:59.595 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2024-03-30T03:49:59.598 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2024-03-30T03:49:59.603 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2024-03-30T03:49:59.605 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2024-03-30T03:49:59.623 INFO:teuthology.orchestra.run.smithi154.stdout:(39/146): libunwind-1.3.1-3.el8.x86_64.rpm 904 kB/s | 75 kB 00:00 2024-03-30T03:49:59.652 INFO:teuthology.orchestra.run.smithi193.stdout:(15/146): librados-devel-18.2.2-1018.g470100a4. 1.6 MB/s | 139 kB 00:00 2024-03-30T03:49:59.728 INFO:teuthology.orchestra.run.smithi193.stdout:(16/146): libradosstriper1-18.2.2-1018.g470100a 6.2 MB/s | 475 kB 00:00 2024-03-30T03:49:59.756 INFO:teuthology.orchestra.run.smithi154.stdout:(40/146): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2024-03-30T03:49:59.824 INFO:teuthology.orchestra.run.smithi154.stdout:(41/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 652 kB/s | 44 kB 00:00 2024-03-30T03:49:59.854 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout:=================================================================================================================================== 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout:=================================================================================================================================== 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-13.el8_9 rhel-8-for-x86_64-baseos-rpms 144 k 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.4 k 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:18.2.2-1018.g470100a4.el8 ceph 5.5 M 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 854 k 2024-03-30T03:49:59.870 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1018.g470100a4.el8 ceph 166 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:18.2.2-1018.g470100a4.el8 ceph 1.6 M 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 138 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 1.8 M 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 7.4 M 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 49 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 13 M 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:18.2.2-1018.g470100a4.el8 ceph 47 M 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 263 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 173 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 261 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 18 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 62 k 2024-03-30T03:49:59.871 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 31 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 775 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:18.2.2-1018.g470100a4.el8 ceph 139 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 100 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 2.3 M 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 1.5 M 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 812 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 10.23-4.module+el8.9.0+21317+d2c6bf8a rhel-8-for-x86_64-appstream-rpms 5.1 M 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:18.2.2-1018.g470100a4.el8 ceph 196 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:18.2.2-1018.g470100a4.el8 ceph 371 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 366 k 2024-03-30T03:49:59.872 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:18.2.2-1018.g470100a4.el8 ceph 107 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 95 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.5 M 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 181 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 73 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 232 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.189-3.el8 rhel-8-for-x86_64-baseos-rpms 303 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: iproute x86_64 6.2.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 854 k 2024-03-30T03:49:59.873 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 254 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 237 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.7 M 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 3.6 M 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 179 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 99 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 711 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:1.1.1k-12.el8_9 rhel-8-for-x86_64-baseos-rpms 1.5 M 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.32.1-43.el8 rhel-8-for-x86_64-baseos-rpms 103 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.8-7.el8_9 rhel-8-for-x86_64-baseos-rpms 333 k 2024-03-30T03:49:59.874 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 5.0.0-12.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.1-8.el8 rhel-8-for-x86_64-appstream-rpms 714 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 22 M 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 23 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:18.2.2-1018.g470100a4.el8 ceph 2.3 M 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 245 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:18.2.2-1018.g470100a4.el8 ceph 4.8 M 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:18.2.2-1018.g470100a4.el8 ceph 18 M 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1018.g470100a4.el8 ceph-noarch 15 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:18.2.2-1018.g470100a4.el8 ceph 24 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.7-17.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-7.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2024-03-30T03:49:59.875 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel x86_64 1.18.2-26.el8_9 rhel-8-for-x86_64-baseos-rpms 562 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:18.2.2-1018.g470100a4.el8 ceph 142 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 645 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libpq x86_64 13.11-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 8.5.0-20.el8 rhel-8-for-x86_64-baseos-rpms 172 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.9.0-5.el8_9 rhel-8-for-x86_64-baseos-rpms 47 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 475 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:18.2.2-1018.g470100a4.el8 ceph 6.5 M 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2024-03-30T03:49:59.876 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.4.4-1.el8 rhel-8-for-x86_64-baseos-rpms 44 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.8.2-2.1.el8_9 rhel-8-for-x86_64-appstream-rpms 188 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.0 M 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads x86_64 0.3.15-6.el8 rhel-8-for-x86_64-appstream-rpms 5.2 M 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-03-30T03:49:59.877 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:18.2.2-1018.g470100a4.el8 ceph 45 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:18.2.2-1018.g470100a4.el8 ceph 129 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.11.5-6.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 3.2.1-7.el8_9 rhel-8-for-x86_64-baseos-rpms 559 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-03-30T03:49:59.878 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.0.6-14.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2024-03-30T03:49:59.879 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 16.8-9.el8 codeready-builder-for-rhel-8-x86_64-rpms 48 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2017.2-11.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-03-30T03:49:59.880 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.0.0-21.module+el8.9.0+20976+d3c38525 rhel-8-for-x86_64-appstream-rpms 14 M 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2024-03-30T03:49:59.881 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling module streams: 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql 10 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout:=================================================================================================================================== 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout:Install 130 Packages 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 16 Packages 2024-03-30T03:49:59.882 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:49:59.883 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 213 M 2024-03-30T03:49:59.883 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-30T03:49:59.890 INFO:teuthology.orchestra.run.smithi154.stdout:(42/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2024-03-30T03:49:59.966 INFO:teuthology.orchestra.run.smithi154.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 440 kB/s | 33 kB 00:00 2024-03-30T03:50:00.016 INFO:teuthology.orchestra.run.smithi154.stdout:(44/146): python3-certifi-2018.10.15-7.el8.noar 328 kB/s | 16 kB 00:00 2024-03-30T03:50:00.084 INFO:teuthology.orchestra.run.smithi154.stdout:(45/146): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2024-03-30T03:50:00.159 INFO:teuthology.orchestra.run.smithi154.stdout:(46/146): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2024-03-30T03:50:00.217 INFO:teuthology.orchestra.run.smithi154.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2024-03-30T03:50:00.268 INFO:teuthology.orchestra.run.smithi154.stdout:(48/146): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2024-03-30T03:50:00.343 INFO:teuthology.orchestra.run.smithi154.stdout:(49/146): python3-jaraco-functools-2.0-4.el8.no 235 kB/s | 18 kB 00:00 2024-03-30T03:50:00.453 INFO:teuthology.orchestra.run.smithi193.stdout:(17/146): librgw2-18.2.2-1018.g470100a4.el8.x86 9.0 MB/s | 6.5 MB 00:00 2024-03-30T03:50:00.460 INFO:teuthology.orchestra.run.smithi154.stdout:(50/146): python3-kubernetes-11.0.0-6.el8.noarc 10 MB/s | 1.2 MB 00:00 2024-03-30T03:50:00.495 INFO:teuthology.orchestra.run.smithi193.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 1.1 MB/s | 45 kB 00:00 2024-03-30T03:50:00.501 INFO:teuthology.orchestra.run.smithi023.stdout:(1/146): ceph-18.2.2-1018.g470100a4.el8.x86_64. 23 kB/s | 6.4 kB 00:00 2024-03-30T03:50:00.570 INFO:teuthology.orchestra.run.smithi193.stdout:(19/146): python3-ceph-common-18.2.2-1018.g4701 1.7 MB/s | 129 kB 00:00 2024-03-30T03:50:00.670 INFO:teuthology.orchestra.run.smithi193.stdout:(20/146): python3-cephfs-18.2.2-1018.g470100a4. 1.9 MB/s | 196 kB 00:00 2024-03-30T03:50:01.055 INFO:teuthology.orchestra.run.smithi154.stdout:(51/146): ceph-test-18.2.2-1018.g470100a4.el8.x 8.3 MB/s | 47 MB 00:05 2024-03-30T03:50:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:(2/146): ceph-base-18.2.2-1018.g470100a4.el8.x8 6.5 MB/s | 5.5 MB 00:00 2024-03-30T03:50:01.102 INFO:teuthology.orchestra.run.smithi154.stdout:(52/146): python3-logutils-0.3.5-11.el8.noarch. 76 kB/s | 49 kB 00:00 2024-03-30T03:50:01.151 INFO:teuthology.orchestra.run.smithi023.stdout:(3/146): ceph-immutable-object-cache-18.2.2-101 2.2 MB/s | 166 kB 00:00 2024-03-30T03:50:01.169 INFO:teuthology.orchestra.run.smithi154.stdout:(53/146): python3-natsort-7.1.1-2.el8.noarch.rp 906 kB/s | 60 kB 00:00 2024-03-30T03:50:01.228 INFO:teuthology.orchestra.run.smithi154.stdout:(54/146): python3-more-itertools-7.2.0-3.el8.no 341 kB/s | 59 kB 00:00 2024-03-30T03:50:01.253 INFO:teuthology.orchestra.run.smithi154.stdout:(55/146): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2024-03-30T03:50:01.286 INFO:teuthology.orchestra.run.smithi154.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2024-03-30T03:50:01.320 INFO:teuthology.orchestra.run.smithi154.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 504 kB/s | 34 kB 00:00 2024-03-30T03:50:01.337 INFO:teuthology.orchestra.run.smithi193.stdout:(21/146): python3-rados-18.2.2-1018.g470100a4.e 557 kB/s | 371 kB 00:00 2024-03-30T03:50:01.376 INFO:teuthology.orchestra.run.smithi023.stdout:(4/146): ceph-fuse-18.2.2-1018.g470100a4.el8.x8 975 kB/s | 854 kB 00:00 2024-03-30T03:50:01.387 INFO:teuthology.orchestra.run.smithi154.stdout:(58/146): python3-rsa-4.9-2.el8.noarch.rpm 967 kB/s | 64 kB 00:00 2024-03-30T03:50:01.412 INFO:teuthology.orchestra.run.smithi154.stdout:(59/146): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2024-03-30T03:50:01.427 INFO:teuthology.orchestra.run.smithi023.stdout:(5/146): ceph-mds-18.2.2-1018.g470100a4.el8.x86 8.3 MB/s | 2.3 MB 00:00 2024-03-30T03:50:01.484 INFO:teuthology.orchestra.run.smithi154.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 193 kB/s | 19 kB 00:00 2024-03-30T03:50:01.488 INFO:teuthology.orchestra.run.smithi193.stdout:(22/146): python3-rbd-18.2.2-1018.g470100a4.el8 2.4 MB/s | 366 kB 00:00 2024-03-30T03:50:01.513 INFO:teuthology.orchestra.run.smithi154.stdout:(61/146): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2024-03-30T03:50:01.555 INFO:teuthology.orchestra.run.smithi154.stdout:(62/146): python3-tempora-1.14.1-5.el8.noarch.r 416 kB/s | 29 kB 00:00 2024-03-30T03:50:01.580 INFO:teuthology.orchestra.run.smithi154.stdout:(63/146): python3-trustme-0.6.0-4.el8.noarch.rp 412 kB/s | 27 kB 00:00 2024-03-30T03:50:01.588 INFO:teuthology.orchestra.run.smithi193.stdout:(23/146): python3-rgw-18.2.2-1018.g470100a4.el8 1.0 MB/s | 107 kB 00:00 2024-03-30T03:50:01.622 INFO:teuthology.orchestra.run.smithi154.stdout:(64/146): python3-waitress-1.4.3-1.el8.noarch.r 3.7 MB/s | 255 kB 00:00 2024-03-30T03:50:01.672 INFO:teuthology.orchestra.run.smithi154.stdout:(65/146): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2024-03-30T03:50:01.696 INFO:teuthology.orchestra.run.smithi193.stdout:(24/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 879 kB/s | 95 kB 00:00 2024-03-30T03:50:01.697 INFO:teuthology.orchestra.run.smithi154.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 815 kB/s | 61 kB 00:00 2024-03-30T03:50:01.731 INFO:teuthology.orchestra.run.smithi154.stdout:(67/146): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2024-03-30T03:50:01.765 INFO:teuthology.orchestra.run.smithi154.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2024-03-30T03:50:01.791 INFO:teuthology.orchestra.run.smithi154.stdout:(69/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2024-03-30T03:50:01.949 INFO:teuthology.orchestra.run.smithi154.stdout:(70/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 113 kB/s | 18 kB 00:00 2024-03-30T03:50:02.002 INFO:teuthology.orchestra.run.smithi023.stdout:(6/146): ceph-mon-18.2.2-1018.g470100a4.el8.x86 8.4 MB/s | 4.8 MB 00:00 2024-03-30T03:50:02.083 INFO:teuthology.orchestra.run.smithi154.stdout:(71/146): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2024-03-30T03:50:02.133 INFO:teuthology.orchestra.run.smithi154.stdout:(72/146): thrift-0.13.0-2.el8.x86_64.rpm 4.3 MB/s | 1.7 MB 00:00 2024-03-30T03:50:02.192 INFO:teuthology.orchestra.run.smithi154.stdout:(73/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.4 MB/s | 155 kB 00:00 2024-03-30T03:50:02.301 INFO:teuthology.orchestra.run.smithi154.stdout:(74/146): python3-pycparser-2.14-14.el8.noarch. 654 kB/s | 109 kB 00:00 2024-03-30T03:50:02.367 INFO:teuthology.orchestra.run.smithi154.stdout:(75/146): python3-requests-oauthlib-1.0.0-1.el8 244 kB/s | 43 kB 00:00 2024-03-30T03:50:02.401 INFO:teuthology.orchestra.run.smithi154.stdout:(76/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 186 kB/s | 19 kB 00:00 2024-03-30T03:50:02.426 INFO:teuthology.orchestra.run.smithi154.stdout:(77/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 749 kB/s | 44 kB 00:00 2024-03-30T03:50:02.460 INFO:teuthology.orchestra.run.smithi154.stdout:(78/146): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2024-03-30T03:50:02.485 INFO:teuthology.orchestra.run.smithi154.stdout:(79/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2024-03-30T03:50:02.485 INFO:teuthology.orchestra.run.smithi023.stdout:(7/146): ceph-mgr-18.2.2-1018.g470100a4.el8.x86 1.4 MB/s | 1.6 MB 00:01 2024-03-30T03:50:02.568 INFO:teuthology.orchestra.run.smithi154.stdout:(80/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-03-30T03:50:02.647 INFO:teuthology.orchestra.run.smithi193.stdout:(25/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 3.6 MB/s | 3.5 MB 00:00 2024-03-30T03:50:02.652 INFO:teuthology.orchestra.run.smithi154.stdout:(81/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2024-03-30T03:50:02.677 INFO:teuthology.orchestra.run.smithi154.stdout:(82/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 443 kB/s | 48 kB 00:00 2024-03-30T03:50:02.736 INFO:teuthology.orchestra.run.smithi154.stdout:(83/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 992 kB/s | 58 kB 00:00 2024-03-30T03:50:02.747 INFO:teuthology.orchestra.run.smithi193.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 1.8 MB/s | 181 kB 00:00 2024-03-30T03:50:02.761 INFO:teuthology.orchestra.run.smithi154.stdout:(84/146): libsepol-devel-2.9-3.el8.x86_64.rpm 801 kB/s | 87 kB 00:00 2024-03-30T03:50:02.845 INFO:teuthology.orchestra.run.smithi154.stdout:(85/146): libcom_err-devel-1.45.6-4.el8.x86_64. 355 kB/s | 39 kB 00:00 2024-03-30T03:50:02.847 INFO:teuthology.orchestra.run.smithi193.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 232 kB/s | 23 kB 00:00 2024-03-30T03:50:02.870 INFO:teuthology.orchestra.run.smithi154.stdout:(86/146): fuse-2.9.7-17.el8.x86_64.rpm 766 kB/s | 83 kB 00:00 2024-03-30T03:50:02.939 INFO:teuthology.orchestra.run.smithi193.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 1.5 MB/s | 138 kB 00:00 2024-03-30T03:50:02.996 INFO:teuthology.orchestra.run.smithi154.stdout:(87/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2024-03-30T03:50:03.029 INFO:teuthology.orchestra.run.smithi154.stdout:(88/146): libgfortran-8.5.0-20.el8.x86_64.rpm 3.4 MB/s | 645 kB 00:00 2024-03-30T03:50:03.154 INFO:teuthology.orchestra.run.smithi154.stdout:(89/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 629 kB/s | 100 kB 00:00 2024-03-30T03:50:03.210 INFO:teuthology.orchestra.run.smithi154.stdout:(90/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2024-03-30T03:50:03.288 INFO:teuthology.orchestra.run.smithi154.stdout:(91/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 544 kB/s | 73 kB 00:00 2024-03-30T03:50:03.311 INFO:teuthology.orchestra.run.smithi023.stdout:(8/146): ceph-common-18.2.2-1018.g470100a4.el8. 7.0 MB/s | 22 MB 00:03 2024-03-30T03:50:03.313 INFO:teuthology.orchestra.run.smithi154.stdout:(92/146): NetworkManager-initscripts-updown-1.4 1.4 MB/s | 144 kB 00:00 2024-03-30T03:50:03.365 INFO:teuthology.orchestra.run.smithi193.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1018.g47010 4.2 MB/s | 1.8 MB 00:00 2024-03-30T03:50:03.378 INFO:teuthology.orchestra.run.smithi023.stdout:(9/146): ceph-selinux-18.2.2-1018.g470100a4.el8 366 kB/s | 24 kB 00:00 2024-03-30T03:50:03.422 INFO:teuthology.orchestra.run.smithi154.stdout:(93/146): elfutils-libelf-devel-0.189-3.el8.x86 463 kB/s | 62 kB 00:00 2024-03-30T03:50:03.481 INFO:teuthology.orchestra.run.smithi154.stdout:(94/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.3 MB/s | 562 kB 00:00 2024-03-30T03:50:03.614 INFO:teuthology.orchestra.run.smithi154.stdout:(95/146): openssl-devel-1.1.1k-12.el8_9.x86_64. 12 MB/s | 2.3 MB 00:00 2024-03-30T03:50:03.640 INFO:teuthology.orchestra.run.smithi154.stdout:(96/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 297 kB/s | 47 kB 00:00 2024-03-30T03:50:03.790 INFO:teuthology.orchestra.run.smithi154.stdout:(97/146): python3-cryptography-3.2.1-7.el8_9.x8 3.1 MB/s | 559 kB 00:00 2024-03-30T03:50:03.815 INFO:teuthology.orchestra.run.smithi154.stdout:(98/146): python3-jsonpointer-1.10-11.el8.noarc 114 kB/s | 20 kB 00:00 2024-03-30T03:50:03.882 INFO:teuthology.orchestra.run.smithi154.stdout:(99/146): python3-prettytable-0.7.2-14.el8.noar 477 kB/s | 44 kB 00:00 2024-03-30T03:50:03.924 INFO:teuthology.orchestra.run.smithi154.stdout:(100/146): python3-pyasn1-modules-0.3.7-6.el8.n 1.0 MB/s | 110 kB 00:00 2024-03-30T03:50:03.983 INFO:teuthology.orchestra.run.smithi154.stdout:(101/146): python3-jsonpatch-1.21-2.el8.noarch. 265 kB/s | 27 kB 00:00 2024-03-30T03:50:04.058 INFO:teuthology.orchestra.run.smithi154.stdout:(102/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 947 kB/s | 126 kB 00:00 2024-03-30T03:50:04.098 INFO:teuthology.orchestra.run.smithi193.stdout:(30/146): ceph-radosgw-18.2.2-1018.g470100a4.el 2.0 MB/s | 13 MB 00:06 2024-03-30T03:50:04.103 INFO:teuthology.orchestra.run.smithi023.stdout:(10/146): ceph-osd-18.2.2-1018.g470100a4.el8.x8 8.8 MB/s | 18 MB 00:02 2024-03-30T03:50:04.142 INFO:teuthology.orchestra.run.smithi154.stdout:(103/146): python3-werkzeug-0.12.2-4.el8.noarch 2.8 MB/s | 457 kB 00:00 2024-03-30T03:50:04.178 INFO:teuthology.orchestra.run.smithi023.stdout:(11/146): cephfs-mirror-18.2.2-1018.g470100a4.e 3.4 MB/s | 261 kB 00:00 2024-03-30T03:50:04.183 INFO:teuthology.orchestra.run.smithi154.stdout:(104/146): bison-3.0.4-10.el8.x86_64.rpm 5.4 MB/s | 688 kB 00:00 2024-03-30T03:50:04.253 INFO:teuthology.orchestra.run.smithi023.stdout:(12/146): libcephfs-devel-18.2.2-1018.g470100a4 414 kB/s | 31 kB 00:00 2024-03-30T03:50:04.290 INFO:teuthology.orchestra.run.smithi193.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 1.3 MB/s | 245 kB 00:00 2024-03-30T03:50:04.309 INFO:teuthology.orchestra.run.smithi154.stdout:(105/146): flex-2.6.1-9.el8.x86_64.rpm 1.9 MB/s | 320 kB 00:00 2024-03-30T03:50:04.368 INFO:teuthology.orchestra.run.smithi154.stdout:(106/146): libtool-2.4.6-25.el8.x86_64.rpm 3.8 MB/s | 709 kB 00:00 2024-03-30T03:50:04.382 INFO:teuthology.orchestra.run.smithi193.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 537 kB/s | 49 kB 00:00 2024-03-30T03:50:04.403 INFO:teuthology.orchestra.run.smithi023.stdout:(13/146): libcephfs2-18.2.2-1018.g470100a4.el8. 5.1 MB/s | 775 kB 00:00 2024-03-30T03:50:04.434 INFO:teuthology.orchestra.run.smithi154.stdout:(107/146): python3-markupsafe-0.23-19.el8.x86_6 311 kB/s | 39 kB 00:00 2024-03-30T03:50:04.449 INFO:teuthology.orchestra.run.smithi193.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 222 kB/s | 15 kB 00:00 2024-03-30T03:50:04.487 INFO:teuthology.orchestra.run.smithi023.stdout:(14/146): libcephsqlite-18.2.2-1018.g470100a4.e 1.7 MB/s | 142 kB 00:00 2024-03-30T03:50:04.535 INFO:teuthology.orchestra.run.smithi154.stdout:(108/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-03-30T03:50:04.562 INFO:teuthology.orchestra.run.smithi023.stdout:(15/146): librados-devel-18.2.2-1018.g470100a4. 1.8 MB/s | 139 kB 00:00 2024-03-30T03:50:04.602 INFO:teuthology.orchestra.run.smithi154.stdout:(109/146): uuid-1.6.2-43.el8.x86_64.rpm 383 kB/s | 64 kB 00:00 2024-03-30T03:50:04.641 INFO:teuthology.orchestra.run.smithi193.stdout:(34/146): ceph-volume-18.2.2-1018.g470100a4.el8 1.3 MB/s | 263 kB 00:00 2024-03-30T03:50:04.663 INFO:teuthology.orchestra.run.smithi023.stdout:(16/146): libradosstriper1-18.2.2-1018.g470100a 4.6 MB/s | 475 kB 00:00 2024-03-30T03:50:04.677 INFO:teuthology.orchestra.run.smithi154.stdout:(110/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 729 kB/s | 103 kB 00:00 2024-03-30T03:50:04.775 INFO:teuthology.orchestra.run.smithi193.stdout:(35/146): cephadm-18.2.2-1018.g470100a4.el8.noa 1.3 MB/s | 173 kB 00:00 2024-03-30T03:50:04.794 INFO:teuthology.orchestra.run.smithi154.stdout:(111/146): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2024-03-30T03:50:04.836 INFO:teuthology.orchestra.run.smithi154.stdout:(112/146): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2024-03-30T03:50:04.900 INFO:teuthology.orchestra.run.smithi193.stdout:(36/146): cephfs-top-18.2.2-1018.g470100a4.el8. 145 kB/s | 18 kB 00:00 2024-03-30T03:50:04.953 INFO:teuthology.orchestra.run.smithi154.stdout:(113/146): socat-1.7.4.1-1.el8.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2024-03-30T03:50:04.992 INFO:teuthology.orchestra.run.smithi193.stdout:(37/146): ceph-mgr-diskprediction-local-18.2.2- 4.5 MB/s | 7.4 MB 00:01 2024-03-30T03:50:05.028 INFO:teuthology.orchestra.run.smithi154.stdout:(114/146): python3-numpy-1.14.3-10.el8.x86_64.r 16 MB/s | 3.7 MB 00:00 2024-03-30T03:50:05.084 INFO:teuthology.orchestra.run.smithi193.stdout:(38/146): liboath-2.6.2-3.el8.x86_64.rpm 643 kB/s | 59 kB 00:00 2024-03-30T03:50:05.109 INFO:teuthology.orchestra.run.smithi193.stdout:(39/146): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2024-03-30T03:50:05.120 INFO:teuthology.orchestra.run.smithi154.stdout:(115/146): ceph-mgr-diskprediction-local-18.2.2 1.1 MB/s | 7.4 MB 00:06 2024-03-30T03:50:05.143 INFO:teuthology.orchestra.run.smithi193.stdout:(40/146): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2024-03-30T03:50:05.145 INFO:teuthology.orchestra.run.smithi154.stdout:(116/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.1 MB/s | 225 kB 00:00 2024-03-30T03:50:05.179 INFO:teuthology.orchestra.run.smithi154.stdout:(117/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.3 MB/s | 354 kB 00:00 2024-03-30T03:50:05.201 INFO:teuthology.orchestra.run.smithi193.stdout:(41/146): python3-asyncssh-2.7.0-2.el8.noarch.r 5.1 MB/s | 476 kB 00:00 2024-03-30T03:50:05.227 INFO:teuthology.orchestra.run.smithi193.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 522 kB/s | 44 kB 00:00 2024-03-30T03:50:05.269 INFO:teuthology.orchestra.run.smithi193.stdout:(43/146): python3-cachetools-3.1.1-4.el8.noarch 801 kB/s | 33 kB 00:00 2024-03-30T03:50:05.294 INFO:teuthology.orchestra.run.smithi193.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2024-03-30T03:50:05.319 INFO:teuthology.orchestra.run.smithi193.stdout:(45/146): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2024-03-30T03:50:05.337 INFO:teuthology.orchestra.run.smithi154.stdout:(118/146): python3-mako-1.0.6-14.el8.noarch.rpm 842 kB/s | 157 kB 00:00 2024-03-30T03:50:05.345 INFO:teuthology.orchestra.run.smithi193.stdout:(46/146): python3-cheroot-8.5.2-1.el8.noarch.rp 3.4 MB/s | 173 kB 00:00 2024-03-30T03:50:05.387 INFO:teuthology.orchestra.run.smithi193.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2024-03-30T03:50:05.412 INFO:teuthology.orchestra.run.smithi193.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 4.1 MB/s | 384 kB 00:00 2024-03-30T03:50:05.421 INFO:teuthology.orchestra.run.smithi023.stdout:(17/146): librgw2-18.2.2-1018.g470100a4.el8.x86 8.6 MB/s | 6.5 MB 00:00 2024-03-30T03:50:05.437 INFO:teuthology.orchestra.run.smithi193.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-03-30T03:50:05.454 INFO:teuthology.orchestra.run.smithi193.stdout:(50/146): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2024-03-30T03:50:05.471 INFO:teuthology.orchestra.run.smithi154.stdout:(119/146): openblas-0.3.15-6.el8.x86_64.rpm 17 MB/s | 5.0 MB 00:00 2024-03-30T03:50:05.480 INFO:teuthology.orchestra.run.smithi023.stdout:(18/146): python3-ceph-argparse-18.2.2-1018.g47 762 kB/s | 45 kB 00:00 2024-03-30T03:50:05.513 INFO:teuthology.orchestra.run.smithi193.stdout:(51/146): python3-logutils-0.3.5-11.el8.noarch. 838 kB/s | 49 kB 00:00 2024-03-30T03:50:05.530 INFO:teuthology.orchestra.run.smithi023.stdout:(19/146): python3-ceph-common-18.2.2-1018.g4701 2.5 MB/s | 129 kB 00:00 2024-03-30T03:50:05.555 INFO:teuthology.orchestra.run.smithi154.stdout:(120/146): openblas-threads-0.3.15-6.el8.x86_64 12 MB/s | 5.2 MB 00:00 2024-03-30T03:50:05.580 INFO:teuthology.orchestra.run.smithi154.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 2.9 MB/s | 714 kB 00:00 2024-03-30T03:50:05.589 INFO:teuthology.orchestra.run.smithi023.stdout:(20/146): python3-cephfs-18.2.2-1018.g470100a4. 3.3 MB/s | 196 kB 00:00 2024-03-30T03:50:05.605 INFO:teuthology.orchestra.run.smithi154.stdout:(122/146): jq-1.6-7.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2024-03-30T03:50:05.630 INFO:teuthology.orchestra.run.smithi154.stdout:(123/146): libpq-13.11-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2024-03-30T03:50:05.656 INFO:teuthology.orchestra.run.smithi154.stdout:(124/146): python3-pytz-2017.2-11.el8.noarch.rp 719 kB/s | 54 kB 00:00 2024-03-30T03:50:05.656 INFO:teuthology.orchestra.run.smithi023.stdout:(21/146): python3-rados-18.2.2-1018.g470100a4.e 5.5 MB/s | 371 kB 00:00 2024-03-30T03:50:05.765 INFO:teuthology.orchestra.run.smithi023.stdout:(22/146): python3-rbd-18.2.2-1018.g470100a4.el8 3.3 MB/s | 366 kB 00:00 2024-03-30T03:50:05.806 INFO:teuthology.orchestra.run.smithi154.stdout:(125/146): postgresql-10.23-4.module+el8.9.0+21 10 MB/s | 1.5 MB 00:00 2024-03-30T03:50:05.823 INFO:teuthology.orchestra.run.smithi023.stdout:(23/146): python3-rgw-18.2.2-1018.g470100a4.el8 1.8 MB/s | 107 kB 00:00 2024-03-30T03:50:05.831 INFO:teuthology.orchestra.run.smithi154.stdout:(126/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-03-30T03:50:05.882 INFO:teuthology.orchestra.run.smithi023.stdout:(24/146): rbd-fuse-18.2.2-1018.g470100a4.el8.x8 1.6 MB/s | 95 kB 00:00 2024-03-30T03:50:05.923 INFO:teuthology.orchestra.run.smithi154.stdout:(127/146): postgresql-contrib-10.23-4.module+el 6.8 MB/s | 812 kB 00:00 2024-03-30T03:50:06.005 INFO:teuthology.orchestra.run.smithi193.stdout:(52/146): ceph-test-18.2.2-1018.g470100a4.el8.x 5.8 MB/s | 47 MB 00:08 2024-03-30T03:50:06.030 INFO:teuthology.orchestra.run.smithi193.stdout:(53/146): python3-more-itertools-7.2.0-3.el8.no 114 kB/s | 59 kB 00:00 2024-03-30T03:50:06.072 INFO:teuthology.orchestra.run.smithi193.stdout:(54/146): python3-kubernetes-11.0.0-6.el8.noarc 1.8 MB/s | 1.2 MB 00:00 2024-03-30T03:50:06.098 INFO:teuthology.orchestra.run.smithi193.stdout:(55/146): python3-natsort-7.1.1-2.el8.noarch.rp 655 kB/s | 60 kB 00:00 2024-03-30T03:50:06.107 INFO:teuthology.orchestra.run.smithi154.stdout:(128/146): python3-scipy-1.0.0-21.module+el8.9. 27 MB/s | 14 MB 00:00 2024-03-30T03:50:06.114 INFO:teuthology.orchestra.run.smithi193.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 373 kB/s | 16 kB 00:00 2024-03-30T03:50:06.124 INFO:teuthology.orchestra.run.smithi154.stdout:(129/146): python3-packaging-16.8-9.el8.noarch. 240 kB/s | 48 kB 00:00 2024-03-30T03:50:06.140 INFO:teuthology.orchestra.run.smithi193.stdout:(57/146): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2024-03-30T03:50:06.157 INFO:teuthology.orchestra.run.smithi193.stdout:(58/146): python3-repoze-lru-0.7-6.el8.noarch.r 572 kB/s | 34 kB 00:00 2024-03-30T03:50:06.182 INFO:teuthology.orchestra.run.smithi193.stdout:(59/146): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2024-03-30T03:50:06.199 INFO:teuthology.orchestra.run.smithi023.stdout:(25/146): rbd-mirror-18.2.2-1018.g470100a4.el8. 11 MB/s | 3.5 MB 00:00 2024-03-30T03:50:06.241 INFO:teuthology.orchestra.run.smithi154.stdout:(130/146): postgresql-server-10.23-4.module+el8 12 MB/s | 5.1 MB 00:00 2024-03-30T03:50:06.249 INFO:teuthology.orchestra.run.smithi023.stdout:(26/146): rbd-nbd-18.2.2-1018.g470100a4.el8.x86 3.5 MB/s | 181 kB 00:00 2024-03-30T03:50:06.250 INFO:teuthology.orchestra.run.smithi193.stdout:(60/146): python3-simplegeneric-0.8.1-17.el8.no 201 kB/s | 19 kB 00:00 2024-03-30T03:50:06.282 INFO:teuthology.orchestra.run.smithi193.stdout:(61/146): python3-routes-2.4.1-12.el8.noarch.rp 1.1 MB/s | 196 kB 00:00 2024-03-30T03:50:06.308 INFO:teuthology.orchestra.run.smithi193.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2024-03-30T03:50:06.308 INFO:teuthology.orchestra.run.smithi023.stdout:(27/146): ceph-grafana-dashboards-18.2.2-1018.g 401 kB/s | 23 kB 00:00 2024-03-30T03:50:06.316 INFO:teuthology.orchestra.run.smithi154.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 3.0 MB/s | 232 kB 00:00 2024-03-30T03:50:06.333 INFO:teuthology.orchestra.run.smithi193.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 353 kB/s | 29 kB 00:00 2024-03-30T03:50:06.358 INFO:teuthology.orchestra.run.smithi193.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2024-03-30T03:50:06.375 INFO:teuthology.orchestra.run.smithi023.stdout:(28/146): ceph-mgr-cephadm-18.2.2-1018.g470100a 2.0 MB/s | 138 kB 00:00 2024-03-30T03:50:06.384 INFO:teuthology.orchestra.run.smithi193.stdout:(65/146): python3-waitress-1.4.3-1.el8.noarch.r 3.3 MB/s | 255 kB 00:00 2024-03-30T03:50:06.409 INFO:teuthology.orchestra.run.smithi193.stdout:(66/146): python3-webob-1.8.5-1.el8.1.noarch.rp 3.3 MB/s | 251 kB 00:00 2024-03-30T03:50:06.434 INFO:teuthology.orchestra.run.smithi193.stdout:(67/146): python3-websocket-client-0.56.0-5.el8 809 kB/s | 61 kB 00:00 2024-03-30T03:50:06.460 INFO:teuthology.orchestra.run.smithi193.stdout:(68/146): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2024-03-30T03:50:06.485 INFO:teuthology.orchestra.run.smithi193.stdout:(69/146): python3-webtest-2.0.33-1.el8.noarch.r 849 kB/s | 85 kB 00:00 2024-03-30T03:50:06.510 INFO:teuthology.orchestra.run.smithi193.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-03-30T03:50:06.594 INFO:teuthology.orchestra.run.smithi193.stdout:(71/146): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2024-03-30T03:50:06.609 INFO:teuthology.orchestra.run.smithi023.stdout:(29/146): ceph-mgr-dashboard-18.2.2-1018.g47010 7.7 MB/s | 1.8 MB 00:00 2024-03-30T03:50:06.652 INFO:teuthology.orchestra.run.smithi193.stdout:(72/146): python3-jsonpointer-1.10-11.el8.noarc 120 kB/s | 20 kB 00:00 2024-03-30T03:50:06.736 INFO:teuthology.orchestra.run.smithi193.stdout:(73/146): python3-prettytable-0.7.2-14.el8.noar 200 kB/s | 44 kB 00:00 2024-03-30T03:50:06.759 INFO:teuthology.orchestra.run.smithi154.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 683 kB/s | 303 kB 00:00 2024-03-30T03:50:06.778 INFO:teuthology.orchestra.run.smithi193.stdout:(74/146): python3-pyasn1-modules-0.3.7-6.el8.no 618 kB/s | 110 kB 00:00 2024-03-30T03:50:06.820 INFO:teuthology.orchestra.run.smithi193.stdout:(75/146): python3-jsonpatch-1.21-2.el8.noarch.r 159 kB/s | 27 kB 00:00 2024-03-30T03:50:06.833 INFO:teuthology.orchestra.run.smithi154.stdout:(133/146): librbd1-18.2.2-1018.g470100a4.el8.x8 5.0 MB/s | 3.6 MB 00:00 2024-03-30T03:50:06.858 INFO:teuthology.orchestra.run.smithi154.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 237 kB 00:00 2024-03-30T03:50:06.887 INFO:teuthology.orchestra.run.smithi193.stdout:(76/146): python3-pyasn1-0.3.7-6.el8.noarch.rpm 840 kB/s | 126 kB 00:00 2024-03-30T03:50:06.929 INFO:teuthology.orchestra.run.smithi193.stdout:(77/146): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2024-03-30T03:50:06.962 INFO:teuthology.orchestra.run.smithi193.stdout:(78/146): python3-werkzeug-0.12.2-4.el8.noarch. 2.4 MB/s | 457 kB 00:00 2024-03-30T03:50:06.984 INFO:teuthology.orchestra.run.smithi154.stdout:(135/146): libuuid-2.32.1-43.el8.x86_64.rpm 792 kB/s | 99 kB 00:00 2024-03-30T03:50:06.996 INFO:teuthology.orchestra.run.smithi193.stdout:(79/146): flex-2.6.1-9.el8.x86_64.rpm 2.9 MB/s | 320 kB 00:00 2024-03-30T03:50:07.026 INFO:teuthology.orchestra.run.smithi154.stdout:(136/146): libsmartcols-2.32.1-43.el8.x86_64.rp 935 kB/s | 179 kB 00:00 2024-03-30T03:50:07.054 INFO:teuthology.orchestra.run.smithi193.stdout:(80/146): libtool-2.4.6-25.el8.x86_64.rpm 5.5 MB/s | 709 kB 00:00 2024-03-30T03:50:07.143 INFO:teuthology.orchestra.run.smithi154.stdout:(137/146): util-linux-user-2.32.1-43.el8.x86_64 881 kB/s | 103 kB 00:00 2024-03-30T03:50:07.165 INFO:teuthology.orchestra.run.smithi193.stdout:(81/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2024-03-30T03:50:07.197 INFO:teuthology.orchestra.run.smithi193.stdout:(82/146): python3-markupsafe-0.23-19.el8.x86_64 166 kB/s | 39 kB 00:00 2024-03-30T03:50:07.231 INFO:teuthology.orchestra.run.smithi193.stdout:(83/146): uuid-1.6.2-43.el8.x86_64.rpm 363 kB/s | 64 kB 00:00 2024-03-30T03:50:07.256 INFO:teuthology.orchestra.run.smithi193.stdout:(84/146): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2024-03-30T03:50:07.267 INFO:teuthology.orchestra.run.smithi023.stdout:(30/146): ceph-mgr-diskprediction-local-18.2.2- 11 MB/s | 7.4 MB 00:00 2024-03-30T03:50:07.268 INFO:teuthology.orchestra.run.smithi154.stdout:(138/146): util-linux-2.32.1-43.el8.x86_64.rpm 8.7 MB/s | 2.5 MB 00:00 2024-03-30T03:50:07.334 INFO:teuthology.orchestra.run.smithi023.stdout:(31/146): ceph-mgr-modules-core-18.2.2-1018.g47 3.6 MB/s | 245 kB 00:00 2024-03-30T03:50:07.385 INFO:teuthology.orchestra.run.smithi023.stdout:(32/146): ceph-mgr-rook-18.2.2-1018.g470100a4.e 988 kB/s | 49 kB 00:00 2024-03-30T03:50:07.401 INFO:teuthology.orchestra.run.smithi154.stdout:(139/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-03-30T03:50:07.435 INFO:teuthology.orchestra.run.smithi154.stdout:(140/146): iproute-6.2.0-5.el8_9.x86_64.rpm 2.9 MB/s | 854 kB 00:00 2024-03-30T03:50:07.439 INFO:teuthology.orchestra.run.smithi193.stdout:(85/146): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2024-03-30T03:50:07.443 INFO:teuthology.orchestra.run.smithi023.stdout:(33/146): ceph-prometheus-alerts-18.2.2-1018.g4 254 kB/s | 15 kB 00:00 2024-03-30T03:50:07.510 INFO:teuthology.orchestra.run.smithi023.stdout:(34/146): ceph-volume-18.2.2-1018.g470100a4.el8 3.9 MB/s | 263 kB 00:00 2024-03-30T03:50:07.531 INFO:teuthology.orchestra.run.smithi193.stdout:(86/146): python3-jinja2-2.10.1-3.el8.noarch.rp 1.7 MB/s | 538 kB 00:00 2024-03-30T03:50:07.568 INFO:teuthology.orchestra.run.smithi023.stdout:(35/146): cephadm-18.2.2-1018.g470100a4.el8.noa 2.9 MB/s | 173 kB 00:00 2024-03-30T03:50:07.615 INFO:teuthology.orchestra.run.smithi193.stdout:(87/146): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-03-30T03:50:07.627 INFO:teuthology.orchestra.run.smithi154.stdout:(141/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.1 MB/s | 254 kB 00:00 2024-03-30T03:50:07.649 INFO:teuthology.orchestra.run.smithi193.stdout:(88/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2024-03-30T03:50:07.652 INFO:teuthology.orchestra.run.smithi023.stdout:(36/146): cephfs-top-18.2.2-1018.g470100a4.el8. 217 kB/s | 18 kB 00:00 2024-03-30T03:50:07.660 INFO:teuthology.orchestra.run.smithi154.stdout:(142/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 3.1 MB/s | 711 kB 00:00 2024-03-30T03:50:07.716 INFO:teuthology.orchestra.run.smithi193.stdout:(89/146): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.2 MB/s | 225 kB 00:00 2024-03-30T03:50:07.769 INFO:teuthology.orchestra.run.smithi154.stdout:(143/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 3.0 MB/s | 333 kB 00:00 2024-03-30T03:50:07.792 INFO:teuthology.orchestra.run.smithi193.stdout:(90/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.0 MB/s | 354 kB 00:00 2024-03-30T03:50:07.853 INFO:teuthology.orchestra.run.smithi154.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 6.5 MB/s | 1.5 MB 00:00 2024-03-30T03:50:07.903 INFO:teuthology.orchestra.run.smithi154.stdout:(145/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-30T03:50:08.000 INFO:teuthology.orchestra.run.smithi193.stdout:(91/146): openblas-threads-0.3.15-6.el8.x86_64. 15 MB/s | 5.2 MB 00:00 2024-03-30T03:50:08.103 INFO:teuthology.orchestra.run.smithi154.stdout:(146/146): librados2-18.2.2-1018.g470100a4.el8. 1.8 MB/s | 3.7 MB 00:01 2024-03-30T03:50:08.106 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:50:08.106 INFO:teuthology.orchestra.run.smithi154.stdout:Total 14 MB/s | 213 MB 00:15 2024-03-30T03:50:08.219 INFO:teuthology.orchestra.run.smithi023.stdout:(37/146): ceph-test-18.2.2-1018.g470100a4.el8.x 9.8 MB/s | 47 MB 00:04 2024-03-30T03:50:08.358 INFO:teuthology.orchestra.run.smithi193.stdout:(92/146): python3-mako-1.0.6-14.el8.noarch.rpm 244 kB/s | 157 kB 00:00 2024-03-30T03:50:08.361 INFO:teuthology.orchestra.run.smithi023.stdout:(38/146): ceph-radosgw-18.2.2-1018.g470100a4.el 2.2 MB/s | 13 MB 00:05 2024-03-30T03:50:08.458 INFO:teuthology.orchestra.run.smithi193.stdout:(93/146): automake-1.16.1-8.el8.noarch.rpm 1.5 MB/s | 714 kB 00:00 2024-03-30T03:50:08.486 INFO:teuthology.orchestra.run.smithi023.stdout:(39/146): liboath-2.6.2-3.el8.x86_64.rpm 221 kB/s | 59 kB 00:00 2024-03-30T03:50:08.517 INFO:teuthology.orchestra.run.smithi193.stdout:(94/146): libpq-13.11-1.el8.x86_64.rpm 1.2 MB/s | 198 kB 00:00 2024-03-30T03:50:08.520 INFO:teuthology.orchestra.run.smithi023.stdout:(40/146): gperftools-libs-2.7-9.el8.x86_64.rpm 353 kB/s | 306 kB 00:00 2024-03-30T03:50:08.545 INFO:teuthology.orchestra.run.smithi023.stdout:(41/146): libunwind-1.3.1-3.el8.x86_64.rpm 410 kB/s | 75 kB 00:00 2024-03-30T03:50:08.584 INFO:teuthology.orchestra.run.smithi193.stdout:(95/146): jq-1.6-7.el8.x86_64.rpm 1.6 MB/s | 202 kB 00:00 2024-03-30T03:50:08.604 INFO:teuthology.orchestra.run.smithi023.stdout:(42/146): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 522 kB/s | 44 kB 00:00 2024-03-30T03:50:08.654 INFO:teuthology.orchestra.run.smithi023.stdout:(43/146): python3-asyncssh-2.7.0-2.el8.noarch.r 2.8 MB/s | 476 kB 00:00 2024-03-30T03:50:08.659 INFO:teuthology.orchestra.run.smithi193.stdout:(96/146): openblas-0.3.15-6.el8.x86_64.rpm 5.8 MB/s | 5.0 MB 00:00 2024-03-30T03:50:08.679 INFO:teuthology.orchestra.run.smithi023.stdout:(44/146): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2024-03-30T03:50:08.684 INFO:teuthology.orchestra.run.smithi193.stdout:(97/146): python3-pytz-2017.2-11.el8.noarch.rpm 324 kB/s | 54 kB 00:00 2024-03-30T03:50:08.705 INFO:teuthology.orchestra.run.smithi023.stdout:(45/146): python3-cachetools-3.1.1-4.el8.noarch 327 kB/s | 33 kB 00:00 2024-03-30T03:50:08.730 INFO:teuthology.orchestra.run.smithi023.stdout:(46/146): python3-certifi-2018.10.15-7.el8.noar 215 kB/s | 16 kB 00:00 2024-03-30T03:50:08.789 INFO:teuthology.orchestra.run.smithi023.stdout:(47/146): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2024-03-30T03:50:08.822 INFO:teuthology.orchestra.run.smithi023.stdout:(48/146): python3-cherrypy-18.4.0-1.el8.noarch. 3.2 MB/s | 384 kB 00:00 2024-03-30T03:50:08.824 INFO:teuthology.orchestra.run.smithi193.stdout:(98/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-03-30T03:50:08.877 INFO:teuthology.orchestra.run.smithi193.stdout:(99/146): postgresql-10.23-4.module+el8.9.0+213 7.8 MB/s | 1.5 MB 00:00 2024-03-30T03:50:08.944 INFO:teuthology.orchestra.run.smithi193.stdout:(100/146): postgresql-contrib-10.23-4.module+el 6.7 MB/s | 812 kB 00:00 2024-03-30T03:50:09.065 INFO:teuthology.orchestra.run.smithi023.stdout:(49/146): python3-jaraco-6.2-6.el8.noarch.rpm 40 kB/s | 11 kB 00:00 2024-03-30T03:50:09.090 INFO:teuthology.orchestra.run.smithi023.stdout:(50/146): python3-cheroot-8.5.2-1.el8.noarch.rp 421 kB/s | 173 kB 00:00 2024-03-30T03:50:09.115 INFO:teuthology.orchestra.run.smithi023.stdout:(51/146): python3-jaraco-functools-2.0-4.el8.no 60 kB/s | 18 kB 00:00 2024-03-30T03:50:09.169 INFO:teuthology.orchestra.run.smithi193.stdout:(101/146): python3-scipy-1.0.0-21.module+el8.9. 23 MB/s | 14 MB 00:00 2024-03-30T03:50:09.174 INFO:teuthology.orchestra.run.smithi023.stdout:(52/146): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2024-03-30T03:50:09.194 INFO:teuthology.orchestra.run.smithi193.stdout:(102/146): python3-packaging-16.8-9.el8.noarch. 193 kB/s | 48 kB 00:00 2024-03-30T03:50:09.199 INFO:teuthology.orchestra.run.smithi023.stdout:(53/146): python3-logutils-0.3.5-11.el8.noarch. 585 kB/s | 49 kB 00:00 2024-03-30T03:50:09.214 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-03-30T03:50:09.241 INFO:teuthology.orchestra.run.smithi023.stdout:(54/146): python3-kubernetes-11.0.0-6.el8.noarc 6.6 MB/s | 1.2 MB 00:00 2024-03-30T03:50:09.266 INFO:teuthology.orchestra.run.smithi023.stdout:(55/146): python3-natsort-7.1.1-2.el8.noarch.rp 657 kB/s | 60 kB 00:00 2024-03-30T03:50:09.277 INFO:teuthology.orchestra.run.smithi193.stdout:(103/146): postgresql-server-10.23-4.module+el8 13 MB/s | 5.1 MB 00:00 2024-03-30T03:50:09.300 INFO:teuthology.orchestra.run.smithi023.stdout:(56/146): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-03-30T03:50:09.303 INFO:teuthology.orchestra.run.smithi193.stdout:(104/146): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2024-03-30T03:50:09.325 INFO:teuthology.orchestra.run.smithi023.stdout:(57/146): python3-repoze-lru-0.7-6.el8.noarch.r 574 kB/s | 34 kB 00:00 2024-03-30T03:50:09.328 INFO:teuthology.orchestra.run.smithi193.stdout:(105/146): python3-jwt-1.6.1-2.el8.noarch.rpm 319 kB/s | 43 kB 00:00 2024-03-30T03:50:09.353 INFO:teuthology.orchestra.run.smithi193.stdout:(106/146): python3-oauthlib-2.1.0-1.el8.noarch. 2.0 MB/s | 155 kB 00:00 2024-03-30T03:50:09.384 INFO:teuthology.orchestra.run.smithi023.stdout:(58/146): python3-routes-2.4.1-12.el8.noarch.rp 2.3 MB/s | 196 kB 00:00 2024-03-30T03:50:09.410 INFO:teuthology.orchestra.run.smithi023.stdout:(59/146): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-03-30T03:50:09.412 INFO:teuthology.orchestra.run.smithi193.stdout:(107/146): python3-pycparser-2.14-14.el8.noarch 1.0 MB/s | 109 kB 00:00 2024-03-30T03:50:09.434 INFO:teuthology.orchestra.run.smithi023.stdout:(60/146): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-03-30T03:50:09.460 INFO:teuthology.orchestra.run.smithi023.stdout:(61/146): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2024-03-30T03:50:09.471 INFO:teuthology.orchestra.run.smithi193.stdout:(108/146): libaio-devel-0.3.112-1.el8.x86_64.rp 160 kB/s | 19 kB 00:00 2024-03-30T03:50:09.485 INFO:teuthology.orchestra.run.smithi023.stdout:(62/146): python3-singledispatch-3.4.0.3-18.el8 317 kB/s | 24 kB 00:00 2024-03-30T03:50:09.496 INFO:teuthology.orchestra.run.smithi193.stdout:(109/146): python3-requests-oauthlib-1.0.0-1.el 254 kB/s | 43 kB 00:00 2024-03-30T03:50:09.510 INFO:teuthology.orchestra.run.smithi023.stdout:(63/146): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2024-03-30T03:50:09.535 INFO:teuthology.orchestra.run.smithi023.stdout:(64/146): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2024-03-30T03:50:09.572 INFO:teuthology.orchestra.run.smithi193.stdout:(110/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 276 kB/s | 44 kB 00:00 2024-03-30T03:50:09.577 INFO:teuthology.orchestra.run.smithi023.stdout:(65/146): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2024-03-30T03:50:09.603 INFO:teuthology.orchestra.run.smithi023.stdout:(66/146): python3-websocket-client-0.56.0-5.el8 916 kB/s | 61 kB 00:00 2024-03-30T03:50:09.609 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-03-30T03:50:09.609 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-03-30T03:50:09.628 INFO:teuthology.orchestra.run.smithi023.stdout:(67/146): python3-waitress-1.4.3-1.el8.noarch.r 1.7 MB/s | 255 kB 00:00 2024-03-30T03:50:09.653 INFO:teuthology.orchestra.run.smithi023.stdout:(68/146): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2024-03-30T03:50:09.663 INFO:teuthology.orchestra.run.smithi193.stdout:(111/146): pcre2-devel-10.32-2.el8.x86_64.rpm 3.1 MB/s | 605 kB 00:00 2024-03-30T03:50:09.678 INFO:teuthology.orchestra.run.smithi023.stdout:(69/146): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2024-03-30T03:50:09.681 INFO:teuthology.orchestra.run.smithi193.stdout:(112/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2024-03-30T03:50:09.704 INFO:teuthology.orchestra.run.smithi023.stdout:(70/146): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2024-03-30T03:50:09.706 INFO:teuthology.orchestra.run.smithi193.stdout:(113/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2024-03-30T03:50:09.798 INFO:teuthology.orchestra.run.smithi193.stdout:(114/146): keyutils-libs-devel-1.5.10-9.el8.x86 411 kB/s | 48 kB 00:00 2024-03-30T03:50:09.815 INFO:teuthology.orchestra.run.smithi193.stdout:(115/146): libselinux-devel-2.9-5.el8.x86_64.rp 1.3 MB/s | 200 kB 00:00 2024-03-30T03:50:09.840 INFO:teuthology.orchestra.run.smithi193.stdout:(116/146): libsepol-devel-2.9-3.el8.x86_64.rpm 650 kB/s | 87 kB 00:00 2024-03-30T03:50:09.897 INFO:teuthology.orchestra.run.smithi023.stdout:(71/146): libverto-devel-0.3.0-5.el8.x86_64.rpm 98 kB/s | 18 kB 00:00 2024-03-30T03:50:09.921 INFO:teuthology.orchestra.run.smithi023.stdout:(72/146): python3-packaging-16.8-9.el8.noarch.r 199 kB/s | 48 kB 00:00 2024-03-30T03:50:09.982 INFO:teuthology.orchestra.run.smithi193.stdout:(117/146): fuse-2.9.7-17.el8.x86_64.rpm 586 kB/s | 83 kB 00:00 2024-03-30T03:50:10.004 INFO:teuthology.orchestra.run.smithi023.stdout:(73/146): python3-jwt-1.6.1-2.el8.noarch.rpm 394 kB/s | 43 kB 00:00 2024-03-30T03:50:10.007 INFO:teuthology.orchestra.run.smithi193.stdout:(118/146): zlib-devel-1.2.11-18.el8_5.x86_64.rp 276 kB/s | 58 kB 00:00 2024-03-30T03:50:10.055 INFO:teuthology.orchestra.run.smithi023.stdout:(74/146): python3-oauthlib-2.1.0-1.el8.noarch.r 1.1 MB/s | 155 kB 00:00 2024-03-30T03:50:10.061 INFO:teuthology.orchestra.run.smithi193.stdout:(119/146): libcom_err-devel-1.45.6-4.el8.x86_64 157 kB/s | 39 kB 00:00 2024-03-30T03:50:10.149 INFO:teuthology.orchestra.run.smithi193.stdout:(120/146): libgfortran-8.5.0-20.el8.x86_64.rpm 3.8 MB/s | 645 kB 00:00 2024-03-30T03:50:10.155 INFO:teuthology.orchestra.run.smithi023.stdout:(75/146): thrift-0.13.0-2.el8.x86_64.rpm 3.3 MB/s | 1.7 MB 00:00 2024-03-30T03:50:10.174 INFO:teuthology.orchestra.run.smithi193.stdout:(121/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.0 MB/s | 172 kB 00:00 2024-03-30T03:50:10.180 INFO:teuthology.orchestra.run.smithi023.stdout:(76/146): python3-requests-oauthlib-1.0.0-1.el8 341 kB/s | 43 kB 00:00 2024-03-30T03:50:10.200 INFO:teuthology.orchestra.run.smithi193.stdout:(122/146): libuuid-devel-2.32.1-43.el8.x86_64.r 721 kB/s | 100 kB 00:00 2024-03-30T03:50:10.214 INFO:teuthology.orchestra.run.smithi023.stdout:(77/146): python3-pycparser-2.14-14.el8.noarch. 521 kB/s | 109 kB 00:00 2024-03-30T03:50:10.234 INFO:teuthology.orchestra.run.smithi193.stdout:(123/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2024-03-30T03:50:10.289 INFO:teuthology.orchestra.run.smithi023.stdout:(78/146): libaio-devel-0.3.112-1.el8.x86_64.rpm 140 kB/s | 19 kB 00:00 2024-03-30T03:50:10.326 INFO:teuthology.orchestra.run.smithi193.stdout:(124/146): NetworkManager-initscripts-updown-1. 1.1 MB/s | 144 kB 00:00 2024-03-30T03:50:10.348 INFO:teuthology.orchestra.run.smithi023.stdout:(79/146): pcre2-devel-10.32-2.el8.x86_64.rpm 4.4 MB/s | 605 kB 00:00 2024-03-30T03:50:10.375 INFO:teuthology.orchestra.run.smithi193.stdout:(125/146): elfutils-libelf-devel-0.189-3.el8.x8 437 kB/s | 62 kB 00:00 2024-03-30T03:50:10.398 INFO:teuthology.orchestra.run.smithi023.stdout:(80/146): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.1 MB/s | 229 kB 00:00 2024-03-30T03:50:10.401 INFO:teuthology.orchestra.run.smithi193.stdout:(126/146): xfsprogs-devel-5.0.0-12.el8.x86_64.r 321 kB/s | 73 kB 00:00 2024-03-30T03:50:10.415 INFO:teuthology.orchestra.run.smithi023.stdout:(81/146): libzstd-devel-1.4.4-1.el8.x86_64.rpm 187 kB/s | 44 kB 00:00 2024-03-30T03:50:10.457 INFO:teuthology.orchestra.run.smithi023.stdout:(82/146): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-03-30T03:50:10.501 INFO:teuthology.orchestra.run.smithi193.stdout:(127/146): krb5-devel-1.18.2-26.el8_9.x86_64.rp 3.1 MB/s | 562 kB 00:00 2024-03-30T03:50:10.516 INFO:teuthology.orchestra.run.smithi023.stdout:(83/146): libselinux-devel-2.9-5.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2024-03-30T03:50:10.527 INFO:teuthology.orchestra.run.smithi193.stdout:(128/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 376 kB/s | 47 kB 00:00 2024-03-30T03:50:10.541 INFO:teuthology.orchestra.run.smithi023.stdout:(84/146): keyutils-libs-devel-1.5.10-9.el8.x86_ 383 kB/s | 48 kB 00:00 2024-03-30T03:50:10.591 INFO:teuthology.orchestra.run.smithi023.stdout:(85/146): libsepol-devel-2.9-3.el8.x86_64.rpm 650 kB/s | 87 kB 00:00 2024-03-30T03:50:10.593 INFO:teuthology.orchestra.run.smithi193.stdout:(129/146): openssl-devel-1.1.1k-12.el8_9.x86_64 11 MB/s | 2.3 MB 00:00 2024-03-30T03:50:10.627 INFO:teuthology.orchestra.run.smithi193.stdout:(130/146): python3-cryptography-3.2.1-7.el8_9.x 4.3 MB/s | 559 kB 00:00 2024-03-30T03:50:10.642 INFO:teuthology.orchestra.run.smithi023.stdout:(86/146): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 460 kB/s | 58 kB 00:00 2024-03-30T03:50:10.667 INFO:teuthology.orchestra.run.smithi023.stdout:(87/146): libcom_err-devel-1.45.6-4.el8.x86_64. 308 kB/s | 39 kB 00:00 2024-03-30T03:50:10.759 INFO:teuthology.orchestra.run.smithi023.stdout:(88/146): fuse-2.9.7-17.el8.x86_64.rpm 496 kB/s | 83 kB 00:00 2024-03-30T03:50:10.761 INFO:teuthology.orchestra.run.smithi193.stdout:(131/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.7 MB/s | 232 kB 00:00 2024-03-30T03:50:10.801 INFO:teuthology.orchestra.run.smithi023.stdout:(89/146): libgfortran-8.5.0-20.el8.x86_64.rpm 4.0 MB/s | 645 kB 00:00 2024-03-30T03:50:10.802 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-03-30T03:50:10.813 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-03-30T03:50:10.826 INFO:teuthology.orchestra.run.smithi023.stdout:(90/146): libquadmath-8.5.0-20.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-03-30T03:50:10.861 INFO:teuthology.orchestra.run.smithi193.stdout:(132/146): elfutils-libs-0.189-3.el8.x86_64.rpm 3.0 MB/s | 303 kB 00:00 2024-03-30T03:50:10.910 INFO:teuthology.orchestra.run.smithi023.stdout:(91/146): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.1 MB/s | 238 kB 00:00 2024-03-30T03:50:10.935 INFO:teuthology.orchestra.run.smithi023.stdout:(92/146): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 669 kB/s | 73 kB 00:00 2024-03-30T03:50:10.960 INFO:teuthology.orchestra.run.smithi023.stdout:(93/146): libuuid-devel-2.32.1-43.el8.x86_64.rp 497 kB/s | 100 kB 00:00 2024-03-30T03:50:11.012 INFO:teuthology.orchestra.run.smithi193.stdout:(133/146): libmount-2.32.1-43.el8.x86_64.rpm 1.5 MB/s | 237 kB 00:00 2024-03-30T03:50:11.035 INFO:teuthology.orchestra.run.smithi023.stdout:(94/146): NetworkManager-initscripts-updown-1.4 1.1 MB/s | 144 kB 00:00 2024-03-30T03:50:11.077 INFO:teuthology.orchestra.run.smithi023.stdout:(95/146): elfutils-libelf-devel-0.189-3.el8.x86 434 kB/s | 62 kB 00:00 2024-03-30T03:50:11.111 INFO:teuthology.orchestra.run.smithi023.stdout:(96/146): krb5-devel-1.18.2-26.el8_9.x86_64.rpm 3.6 MB/s | 562 kB 00:00 2024-03-30T03:50:11.137 INFO:teuthology.orchestra.run.smithi193.stdout:(134/146): librbd1-18.2.2-1018.g470100a4.el8.x8 6.6 MB/s | 3.6 MB 00:00 2024-03-30T03:50:11.170 INFO:teuthology.orchestra.run.smithi023.stdout:(97/146): librabbitmq-0.9.0-5.el8_9.x86_64.rpm 513 kB/s | 47 kB 00:00 2024-03-30T03:50:11.187 INFO:teuthology.orchestra.run.smithi193.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 1.0 MB/s | 179 kB 00:00 2024-03-30T03:50:11.212 INFO:teuthology.orchestra.run.smithi193.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 1.3 MB/s | 99 kB 00:00 2024-03-30T03:50:11.278 INFO:teuthology.orchestra.run.smithi023.stdout:(98/146): python3-cryptography-3.2.1-7.el8_9.x8 3.3 MB/s | 559 kB 00:00 2024-03-30T03:50:11.304 INFO:teuthology.orchestra.run.smithi023.stdout:(99/146): python3-jsonpointer-1.10-11.el8.noarc 150 kB/s | 20 kB 00:00 2024-03-30T03:50:11.346 INFO:teuthology.orchestra.run.smithi193.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 16 MB/s | 2.5 MB 00:00 2024-03-30T03:50:11.362 INFO:teuthology.orchestra.run.smithi023.stdout:(100/146): openssl-devel-1.1.1k-12.el8_9.x86_64 7.1 MB/s | 2.3 MB 00:00 2024-03-30T03:50:11.363 INFO:teuthology.orchestra.run.smithi193.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 684 kB/s | 103 kB 00:00 2024-03-30T03:50:11.429 INFO:teuthology.orchestra.run.smithi023.stdout:(101/146): python3-prettytable-0.7.2-14.el8.noa 292 kB/s | 44 kB 00:00 2024-03-30T03:50:11.446 INFO:teuthology.orchestra.run.smithi193.stdout:(139/146): librados2-18.2.2-1018.g470100a4.el8. 4.0 MB/s | 3.7 MB 00:00 2024-03-30T03:50:11.454 INFO:teuthology.orchestra.run.smithi023.stdout:(102/146): python3-pyasn1-modules-0.3.7-6.el8.n 733 kB/s | 110 kB 00:00 2024-03-30T03:50:11.472 INFO:teuthology.orchestra.run.smithi193.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2024-03-30T03:50:11.505 INFO:teuthology.orchestra.run.smithi023.stdout:(103/146): python3-jsonpatch-1.21-2.el8.noarch. 187 kB/s | 27 kB 00:00 2024-03-30T03:50:11.514 INFO:teuthology.orchestra.run.smithi193.stdout:(141/146): iproute-6.2.0-5.el8_9.x86_64.rpm 5.0 MB/s | 854 kB 00:00 2024-03-30T03:50:11.556 INFO:teuthology.orchestra.run.smithi193.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 2.3 MB/s | 254 kB 00:00 2024-03-30T03:50:11.631 INFO:teuthology.orchestra.run.smithi193.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.4 MB/s | 711 kB 00:00 2024-03-30T03:50:11.655 INFO:teuthology.orchestra.run.smithi023.stdout:(104/146): python3-pyasn1-0.3.7-6.el8.noarch.rp 631 kB/s | 126 kB 00:00 2024-03-30T03:50:11.681 INFO:teuthology.orchestra.run.smithi193.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 8.8 MB/s | 1.5 MB 00:00 2024-03-30T03:50:11.689 INFO:teuthology.orchestra.run.smithi023.stdout:(105/146): python3-werkzeug-0.12.2-4.el8.noarch 1.9 MB/s | 457 kB 00:00 2024-03-30T03:50:11.723 INFO:teuthology.orchestra.run.smithi193.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 1.9 MB/s | 333 kB 00:00 2024-03-30T03:50:11.731 INFO:teuthology.orchestra.run.smithi023.stdout:(106/146): bison-3.0.4-10.el8.x86_64.rpm 3.0 MB/s | 688 kB 00:00 2024-03-30T03:50:11.765 INFO:teuthology.orchestra.run.smithi193.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-03-30T03:50:11.767 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:50:11.768 INFO:teuthology.orchestra.run.smithi193.stdout:Total 12 MB/s | 213 MB 00:17 2024-03-30T03:50:11.797 INFO:teuthology.orchestra.run.smithi023.stdout:(107/146): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2024-03-30T03:50:11.823 INFO:teuthology.orchestra.run.smithi023.stdout:(108/146): python3-markupsafe-0.23-19.el8.x86_6 423 kB/s | 39 kB 00:00 2024-03-30T03:50:11.865 INFO:teuthology.orchestra.run.smithi023.stdout:(109/146): libtool-2.4.6-25.el8.x86_64.rpm 3.9 MB/s | 709 kB 00:00 2024-03-30T03:50:11.932 INFO:teuthology.orchestra.run.smithi023.stdout:(110/146): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.9 MB/s | 259 kB 00:00 2024-03-30T03:50:12.024 INFO:teuthology.orchestra.run.smithi023.stdout:(111/146): uuid-1.6.2-43.el8.x86_64.rpm 319 kB/s | 64 kB 00:00 2024-03-30T03:50:12.049 INFO:teuthology.orchestra.run.smithi023.stdout:(112/146): python3-pyOpenSSL-19.0.0-1.el8.noarc 562 kB/s | 103 kB 00:00 2024-03-30T03:50:12.108 INFO:teuthology.orchestra.run.smithi023.stdout:(113/146): python3-jinja2-2.10.1-3.el8.noarch.r 6.3 MB/s | 538 kB 00:00 2024-03-30T03:50:12.250 INFO:teuthology.orchestra.run.smithi023.stdout:(114/146): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2024-03-30T03:50:12.283 INFO:teuthology.orchestra.run.smithi023.stdout:(115/146): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2024-03-30T03:50:12.372 INFO:teuthology.orchestra.run.smithi023.stdout:(116/146): python3-babel-2.5.1-7.el8.noarch.rpm 11 MB/s | 4.8 MB 00:00 2024-03-30T03:50:12.417 INFO:teuthology.orchestra.run.smithi023.stdout:(117/146): python3-numpy-f2py-1.14.3-10.el8.x86 1.3 MB/s | 225 kB 00:00 2024-03-30T03:50:12.442 INFO:teuthology.orchestra.run.smithi023.stdout:(118/146): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2024-03-30T03:50:12.542 INFO:teuthology.orchestra.run.smithi023.stdout:(119/146): python3-mako-1.0.6-14.el8.noarch.rpm 1.2 MB/s | 157 kB 00:00 2024-03-30T03:50:12.676 INFO:teuthology.orchestra.run.smithi023.stdout:(120/146): openblas-threads-0.3.15-6.el8.x86_64 17 MB/s | 5.2 MB 00:00 2024-03-30T03:50:12.726 INFO:teuthology.orchestra.run.smithi023.stdout:(121/146): automake-1.16.1-8.el8.noarch.rpm 3.8 MB/s | 714 kB 00:00 2024-03-30T03:50:12.785 INFO:teuthology.orchestra.run.smithi023.stdout:(122/146): libpq-13.11-1.el8.x86_64.rpm 1.8 MB/s | 198 kB 00:00 2024-03-30T03:50:12.797 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-30T03:50:12.869 INFO:teuthology.orchestra.run.smithi023.stdout:(123/146): openblas-0.3.15-6.el8.x86_64.rpm 12 MB/s | 5.0 MB 00:00 2024-03-30T03:50:12.893 INFO:teuthology.orchestra.run.smithi023.stdout:(124/146): jq-1.6-7.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2024-03-30T03:50:12.952 INFO:teuthology.orchestra.run.smithi023.stdout:(125/146): python3-pytz-2017.2-11.el8.noarch.rp 325 kB/s | 54 kB 00:00 2024-03-30T03:50:13.061 INFO:teuthology.orchestra.run.smithi023.stdout:(126/146): postgresql-10.23-4.module+el8.9.0+21 14 MB/s | 1.5 MB 00:00 2024-03-30T03:50:13.086 INFO:teuthology.orchestra.run.smithi023.stdout:(127/146): oniguruma-6.8.2-2.1.el8_9.x86_64.rpm 979 kB/s | 188 kB 00:00 2024-03-30T03:50:13.191 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-30T03:50:13.191 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-30T03:50:13.211 INFO:teuthology.orchestra.run.smithi023.stdout:(128/146): postgresql-contrib-10.23-4.module+el 5.3 MB/s | 812 kB 00:00 2024-03-30T03:50:13.379 INFO:teuthology.orchestra.run.smithi023.stdout:(129/146): postgresql-server-10.23-4.module+el8 17 MB/s | 5.1 MB 00:00 2024-03-30T03:50:13.570 INFO:teuthology.orchestra.run.smithi023.stdout:(130/146): python3-scipy-1.0.0-21.module+el8.9. 20 MB/s | 14 MB 00:00 2024-03-30T03:50:13.771 INFO:teuthology.orchestra.run.smithi023.stdout:(131/146): librbd1-18.2.2-1018.g470100a4.el8.x8 9.1 MB/s | 3.6 MB 00:00 2024-03-30T03:50:13.796 INFO:teuthology.orchestra.run.smithi023.stdout:(132/146): elfutils-libelf-0.189-3.el8.x86_64.r 1.0 MB/s | 232 kB 00:00 2024-03-30T03:50:13.903 INFO:teuthology.orchestra.run.smithi023.stdout:(133/146): elfutils-libs-0.189-3.el8.x86_64.rpm 2.2 MB/s | 303 kB 00:00 2024-03-30T03:50:14.067 INFO:teuthology.orchestra.run.smithi023.stdout:(134/146): libmount-2.32.1-43.el8.x86_64.rpm 874 kB/s | 237 kB 00:00 2024-03-30T03:50:14.097 INFO:teuthology.orchestra.run.smithi023.stdout:(135/146): libsmartcols-2.32.1-43.el8.x86_64.rp 928 kB/s | 179 kB 00:00 2024-03-30T03:50:14.239 INFO:teuthology.orchestra.run.smithi023.stdout:(136/146): libuuid-2.32.1-43.el8.x86_64.rpm 575 kB/s | 99 kB 00:00 2024-03-30T03:50:14.323 INFO:teuthology.orchestra.run.smithi023.stdout:(137/146): util-linux-2.32.1-43.el8.x86_64.rpm 11 MB/s | 2.5 MB 00:00 2024-03-30T03:50:14.348 INFO:teuthology.orchestra.run.smithi023.stdout:(138/146): util-linux-user-2.32.1-43.el8.x86_64 947 kB/s | 103 kB 00:00 2024-03-30T03:50:14.394 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-30T03:50:14.404 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-30T03:50:14.490 INFO:teuthology.orchestra.run.smithi023.stdout:(139/146): librados2-18.2.2-1018.g470100a4.el8. 2.9 MB/s | 3.7 MB 00:01 2024-03-30T03:50:14.523 INFO:teuthology.orchestra.run.smithi023.stdout:(140/146): libblkid-2.32.1-43.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2024-03-30T03:50:14.565 INFO:teuthology.orchestra.run.smithi023.stdout:(141/146): iproute-6.2.0-5.el8_9.x86_64.rpm 3.4 MB/s | 854 kB 00:00 2024-03-30T03:50:14.633 INFO:teuthology.orchestra.run.smithi023.stdout:(142/146): libfdisk-2.32.1-43.el8.x86_64.rpm 1.7 MB/s | 254 kB 00:00 2024-03-30T03:50:14.699 INFO:teuthology.orchestra.run.smithi023.stdout:(143/146): openssl-1.1.1k-12.el8_9.x86_64.rpm 4.0 MB/s | 711 kB 00:00 2024-03-30T03:50:14.758 INFO:teuthology.orchestra.run.smithi023.stdout:(144/146): openssl-libs-1.1.1k-12.el8_9.x86_64. 7.7 MB/s | 1.5 MB 00:00 2024-03-30T03:50:14.791 INFO:teuthology.orchestra.run.smithi023.stdout:(145/146): xfsdump-3.1.8-7.el8_9.x86_64.rpm 2.1 MB/s | 333 kB 00:00 2024-03-30T03:50:14.883 INFO:teuthology.orchestra.run.smithi023.stdout:(146/146): xfsprogs-5.0.0-12.el8.x86_64.rpm 5.8 MB/s | 1.1 MB 00:00 2024-03-30T03:50:14.886 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:50:14.886 INFO:teuthology.orchestra.run.smithi023.stdout:Total 14 MB/s | 213 MB 00:15 2024-03-30T03:50:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-03-30T03:50:16.830 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T03:50:17.118 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-30T03:50:17.125 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:17.326 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:17.489 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T03:50:17.595 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-30T03:50:17.595 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-30T03:50:17.847 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:18.089 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:18.150 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T03:50:18.538 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-30T03:50:18.772 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T03:50:18.874 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T03:50:19.047 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:19.292 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-30T03:50:19.305 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-30T03:50:19.439 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:19.630 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:20.027 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:20.205 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:20.331 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:20.471 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-30T03:50:20.555 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:20.606 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T03:50:20.653 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:20.869 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:20.957 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:50:21.071 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:21.238 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:21.307 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T03:50:21.326 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:21.504 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:21.528 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T03:50:21.578 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:21.771 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:21.808 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T03:50:21.845 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T03:50:22.102 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T03:50:22.312 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-30T03:50:22.350 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T03:50:22.378 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 6/162 2024-03-30T03:50:22.587 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T03:50:22.589 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 7/162 2024-03-30T03:50:22.737 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:22.738 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:22.896 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:23.038 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:23.118 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:23.141 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:23.283 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-30T03:50:23.378 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:23.557 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-30T03:50:23.795 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:23.801 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-30T03:50:23.970 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:23.988 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T03:50:24.094 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:24.134 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T03:50:24.289 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T03:50:24.295 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:24.631 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:24.911 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T03:50:24.971 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:50:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T03:50:25.131 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-30T03:50:25.253 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:25.322 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:25.330 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/1 2024-03-30T03:50:25.585 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T03:50:25.586 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:25.717 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T03:50:25.871 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openssl-libs-1:1.1.1k-12.el8_9.x86_64 1/162 2024-03-30T03:50:25.874 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T03:50:25.888 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T03:50:26.057 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T03:50:26.067 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.13.0-2.el8.x86_64 2/162 2024-03-30T03:50:26.259 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:26.423 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libuuid-2.32.1-43.el8.x86_64 3/162 2024-03-30T03:50:26.466 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T03:50:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T03:50:26.626 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:26.710 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.32.1-43.el8.x86_64 4/162 2024-03-30T03:50:26.749 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T03:50:26.835 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-30T03:50:26.890 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/162 2024-03-30T03:50:26.903 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:27.009 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:27.025 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-30T03:50:27.102 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.9.0-5.el8_9.x86_64 6/162 2024-03-30T03:50:27.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:27.272 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:27.281 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T03:50:27.343 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T03:50:27.468 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T03:50:27.478 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-30T03:50:27.481 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 7/162 2024-03-30T03:50:27.606 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T03:50:27.615 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 21/162 2024-03-30T03:50:27.701 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:27.817 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-30T03:50:27.868 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T03:50:28.062 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T03:50:28.183 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 8/162 2024-03-30T03:50:28.436 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:28.440 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 22/162 2024-03-30T03:50:28.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-30T03:50:28.897 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:50:29.067 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:29.247 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 23/162 2024-03-30T03:50:29.264 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:50:29.428 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.15-6.el8.x86_64 24/162 2024-03-30T03:50:29.650 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:29.774 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:50:29.805 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T03:50:30.108 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 24/162 2024-03-30T03:50:30.122 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-30T03:50:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:50:30.400 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/162 2024-03-30T03:50:30.583 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 26/162 2024-03-30T03:50:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 13/162 2024-03-30T03:50:30.747 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 27/162 2024-03-30T03:50:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpq-13.11-1.el8.x86_64 14/162 2024-03-30T03:50:31.071 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/162 2024-03-30T03:50:31.116 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-30T03:50:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.189-3.el8.x86_64 15/162 2024-03-30T03:50:31.383 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 29/162 2024-03-30T03:50:31.450 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-30T03:50:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 16/162 2024-03-30T03:50:31.621 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T03:50:31.687 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-30T03:50:31.825 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T03:50:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 17/162 2024-03-30T03:50:31.939 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T03:50:31.963 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T03:50:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 18/162 2024-03-30T03:50:32.197 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T03:50:32.256 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T03:50:32.333 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:32.384 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T03:50:32.451 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:50:32.601 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T03:50:32.650 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:32.737 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 20/162 2024-03-30T03:50:32.816 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:32.893 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:32.899 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 21/162 2024-03-30T03:50:33.239 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 22/162 2024-03-30T03:50:33.267 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 34/162 2024-03-30T03:50:33.437 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:33.495 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:33.616 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 35/162 2024-03-30T03:50:33.623 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/162 2024-03-30T03:50:33.692 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T03:50:33.833 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T03:50:33.922 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 36/162 2024-03-30T03:50:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T03:50:34.001 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 24/162 2024-03-30T03:50:34.126 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-30T03:50:34.127 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:34.167 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T03:50:34.617 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:34.899 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 25/162 2024-03-30T03:50:34.911 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:34.938 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:35.066 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T03:50:35.147 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-16.8-9.el8.noarch 49/162 2024-03-30T03:50:35.303 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T03:50:35.390 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 50/162 2024-03-30T03:50:35.691 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-7.el8.x86_64 51/162 2024-03-30T03:50:35.785 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-threads-0.3.15-6.el8.x86_64 26/162 2024-03-30T03:50:35.870 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 37/162 2024-03-30T03:50:35.910 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : automake-1.16.1-8.el8.noarch 52/162 2024-03-30T03:50:35.949 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T03:50:36.136 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T03:50:36.202 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 38/162 2024-03-30T03:50:36.208 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 53/162 2024-03-30T03:50:36.359 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 54/162 2024-03-30T03:50:36.454 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T03:50:36.551 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 55/162 2024-03-30T03:50:36.721 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T03:50:36.746 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 56/162 2024-03-30T03:50:36.863 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openblas-0.3.15-6.el8.x86_64 27/162 2024-03-30T03:50:36.924 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T03:50:36.940 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.7-17.el8.x86_64 57/162 2024-03-30T03:50:37.118 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 58/162 2024-03-30T03:50:37.124 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 28/162 2024-03-30T03:50:37.124 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T03:50:37.324 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 59/162 2024-03-30T03:50:37.369 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 29/162 2024-03-30T03:50:37.373 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:37.450 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:37.506 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 60/162 2024-03-30T03:50:37.655 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 30/162 2024-03-30T03:50:37.698 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 61/162 2024-03-30T03:50:37.709 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:37.785 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:37.857 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 31/162 2024-03-30T03:50:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 32/162 2024-03-30T03:50:38.030 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 62/162 2024-03-30T03:50:38.049 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T03:50:38.237 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T03:50:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 63/162 2024-03-30T03:50:38.327 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 33/162 2024-03-30T03:50:38.434 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:38.532 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 64/162 2024-03-30T03:50:38.540 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T03:50:38.731 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 65/162 2024-03-30T03:50:38.872 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:38.946 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 66/162 2024-03-30T03:50:39.218 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:39.236 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:39.326 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 67/162 2024-03-30T03:50:39.438 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.7-17.el8.x86_64 49/162 2024-03-30T03:50:39.647 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 68/162 2024-03-30T03:50:39.665 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 50/162 2024-03-30T03:50:39.898 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 51/162 2024-03-30T03:50:39.934 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 69/162 2024-03-30T03:50:40.071 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 52/162 2024-03-30T03:50:40.176 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 70/162 2024-03-30T03:50:40.313 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 53/162 2024-03-30T03:50:40.362 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 71/162 2024-03-30T03:50:40.606 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 72/162 2024-03-30T03:50:40.638 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 54/162 2024-03-30T03:50:40.705 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 34/162 2024-03-30T03:50:40.872 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 73/162 2024-03-30T03:50:40.930 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 55/162 2024-03-30T03:50:41.052 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 74/162 2024-03-30T03:50:41.092 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 35/162 2024-03-30T03:50:41.181 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 56/162 2024-03-30T03:50:41.296 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 75/162 2024-03-30T03:50:41.330 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 57/162 2024-03-30T03:50:41.490 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 76/162 2024-03-30T03:50:41.579 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 58/162 2024-03-30T03:50:41.752 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 77/162 2024-03-30T03:50:41.942 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T03:50:41.958 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 59/162 2024-03-30T03:50:42.129 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T03:50:42.212 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 36/162 2024-03-30T03:50:42.307 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 60/162 2024-03-30T03:50:42.418 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T03:50:42.583 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 37/162 2024-03-30T03:50:42.616 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 61/162 2024-03-30T03:50:42.649 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T03:50:42.805 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T03:50:42.816 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 38/162 2024-03-30T03:50:42.875 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 62/162 2024-03-30T03:50:42.962 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T03:50:43.079 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el8.noarch 39/162 2024-03-30T03:50:43.102 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/162 2024-03-30T03:50:43.202 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T03:50:43.333 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 40/162 2024-03-30T03:50:43.339 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 64/162 2024-03-30T03:50:43.405 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T03:50:43.531 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 41/162 2024-03-30T03:50:43.588 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 65/162 2024-03-30T03:50:43.671 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T03:50:43.777 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 42/162 2024-03-30T03:50:43.818 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 66/162 2024-03-30T03:50:43.896 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T03:50:44.025 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:44.045 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 67/162 2024-03-30T03:50:44.048 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T03:50:44.135 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libmount-2.32.1-43.el8.x86_64 43/162 2024-03-30T03:50:44.242 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 68/162 2024-03-30T03:50:44.322 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T03:50:44.373 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:44.487 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 69/162 2024-03-30T03:50:44.489 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libfdisk-2.32.1-43.el8.x86_64 44/162 2024-03-30T03:50:44.538 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T03:50:44.694 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.32.1-43.el8.x86_64 45/162 2024-03-30T03:50:44.729 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-16.8-9.el8.noarch 70/162 2024-03-30T03:50:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T03:50:44.920 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 46/162 2024-03-30T03:50:44.959 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-30T03:50:44.997 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T03:50:45.027 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 71/162 2024-03-30T03:50:45.106 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:45.208 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T03:50:45.350 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-7.el8.x86_64 72/162 2024-03-30T03:50:45.407 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T03:50:45.623 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T03:50:45.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.1-8.el8.noarch 73/162 2024-03-30T03:50:45.754 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libsmartcols-2.32.1-43.el8.x86_64 47/162 2024-03-30T03:50:45.850 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-43.el8.x86_64 74/162 2024-03-30T03:50:45.913 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 74/162 2024-03-30T03:50:46.082 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 75/162 2024-03-30T03:50:46.115 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:46.161 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-2.32.1-43.el8.x86_64 48/162 2024-03-30T03:50:46.308 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 76/162 2024-03-30T03:50:46.315 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-30T03:50:46.420 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: oniguruma-6.8.2-2.1.el8_9.x86_64 49/162 2024-03-30T03:50:46.551 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 77/162 2024-03-30T03:50:46.671 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-7.el8.x86_64 50/162 2024-03-30T03:50:46.691 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T03:50:46.717 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T03:50:46.938 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T03:50:46.961 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.1-8.el8.noarch 51/162 2024-03-30T03:50:46.996 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T03:50:47.181 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-43.el8.x86_64 52/162 2024-03-30T03:50:47.233 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T03:50:47.280 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 52/162 2024-03-30T03:50:47.288 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T03:50:47.449 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/162 2024-03-30T03:50:47.508 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T03:50:47.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T03:50:47.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 54/162 2024-03-30T03:50:47.754 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T03:50:47.776 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T03:50:47.803 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 55/162 2024-03-30T03:50:47.919 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T03:50:47.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T03:50:48.012 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.7-17.el8.x86_64 56/162 2024-03-30T03:50:48.159 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T03:50:48.197 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 57/162 2024-03-30T03:50:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T03:50:48.416 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 58/162 2024-03-30T03:50:48.594 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T03:50:48.604 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 59/162 2024-03-30T03:50:48.784 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T03:50:48.828 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T03:50:48.830 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 60/162 2024-03-30T03:50:49.022 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T03:50:49.125 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:50:49.160 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 61/162 2024-03-30T03:50:49.200 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:50:49.236 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T03:50:49.401 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T03:50:49.438 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 62/162 2024-03-30T03:50:49.494 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T03:50:49.696 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 63/162 2024-03-30T03:50:49.696 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T03:50:49.911 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 64/162 2024-03-30T03:50:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T03:50:50.120 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 65/162 2024-03-30T03:50:50.198 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T03:50:50.456 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T03:50:50.596 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 66/162 2024-03-30T03:50:50.714 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T03:50:50.922 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-3.2.1-7.el8_9.x86_64 67/162 2024-03-30T03:50:51.272 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 68/162 2024-03-30T03:50:51.524 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 69/162 2024-03-30T03:50:51.713 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 70/162 2024-03-30T03:50:51.819 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T03:50:52.005 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 71/162 2024-03-30T03:50:52.169 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T03:50:52.291 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 72/162 2024-03-30T03:50:52.423 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T03:50:52.525 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 73/162 2024-03-30T03:50:52.750 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T03:50:52.792 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 74/162 2024-03-30T03:50:52.993 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 75/162 2024-03-30T03:50:53.008 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T03:50:53.085 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:50:53.232 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T03:50:53.291 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:50:53.312 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : krb5-devel-1.18.2-26.el8_9.x86_64 76/162 2024-03-30T03:50:53.526 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-16.8-9.el8.noarch 77/162 2024-03-30T03:50:53.725 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 78/162 2024-03-30T03:50:53.883 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:50:53.977 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 79/162 2024-03-30T03:50:54.063 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T03:50:54.186 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:50:54.231 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:50:54.260 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:50:54.298 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 80/162 2024-03-30T03:50:54.349 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:50:54.399 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:50:54.544 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 81/162 2024-03-30T03:50:54.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T03:50:54.728 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 82/162 2024-03-30T03:50:54.945 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 83/162 2024-03-30T03:50:55.191 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 84/162 2024-03-30T03:50:55.408 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 85/162 2024-03-30T03:50:55.598 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 86/162 2024-03-30T03:50:55.785 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 87/162 2024-03-30T03:50:55.946 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 88/162 2024-03-30T03:50:56.224 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 89/162 2024-03-30T03:50:56.403 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 90/162 2024-03-30T03:50:56.612 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 91/162 2024-03-30T03:50:56.900 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-2.6-1.el8.noarch 92/162 2024-03-30T03:50:57.131 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 93/162 2024-03-30T03:50:57.343 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 94/162 2024-03-30T03:50:57.601 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 95/162 2024-03-30T03:50:58.977 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 96/162 2024-03-30T03:50:59.317 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 97/162 2024-03-30T03:50:59.324 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:50:59.531 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:50:59.577 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 98/162 2024-03-30T03:50:59.909 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 99/162 2024-03-30T03:51:00.120 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:51:00.412 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:51:00.517 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:51:00.680 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:51:00.990 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 100/162 2024-03-30T03:51:01.251 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 101/162 2024-03-30T03:51:02.110 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 102/162 2024-03-30T03:51:02.337 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:51:02.438 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 103/162 2024-03-30T03:51:02.630 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 104/162 2024-03-30T03:51:07.770 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:51:08.054 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:51:08.783 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 105/162 2024-03-30T03:51:09.056 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:51:09.229 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 106/162 2024-03-30T03:51:09.361 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /run 2024-03-30T03:51:13.850 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-03-30T03:51:13.851 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-03-30T03:51:13.851 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-03-30T03:51:13.851 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-03-30T03:51:13.851 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2024-03-30T03:51:13.851 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:51:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:15.840 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:16.016 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:16.235 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:16.896 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:17.054 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:17.072 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:19.001 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:19.118 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T03:51:19.309 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T03:51:19.525 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:19.753 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:20.056 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:20.381 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /run 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2024-03-30T03:51:20.382 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:51:20.753 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:21.021 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:21.178 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:21.272 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:21.669 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T03:51:21.848 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:21.887 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:22.069 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:22.081 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:22.380 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:22.608 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:22.829 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:23.019 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:23.345 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T03:51:23.494 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:23.539 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:23.574 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:23.694 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:23.712 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:23.744 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:23.788 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:23.944 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:24.291 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:24.494 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:25.495 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:25.644 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:25.789 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T03:51:26.006 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T03:51:26.272 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:26.499 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:26.766 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:27.730 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:27.901 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:27.995 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:28.353 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T03:51:28.524 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:28.562 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:28.759 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:28.772 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:28.960 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:29.913 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T03:51:30.049 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T03:51:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-30T03:51:30.282 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:30.317 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:30.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-30T03:51:30.526 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:30.569 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:30.776 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:30.895 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 129/162 2024-03-30T03:51:31.123 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 107/162 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /run 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /dev 2024-03-30T03:51:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:51:31.212 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-30T03:51:31.309 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:31.451 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T03:51:31.464 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T03:51:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:31.704 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T03:51:31.939 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T03:51:32.102 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T03:51:33.467 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 108/162 2024-03-30T03:51:33.685 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:33.966 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 109/162 2024-03-30T03:51:34.133 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:34.403 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:34.676 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:34.706 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 110/162 2024-03-30T03:51:34.716 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:34.875 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:34.919 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 111/162 2024-03-30T03:51:35.095 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T03:51:35.466 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T03:51:35.679 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T03:51:35.780 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:35.886 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T03:51:36.092 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T03:51:36.294 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T03:51:36.352 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T03:51:36.635 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 127/162 2024-03-30T03:51:36.652 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T03:51:36.866 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T03:51:36.866 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-25.el8.x86_64 128/162 2024-03-30T03:51:36.943 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T03:51:37.167 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T03:51:37.182 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T03:51:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T03:51:37.359 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T03:51:37.494 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-30T03:51:37.551 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 128/162 2024-03-30T03:51:37.699 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T03:51:37.846 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 129/162 2024-03-30T03:51:37.879 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:37.885 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T03:51:38.046 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T03:51:38.046 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:51:38.074 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 112/162 2024-03-30T03:51:38.149 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 130/162 2024-03-30T03:51:38.205 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 113/162 2024-03-30T03:51:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:51:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:51:38.372 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T03:51:38.402 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 114/162 2024-03-30T03:51:38.545 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:51:38.545 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:51:38.569 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T03:51:38.677 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:38.787 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T03:51:38.798 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:51:38.960 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 115/162 2024-03-30T03:51:39.002 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T03:51:39.025 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T03:51:39.177 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:51:39.200 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:39.426 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T03:51:39.427 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:51:39.669 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:51:39.669 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:51:39.935 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:51:39.936 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:51:40.177 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:51:40.177 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:51:40.357 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 116/162 2024-03-30T03:51:40.402 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:51:40.402 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:51:40.515 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:40.644 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:51:40.653 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 117/162 2024-03-30T03:51:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T03:51:40.794 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:51:41.101 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 118/162 2024-03-30T03:51:41.268 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:41.326 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:41.340 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 119/162 2024-03-30T03:51:41.523 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:41.555 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 120/162 2024-03-30T03:51:41.650 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T03:51:41.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T03:51:41.669 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T03:51:41.670 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T03:51:41.682 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T03:51:41.683 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T03:51:41.684 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T03:51:41.685 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 72/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 75/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 77/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 78/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 79/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 81/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 82/162 2024-03-30T03:51:41.686 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 83/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 84/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 85/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 86/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 87/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 88/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 89/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 90/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 91/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 92/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 93/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 94/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 95/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 96/162 2024-03-30T03:51:41.687 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 97/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 98/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 99/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 100/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 101/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 102/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 103/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 104/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : bison-3.0.4-10.el8.x86_64 105/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flex-2.6.1-9.el8.x86_64 106/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 107/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 108/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 109/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 110/162 2024-03-30T03:51:41.688 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 111/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 112/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 113/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 114/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 115/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 116/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 117/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 118/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 119/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 120/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : automake-1.16.1-8.el8.noarch 121/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpq-13.11-1.el8.x86_64 122/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-7.el8.x86_64 123/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 124/162 2024-03-30T03:51:41.689 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 125/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 126/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 127/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 128/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 129/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 130/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T03:51:41.690 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T03:51:41.691 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T03:51:41.692 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T03:51:41.693 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:42.126 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T03:51:42.573 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T03:51:42.828 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T03:51:42.995 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 121/162 2024-03-30T03:51:43.024 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T03:51:43.192 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:43.257 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 122/162 2024-03-30T03:51:43.266 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T03:51:43.490 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:43.565 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1018.g47010 123/162 2024-03-30T03:51:43.569 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T03:51:43.757 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:43.884 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T03:51:44.099 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-30T03:51:44.110 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 143/162 2024-03-30T03:51:44.184 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 124/162 2024-03-30T03:51:44.295 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 144/162 2024-03-30T03:51:44.386 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:44.554 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.1-9.el8.x86_64 145/162 2024-03-30T03:51:44.636 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 145/162 2024-03-30T03:51:44.882 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.0.4-10.el8.x86_64 146/162 2024-03-30T03:51:44.896 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 146/162 2024-03-30T03:51:45.126 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T03:51:45.317 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T03:51:45.511 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T03:51:45.511 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:51:45.810 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:51:45.810 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:51:46.102 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:51:46.102 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:51:46.371 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:51:46.600 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T03:51:46.825 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:51:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T03:51:47.108 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:51:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:51:47.368 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:51:47.642 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:51:47.642 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:51:47.892 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:51:47.892 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:51:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:51:48.150 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:51:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:51:48.600 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T03:51:48.600 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:51:49.505 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:51:49.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T03:51:49.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T03:51:49.506 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T03:51:49.507 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T03:51:49.508 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T03:51:49.520 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T03:51:49.521 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T03:51:49.522 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 72/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 74/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 75/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.1-9.el8.x86_64 79/162 2024-03-30T03:51:49.523 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 80/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 81/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 82/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 83/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 84/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 85/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 86/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 88/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 90/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 91/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 92/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 93/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.1-8.el8.noarch 94/162 2024-03-30T03:51:49.524 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpq-13.11-1.el8.x86_64 95/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-7.el8.x86_64 96/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 97/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 98/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 99/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 100/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 101/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 102/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 103/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 104/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 106/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 107/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 108/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 109/162 2024-03-30T03:51:49.525 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 110/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 111/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 112/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 113/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 114/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 115/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 116/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 117/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 118/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 119/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 120/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 121/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 122/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 123/162 2024-03-30T03:51:49.526 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 124/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 125/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 126/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 127/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 128/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 129/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 130/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T03:51:49.527 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T03:51:49.528 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T03:51:49.529 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T03:51:49.561 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 125/162 2024-03-30T03:51:50.913 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 126/162 2024-03-30T03:51:51.253 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:1.1.1k-12.el8_9.x86_64 127/162 2024-03-30T03:51:51.547 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.189-3.el8.x86_64 128/162 2024-03-30T03:51:52.009 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 129/162 2024-03-30T03:51:52.410 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 130/162 2024-03-30T03:51:52.663 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T03:51:52.695 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 131/162 2024-03-30T03:51:53.062 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.32.1-43.el8.x86_64 132/162 2024-03-30T03:51:53.252 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 133/162 2024-03-30T03:51:53.473 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 134/162 2024-03-30T03:51:57.203 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:57.772 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-10.23-4.module+el8.9.0+21317+d 135/162 2024-03-30T03:51:58.231 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.189-3.el8.x86_64 136/162 2024-03-30T03:51:58.602 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : iproute-6.2.0-5.el8_9.x86_64 137/162 2024-03-30T03:51:58.650 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T03:51:58.651 INFO:teuthology.orchestra.run.smithi154.stdout:Installed products updated. 2024-03-30T03:51:58.893 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.8-7.el8_9.x86_64 138/162 2024-03-30T03:51:59.081 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 139/162 2024-03-30T03:51:59.265 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:51:59.265 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:51:59.266 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.267 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.268 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: jq-1.6-7.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.269 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T03:51:59.270 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:51:59.271 INFO:teuthology.orchestra.run.smithi154.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T03:51:59.272 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T03:51:59.273 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T03:51:59.274 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T03:51:59.275 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T03:51:59.276 INFO:teuthology.orchestra.run.smithi154.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T03:51:59.277 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-03-30T03:51:59.277 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-03-30T03:51:59.374 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 140/162 2024-03-30T03:51:59.495 DEBUG:teuthology.parallel:result is None 2024-03-30T03:51:59.785 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 141/162 2024-03-30T03:52:00.156 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:1.1.1k-12.el8_9.x86_64 142/162 2024-03-30T03:52:00.427 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T03:52:00.543 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 143/162 2024-03-30T03:52:00.779 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T03:52:00.811 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 144/162 2024-03-30T03:52:00.979 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T03:52:01.006 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-13.e 145/162 2024-03-30T03:52:01.174 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 146/162 2024-03-30T03:52:01.378 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.186-1.el8.x86_64 147/162 2024-03-30T03:52:01.613 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:1.1.1k-6.el8_5.x86_64 148/162 2024-03-30T03:52:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.8-4.el8.x86_64 149/162 2024-03-30T03:52:01.818 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:52:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 150/162 2024-03-30T03:52:02.134 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:52:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 151/162 2024-03-30T03:52:02.445 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:52:02.792 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 152/162 2024-03-30T03:52:03.089 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 153/162 2024-03-30T03:52:03.274 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:52:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.32.1-35.el8.x86_64 155/162 2024-03-30T03:52:03.547 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:52:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libfdisk-2.32.1-35.el8.x86_64 156/162 2024-03-30T03:52:03.866 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:52:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libmount-2.32.1-35.el8.x86_64 157/162 2024-03-30T03:52:04.091 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:52:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.32.1-35.el8.x86_64 158/162 2024-03-30T03:52:04.374 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:52:05.582 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libuuid-2.32.1-35.el8.x86_64 159/162 2024-03-30T03:52:05.583 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:52:05.865 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libsmartcols-2.32.1-35.el8.x86_64 160/162 2024-03-30T03:52:06.029 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T03:52:06.029 INFO:teuthology.orchestra.run.smithi193.stdout:Installed products updated. 2024-03-30T03:52:06.065 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.186-1.el8.x86_64 161/162 2024-03-30T03:52:06.065 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:52:06.649 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: openssl-libs-1:1.1.1k-6.el8_5.x86_64 162/162 2024-03-30T03:52:06.649 INFO:teuthology.orchestra.run.smithi023.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-30T03:52:06.650 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:52:06.661 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:52:06.661 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:06.662 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.663 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.664 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-7.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T03:52:06.665 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T03:52:06.666 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:06.667 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T03:52:06.668 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T03:52:06.669 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.670 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T03:52:06.671 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:52:06.672 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-30T03:52:06.878 DEBUG:teuthology.parallel:result is None 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:18.2.2-1018.g470100a4.el8.x86_64 1/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 3/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 4/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1018.g47010 5/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 6/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 7/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 8/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 9/162 2024-03-30T03:52:07.343 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 10/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 11/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 12/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 13/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_ 14/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 15/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 16/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:18.2.2-1018.g470100a4.el8.x86_6 17/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86 18/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 19/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1018.g470100a4.el 20/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:18.2.2-1018.g470100a4.el8. 21/162 2024-03-30T03:52:07.344 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_6 22/162 2024-03-30T03:52:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 23/162 2024-03-30T03:52:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 24/162 2024-03-30T03:52:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 25/162 2024-03-30T03:52:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 26/162 2024-03-30T03:52:07.345 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 27/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 28/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1018.g470100a4. 29/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noa 30/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.n 31/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1018.g470 32/162 2024-03-30T03:52:07.359 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el 33/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 34/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.e 35/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 36/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:18.2.2-1018.g470100a4.el8.noarch 37/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 38/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/162 2024-03-30T03:52:07.360 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/162 2024-03-30T03:52:07.361 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/162 2024-03-30T03:52:07.362 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-16.8-9.el8.noarch 72/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 73/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 75/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 76/162 2024-03-30T03:52:07.363 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 77/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 78/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 80/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 81/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 82/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 83/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 84/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 85/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 86/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 87/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 88/162 2024-03-30T03:52:07.364 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 89/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 90/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.32.1-43.el8.x86_64 91/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 92/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 93/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-13.e 94/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.189-3.el8.x86_64 95/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : krb5-devel-1.18.2-26.el8_9.x86_64 96/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:1.1.1k-12.el8_9.x86_64 97/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.9.0-5.el8_9.x86_64 98/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-3.2.1-7.el8_9.x86_64 99/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 100/162 2024-03-30T03:52:07.365 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 101/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 103/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 104/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 105/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.0.4-10.el8.x86_64 106/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.1-9.el8.x86_64 107/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 108/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 109/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 110/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 111/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 112/162 2024-03-30T03:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 113/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 114/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 115/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 116/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 117/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 118/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-threads-0.3.15-6.el8.x86_64 119/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 120/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.15-6.el8.x86_64 121/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.1-8.el8.noarch 122/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpq-13.11-1.el8.x86_64 123/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-7.el8.x86_64 124/162 2024-03-30T03:52:07.367 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 125/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c3 126/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.8.2-2.1.el8_9.x86_64 127/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a 128/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-10.23-4.module+el8.9.0+21317+ 129/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-10.23-4.module+el8.9.0+21317+d 130/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:18.2.2-1018.g470100a4.el8.x86_64 131/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 132/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 133/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 134/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el8.x86_64 135/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.186-1.el8.x86_64 136/162 2024-03-30T03:52:07.368 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el8.x86_64 137/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.186-1.el8.x86_64 138/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.32.1-43.el8.x86_64 139/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.32.1-35.el8.x86_64 140/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.32.1-43.el8.x86_64 141/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.32.1-35.el8.x86_64 142/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.32.1-43.el8.x86_64 143/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.32.1-35.el8.x86_64 144/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.32.1-43.el8.x86_64 145/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.32.1-35.el8.x86_64 146/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.32.1-43.el8.x86_64 147/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.32.1-35.el8.x86_64 148/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-6.2.0-5.el8_9.x86_64 149/162 2024-03-30T03:52:07.369 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 150/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.32.1-43.el8.x86_64 151/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.32.1-35.el8.x86_64 152/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.32.1-43.el8.x86_64 153/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.32.1-35.el8.x86_64 154/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:1.1.1k-12.el8_9.x86_64 155/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:1.1.1k-6.el8_5.x86_64 156/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:1.1.1k-12.el8_9.x86_64 157/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:1.1.1k-6.el8_5.x86_64 158/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.8-7.el8_9.x86_64 159/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.8-4.el8.x86_64 160/162 2024-03-30T03:52:07.370 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 161/162 2024-03-30T03:52:20.680 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 162/162 2024-03-30T03:52:20.680 INFO:teuthology.orchestra.run.smithi023.stdout:Installed products updated. 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.189-3.el8.x86_64 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.189-3.el8.x86_64 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: iproute-6.2.0-5.el8_9.x86_64 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.359 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.8-7.el8_9.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-5.0.0-12.el8.x86_64 2024-03-30T03:52:21.360 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-initscripts-updown-1:1.40.16-13.el8_9.noarch 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.1-8.el8.noarch 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.0.4-10.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.361 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.362 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:18.2.2-1018.g470100a4.el8.noarch 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.189-3.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.1-9.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.7-17.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-7.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: krb5-devel-1.18.2-26.el8_9.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-03-30T03:52:21.363 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-8.5.0-20.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.2-3.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libpq-13.11-1.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-8.5.0-20.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.9.0-5.el8_9.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-0.11.4-3.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.364 INFO:teuthology.orchestra.run.smithi023.stdout: libselinux-devel-2.9-5.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libsepol-devel-2.9-3.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-25.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.3.1-3.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.32.1-43.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libverto-devel-0.3.0-5.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.8.2-2.1.el8_9.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.15-6.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-threads-0.3.15-6.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:1.1.1k-12.el8_9.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-devel-10.32-2.el8.x86_64 2024-03-30T03:52:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf16-10.32-2.el8.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: pcre2-utf32-10.32-2.el8.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-10.23-4.module+el8.9.0+21317+d2c6bf8a.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.5.1-7.el8.noarch 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.366 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-3.2.1-7.el8_9.x86_64 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-6.2-6.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.10.1-3.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-03-30T03:52:21.367 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.0.6-14.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-2.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-16.8-9.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-2.6-1.el8.noarch 2024-03-30T03:52:21.368 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.14-14.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2017.2-11.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.4.1-12.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el8.noarch 2024-03-30T03:52:21.369 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.0.0-21.module+el8.9.0+20976+d3c38525.x86_64 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.370 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:18.2.2-1018.g470100a4.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-1.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.13.0-2.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-43.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2024-03-30T03:52:21.371 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2024-03-30T03:52:21.372 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2024-03-30T03:52:21.372 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:52:21.372 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-30T03:52:21.756 DEBUG:teuthology.parallel:result is None 2024-03-30T03:52:21.756 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:21.996 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T03:52:22.050 INFO:teuthology.orchestra.run.smithi023.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.051 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.051 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T03:52:22.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:22.195 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T03:52:22.226 INFO:teuthology.orchestra.run.smithi154.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.226 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.226 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T03:52:22.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:22.447 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-30T03:52:22.477 INFO:teuthology.orchestra.run.smithi193.stdout:18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.477 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1018.g470100a4.el8 2024-03-30T03:52:22.477 INFO:teuthology.task.install:The correct ceph version 18.2.2-1018.g470100a4 is installed. 2024-03-30T03:52:22.479 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T03:52:22.479 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:52:22.479 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T03:52:22.569 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:52:22.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T03:52:22.618 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:52:22.618 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T03:52:22.670 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T03:52:22.671 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:52:22.671 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T03:52:22.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T03:52:22.839 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:52:22.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T03:52:22.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T03:52:22.943 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:52:22.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T03:52:22.976 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T03:52:23.046 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T03:52:23.047 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:52:23.047 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T03:52:23.112 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T03:52:23.222 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:52:23.222 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T03:52:23.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T03:52:23.326 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:52:23.327 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T03:52:23.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T03:52:23.426 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T03:52:23.426 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:52:23.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T03:52:23.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T03:52:23.599 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:52:23.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T03:52:23.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T03:52:23.703 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:52:23.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T03:52:23.734 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T03:52:23.804 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-30T03:52:23.908 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-30T03:52:23.908 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:23.908 INFO:tasks.cephadm:Cluster fsid is eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:52:23.909 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-30T03:52:23.909 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.154', 'mon.c': '172.21.15.193'} 2024-03-30T03:52:23.909 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-30T03:52:23.909 INFO:tasks.cephadm:First mgr is x 2024-03-30T03:52:23.909 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-30T03:52:23.909 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-30T03:52:23.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-03-30T03:52:24.009 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-03-30T03:52:24.041 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-30T03:52:24.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:24.184 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '470100a4ccbcf7769052bf09a5413881ccf1b97f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/78078/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.2-1018-g470100a4', 'node_name': '172.21.2.5+braggi05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.2-1018.g470100a4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2024-03-29 13:05:59.448245', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240327.051042-reef', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-30T03:52:24.324 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-rishabh-testing-20240327.051042-reef, sha1 470100a4ccbcf7769052bf09a5413881ccf1b97f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T03:52:24.325 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm 2024-03-30T03:52:24.326 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:25.890 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 03:52 /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:25.890 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:26.453 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 03:52 /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:26.453 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240327.051042-reef/470100a4ccbcf7769052bf09a5413881ccf1b97f/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:27.030 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-rw-r--. 1 ubuntu ubuntu 703022 Mar 30 03:52 /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:27.031 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:27.080 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:27.102 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-30T03:52:27.141 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f on all hosts... 2024-03-30T03:52:27.141 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T03:52:27.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T03:52:27.148 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f pull 2024-03-30T03:52:28.018 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T03:52:28.119 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T03:52:28.954 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T03:53:00.415 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-30T03:53:00.416 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-03-30T03:53:04.718 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-30T03:53:06.584 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T03:53:06.584 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T03:53:06.585 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d", 2024-03-30T03:53:06.585 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-30T03:53:06.585 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c123be959ad18653301727c9f149d0b2813c48e5bad48157ed6a183a20cecd35" 2024-03-30T03:53:06.585 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T03:53:06.585 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T03:53:06.612 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-30T03:53:06.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-03-30T03:53:06.719 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-03-30T03:53:06.758 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-30T03:53:06.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-03-30T03:53:06.856 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-03-30T03:53:06.888 INFO:tasks.cephadm:Writing seed config... 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-30T03:53:06.889 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-30T03:53:06.890 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-30T03:53:06.891 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-30T03:53:06.892 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-30T03:53:06.893 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:53:06.893 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-30T03:53:06.934 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eac7e298-ee48-11ee-b647-cb9ed24678a4 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-30T03:53:06.934 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service 2024-03-30T03:53:06.978 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x.service 2024-03-30T03:53:07.022 INFO:tasks.cephadm:Bootstrapping... 2024-03-30T03:53:07.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f -v bootstrap --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-30T03:53:07.043 INFO:journalctl@ceph.mon.a.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:53:07.090 INFO:journalctl@ceph.mgr.x.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:53:07.247 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-30T03:53:07.247 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f', '-v', 'bootstrap', '--fsid', 'eac7e298-ee48-11ee-b647-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-30T03:53:07.295 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.6.1 2024-03-30T03:53:07.295 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-30T03:53:07.295 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-30T03:53:07.339 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.6.1 2024-03-30T03:53:07.339 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-30T03:53:07.339 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-30T03:53:07.353 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T03:53:07.353 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T03:53:07.366 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T03:53:07.366 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:53:07.378 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-30T03:53:07.391 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-30T03:53:07.391 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-30T03:53:07.391 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-30T03:53:07.434 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.6.1 2024-03-30T03:53:07.434 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.6.1 is present 2024-03-30T03:53:07.434 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-30T03:53:07.434 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-30T03:53:07.447 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-30T03:53:07.447 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-30T03:53:07.459 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-30T03:53:07.459 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:53:07.472 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-30T03:53:07.484 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-30T03:53:07.484 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-30T03:53:07.484 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-30T03:53:07.485 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:07.485 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 140050238646760 on /run/cephadm/eac7e298-ee48-11ee-b647-cb9ed24678a4.lock 2024-03-30T03:53:07.485 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 140050238646760 acquired on /run/cephadm/eac7e298-ee48-11ee-b647-cb9ed24678a4.lock 2024-03-30T03:53:07.485 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-30T03:53:07.486 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-30T03:53:07.486 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-30T03:53:07.494 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2024-03-30T03:53:07.494 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-30T03:53:07.501 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-30T03:53:07.501 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-03-30T03:53:07.501 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1756sec hoplimit 64 pref medium 2024-03-30T03:53:07.507 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-30T03:53:07.507 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-30T03:53:07.507 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T03:53:07.507 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-30T03:53:07.508 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2024-03-30T03:53:07.508 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-30T03:53:07.509 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-30T03:53:07.509 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-30T03:53:07.509 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-30T03:53:07.509 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-30T03:53:07.510 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f... 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:86ca209f044bd2e3705adc9da4fc1145309402ec2e51b9b9b051617a85f31417 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:11da15500eca1f826c328eb3265caa30b0a3730510e2a32023084d6fe9cee20c 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:e7c444a08cf749270ac19a49d6d49787223f88c26222d2dfc1d77b830f40791d 2024-03-30T03:53:08.037 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-30T03:53:08.890 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable) 2024-03-30T03:53:08.890 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable) 2024-03-30T03:53:08.890 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-30T03:53:09.435 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-30T03:53:09.435 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-30T03:53:10.003 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQCljAdm/gJpKRAAsbWYe6DStHOt4fHfdKwEsg== 2024-03-30T03:53:10.579 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQCmjAdmJxuFERAAYWTwfg7e+F9WHS3XYImzbA== 2024-03-30T03:53:11.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQCnjAdmZasNFBAARi9rl9n12+kTtgzzRF0Zkw== 2024-03-30T03:53:11.653 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-30T03:53:12.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T03:53:12.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = pacific 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:53:12.228 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-30T03:53:12.845 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-30T03:53:13.091 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-03-30T03:53:13.283 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-30T03:53:13.636 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target. 2024-03-30T03:53:13.637 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target → /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target. 2024-03-30T03:53:13.978 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T03:53:13.979 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service: Unit ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service not loaded. 2024-03-30T03:53:14.179 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target.wants/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service → /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@.service. 2024-03-30T03:53:14.261 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:14 smithi023 systemd[1]: Starting Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:53:14.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:14 smithi023 bash[37724]: a2a15e899bac4169be43a3d1e53a3f5aaa93c5b309f012af4789bda58f63ab9e 2024-03-30T03:53:14.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:14 smithi023 systemd[1]: Started Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:53:14.704 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:14 smithi023 ceph-mon[37762]: mkfs eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:14.709 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T03:53:14.709 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-30T03:53:14.725 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T03:53:14.726 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:53:14.726 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-30T03:53:14.726 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T03:53:14.726 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-30T03:53:14.726 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-30T03:53:14.952 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:14 smithi023 ceph-mon[37762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T03:53:15.643 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.626786s) 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-30T03:53:15.650 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.235+0000 7fe3e1200700 1 Processor -- start 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.235+0000 7fe3e1200700 1 -- start start 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.236+0000 7fe3e1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.236+0000 7fe3e1200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc07a810 con 0x7fe3dc07be20 2024-03-30T03:53:15.651 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.236+0000 7fe3d9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.236+0000 7fe3d9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33244/0 (socket says 172.21.15.23:33244) 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.236+0000 7fe3d9e00700 1 -- 172.21.15.23:0/1747329633 learned_addr learned my addr 172.21.15.23:0/1747329633 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.237+0000 7fe3d9e00700 1 -- 172.21.15.23:0/1747329633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc07a950 con 0x7fe3dc07be20 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.237+0000 7fe3d9e00700 1 --2- 172.21.15.23:0/1747329633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0098a0 tx=0x7fe3cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79702346fc794c55 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.237+0000 7fe3d9400700 1 -- 172.21.15.23:0/1747329633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3cc004160 con 0x7fe3dc07be20 2024-03-30T03:53:15.652 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.237+0000 7fe3d9400700 1 -- 172.21.15.23:0/1747329633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc0042c0 con 0x7fe3dc07be20 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.237+0000 7fe3d9400700 1 -- 172.21.15.23:0/1747329633 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3cc0045a0 con 0x7fe3dc07be20 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 -- 172.21.15.23:0/1747329633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 msgr2=0x7fe3dc07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 --2- 172.21.15.23:0/1747329633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc0098a0 tx=0x7fe3cc009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 -- 172.21.15.23:0/1747329633 shutdown_connections 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 --2- 172.21.15.23:0/1747329633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc07a2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 -- 172.21.15.23:0/1747329633 >> 172.21.15.23:0/1747329633 conn(0x7fe3dc101ab0 msgr2=0x7fe3dc103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 -- 172.21.15.23:0/1747329633 shutdown_connections 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.238+0000 7fe3e1200700 1 -- 172.21.15.23:0/1747329633 wait complete. 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.239+0000 7fe3e1200700 1 Processor -- start 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.239+0000 7fe3e1200700 1 -- start start 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3e1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3e1200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3dc19c090 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3d9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3d9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33246/0 (socket says 172.21.15.23:33246) 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3d9e00700 1 -- 172.21.15.23:0/1205575648 learned_addr learned my addr 172.21.15.23:0/1205575648 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.240+0000 7fe3d9e00700 1 -- 172.21.15.23:0/1205575648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc009550 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d9e00700 1 --2- 172.21.15.23:0/1205575648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc07b340 tx=0x7fe3cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3cc011630 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3cc011790 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3dc19c290 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3cc011a30 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3dc19c730 con 0x7fe3dc07be20 2024-03-30T03:53:15.654 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe3cc022070 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.241+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3cc01c450 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.242+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3dc195970 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.245+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fe3cc01b450 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.321+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe3dc07aef0 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.321+0000 7fe3d2a00700 1 -- 172.21.15.23:0/1205575648 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fe3cc01b5f0 con 0x7fe3dc07be20 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.322+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 msgr2=0x7fe3dc19bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.322+0000 7fe3e1200700 1 --2- 172.21.15.23:0/1205575648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe3dc07b340 tx=0x7fe3cc004750 comp rx=0 tx=0).stop 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.322+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 shutdown_connections 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.322+0000 7fe3e1200700 1 --2- 172.21.15.23:0/1205575648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3dc07be20 0x7fe3dc19bb50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.322+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 >> 172.21.15.23:0/1205575648 conn(0x7fe3dc101ab0 msgr2=0x7fe3dc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.323+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 shutdown_connections 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:15.323+0000 7fe3e1200700 1 -- 172.21.15.23:0/1205575648 wait complete. 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-30T03:53:15.655 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: fsmap 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: mgrmap e1: no daemons active 2024-03-30T03:53:16.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:15 smithi023 ceph-mon[37762]: from='client.? 172.21.15.23:0/1205575648' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-30T03:53:18.199 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:17 smithi023 ceph-mon[37762]: from='client.? 172.21.15.23:0/1067490461' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T03:53:18.199 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:17 smithi023 ceph-mon[37762]: from='client.? 172.21.15.23:0/1067490461' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-30T03:53:18.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:18.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.760+0000 7efcea600700 1 Processor -- start 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.760+0000 7efcea600700 1 -- start start 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.761+0000 7efcea600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.761+0000 7efcea600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce407a810 con 0x7efce407be20 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.761+0000 7efce3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.761+0000 7efce3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33250/0 (socket says 172.21.15.23:33250) 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.761+0000 7efce3400700 1 -- 172.21.15.23:0/2131640103 learned_addr learned my addr 172.21.15.23:0/2131640103 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:18.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.762+0000 7efce3400700 1 -- 172.21.15.23:0/2131640103 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce407a950 con 0x7efce407be20 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.762+0000 7efce3400700 1 --2- 172.21.15.23:0/2131640103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efcd40098a0 tx=0x7efcd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f1d700ccee13ea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.762+0000 7efce2a00700 1 -- 172.21.15.23:0/2131640103 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd4004160 con 0x7efce407be20 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.762+0000 7efce2a00700 1 -- 172.21.15.23:0/2131640103 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd40042c0 con 0x7efce407be20 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.762+0000 7efce2a00700 1 -- 172.21.15.23:0/2131640103 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd40045a0 con 0x7efce407be20 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 -- 172.21.15.23:0/2131640103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 msgr2=0x7efce407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 --2- 172.21.15.23:0/2131640103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7efcd40098a0 tx=0x7efcd4009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 -- 172.21.15.23:0/2131640103 shutdown_connections 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 --2- 172.21.15.23:0/2131640103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce407a2d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 -- 172.21.15.23:0/2131640103 >> 172.21.15.23:0/2131640103 conn(0x7efce4101ab0 msgr2=0x7efce4103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 -- 172.21.15.23:0/2131640103 shutdown_connections 2024-03-30T03:53:18.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.763+0000 7efcea600700 1 -- 172.21.15.23:0/2131640103 wait complete. 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.764+0000 7efcea600700 1 Processor -- start 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.764+0000 7efcea600700 1 -- start start 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efcea600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efcea600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce419c090 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efce3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efce3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33256/0 (socket says 172.21.15.23:33256) 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efce3400700 1 -- 172.21.15.23:0/1067490461 learned_addr learned my addr 172.21.15.23:0/1067490461 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efce3400700 1 -- 172.21.15.23:0/1067490461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4009550 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.765+0000 7efce3400700 1 --2- 172.21.15.23:0/1067490461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efce407b340 tx=0x7efcd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd4011630 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd4011790 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd4011a30 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce419c290 con 0x7efce407be20 2024-03-30T03:53:18.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce419c730 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efcd4022070 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.766+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efcd401c790 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.767+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce4195970 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.769+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7efcd401bcf0 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.842+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7efce407aef0 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.847+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcd401be90 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.847+0000 7efce0c00700 1 -- 172.21.15.23:0/1067490461 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7efcd4037610 con 0x7efce407be20 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 msgr2=0x7efce419bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 --2- 172.21.15.23:0/1067490461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efce407b340 tx=0x7efcd4004750 comp rx=0 tx=0).stop 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 shutdown_connections 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 --2- 172.21.15.23:0/1067490461 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efce407be20 0x7efce419bb50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 >> 172.21.15.23:0/1067490461 conn(0x7efce4101ab0 msgr2=0x7efce4103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 shutdown_connections 2024-03-30T03:53:18.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:17.848+0000 7efcea600700 1 -- 172.21.15.23:0/1067490461 wait complete. 2024-03-30T03:53:18.366 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-30T03:53:19.185 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:18 smithi023 ceph-mon[37762]: from='client.? 172.21.15.23:0/3030091912' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.756+0000 7ff6ffa00700 1 Processor -- start 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6ffa00700 1 -- start start 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6ffa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6ffa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f807a830 con 0x7ff6f807be40 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6fd000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6fd000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52348/0 (socket says 172.21.15.23:52348) 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.757+0000 7ff6fd000700 1 -- 172.21.15.23:0/337732044 learned_addr learned my addr 172.21.15.23:0/337732044 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:19.360 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.758+0000 7ff6fd000700 1 -- 172.21.15.23:0/337732044 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f807a970 con 0x7ff6f807be40 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.758+0000 7ff6fd000700 1 --2- 172.21.15.23:0/337732044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0098a0 tx=0x7ff6ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9754f5c03aeba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.758+0000 7ff6f7400700 1 -- 172.21.15.23:0/337732044 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec004160 con 0x7ff6f807be40 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.758+0000 7ff6f7400700 1 -- 172.21.15.23:0/337732044 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6ec0042c0 con 0x7ff6f807be40 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.758+0000 7ff6f7400700 1 -- 172.21.15.23:0/337732044 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec011a00 con 0x7ff6f807be40 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 -- 172.21.15.23:0/337732044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 msgr2=0x7ff6f807a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 --2- 172.21.15.23:0/337732044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec0098a0 tx=0x7ff6ec009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 -- 172.21.15.23:0/337732044 shutdown_connections 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 --2- 172.21.15.23:0/337732044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f807a2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 -- 172.21.15.23:0/337732044 >> 172.21.15.23:0/337732044 conn(0x7ff6f8101ab0 msgr2=0x7ff6f8103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:19.361 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 -- 172.21.15.23:0/337732044 shutdown_connections 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.759+0000 7ff6ffa00700 1 -- 172.21.15.23:0/337732044 wait complete. 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.760+0000 7ff6ffa00700 1 Processor -- start 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.760+0000 7ff6ffa00700 1 -- start start 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.760+0000 7ff6ffa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.760+0000 7ff6ffa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f819c8e0 con 0x7ff6f807be40 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6fd000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6fd000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52364/0 (socket says 172.21.15.23:52364) 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6fd000700 1 -- 172.21.15.23:0/3030091912 learned_addr learned my addr 172.21.15.23:0/3030091912 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6fd000700 1 -- 172.21.15.23:0/3030091912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec009550 con 0x7ff6f807be40 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6fd000700 1 --2- 172.21.15.23:0/3030091912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009130 tx=0x7ff6ec011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.761+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec003b70 con 0x7ff6f807be40 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f819cae0 con 0x7ff6f807be40 2024-03-30T03:53:19.362 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6ec003cd0 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6ec0197d0 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f819cf80 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff6ec019a10 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.762+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff6ec00b720 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.763+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f810c7e0 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.766+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff6ec02a060 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.838+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff6f810c9a0 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.838+0000 7ff6f5600700 1 -- 172.21.15.23:0/3030091912 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ff6ec019e50 con 0x7ff6f807be40 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.839+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 msgr2=0x7ff6f819c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.839+0000 7ff6ffa00700 1 --2- 172.21.15.23:0/3030091912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec009130 tx=0x7ff6ec011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.839+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 shutdown_connections 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.839+0000 7ff6ffa00700 1 --2- 172.21.15.23:0/3030091912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6f807be40 0x7ff6f819c3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:19.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.839+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 >> 172.21.15.23:0/3030091912 conn(0x7ff6f8101ab0 msgr2=0x7ff6f8102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:19.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.840+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 shutdown_connections 2024-03-30T03:53:19.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:18.840+0000 7ff6ffa00700 1 -- 172.21.15.23:0/3030091912 wait complete. 2024-03-30T03:53:19.364 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-30T03:53:19.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:19 smithi023 systemd[1]: Stopping Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:53:19.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a[37739]: 2024-03-30T03:53:19.659+0000 7f49bc000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T03:53:19.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a[37739]: 2024-03-30T03:53:19.659+0000 7f49bc000700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-30T03:53:20.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:19 smithi023 bash[38094]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T03:53:20.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 systemd[1]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service: Succeeded. 2024-03-30T03:53:20.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 systemd[1]: Stopped Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:53:20.701 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 systemd[1]: Starting Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:53:20.775 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-30T03:53:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 bash[38217]: 8d24ba58831e61b5c3b63c0cf9871c23cd58eb989869b1e43e48316954165513 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 systemd[1]: Started Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: set uid:gid to 167:167 (ceph:ceph) 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable), process ceph-mon, pid 7 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: pidfile_write: ignore empty --pid-file 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: load: jerasure load: lrc 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: RocksDB version: 7.9.2 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Git sha 0 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Compile date 2024-03-29 11:49:16 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: DB SUMMARY 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: DB Session ID: 902OCQLHMWZ0JY651F37 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: CURRENT file: CURRENT 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: IDENTITY file: IDENTITY 2024-03-30T03:53:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98859 ; 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.error_if_exists: 0 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.create_if_missing: 0 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.paranoid_checks: 1 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.env: 0x55de37ef9720 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.fs: PosixFileSystem 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.info_log: 0x55de3a4ad520 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_file_opening_threads: 16 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.statistics: (nil) 2024-03-30T03:53:21.024 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.use_fsync: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_log_file_size: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.keep_log_file_num: 1000 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.recycle_log_file_num: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_fallocate: 1 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_mmap_reads: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_mmap_writes: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.use_direct_reads: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.create_missing_column_families: 0 2024-03-30T03:53:21.025 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.db_log_dir: 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.wal_dir: 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.advise_random_on_open: 1 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.db_write_buffer_size: 0 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.write_buffer_manager: 0x55de39744320 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-30T03:53:21.027 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.rate_limiter: (nil) 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.wal_recovery_mode: 2 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enable_thread_tracking: 0 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enable_pipelined_write: 0 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.unordered_write: 0 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.row_cache: None 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.wal_filter: None 2024-03-30T03:53:21.028 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_ingest_behind: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.two_write_queues: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.manual_wal_flush: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.wal_compression: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.atomic_flush: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.log_readahead_size: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.best_efforts_recovery: 0 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-30T03:53:21.029 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.allow_data_in_errors: 0 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.db_host_id: __hostname__ 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_background_jobs: 2 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_background_compactions: -1 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_subcompactions: 1 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_total_wal_size: 0 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-30T03:53:21.030 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_open_files: -1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bytes_per_sync: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_readahead_size: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_background_flushes: -1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Compression algorithms supported: 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kZSTD supported: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kXpressCompression supported: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kLZ4HCCompression supported: 1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kZlibCompression supported: 1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kSnappyCompression supported: 1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kLZ4Compression supported: 1 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: kBZip2Compression supported: 0 2024-03-30T03:53:21.031 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.merge_operator: 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_filter: None 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55de3a4ad620) 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T03:53:21.032 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x55de397c5350 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-30T03:53:21.033 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-30T03:53:21.034 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression: NoCompression 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.num_levels: 7 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T03:53:21.035 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T03:53:21.036 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T03:53:21.037 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T03:53:21.038 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.table_properties_collectors: 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.inplace_update_support: 0 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.bloom_locality: 0 2024-03-30T03:53:21.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.max_successive_merges: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.ttl: 2592000 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enable_blob_files: false 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.min_blob_size: 0 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T03:53:21.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d94b5f27-f8b4-47ef-b89f-73f5c4094a17 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770800794535, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770800796621, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92088, "index_size": 271, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13650, "raw_average_key_size": 50, "raw_value_size": 84660, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711770800, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d94b5f27-f8b4-47ef-b89f-73f5c4094a17", "db_session_id": "902OCQLHMWZ0JY651F37", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770800796990, "job": 1, "event": "recovery_finished"} 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55de39862000 2024-03-30T03:53:21.041 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: DB pointer 0x55de39850000 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T03:53:21.042 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 93.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 93.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 65.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 65.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T03:53:21.043 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 11.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 11.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x55de397c5350#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???) e1 preinit fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).mds e1 new map 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).mds e1 print_map 2024-03-30T03:53:21.044 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: fsmap 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: osdmap e1: 0 total, 0 up, 0 in 2024-03-30T03:53:21.045 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:20 smithi023 ceph-mon[38254]: mgrmap e1: no daemons active 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.207+0000 7f0682400700 1 Processor -- start 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.207+0000 7f0682400700 1 -- start start 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.207+0000 7f0682400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.207+0000 7f0682400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c079ec0 con 0x7f067c106f30 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.208+0000 7f067b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:21.659 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.208+0000 7f067b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52376/0 (socket says 172.21.15.23:52376) 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.208+0000 7f067b400700 1 -- 172.21.15.23:0/4117462511 learned_addr learned my addr 172.21.15.23:0/4117462511 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.208+0000 7f067b400700 1 -- 172.21.15.23:0/4117462511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c109860 con 0x7f067c106f30 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.208+0000 7f067b400700 1 --2- 172.21.15.23:0/4117462511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f066c0098a0 tx=0x7f066c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2a044f105c1583 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f067aa00700 1 -- 172.21.15.23:0/4117462511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f066c004160 con 0x7f067c106f30 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f067aa00700 1 -- 172.21.15.23:0/4117462511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f066c0042c0 con 0x7f067c106f30 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f067aa00700 1 -- 172.21.15.23:0/4117462511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f066c011a00 con 0x7f067c106f30 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 -- 172.21.15.23:0/4117462511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 msgr2=0x7f067c109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 --2- 172.21.15.23:0/4117462511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f066c0098a0 tx=0x7f066c009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 -- 172.21.15.23:0/4117462511 shutdown_connections 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 --2- 172.21.15.23:0/4117462511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c109320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 -- 172.21.15.23:0/4117462511 >> 172.21.15.23:0/4117462511 conn(0x7f067c1009c0 msgr2=0x7f067c102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.209+0000 7f0682400700 1 -- 172.21.15.23:0/4117462511 shutdown_connections 2024-03-30T03:53:21.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.210+0000 7f0682400700 1 -- 172.21.15.23:0/4117462511 wait complete. 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f0682400700 1 Processor -- start 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f0682400700 1 -- start start 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f0682400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f0682400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c106ca0 con 0x7f067c106f30 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f067b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f067b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52388/0 (socket says 172.21.15.23:52388) 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.211+0000 7f067b400700 1 -- 172.21.15.23:0/2111641243 learned_addr learned my addr 172.21.15.23:0/2111641243 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f067b400700 1 -- 172.21.15.23:0/2111641243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c009550 con 0x7f067c106f30 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f067b400700 1 --2- 172.21.15.23:0/2111641243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f066c009130 tx=0x7f066c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f066c003b70 con 0x7f067c106f30 2024-03-30T03:53:21.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f066c003cd0 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c104ee0 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f066c0197d0 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.212+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c105380 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.213+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f066c0199c0 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.213+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f066c00b720 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.213+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c10a660 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.217+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f066c02a060 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.289+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f067c10a850 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.293+0000 7f0678c00700 1 -- 172.21.15.23:0/2111641243 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f066c019e00 con 0x7f067c106f30 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 msgr2=0x7f067c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 --2- 172.21.15.23:0/2111641243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f066c009130 tx=0x7f066c011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 shutdown_connections 2024-03-30T03:53:21.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 --2- 172.21.15.23:0/2111641243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f067c106f30 0x7f067c106760 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:21.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 >> 172.21.15.23:0/2111641243 conn(0x7f067c1009c0 msgr2=0x7f067c101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:21.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 shutdown_connections 2024-03-30T03:53:21.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:21.294+0000 7f0682400700 1 -- 172.21.15.23:0/2111641243 wait complete. 2024-03-30T03:53:21.663 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-30T03:53:21.699 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-30T03:53:21.700 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-30T03:53:21.700 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-30T03:53:21.700 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-30T03:53:22.060 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T03:53:22.060 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x.service: Unit ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x.service not loaded. 2024-03-30T03:53:22.260 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4.target.wants/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x.service → /etc/systemd/system/ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@.service. 2024-03-30T03:53:22.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:22 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2111641243' entity='client.admin' 2024-03-30T03:53:22.523 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:22 smithi023 systemd[1]: Starting Ceph mgr.x for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:53:22.800 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T03:53:22.800 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-30T03:53:22.818 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T03:53:22.818 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:53:22.818 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-30T03:53:22.818 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-30T03:53:22.832 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T03:53:22.833 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-30T03:53:22.848 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T03:53:22.849 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:53:22.849 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-30T03:53:22.849 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-30T03:53:22.849 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-30T03:53:22.849 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-30T03:53:23.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:22 smithi023 bash[38488]: 5412540cb7c9aaf04f212d1b17d0415856083d2912c53c66c70728c3ad3b8ad0 2024-03-30T03:53:23.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:22 smithi023 systemd[1]: Started Ceph mgr.x for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:53:23.588 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3703586819' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T03:53:23.588 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:23 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:23.333+0000 7f61f66d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T03:53:23.589 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:23 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:23.469+0000 7f61f66d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T03:53:23.589 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:23 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:23.588+0000 7f61f66d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T03:53:23.726 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T03:53:23.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T03:53:23.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:23.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T03:53:23.736 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T03:53:23.737 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:23.738 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T03:53:14.699029+0000", 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.273+0000 7f8207800700 1 Processor -- start 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.274+0000 7f8207800700 1 -- start start 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.274+0000 7f8207800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.274+0000 7f8207800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820007a8e0 con 0x7f820007be60 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.274+0000 7f8204e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.275+0000 7f8204e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52418/0 (socket says 172.21.15.23:52418) 2024-03-30T03:53:23.739 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.275+0000 7f8204e00700 1 -- 172.21.15.23:0/2194471225 learned_addr learned my addr 172.21.15.23:0/2194471225 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:23.741 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.275+0000 7f8204e00700 1 -- 172.21.15.23:0/2194471225 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820007aa20 con 0x7f820007be60 2024-03-30T03:53:23.741 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.275+0000 7f8204e00700 1 --2- 172.21.15.23:0/2194471225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098a0 tx=0x7f81f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dc054e90542426 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:23.741 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.275+0000 7f81ff400700 1 -- 172.21.15.23:0/2194471225 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f4004160 con 0x7f820007be60 2024-03-30T03:53:23.741 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f81ff400700 1 -- 172.21.15.23:0/2194471225 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81f40042c0 con 0x7f820007be60 2024-03-30T03:53:23.741 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f81ff400700 1 -- 172.21.15.23:0/2194471225 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f4011a00 con 0x7f820007be60 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 -- 172.21.15.23:0/2194471225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 msgr2=0x7f820007a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 --2- 172.21.15.23:0/2194471225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f81f40098a0 tx=0x7f81f4009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 -- 172.21.15.23:0/2194471225 shutdown_connections 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 --2- 172.21.15.23:0/2194471225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820007a310 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 -- 172.21.15.23:0/2194471225 >> 172.21.15.23:0/2194471225 conn(0x7f8200101ab0 msgr2=0x7f8200103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 -- 172.21.15.23:0/2194471225 shutdown_connections 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.276+0000 7f8207800700 1 -- 172.21.15.23:0/2194471225 wait complete. 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8207800700 1 Processor -- start 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8207800700 1 -- start start 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8207800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8207800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f820019c8e0 con 0x7f820007be60 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8204e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:23.742 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8204e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52426/0 (socket says 172.21.15.23:52426) 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.278+0000 7f8204e00700 1 -- 172.21.15.23:0/3703586819 learned_addr learned my addr 172.21.15.23:0/3703586819 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f8204e00700 1 -- 172.21.15.23:0/3703586819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f4009550 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f8204e00700 1 --2- 172.21.15.23:0/3703586819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009130 tx=0x7f81f4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f4003b70 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81f4003cd0 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820019cae0 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f40197d0 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.279+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820019cf80 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.280+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f81f4019a10 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.280+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f81f400b720 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.281+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820010c820 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.283+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f81f402a060 con 0x7f820007be60 2024-03-30T03:53:23.743 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.380+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f820010c9d0 con 0x7f820007be60 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.380+0000 7f81fd600700 1 -- 172.21.15.23:0/3703586819 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f81f4019e50 con 0x7f820007be60 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 msgr2=0x7f820019c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 --2- 172.21.15.23:0/3703586819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f81f4009130 tx=0x7f81f4011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 shutdown_connections 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 --2- 172.21.15.23:0/3703586819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f820007be60 0x7f820019c3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 >> 172.21.15.23:0/3703586819 conn(0x7f8200101ab0 msgr2=0x7f8200102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 shutdown_connections 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:23.382+0000 7f8207800700 1 -- 172.21.15.23:0/3703586819 wait complete. 2024-03-30T03:53:23.744 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-30T03:53:24.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:23 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:23.704+0000 7f61f66d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T03:53:24.924 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:24 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:24.602+0000 7f61f66d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:53:25.182 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:24 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:24.925+0000 7f61f66d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T03:53:25.182 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:25.038+0000 7f61f66d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T03:53:25.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:25.182+0000 7f61f66d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T03:53:25.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:25.383+0000 7f61f66d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T03:53:25.878 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:25.558+0000 7f61f66d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T03:53:25.879 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:25.673+0000 7f61f66d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T03:53:26.472 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:26 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:26.210+0000 7f61f66d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T03:53:26.622 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:26.622 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:26.622 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T03:53:26.623 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T03:53:26.624 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.632 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T03:53:26.633 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T03:53:14.699029+0000", 2024-03-30T03:53:26.634 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.159+0000 7fad9aa00700 1 Processor -- start 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.159+0000 7fad9aa00700 1 -- start start 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.159+0000 7fad9aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.159+0000 7fad9aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9407a810 con 0x7fad9407be20 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.160+0000 7fad93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.160+0000 7fad93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52432/0 (socket says 172.21.15.23:52432) 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.160+0000 7fad93e00700 1 -- 172.21.15.23:0/3095422401 learned_addr learned my addr 172.21.15.23:0/3095422401 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.160+0000 7fad93e00700 1 -- 172.21.15.23:0/3095422401 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9407a950 con 0x7fad9407be20 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad93e00700 1 --2- 172.21.15.23:0/3095422401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad840098a0 tx=0x7fad84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c254ad51814fbde server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad92a00700 1 -- 172.21.15.23:0/3095422401 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad84004160 con 0x7fad9407be20 2024-03-30T03:53:26.635 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad92a00700 1 -- 172.21.15.23:0/3095422401 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad840042c0 con 0x7fad9407be20 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad92a00700 1 -- 172.21.15.23:0/3095422401 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad84011a00 con 0x7fad9407be20 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad9aa00700 1 -- 172.21.15.23:0/3095422401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 msgr2=0x7fad9407a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.161+0000 7fad9aa00700 1 --2- 172.21.15.23:0/3095422401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad840098a0 tx=0x7fad84009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.162+0000 7fad9aa00700 1 -- 172.21.15.23:0/3095422401 shutdown_connections 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.162+0000 7fad9aa00700 1 --2- 172.21.15.23:0/3095422401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9407a2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.162+0000 7fad9aa00700 1 -- 172.21.15.23:0/3095422401 >> 172.21.15.23:0/3095422401 conn(0x7fad94101ab0 msgr2=0x7fad94103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.162+0000 7fad9aa00700 1 -- 172.21.15.23:0/3095422401 shutdown_connections 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.162+0000 7fad9aa00700 1 -- 172.21.15.23:0/3095422401 wait complete. 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.163+0000 7fad9aa00700 1 Processor -- start 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.163+0000 7fad9aa00700 1 -- start start 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad9aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad9aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9419c8e0 con 0x7fad9407be20 2024-03-30T03:53:26.636 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52444/0 (socket says 172.21.15.23:52444) 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad93e00700 1 -- 172.21.15.23:0/126387655 learned_addr learned my addr 172.21.15.23:0/126387655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.164+0000 7fad93e00700 1 -- 172.21.15.23:0/126387655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84009550 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad93e00700 1 --2- 172.21.15.23:0/126387655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fad84009130 tx=0x7fad84011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad84003b70 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad84003cd0 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9419cae0 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad840197d0 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.165+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9419cf80 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.166+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fad840199c0 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.166+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fad8400b720 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.166+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9410c800 con 0x7fad9407be20 2024-03-30T03:53:26.637 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.171+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fad8402a060 con 0x7fad9407be20 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.247+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fad9410c9c0 con 0x7fad9407be20 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.248+0000 7fad90c00700 1 -- 172.21.15.23:0/126387655 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fad84019e00 con 0x7fad9407be20 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 msgr2=0x7fad9419c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 --2- 172.21.15.23:0/126387655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fad84009130 tx=0x7fad84011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 shutdown_connections 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 --2- 172.21.15.23:0/126387655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad9407be20 0x7fad9419c3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 >> 172.21.15.23:0/126387655 conn(0x7fad94101ab0 msgr2=0x7fad94102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 shutdown_connections 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:26.249+0000 7fad9aa00700 1 -- 172.21.15.23:0/126387655 wait complete. 2024-03-30T03:53:26.638 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-30T03:53:26.995 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:26 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/126387655' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T03:53:26.995 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:26 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:26.735+0000 7f61f66d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T03:53:26.995 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:26 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:26.995+0000 7f61f66d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T03:53:27.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:27 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:27.122+0000 7f61f66d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T03:53:28.452 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:28 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:28.119+0000 7f61f66d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T03:53:28.453 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:28 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:28.251+0000 7f61f66d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T03:53:28.453 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:28 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:28.453+0000 7f61f66d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T03:53:29.559 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:29 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:29.301+0000 7f61f66d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T03:53:29.816 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2759956826' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T03:53:29.816 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:29 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:29.559+0000 7f61f66d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T03:53:29.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:29.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:29.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-30T03:53:29.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T03:53:29.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T03:53:29.966 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T03:53:29.967 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T03:53:14.699029+0000", 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:29.968 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.546+0000 7fa98d000700 1 Processor -- start 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa98d000700 1 -- start start 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa98d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa98d000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa988106ea0 con 0x7fa9881064b0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa985e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa985e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37396/0 (socket says 172.21.15.23:37396) 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.547+0000 7fa985e00700 1 -- 172.21.15.23:0/874369241 learned_addr learned my addr 172.21.15.23:0/874369241 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.548+0000 7fa985e00700 1 -- 172.21.15.23:0/874369241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa988107660 con 0x7fa9881064b0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.548+0000 7fa985e00700 1 --2- 172.21.15.23:0/874369241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa9780098a0 tx=0x7fa978009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3674ee8f97eff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.548+0000 7fa984a00700 1 -- 172.21.15.23:0/874369241 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa978004160 con 0x7fa9881064b0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.548+0000 7fa984a00700 1 -- 172.21.15.23:0/874369241 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9780042c0 con 0x7fa9881064b0 2024-03-30T03:53:29.969 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.548+0000 7fa984a00700 1 -- 172.21.15.23:0/874369241 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa978011a00 con 0x7fa9881064b0 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 -- 172.21.15.23:0/874369241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 msgr2=0x7fa9881068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 --2- 172.21.15.23:0/874369241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa9780098a0 tx=0x7fa978009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 -- 172.21.15.23:0/874369241 shutdown_connections 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 --2- 172.21.15.23:0/874369241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa9881068d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 -- 172.21.15.23:0/874369241 >> 172.21.15.23:0/874369241 conn(0x7fa988101ab0 msgr2=0x7fa988103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 -- 172.21.15.23:0/874369241 shutdown_connections 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.549+0000 7fa98d000700 1 -- 172.21.15.23:0/874369241 wait complete. 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.550+0000 7fa98d000700 1 Processor -- start 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa98d000700 1 -- start start 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa98d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa98d000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98807c070 con 0x7fa9881064b0 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa985e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:29.970 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa985e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37408/0 (socket says 172.21.15.23:37408) 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.551+0000 7fa985e00700 1 -- 172.21.15.23:0/2759956826 learned_addr learned my addr 172.21.15.23:0/2759956826 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa985e00700 1 -- 172.21.15.23:0/2759956826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978009550 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa985e00700 1 --2- 172.21.15.23:0/2759956826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa978009130 tx=0x7fa978011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa978003ab0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa978003c10 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98807a2b0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.552+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa9780247d0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.553+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98807a750 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.553+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa978024d50 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.553+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa9780193c0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.554+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa988067de0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.556+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa97801abb0 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.632+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa98810a800 con 0x7fa9881064b0 2024-03-30T03:53:29.971 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.633+0000 7fa97ea00700 1 -- 172.21.15.23:0/2759956826 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fa978035090 con 0x7fa9881064b0 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 msgr2=0x7fa98807bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 --2- 172.21.15.23:0/2759956826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa978009130 tx=0x7fa978011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 shutdown_connections 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 --2- 172.21.15.23:0/2759956826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9881064b0 0x7fa98807bb30 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 >> 172.21.15.23:0/2759956826 conn(0x7fa988101ab0 msgr2=0x7fa988102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 shutdown_connections 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:29.634+0000 7fa98d000700 1 -- 172.21.15.23:0/2759956826 wait complete. 2024-03-30T03:53:29.972 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-30T03:53:30.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:29 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:29.831+0000 7f61f66d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T03:53:31.410 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:31.023+0000 7f61f66d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T03:53:31.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:31.410+0000 7f61f66d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:53:31.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:31.530+0000 7f61f66d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T03:53:32.100 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:31.794+0000 7f61f66d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: Activating manager daemon x 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: mgrmap e2: x(active, starting, since 0.00913807s) 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T03:53:32.101 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:31 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:53:32.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-30T03:53:32.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T03:53:32.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T03:53:32.810 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.811 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T03:53:14.699029+0000", 2024-03-30T03:53:32.812 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.356+0000 7f4d5de00700 1 Processor -- start 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.356+0000 7f4d5de00700 1 -- start start 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.357+0000 7f4d5de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.357+0000 7f4d5de00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d58106ea0 con 0x7f4d581064b0 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.357+0000 7f4d56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.357+0000 7f4d56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37504/0 (socket says 172.21.15.23:37504) 2024-03-30T03:53:32.813 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.357+0000 7f4d56a00700 1 -- 172.21.15.23:0/3558204148 learned_addr learned my addr 172.21.15.23:0/3558204148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.358+0000 7f4d56a00700 1 -- 172.21.15.23:0/3558204148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58107660 con 0x7f4d581064b0 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.358+0000 7f4d56a00700 1 --2- 172.21.15.23:0/3558204148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d480098a0 tx=0x7f4d48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6524e1699c03d4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.358+0000 7f4d55600700 1 -- 172.21.15.23:0/3558204148 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d48004160 con 0x7f4d581064b0 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.358+0000 7f4d55600700 1 -- 172.21.15.23:0/3558204148 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d480042c0 con 0x7f4d581064b0 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.358+0000 7f4d55600700 1 -- 172.21.15.23:0/3558204148 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d48011a00 con 0x7f4d581064b0 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 -- 172.21.15.23:0/3558204148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 msgr2=0x7f4d581068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 --2- 172.21.15.23:0/3558204148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4d480098a0 tx=0x7f4d48009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 -- 172.21.15.23:0/3558204148 shutdown_connections 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 --2- 172.21.15.23:0/3558204148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d581068d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 -- 172.21.15.23:0/3558204148 >> 172.21.15.23:0/3558204148 conn(0x7f4d58101ab0 msgr2=0x7f4d58103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 -- 172.21.15.23:0/3558204148 shutdown_connections 2024-03-30T03:53:32.814 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.359+0000 7f4d5de00700 1 -- 172.21.15.23:0/3558204148 wait complete. 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.360+0000 7f4d5de00700 1 Processor -- start 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.360+0000 7f4d5de00700 1 -- start start 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.360+0000 7f4d5de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.360+0000 7f4d5de00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5807c070 con 0x7f4d581064b0 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.361+0000 7f4d56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.361+0000 7f4d56a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37508/0 (socket says 172.21.15.23:37508) 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.361+0000 7f4d56a00700 1 -- 172.21.15.23:0/3619099342 learned_addr learned my addr 172.21.15.23:0/3619099342 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.361+0000 7f4d56a00700 1 -- 172.21.15.23:0/3619099342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48009550 con 0x7f4d581064b0 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.361+0000 7f4d56a00700 1 --2- 172.21.15.23:0/3619099342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009130 tx=0x7f4d48011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:32.815 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.362+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d48003ab0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.362+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d48003c10 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.362+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d480247d0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.362+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d5807a2b0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.362+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d5807a750 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.363+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d58067de0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.363+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 45289+0+0 (secure 0 0 0) 0x7f4d4801abb0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.363+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4d4804c0a0 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.366+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f4d48018530 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.442+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4d5810a800 con 0x7f4d581064b0 2024-03-30T03:53:32.816 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.442+0000 7f4d5ca00700 1 -- 172.21.15.23:0/3619099342 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4d48018350 con 0x7f4d581064b0 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 msgr2=0x7f4d5807bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 --2- 172.21.15.23:0/3619099342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009130 tx=0x7f4d48011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 shutdown_connections 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 --2- 172.21.15.23:0/3619099342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d581064b0 0x7f4d5807bb30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 >> 172.21.15.23:0/3619099342 conn(0x7f4d58101ab0 msgr2=0x7f4d58102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 shutdown_connections 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:32.443+0000 7f4d5de00700 1 -- 172.21.15.23:0/3619099342 wait complete. 2024-03-30T03:53:32.817 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (4/15)... 2024-03-30T03:53:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T03:53:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: Manager daemon x is now available 2024-03-30T03:53:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T03:53:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' 2024-03-30T03:53:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T03:53:33.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' 2024-03-30T03:53:33.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='mgr.14100 172.21.15.23:0/4025568511' entity='mgr.x' 2024-03-30T03:53:33.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:32 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3619099342' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T03:53:34.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:33 smithi023 ceph-mon[38254]: mgrmap e3: x(active, since 1.02945s) 2024-03-30T03:53:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:34 smithi023 ceph-mon[38254]: mgrmap e4: x(active, since 2s) 2024-03-30T03:53:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3667028058' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-30T03:53:40.566 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-30T03:53:40.567 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-30T03:53:40.568 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:40.569 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-30T03:53:14.699029+0000", 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.917+0000 7f84e5600700 1 Processor -- start 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.917+0000 7f84e5600700 1 -- start start 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84e5600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84e5600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e0106ed0 con 0x7f84e01064e0 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84dea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84dea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45736/0 (socket says 172.21.15.23:45736) 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84dea00700 1 -- 172.21.15.23:0/3175140326 learned_addr learned my addr 172.21.15.23:0/3175140326 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:40.571 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.918+0000 7f84dea00700 1 -- 172.21.15.23:0/3175140326 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e0107690 con 0x7f84e01064e0 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.919+0000 7f84dea00700 1 --2- 172.21.15.23:0/3175140326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84d00098a0 tx=0x7f84d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807603fe062a70bc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.919+0000 7f84dd600700 1 -- 172.21.15.23:0/3175140326 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84d0004160 con 0x7f84e01064e0 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.919+0000 7f84dd600700 1 -- 172.21.15.23:0/3175140326 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d00042c0 con 0x7f84e01064e0 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.919+0000 7f84dd600700 1 -- 172.21.15.23:0/3175140326 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84d0011a00 con 0x7f84e01064e0 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 -- 172.21.15.23:0/3175140326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 msgr2=0x7f84e0106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 --2- 172.21.15.23:0/3175140326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f84d00098a0 tx=0x7f84d0009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 -- 172.21.15.23:0/3175140326 shutdown_connections 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 --2- 172.21.15.23:0/3175140326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e0106900 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 -- 172.21.15.23:0/3175140326 >> 172.21.15.23:0/3175140326 conn(0x7f84e0101a80 msgr2=0x7f84e0103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 -- 172.21.15.23:0/3175140326 shutdown_connections 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.920+0000 7f84e5600700 1 -- 172.21.15.23:0/3175140326 wait complete. 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84e5600700 1 Processor -- start 2024-03-30T03:53:40.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84e5600700 1 -- start start 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84e5600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84e5600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e019a740 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84dea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.921+0000 7f84dea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45740/0 (socket says 172.21.15.23:45740) 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84dea00700 1 -- 172.21.15.23:0/3667028058 learned_addr learned my addr 172.21.15.23:0/3667028058 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84dea00700 1 -- 172.21.15.23:0/3667028058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009550 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84dea00700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009130 tx=0x7f84d0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84d0003ab0 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d0003c10 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e019a940 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.922+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f84d00247d0 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.923+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e019ade0 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.923+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e010a8c0 con 0x7f84e01064e0 2024-03-30T03:53:40.573 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.924+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f84d001abb0 con 0x7f84e01064e0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.924+0000 7f84d7400700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 0x7f84c403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.924+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f84d004c3d0 con 0x7f84e01064e0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.924+0000 7f84de000700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 0x7f84c403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.925+0000 7f84de000700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 0x7f84c403abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:38.930+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f84d0018710 con 0x7f84e01064e0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.227+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f84e019d670 con 0x7f84e01064e0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.227+0000 7f84d7400700 1 -- 172.21.15.23:0/3667028058 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f84d0018350 con 0x7f84e01064e0 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 msgr2=0x7f84c403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 0x7f84c403abc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f84cc006fd0 tx=0x7f84cc006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 msgr2=0x7f84e019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009130 tx=0x7f84d0011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 shutdown_connections 2024-03-30T03:53:40.574 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f84c4038700 0x7f84c403abc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 --2- 172.21.15.23:0/3667028058 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84e01064e0 0x7f84e019a200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.229+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 >> 172.21.15.23:0/3667028058 conn(0x7f84e0101a80 msgr2=0x7f84e01026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.230+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 shutdown_connections 2024-03-30T03:53:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:39.230+0000 7f84e5600700 1 -- 172.21.15.23:0/3667028058 wait complete. 2024-03-30T03:53:40.575 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-30T03:53:41.529 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.970+0000 7f0299c00700 1 Processor -- start 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.970+0000 7f0299c00700 1 -- start start 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0299c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:41.530 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0299c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029407a8e0 con 0x7f029407be60 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0292a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0292a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45752/0 (socket says 172.21.15.23:45752) 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0292a00700 1 -- 172.21.15.23:0/1961957787 learned_addr learned my addr 172.21.15.23:0/1961957787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.971+0000 7f0292a00700 1 -- 172.21.15.23:0/1961957787 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f029407aa20 con 0x7f029407be60 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0292a00700 1 --2- 172.21.15.23:0/1961957787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02840098a0 tx=0x7f0284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63afadd4b6bdab6c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0292000700 1 -- 172.21.15.23:0/1961957787 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0284004160 con 0x7f029407be60 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0292000700 1 -- 172.21.15.23:0/1961957787 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02840042c0 con 0x7f029407be60 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0292000700 1 -- 172.21.15.23:0/1961957787 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0284011a00 con 0x7f029407be60 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0299c00700 1 -- 172.21.15.23:0/1961957787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 msgr2=0x7f029407a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.972+0000 7f0299c00700 1 --2- 172.21.15.23:0/1961957787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02840098a0 tx=0x7f0284009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.973+0000 7f0299c00700 1 -- 172.21.15.23:0/1961957787 shutdown_connections 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.973+0000 7f0299c00700 1 --2- 172.21.15.23:0/1961957787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029407a310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.973+0000 7f0299c00700 1 -- 172.21.15.23:0/1961957787 >> 172.21.15.23:0/1961957787 conn(0x7f0294101ab0 msgr2=0x7f0294103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.973+0000 7f0299c00700 1 -- 172.21.15.23:0/1961957787 shutdown_connections 2024-03-30T03:53:41.531 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.973+0000 7f0299c00700 1 -- 172.21.15.23:0/1961957787 wait complete. 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0299c00700 1 Processor -- start 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0299c00700 1 -- start start 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0299c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0299c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029419c8e0 con 0x7f029407be60 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0292a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0292a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45760/0 (socket says 172.21.15.23:45760) 2024-03-30T03:53:41.532 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.974+0000 7f0292a00700 1 -- 172.21.15.23:0/2626664885 learned_addr learned my addr 172.21.15.23:0/2626664885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f0292a00700 1 -- 172.21.15.23:0/2626664885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0284009550 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f0292a00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0284009130 tx=0x7f0284011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0284003b70 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0284003cd0 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029419cae0 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02840197d0 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.975+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029419cf80 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.976+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f029410c820 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.976+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f028401abb0 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.977+0000 7f028b400700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 0x7f027803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.977+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f028404c8d0 con 0x7f029407be60 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.977+0000 7f028be00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 0x7f027803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:41.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.978+0000 7f028be00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 0x7f027803ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f027c006fd0 tx=0x7f027c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:40.982+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f028401a430 con 0x7f029407be60 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.192+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f029410ca30 con 0x7f029407be60 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.193+0000 7f028b400700 1 -- 172.21.15.23:0/2626664885 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f028402db40 con 0x7f029407be60 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 msgr2=0x7f027803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 0x7f027803ab70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f027c006fd0 tx=0x7f027c006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 msgr2=0x7f029419c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0284009130 tx=0x7f0284011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 shutdown_connections 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f02780386b0 0x7f027803ab70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 --2- 172.21.15.23:0/2626664885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f029407be60 0x7f029419c3a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 >> 172.21.15.23:0/2626664885 conn(0x7f0294101ab0 msgr2=0x7f0294102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 shutdown_connections 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.195+0000 7f0299c00700 1 -- 172.21.15.23:0/2626664885 wait complete. 2024-03-30T03:53:41.534 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-30T03:53:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:41 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2626664885' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-30T03:53:43.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:42 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/112685713' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-30T03:53:43.157 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:42 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setuser ceph since I am not root 2024-03-30T03:53:43.157 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:42 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setgroup ceph since I am not root 2024-03-30T03:53:43.158 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.887+0000 7f0724200700 1 Processor -- start 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.887+0000 7f0724200700 1 -- start start 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0724200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0724200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c07a810 con 0x7f071c07be20 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0721800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0721800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45766/0 (socket says 172.21.15.23:45766) 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0721800700 1 -- 172.21.15.23:0/1596843187 learned_addr learned my addr 172.21.15.23:0/1596843187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.888+0000 7f0721800700 1 -- 172.21.15.23:0/1596843187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c07a950 con 0x7f071c07be20 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f0721800700 1 --2- 172.21.15.23:0/1596843187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f07100098a0 tx=0x7f0710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404fc5836923f36 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f071be00700 1 -- 172.21.15.23:0/1596843187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0710004160 con 0x7f071c07be20 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f071be00700 1 -- 172.21.15.23:0/1596843187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f07100042c0 con 0x7f071c07be20 2024-03-30T03:53:43.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f071be00700 1 -- 172.21.15.23:0/1596843187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0710011a00 con 0x7f071c07be20 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f0724200700 1 -- 172.21.15.23:0/1596843187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 msgr2=0x7f071c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.889+0000 7f0724200700 1 --2- 172.21.15.23:0/1596843187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f07100098a0 tx=0x7f0710009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.890+0000 7f0724200700 1 -- 172.21.15.23:0/1596843187 shutdown_connections 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.890+0000 7f0724200700 1 --2- 172.21.15.23:0/1596843187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c07a2d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.890+0000 7f0724200700 1 -- 172.21.15.23:0/1596843187 >> 172.21.15.23:0/1596843187 conn(0x7f071c101ae0 msgr2=0x7f071c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.890+0000 7f0724200700 1 -- 172.21.15.23:0/1596843187 shutdown_connections 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.890+0000 7f0724200700 1 -- 172.21.15.23:0/1596843187 wait complete. 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0724200700 1 Processor -- start 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0724200700 1 -- start start 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0724200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0724200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071c19c920 con 0x7f071c07be20 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0721800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0721800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45782/0 (socket says 172.21.15.23:45782) 2024-03-30T03:53:43.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.891+0000 7f0721800700 1 -- 172.21.15.23:0/112685713 learned_addr learned my addr 172.21.15.23:0/112685713 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f0721800700 1 -- 172.21.15.23:0/112685713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0710009550 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f0721800700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0710009130 tx=0x7f0710011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0710003b70 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0710003cd0 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07100197d0 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071c19cb20 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.892+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071c19cfc0 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.893+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071c10c830 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.893+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 45522+0+0 (secure 0 0 0) 0x7f071001abb0 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.894+0000 7f071a000700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 0x7f070403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.894+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f071004c8e0 con 0x7f071c07be20 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.894+0000 7f0720e00700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 0x7f070403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.895+0000 7f0720e00700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 0x7f070403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0708006fd0 tx=0x7f0708006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:43.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:41.899+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f071001a430 con 0x7f071c07be20 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.165+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f071c19f8b0 con 0x7f071c07be20 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.834+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f071001afe0 con 0x7f071c07be20 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.836+0000 7f071a000700 1 -- 172.21.15.23:0/112685713 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f0710017ca0 con 0x7f071c07be20 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 msgr2=0x7f070403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 0x7f070403abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0708006fd0 tx=0x7f0708006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 msgr2=0x7f071c19c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0710009130 tx=0x7f0710011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 shutdown_connections 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0704038700 0x7f070403abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 --2- 172.21.15.23:0/112685713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f071c07be20 0x7f071c19c3e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 >> 172.21.15.23:0/112685713 conn(0x7f071c101ae0 msgr2=0x7f071c1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.839+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 shutdown_connections 2024-03-30T03:53:43.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:42.840+0000 7f0724200700 1 -- 172.21.15.23:0/112685713 wait complete. 2024-03-30T03:53:43.714 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:43.462+0000 7ff4cfbef200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T03:53:43.714 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:43.589+0000 7ff4cfbef200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T03:53:43.714 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:43.713+0000 7ff4cfbef200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T03:53:44.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:43.827+0000 7ff4cfbef200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T03:53:44.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/112685713' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-30T03:53:44.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:43 smithi023 ceph-mon[38254]: mgrmap e5: x(active, since 11s) 2024-03-30T03:53:44.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.606+0000 7f0b33400700 1 Processor -- start 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.606+0000 7f0b33400700 1 -- start start 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.606+0000 7f0b33400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.606+0000 7f0b33400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c07a830 con 0x7f0b2c07be40 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b30a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b30a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45820/0 (socket says 172.21.15.23:45820) 2024-03-30T03:53:44.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b30a00700 1 -- 172.21.15.23:0/2459018388 learned_addr learned my addr 172.21.15.23:0/2459018388 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b30a00700 1 -- 172.21.15.23:0/2459018388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c07a970 con 0x7f0b2c07be40 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b30a00700 1 --2- 172.21.15.23:0/2459018388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098a0 tx=0x7f0b20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f947599d9425dc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.607+0000 7f0b2b400700 1 -- 172.21.15.23:0/2459018388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b20004160 con 0x7f0b2c07be40 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b2b400700 1 -- 172.21.15.23:0/2459018388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b200042c0 con 0x7f0b2c07be40 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b2b400700 1 -- 172.21.15.23:0/2459018388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b20011a00 con 0x7f0b2c07be40 2024-03-30T03:53:44.241 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b33400700 1 -- 172.21.15.23:0/2459018388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 msgr2=0x7f0b2c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b33400700 1 --2- 172.21.15.23:0/2459018388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098a0 tx=0x7f0b20009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b33400700 1 -- 172.21.15.23:0/2459018388 shutdown_connections 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b33400700 1 --2- 172.21.15.23:0/2459018388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c07a2f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.608+0000 7f0b33400700 1 -- 172.21.15.23:0/2459018388 >> 172.21.15.23:0/2459018388 conn(0x7f0b2c101ab0 msgr2=0x7f0b2c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.609+0000 7f0b33400700 1 -- 172.21.15.23:0/2459018388 shutdown_connections 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.609+0000 7f0b33400700 1 -- 172.21.15.23:0/2459018388 wait complete. 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.610+0000 7f0b33400700 1 Processor -- start 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.610+0000 7f0b33400700 1 -- start start 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b33400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:44.242 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b33400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c19c8e0 con 0x7f0b2c07be40 2024-03-30T03:53:44.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b30a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:44.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b30a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45836/0 (socket says 172.21.15.23:45836) 2024-03-30T03:53:44.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b30a00700 1 -- 172.21.15.23:0/3800566567 learned_addr learned my addr 172.21.15.23:0/3800566567 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:44.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.611+0000 7f0b30a00700 1 -- 172.21.15.23:0/3800566567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009550 con 0x7f0b2c07be40 2024-03-30T03:53:44.243 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b30a00700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009fb0 tx=0x7f0b20011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:44.245 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b20003a40 con 0x7f0b2c07be40 2024-03-30T03:53:44.245 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b20003ba0 con 0x7f0b2c07be40 2024-03-30T03:53:44.245 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b2001a750 con 0x7f0b2c07be40 2024-03-30T03:53:44.245 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c19cae0 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.612+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c19cf80 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.613+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f0b20024430 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.613+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2c10c7e0 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.614+0000 7f0b29600700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 0x7f0b1403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.614+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0b2004c3f0 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.614+0000 7f0b2be00700 1 -- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 msgr2=0x7f0b1403ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.614+0000 7f0b2be00700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 0x7f0b1403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.623+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0b200189a0 con 0x7f0b2c07be40 2024-03-30T03:53:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.815+0000 7f0b2be00700 1 -- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 msgr2=0x7f0b1403ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.815+0000 7f0b2be00700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 0x7f0b1403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.900+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0b2c19f810 con 0x7f0b2c07be40 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.900+0000 7f0b29600700 1 -- 172.21.15.23:0/3800566567 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f0b20018350 con 0x7f0b2c07be40 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 msgr2=0x7f0b1403ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 0x7f0b1403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 msgr2=0x7f0b2c19c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009fb0 tx=0x7f0b20011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 shutdown_connections 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f0b140386b0 0x7f0b1403ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 --2- 172.21.15.23:0/3800566567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0b2c07be40 0x7f0b2c19c3a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 >> 172.21.15.23:0/3800566567 conn(0x7f0b2c101ab0 msgr2=0x7f0b2c102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 shutdown_connections 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:43.903+0000 7f0b33400700 1 -- 172.21.15.23:0/3800566567 wait complete. 2024-03-30T03:53:44.247 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-30T03:53:44.248 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-30T03:53:45.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3800566567' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T03:53:45.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:44 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:44.711+0000 7ff4cfbef200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:53:45.291 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.033+0000 7ff4cfbef200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T03:53:45.291 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.147+0000 7ff4cfbef200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T03:53:45.664 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.290+0000 7ff4cfbef200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T03:53:45.664 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.491+0000 7ff4cfbef200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T03:53:46.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.663+0000 7ff4cfbef200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T03:53:46.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:45.777+0000 7ff4cfbef200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T03:53:46.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:46 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:46.256+0000 7ff4cfbef200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T03:53:47.016 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:46 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:46.757+0000 7ff4cfbef200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T03:53:47.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:47.016+0000 7ff4cfbef200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T03:53:47.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:47.143+0000 7ff4cfbef200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T03:53:48.453 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:48.124+0000 7ff4cfbef200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T03:53:48.454 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:48.254+0000 7ff4cfbef200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T03:53:49.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:48.453+0000 7ff4cfbef200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T03:53:49.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:49.290+0000 7ff4cfbef200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T03:53:49.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:49.536+0000 7ff4cfbef200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T03:53:50.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:49.799+0000 7ff4cfbef200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T03:53:51.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:50.979+0000 7ff4cfbef200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T03:53:51.746 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:51.364+0000 7ff4cfbef200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:53:51.746 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:51.485+0000 7ff4cfbef200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T03:53:52.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:53:51.746+0000 7ff4cfbef200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T03:53:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: Active manager daemon x restarted 2024-03-30T03:53:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: Activating manager daemon x 2024-03-30T03:53:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: osdmap e2: 0 total, 0 up, 0 in 2024-03-30T03:53:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: mgrmap e6: x(active, starting, since 0.0131192s) 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: Manager daemon x is now available 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:53:52.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:51 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.679+0000 7f1c44200700 1 Processor -- start 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.679+0000 7f1c44200700 1 -- start start 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.680+0000 7f1c44200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.680+0000 7f1c44200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c106ea0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.133 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.680+0000 7f1c41800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.680+0000 7f1c41800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45842/0 (socket says 172.21.15.23:45842) 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.680+0000 7f1c41800700 1 -- 172.21.15.23:0/1627202238 learned_addr learned my addr 172.21.15.23:0/1627202238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.681+0000 7f1c41800700 1 -- 172.21.15.23:0/1627202238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3c107660 con 0x7f1c3c1064b0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.681+0000 7f1c41800700 1 --2- 172.21.15.23:0/1627202238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c300098a0 tx=0x7f1c30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b9733a0b9464f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.681+0000 7f1c3be00700 1 -- 172.21.15.23:0/1627202238 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c30004160 con 0x7f1c3c1064b0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.681+0000 7f1c3be00700 1 -- 172.21.15.23:0/1627202238 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c300042c0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.681+0000 7f1c3be00700 1 -- 172.21.15.23:0/1627202238 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c30011a00 con 0x7f1c3c1064b0 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 -- 172.21.15.23:0/1627202238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 msgr2=0x7f1c3c1068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 --2- 172.21.15.23:0/1627202238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c300098a0 tx=0x7f1c30009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 -- 172.21.15.23:0/1627202238 shutdown_connections 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 --2- 172.21.15.23:0/1627202238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c1068d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.134 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 -- 172.21.15.23:0/1627202238 >> 172.21.15.23:0/1627202238 conn(0x7f1c3c101ab0 msgr2=0x7f1c3c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 -- 172.21.15.23:0/1627202238 shutdown_connections 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.682+0000 7f1c44200700 1 -- 172.21.15.23:0/1627202238 wait complete. 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.684+0000 7f1c44200700 1 Processor -- start 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.684+0000 7f1c44200700 1 -- start start 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.684+0000 7f1c44200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.684+0000 7f1c44200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c07c070 con 0x7f1c3c1064b0 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c41800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c41800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45854/0 (socket says 172.21.15.23:45854) 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c41800700 1 -- 172.21.15.23:0/4265738557 learned_addr learned my addr 172.21.15.23:0/4265738557 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c41800700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30009550 con 0x7f1c3c1064b0 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c41800700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009130 tx=0x7f1c30011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.685+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c30003ab0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.135 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.686+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c30003c10 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.686+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c300247d0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.686+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c07a2b0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.686+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c07a750 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.687+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f1c3001abb0 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.687+0000 7f1c3a000700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.687+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c2403b320 con 0x7f1c24038750 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.688+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1c3004c440 con 0x7f1c3c1064b0 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.688+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.688+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.889+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:44.889+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:45.290+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.136 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:45.290+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:46.091+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:46.091+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:47.692+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:47.692+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:50.895+0000 7f1c40e00700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2151815679 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:50.895+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:51.760+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 45300+0+0 (secure 0 0 0) 0x7f1c30021d60 con 0x7f1c3c1064b0 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:51.760+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 msgr2=0x7f1c2403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:51.760+0000 7f1c3a000700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.762+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f1c3004e150 con 0x7f1c3c1064b0 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.762+0000 7f1c3a000700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 0x7f1c2403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.762+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c2403b320 con 0x7f1c2403bfe0 2024-03-30T03:53:53.137 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.762+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 0x7f1c2403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.764+0000 7f1c40e00700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 0x7f1c2403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1c28003a10 tx=0x7f1c280092b0 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.764+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f1c2403b320 con 0x7f1c2403bfe0 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1c3c1084d0 con 0x7f1c2403bfe0 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c3a000700 1 -- 172.21.15.23:0/4265738557 <== mgr.14122 v2:172.21.15.23:6800/604121731 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f1c2800a050 con 0x7f1c2403bfe0 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 msgr2=0x7f1c2403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c44200700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 0x7f1c2403e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1c28003a10 tx=0x7f1c280092b0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 msgr2=0x7f1c3c07bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.772+0000 7f1c44200700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1c30009130 tx=0x7f1c30011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 shutdown_connections 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f1c2403bfe0 0x7f1c2403e3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1c3c1064b0 0x7f1c3c07bb30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 --2- 172.21.15.23:0/4265738557 >> [v2:172.21.15.23:6800/2151815679,v1:172.21.15.23:6801/2151815679] conn(0x7f1c24038750 0x7f1c2403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:53.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 >> 172.21.15.23:0/4265738557 conn(0x7f1c3c101ab0 msgr2=0x7f1c3c102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:53.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 shutdown_connections 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: Found migration_current of "None". Setting to last migration. 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:53:53.139 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:52 smithi023 ceph-mon[38254]: mgrmap e7: x(active, since 1.01578s) 2024-03-30T03:53:53.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:52.773+0000 7f1c44200700 1 -- 172.21.15.23:0/4265738557 wait complete. 2024-03-30T03:53:53.140 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-30T03:53:53.140 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: [30/Mar/2024:03:53:52] ENGINE Bus STARTING 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: [30/Mar/2024:03:53:52] ENGINE Serving on http://172.21.15.23:8765 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: [30/Mar/2024:03:53:52] ENGINE Serving on https://172.21.15.23:7150 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: [30/Mar/2024:03:53:52] ENGINE Bus STARTED 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: from='client.14126 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T03:53:54.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: from='client.14126 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T03:53:54.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:54.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:53 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.528+0000 7f3751a00700 1 Processor -- start 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.528+0000 7f3751a00700 1 -- start start 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f3751a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f3751a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c07a810 con 0x7f374c07be20 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f374be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f374be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32956/0 (socket says 172.21.15.23:32956) 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f374be00700 1 -- 172.21.15.23:0/1931128250 learned_addr learned my addr 172.21.15.23:0/1931128250 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.529+0000 7f374be00700 1 -- 172.21.15.23:0/1931128250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c07a950 con 0x7f374c07be20 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f374be00700 1 --2- 172.21.15.23:0/1931128250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f373c0098a0 tx=0x7f373c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59bef0dea6a9f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f374aa00700 1 -- 172.21.15.23:0/1931128250 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f373c004160 con 0x7f374c07be20 2024-03-30T03:53:54.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f374aa00700 1 -- 172.21.15.23:0/1931128250 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f373c0042c0 con 0x7f374c07be20 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f374aa00700 1 -- 172.21.15.23:0/1931128250 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f373c011a00 con 0x7f374c07be20 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f3751a00700 1 -- 172.21.15.23:0/1931128250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 msgr2=0x7f374c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f3751a00700 1 --2- 172.21.15.23:0/1931128250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f373c0098a0 tx=0x7f373c009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f3751a00700 1 -- 172.21.15.23:0/1931128250 shutdown_connections 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.530+0000 7f3751a00700 1 --2- 172.21.15.23:0/1931128250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c07a2d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.531+0000 7f3751a00700 1 -- 172.21.15.23:0/1931128250 >> 172.21.15.23:0/1931128250 conn(0x7f374c1018c0 msgr2=0x7f374c103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.531+0000 7f3751a00700 1 -- 172.21.15.23:0/1931128250 shutdown_connections 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.531+0000 7f3751a00700 1 -- 172.21.15.23:0/1931128250 wait complete. 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.532+0000 7f3751a00700 1 Processor -- start 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.532+0000 7f3751a00700 1 -- start start 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f3751a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f3751a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c19cb00 con 0x7f374c07be20 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f374be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:54.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f374be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32960/0 (socket says 172.21.15.23:32960) 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f374be00700 1 -- 172.21.15.23:0/64360155 learned_addr learned my addr 172.21.15.23:0/64360155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.533+0000 7f374be00700 1 -- 172.21.15.23:0/64360155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009550 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f374be00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f373c011a20 tx=0x7f373c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f373c003af0 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f373c003c50 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f373c019760 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374c19cd00 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.534+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374c19d1a0 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.535+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 7) v1 ==== 45427+0+0 (secure 0 0 0) 0x7f373c01ab20 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.535+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c196290 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.535+0000 7f3748c00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 0x7f373003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.535+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f373c04c730 con 0x7f374c07be20 2024-03-30T03:53:54.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.536+0000 7f374b400700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 0x7f373003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.536+0000 7f374b400700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 0x7f373003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3738006fd0 tx=0x7f3738006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.541+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f373c02db90 con 0x7f374c07be20 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.770+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f374c0611f0 con 0x7f3730038620 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.781+0000 7f3748c00700 1 -- 172.21.15.23:0/64360155 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f374c0611f0 con 0x7f3730038620 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 msgr2=0x7f373003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 0x7f373003aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3738006fd0 tx=0x7f3738006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 msgr2=0x7f374c19c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f373c011a20 tx=0x7f373c011b00 comp rx=0 tx=0).stop 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 shutdown_connections 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f3730038620 0x7f373003aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 --2- 172.21.15.23:0/64360155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f374c07be20 0x7f374c19c5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 >> 172.21.15.23:0/64360155 conn(0x7f374c1018c0 msgr2=0x7f374c1027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 shutdown_connections 2024-03-30T03:53:54.174 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:53.784+0000 7f3751a00700 1 -- 172.21.15.23:0/64360155 wait complete. 2024-03-30T03:53:55.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-30T03:53:55.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.576+0000 7fac16000700 1 Processor -- start 2024-03-30T03:53:55.165 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac16000700 1 -- start start 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac16000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac16000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1007a830 con 0x7fac1007be40 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac0f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac0f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32976/0 (socket says 172.21.15.23:32976) 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.577+0000 7fac0f400700 1 -- 172.21.15.23:0/3453194689 learned_addr learned my addr 172.21.15.23:0/3453194689 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.578+0000 7fac0f400700 1 -- 172.21.15.23:0/3453194689 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1007a970 con 0x7fac1007be40 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.578+0000 7fac0f400700 1 --2- 172.21.15.23:0/3453194689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac000098a0 tx=0x7fac00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3d4b47c73db3ada server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.578+0000 7fac0e000700 1 -- 172.21.15.23:0/3453194689 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac00004160 con 0x7fac1007be40 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.578+0000 7fac0e000700 1 -- 172.21.15.23:0/3453194689 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac000042c0 con 0x7fac1007be40 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.578+0000 7fac0e000700 1 -- 172.21.15.23:0/3453194689 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac00011a00 con 0x7fac1007be40 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 -- 172.21.15.23:0/3453194689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 msgr2=0x7fac1007a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:55.166 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 --2- 172.21.15.23:0/3453194689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fac000098a0 tx=0x7fac00009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 -- 172.21.15.23:0/3453194689 shutdown_connections 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 --2- 172.21.15.23:0/3453194689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1007a2f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 -- 172.21.15.23:0/3453194689 >> 172.21.15.23:0/3453194689 conn(0x7fac10101ab0 msgr2=0x7fac10103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 -- 172.21.15.23:0/3453194689 shutdown_connections 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.579+0000 7fac16000700 1 -- 172.21.15.23:0/3453194689 wait complete. 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.580+0000 7fac16000700 1 Processor -- start 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.580+0000 7fac16000700 1 -- start start 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.580+0000 7fac16000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.580+0000 7fac16000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1019c8e0 con 0x7fac1007be40 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac0f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac0f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32988/0 (socket says 172.21.15.23:32988) 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac0f400700 1 -- 172.21.15.23:0/3181604081 learned_addr learned my addr 172.21.15.23:0/3181604081 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:55.167 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac0f400700 1 -- 172.21.15.23:0/3181604081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac00009550 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac0f400700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac00009130 tx=0x7fac00011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac00003b70 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac00003cd0 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fac000197d0 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.581+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1019cae0 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.582+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1019cf80 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.583+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fac0001abb0 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.583+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1010c7e0 con 0x7fac1007be40 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.583+0000 7fac07e00700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 0x7fabf403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:55.168 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.583+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fac0004c8e0 con 0x7fac1007be40 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.583+0000 7fac0ea00700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 0x7fabf403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.584+0000 7fac0ea00700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 0x7fabf403ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabfc006fd0 tx=0x7fabfc006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.589+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fac0001a430 con 0x7fac1007be40 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.802+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fac10102d60 con 0x7fabf4038750 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.803+0000 7fac07e00700 1 -- 172.21.15.23:0/3181604081 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fac10102d60 con 0x7fabf4038750 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 msgr2=0x7fabf403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 0x7fabf403ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fabfc006fd0 tx=0x7fabfc006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 msgr2=0x7fac1019c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fac00009130 tx=0x7fac00011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 shutdown_connections 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fabf4038750 0x7fabf403ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:55.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 --2- 172.21.15.23:0/3181604081 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fac1007be40 0x7fac1019c3a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:55.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 >> 172.21.15.23:0/3181604081 conn(0x7fac10101ab0 msgr2=0x7fac10102650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:55.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 shutdown_connections 2024-03-30T03:53:55.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:54.805+0000 7fac16000700 1 -- 172.21.15.23:0/3181604081 wait complete. 2024-03-30T03:53:55.170 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-30T03:53:55.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:54 smithi023 ceph-mon[38254]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:53:55.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:54 smithi023 ceph-mon[38254]: mgrmap e8: x(active, since 2s) 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.551+0000 7f7437200700 1 Processor -- start 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.551+0000 7f7437200700 1 -- start start 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.551+0000 7f7437200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.551+0000 7f7437200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743807a8a0 con 0x7f743807be70 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7435e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7435e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33002/0 (socket says 172.21.15.23:33002) 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7435e00700 1 -- 172.21.15.23:0/1867414212 learned_addr learned my addr 172.21.15.23:0/1867414212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7435e00700 1 -- 172.21.15.23:0/1867414212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743807a9e0 con 0x7f743807be70 2024-03-30T03:53:56.196 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7435e00700 1 --2- 172.21.15.23:0/1867414212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74280098a0 tx=0x7f7428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e7b39dede72544 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:56.197 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7434a00700 1 -- 172.21.15.23:0/1867414212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7428004160 con 0x7f743807be70 2024-03-30T03:53:56.197 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-mon[38254]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: Generating public/private rsa key pair. 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: Your identification has been saved in /tmp/tmpamqycahj/key. 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: Your public key has been saved in /tmp/tmpamqycahj/key.pub. 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: The key fingerprint is: 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: SHA256:IVABTzFTNDt6RLPAqCyBHSAEWBME8giVQBUvDnacUp0 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: The key's randomart image is: 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: +---[RSA 3072]----+ 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: |#@OB==X=* | 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: |Bo=.+E.= = | 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: |.*.* .o * | 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: |o B . + o | 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: | . . . S | 2024-03-30T03:53:56.198 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: | . | 2024-03-30T03:53:56.199 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: | | 2024-03-30T03:53:56.199 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: | | 2024-03-30T03:53:56.199 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: | | 2024-03-30T03:53:56.199 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:53:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: +----[SHA256]-----+ 2024-03-30T03:53:56.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7434a00700 1 -- 172.21.15.23:0/1867414212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74280042c0 con 0x7f743807be70 2024-03-30T03:53:56.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.552+0000 7f7434a00700 1 -- 172.21.15.23:0/1867414212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7428011a00 con 0x7f743807be70 2024-03-30T03:53:56.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 -- 172.21.15.23:0/1867414212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 msgr2=0x7f743807a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:56.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 --2- 172.21.15.23:0/1867414212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f74280098a0 tx=0x7f7428009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:56.202 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 -- 172.21.15.23:0/1867414212 shutdown_connections 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 --2- 172.21.15.23:0/1867414212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743807a2d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 -- 172.21.15.23:0/1867414212 >> 172.21.15.23:0/1867414212 conn(0x7f7438101900 msgr2=0x7f7438103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 -- 172.21.15.23:0/1867414212 shutdown_connections 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.553+0000 7f7437200700 1 -- 172.21.15.23:0/1867414212 wait complete. 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.554+0000 7f7437200700 1 Processor -- start 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.554+0000 7f7437200700 1 -- start start 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7437200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7437200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743819caa0 con 0x7f743807be70 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7435e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7435e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33006/0 (socket says 172.21.15.23:33006) 2024-03-30T03:53:56.203 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7435e00700 1 -- 172.21.15.23:0/3809186882 learned_addr learned my addr 172.21.15.23:0/3809186882 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7435e00700 1 -- 172.21.15.23:0/3809186882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7428009550 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.555+0000 7f7435e00700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7428011a20 tx=0x7f7428011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.556+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7428003af0 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.556+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7428003c50 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.556+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743819cca0 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.556+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7428019760 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.556+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743819d140 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.557+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f742801ab20 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.557+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7438196290 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.557+0000 7f742ea00700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 0x7f741c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.557+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f742804c9a0 con 0x7f743807be70 2024-03-30T03:53:56.204 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.558+0000 7f7435400700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 0x7f741c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.558+0000 7f7435400700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 0x7f741c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7424006fd0 tx=0x7f7424006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.563+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f742801a7b0 con 0x7f743807be70 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.776+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f74380611f0 con 0x7f741c038750 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.847+0000 7f742ea00700 1 -- 172.21.15.23:0/3809186882 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f74380611f0 con 0x7f741c038750 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 msgr2=0x7f741c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 0x7f741c03ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7424006fd0 tx=0x7f7424006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 msgr2=0x7f743819c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7428011a20 tx=0x7f7428011b00 comp rx=0 tx=0).stop 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 shutdown_connections 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f741c038750 0x7f741c03ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 --2- 172.21.15.23:0/3809186882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f743807be70 0x7f743819c560 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 >> 172.21.15.23:0/3809186882 conn(0x7f7438101900 msgr2=0x7f74381027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:56.205 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 shutdown_connections 2024-03-30T03:53:56.206 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:55.849+0000 7f7437200700 1 -- 172.21.15.23:0/3809186882 wait complete. 2024-03-30T03:53:57.112 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:53:57.112 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.566+0000 7f93a4800700 1 Processor -- start 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.566+0000 7f93a4800700 1 -- start start 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a4800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a4800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c106eb0 con 0x7f939c1064c0 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33016/0 (socket says 172.21.15.23:33016) 2024-03-30T03:53:57.113 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a1e00700 1 -- 172.21.15.23:0/3562384195 learned_addr learned my addr 172.21.15.23:0/3562384195 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:57.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:56 smithi023 ceph-mon[38254]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:53:57.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:56 smithi023 ceph-mon[38254]: Generating ssh key... 2024-03-30T03:53:57.113 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:56 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:57.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:56 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:53:57.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.567+0000 7f93a1e00700 1 -- 172.21.15.23:0/3562384195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c107670 con 0x7f939c1064c0 2024-03-30T03:53:57.118 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a1e00700 1 --2- 172.21.15.23:0/3562384195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93900098a0 tx=0x7f9390009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33df757c57ef09c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a0a00700 1 -- 172.21.15.23:0/3562384195 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9390004160 con 0x7f939c1064c0 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a0a00700 1 -- 172.21.15.23:0/3562384195 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93900042c0 con 0x7f939c1064c0 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a0a00700 1 -- 172.21.15.23:0/3562384195 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9390011a00 con 0x7f939c1064c0 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a4800700 1 -- 172.21.15.23:0/3562384195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 msgr2=0x7f939c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.568+0000 7f93a4800700 1 --2- 172.21.15.23:0/3562384195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f93900098a0 tx=0x7f9390009bb0 comp rx=0 tx=0).stop 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.569+0000 7f93a4800700 1 -- 172.21.15.23:0/3562384195 shutdown_connections 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.569+0000 7f93a4800700 1 --2- 172.21.15.23:0/3562384195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c1068e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.569+0000 7f93a4800700 1 -- 172.21.15.23:0/3562384195 >> 172.21.15.23:0/3562384195 conn(0x7f939c101ae0 msgr2=0x7f939c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.569+0000 7f93a4800700 1 -- 172.21.15.23:0/3562384195 shutdown_connections 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.569+0000 7f93a4800700 1 -- 172.21.15.23:0/3562384195 wait complete. 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a4800700 1 Processor -- start 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a4800700 1 -- start start 2024-03-30T03:53:57.119 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a4800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a4800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c07c070 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33032/0 (socket says 172.21.15.23:33032) 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.570+0000 7f93a1e00700 1 -- 172.21.15.23:0/860338857 learned_addr learned my addr 172.21.15.23:0/860338857 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f93a1e00700 1 -- 172.21.15.23:0/860338857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9390009550 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f93a1e00700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9390009130 tx=0x7f9390011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9390003ab0 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9390003c10 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c07a2b0 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93900247d0 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.571+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c07a750 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.572+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f939001abb0 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.572+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c067de0 con 0x7f939c1064c0 2024-03-30T03:53:57.120 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.573+0000 7f9396a00700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 0x7f938403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.573+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f939004c860 con 0x7f939c1064c0 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.573+0000 7f93a1400700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 0x7f938403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.574+0000 7f93a1400700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 0x7f938403ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f938c006fd0 tx=0x7f938c006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.578+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9390018920 con 0x7f939c1064c0 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.791+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f939c10a870 con 0x7f9384038750 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.792+0000 7f9396a00700 1 -- 172.21.15.23:0/860338857 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f939c10a870 con 0x7f9384038750 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.794+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 msgr2=0x7f938403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.794+0000 7f93a4800700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 0x7f938403ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f938c006fd0 tx=0x7f938c006e40 comp rx=0 tx=0).stop 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.794+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 msgr2=0x7f939c07bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.794+0000 7f93a4800700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9390009130 tx=0x7f9390011ad0 comp rx=0 tx=0).stop 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 shutdown_connections 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f9384038750 0x7f938403ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:57.121 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 --2- 172.21.15.23:0/860338857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f939c1064c0 0x7f939c07bb30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 >> 172.21.15.23:0/860338857 conn(0x7f939c101ae0 msgr2=0x7f939c102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 shutdown_connections 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:56.795+0000 7f93a4800700 1 -- 172.21.15.23:0/860338857 wait complete. 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-30T03:53:57.122 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-30T03:53:58.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:57 smithi023 ceph-mon[38254]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:53:59.016 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:58 smithi023 ceph-mon[38254]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:00.116 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:53:59 smithi023 ceph-mon[38254]: Deploying cephadm binary to smithi023 2024-03-30T03:54:01.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-30T03:54:01.138 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f5082800700 1 Processor -- start 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f5082800700 1 -- start start 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f5082800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f5082800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c106ed0 con 0x7f507c1064e0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f507b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f507b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33038/0 (socket says 172.21.15.23:33038) 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.512+0000 7f507b400700 1 -- 172.21.15.23:0/4041054015 learned_addr learned my addr 172.21.15.23:0/4041054015 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.513+0000 7f507b400700 1 -- 172.21.15.23:0/4041054015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c107690 con 0x7f507c1064e0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.513+0000 7f507b400700 1 --2- 172.21.15.23:0/4041054015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f506c0098a0 tx=0x7f506c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173f29f18ab90dce server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.513+0000 7f507a000700 1 -- 172.21.15.23:0/4041054015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f506c004160 con 0x7f507c1064e0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.513+0000 7f507a000700 1 -- 172.21.15.23:0/4041054015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f506c0042c0 con 0x7f507c1064e0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.513+0000 7f507a000700 1 -- 172.21.15.23:0/4041054015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f506c011a00 con 0x7f507c1064e0 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 -- 172.21.15.23:0/4041054015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 msgr2=0x7f507c106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:01.139 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 --2- 172.21.15.23:0/4041054015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f506c0098a0 tx=0x7f506c009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 -- 172.21.15.23:0/4041054015 shutdown_connections 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 --2- 172.21.15.23:0/4041054015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c106900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 -- 172.21.15.23:0/4041054015 >> 172.21.15.23:0/4041054015 conn(0x7f507c101ae0 msgr2=0x7f507c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 -- 172.21.15.23:0/4041054015 shutdown_connections 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.514+0000 7f5082800700 1 -- 172.21.15.23:0/4041054015 wait complete. 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.515+0000 7f5082800700 1 Processor -- start 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f5082800700 1 -- start start 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f5082800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f5082800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c19c940 con 0x7f507c1064e0 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f507b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f507b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33052/0 (socket says 172.21.15.23:33052) 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.516+0000 7f507b400700 1 -- 172.21.15.23:0/1454979888 learned_addr learned my addr 172.21.15.23:0/1454979888 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:01.140 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f507b400700 1 -- 172.21.15.23:0/1454979888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c009550 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f507b400700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f506c009130 tx=0x7f506c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f506c003ab0 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f506c003c10 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f506c0247d0 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c19cb40 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.517+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c19cfe0 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.518+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f506c01abb0 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.518+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c10a8c0 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.519+0000 7f5073e00700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 0x7f506003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.519+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f506c04c840 con 0x7f507c1064e0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.519+0000 7f507aa00700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 0x7f506003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.520+0000 7f507aa00700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 0x7f506003ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5068006fd0 tx=0x7f5068006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:01.141 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.524+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f506c018530 con 0x7f507c1064e0 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:53:57.739+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7f507c102d30 con 0x7f5060038750 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.733+0000 7f5073e00700 1 -- 172.21.15.23:0/1454979888 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f507c102d30 con 0x7f5060038750 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 msgr2=0x7f506003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 0x7f506003ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5068006fd0 tx=0x7f5068006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 msgr2=0x7f507c19c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f506c009130 tx=0x7f506c011ad0 comp rx=0 tx=0).stop 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 shutdown_connections 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5060038750 0x7f506003ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 --2- 172.21.15.23:0/1454979888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f507c1064e0 0x7f507c19c400 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.736+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 >> 172.21.15.23:0/1454979888 conn(0x7f507c101ae0 msgr2=0x7f507c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.737+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 shutdown_connections 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:00.737+0000 7f5082800700 1 -- 172.21.15.23:0/1454979888 wait complete. 2024-03-30T03:54:01.142 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-30T03:54:01.936 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:01 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:01.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:01 smithi023 ceph-mon[38254]: Added host smithi023 2024-03-30T03:54:01.937 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:01 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.598+0000 7f009d600700 1 Processor -- start 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.598+0000 7f009d600700 1 -- start start 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.599+0000 7f009d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.599+0000 7f009d600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0098106ed0 con 0x7f00981064e0 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.599+0000 7f0096a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.599+0000 7f0096a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55800/0 (socket says 172.21.15.23:55800) 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.599+0000 7f0096a00700 1 -- 172.21.15.23:0/4029532010 learned_addr learned my addr 172.21.15.23:0/4029532010 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.600+0000 7f0096a00700 1 -- 172.21.15.23:0/4029532010 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098107690 con 0x7f00981064e0 2024-03-30T03:54:02.278 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.600+0000 7f0096a00700 1 --2- 172.21.15.23:0/4029532010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00880098a0 tx=0x7f0088009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f012febb1fbcf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.600+0000 7f0095600700 1 -- 172.21.15.23:0/4029532010 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0088004160 con 0x7f00981064e0 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.600+0000 7f0095600700 1 -- 172.21.15.23:0/4029532010 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00880042c0 con 0x7f00981064e0 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.600+0000 7f0095600700 1 -- 172.21.15.23:0/4029532010 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0088011a00 con 0x7f00981064e0 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 -- 172.21.15.23:0/4029532010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 msgr2=0x7f0098106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 --2- 172.21.15.23:0/4029532010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00880098a0 tx=0x7f0088009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 -- 172.21.15.23:0/4029532010 shutdown_connections 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 --2- 172.21.15.23:0/4029532010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f0098106900 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 -- 172.21.15.23:0/4029532010 >> 172.21.15.23:0/4029532010 conn(0x7f0098101ae0 msgr2=0x7f0098103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 -- 172.21.15.23:0/4029532010 shutdown_connections 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.601+0000 7f009d600700 1 -- 172.21.15.23:0/4029532010 wait complete. 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.602+0000 7f009d600700 1 Processor -- start 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f009d600700 1 -- start start 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f009d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f009d600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009807c070 con 0x7f00981064e0 2024-03-30T03:54:02.279 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f0096a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f0096a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55810/0 (socket says 172.21.15.23:55810) 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.603+0000 7f0096a00700 1 -- 172.21.15.23:0/1382949375 learned_addr learned my addr 172.21.15.23:0/1382949375 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f0096a00700 1 -- 172.21.15.23:0/1382949375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0088009550 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f0096a00700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0088009130 tx=0x7f0088011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0088003ab0 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0088003c10 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009807a2b0 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00880247d0 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.604+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009807a750 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.605+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f008802d3f0 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.605+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009804f2d0 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.606+0000 7f008f400700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 0x7f007c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.606+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f008804c940 con 0x7f00981064e0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.606+0000 7f0096000700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 0x7f007c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:02.280 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.607+0000 7f0096000700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 0x7f007c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0084006fd0 tx=0x7f0084006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.611+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0088010970 con 0x7f00981064e0 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.844+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f009810a830 con 0x7f007c038750 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.849+0000 7f008f400700 1 -- 172.21.15.23:0/1382949375 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f009810a830 con 0x7f007c038750 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 msgr2=0x7f007c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 0x7f007c03ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0084006fd0 tx=0x7f0084006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 msgr2=0x7f009807bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0088009130 tx=0x7f0088011ad0 comp rx=0 tx=0).stop 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 shutdown_connections 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f007c038750 0x7f007c03ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 --2- 172.21.15.23:0/1382949375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f00981064e0 0x7f009807bb30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 >> 172.21.15.23:0/1382949375 conn(0x7f0098101ae0 msgr2=0x7f00981026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.851+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 shutdown_connections 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:01.852+0000 7f009d600700 1 -- 172.21.15.23:0/1382949375 wait complete. 2024-03-30T03:54:02.281 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-30T03:54:03.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:02 smithi023 ceph-mon[38254]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:03.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:02 smithi023 ceph-mon[38254]: Saving service mon spec with placement count:5 2024-03-30T03:54:03.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:02 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:03.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:02 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:03.137 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:02 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.702+0000 7f5a5ec00700 1 Processor -- start 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.702+0000 7f5a5ec00700 1 -- start start 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.702+0000 7f5a5ec00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.702+0000 7f5a5ec00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a58106ed0 con 0x7f5a581064e0 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.703+0000 7f5a57e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.703+0000 7f5a57e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55816/0 (socket says 172.21.15.23:55816) 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.703+0000 7f5a57e00700 1 -- 172.21.15.23:0/3606101319 learned_addr learned my addr 172.21.15.23:0/3606101319 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:03.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.703+0000 7f5a57e00700 1 -- 172.21.15.23:0/3606101319 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a58107690 con 0x7f5a581064e0 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a57e00700 1 --2- 172.21.15.23:0/3606101319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a480098a0 tx=0x7f5a48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb961b480995f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a56a00700 1 -- 172.21.15.23:0/3606101319 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a48004160 con 0x7f5a581064e0 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a56a00700 1 -- 172.21.15.23:0/3606101319 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a480042c0 con 0x7f5a581064e0 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a56a00700 1 -- 172.21.15.23:0/3606101319 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a480119b0 con 0x7f5a581064e0 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a5ec00700 1 -- 172.21.15.23:0/3606101319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 msgr2=0x7f5a58106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.704+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/3606101319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5a480098a0 tx=0x7f5a48009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.705+0000 7f5a5ec00700 1 -- 172.21.15.23:0/3606101319 shutdown_connections 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.705+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/3606101319 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a58106900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.705+0000 7f5a5ec00700 1 -- 172.21.15.23:0/3606101319 >> 172.21.15.23:0/3606101319 conn(0x7f5a58101ae0 msgr2=0x7f5a58103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.705+0000 7f5a5ec00700 1 -- 172.21.15.23:0/3606101319 shutdown_connections 2024-03-30T03:54:03.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.705+0000 7f5a5ec00700 1 -- 172.21.15.23:0/3606101319 wait complete. 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.706+0000 7f5a5ec00700 1 Processor -- start 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.706+0000 7f5a5ec00700 1 -- start start 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.706+0000 7f5a5ec00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.706+0000 7f5a5ec00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5819c960 con 0x7f5a581064e0 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.706+0000 7f5a57e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a57e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55828/0 (socket says 172.21.15.23:55828) 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a57e00700 1 -- 172.21.15.23:0/1453523163 learned_addr learned my addr 172.21.15.23:0/1453523163 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a57e00700 1 -- 172.21.15.23:0/1453523163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48009550 con 0x7f5a581064e0 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a57e00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a48003e60 tx=0x7f5a48011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a48003a60 con 0x7f5a581064e0 2024-03-30T03:54:03.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a48003bc0 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.707+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a480196c0 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.708+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5819cb60 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.708+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5819d000 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.709+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f5a4801a430 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.709+0000 7f5a54c00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 0x7f5a3c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.709+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5a4804c9a0 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.709+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5810a8c0 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.709+0000 7f5a57400700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 0x7f5a3c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.710+0000 7f5a57400700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 0x7f5a3c03abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a44006fd0 tx=0x7f5a44006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.714+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5a4802c3a0 con 0x7f5a581064e0 2024-03-30T03:54:03.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.945+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5a5819f890 con 0x7f5a3c038730 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.951+0000 7f5a54c00700 1 -- 172.21.15.23:0/1453523163 <== mgr.14122 v2:172.21.15.23:6800/604121731 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5a5819f890 con 0x7f5a3c038730 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 msgr2=0x7f5a3c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 0x7f5a3c03abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5a44006fd0 tx=0x7f5a44006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 msgr2=0x7f5a5819c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5a48003e60 tx=0x7f5a48011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 shutdown_connections 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f5a3c038730 0x7f5a3c03abf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 --2- 172.21.15.23:0/1453523163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a581064e0 0x7f5a5819c420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.953+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 >> 172.21.15.23:0/1453523163 conn(0x7f5a58101ae0 msgr2=0x7f5a581026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.954+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 shutdown_connections 2024-03-30T03:54:03.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:02.954+0000 7f5a5ec00700 1 -- 172.21.15.23:0/1453523163 wait complete. 2024-03-30T03:54:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:03 smithi023 ceph-mon[38254]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:03 smithi023 ceph-mon[38254]: Saving service mgr spec with placement count:2 2024-03-30T03:54:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:03 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:03 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:04.497 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.889+0000 7ff144400700 1 Processor -- start 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.889+0000 7ff144400700 1 -- start start 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.890+0000 7ff144400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.890+0000 7ff144400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c07a830 con 0x7ff13c07be40 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.890+0000 7ff141a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.890+0000 7ff141a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55842/0 (socket says 172.21.15.23:55842) 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.890+0000 7ff141a00700 1 -- 172.21.15.23:0/4162010358 learned_addr learned my addr 172.21.15.23:0/4162010358 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.891+0000 7ff141a00700 1 -- 172.21.15.23:0/4162010358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13c07a970 con 0x7ff13c07be40 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.891+0000 7ff141a00700 1 --2- 172.21.15.23:0/4162010358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1300098a0 tx=0x7ff130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ff941144a62a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.891+0000 7ff13be00700 1 -- 172.21.15.23:0/4162010358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff130004160 con 0x7ff13c07be40 2024-03-30T03:54:04.498 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.891+0000 7ff13be00700 1 -- 172.21.15.23:0/4162010358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff1300042c0 con 0x7ff13c07be40 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.891+0000 7ff13be00700 1 -- 172.21.15.23:0/4162010358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1300119b0 con 0x7ff13c07be40 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 -- 172.21.15.23:0/4162010358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 msgr2=0x7ff13c07a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 --2- 172.21.15.23:0/4162010358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff1300098a0 tx=0x7ff130009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 -- 172.21.15.23:0/4162010358 shutdown_connections 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 --2- 172.21.15.23:0/4162010358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c07a2f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 -- 172.21.15.23:0/4162010358 >> 172.21.15.23:0/4162010358 conn(0x7ff13c101ab0 msgr2=0x7ff13c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 -- 172.21.15.23:0/4162010358 shutdown_connections 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.892+0000 7ff144400700 1 -- 172.21.15.23:0/4162010358 wait complete. 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.893+0000 7ff144400700 1 Processor -- start 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.893+0000 7ff144400700 1 -- start start 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff144400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff144400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13c19c8f0 con 0x7ff13c07be40 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff141a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:04.499 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff141a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55844/0 (socket says 172.21.15.23:55844) 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff141a00700 1 -- 172.21.15.23:0/432880650 learned_addr learned my addr 172.21.15.23:0/432880650 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff141a00700 1 -- 172.21.15.23:0/432880650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff130009550 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.894+0000 7ff141a00700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff130011670 tx=0x7ff130011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.895+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff130003bf0 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.895+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13c19caf0 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.895+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff13001a8f0 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.895+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff130019840 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.895+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13c19cf90 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.896+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7ff13001a430 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.896+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13c10c830 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.896+0000 7ff13a000700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 0x7ff12403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.896+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff13004caa0 con 0x7ff13c07be40 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.897+0000 7ff141000700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 0x7ff12403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:04.500 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.897+0000 7ff141000700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 0x7ff12403abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff128006fd0 tx=0x7ff128006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:03.902+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff13001ad40 con 0x7ff13c07be40 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.119+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff13c10ca70 con 0x7ff13c07be40 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.123+0000 7ff13a000700 1 -- 172.21.15.23:0/432880650 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff13002d960 con 0x7ff13c07be40 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 msgr2=0x7ff12403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 0x7ff12403abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff128006fd0 tx=0x7ff128006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 msgr2=0x7ff13c19c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff130011670 tx=0x7ff130011aa0 comp rx=0 tx=0).stop 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 shutdown_connections 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7ff124038700 0x7ff12403abc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 --2- 172.21.15.23:0/432880650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff13c07be40 0x7ff13c19c3b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.126+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 >> 172.21.15.23:0/432880650 conn(0x7ff13c101ab0 msgr2=0x7ff13c102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.127+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 shutdown_connections 2024-03-30T03:54:04.501 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.127+0000 7ff144400700 1 -- 172.21.15.23:0/432880650 wait complete. 2024-03-30T03:54:05.386 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:05 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/432880650' entity='client.admin' 2024-03-30T03:54:05.386 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:05 smithi023 ceph-mon[38254]: from='mgr.14122 172.21.15.23:0/605098029' entity='mgr.x' 2024-03-30T03:54:05.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.920+0000 7fee7a200700 1 Processor -- start 2024-03-30T03:54:05.533 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.920+0000 7fee7a200700 1 -- start start 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.920+0000 7fee7a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.920+0000 7fee7a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74079e40 con 0x7fee74106fb0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.921+0000 7fee73400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.921+0000 7fee73400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55850/0 (socket says 172.21.15.23:55850) 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.921+0000 7fee73400700 1 -- 172.21.15.23:0/3451585156 learned_addr learned my addr 172.21.15.23:0/3451585156 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.921+0000 7fee73400700 1 -- 172.21.15.23:0/3451585156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee741098e0 con 0x7fee74106fb0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.921+0000 7fee73400700 1 --2- 172.21.15.23:0/3451585156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5834bd74fc3303e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee72000700 1 -- 172.21.15.23:0/3451585156 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee64004160 con 0x7fee74106fb0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee72000700 1 -- 172.21.15.23:0/3451585156 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee640042c0 con 0x7fee74106fb0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee72000700 1 -- 172.21.15.23:0/3451585156 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee640119b0 con 0x7fee74106fb0 2024-03-30T03:54:05.534 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee7a200700 1 -- 172.21.15.23:0/3451585156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 msgr2=0x7fee741093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee7a200700 1 --2- 172.21.15.23:0/3451585156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee640098a0 tx=0x7fee64009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee7a200700 1 -- 172.21.15.23:0/3451585156 shutdown_connections 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee7a200700 1 --2- 172.21.15.23:0/3451585156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee741093a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.922+0000 7fee7a200700 1 -- 172.21.15.23:0/3451585156 >> 172.21.15.23:0/3451585156 conn(0x7fee741009c0 msgr2=0x7fee74102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.923+0000 7fee7a200700 1 -- 172.21.15.23:0/3451585156 shutdown_connections 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.923+0000 7fee7a200700 1 -- 172.21.15.23:0/3451585156 wait complete. 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee7a200700 1 Processor -- start 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee7a200700 1 -- start start 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee7a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee7a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee7419c950 con 0x7fee74106fb0 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee73400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee73400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55862/0 (socket says 172.21.15.23:55862) 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.924+0000 7fee73400700 1 -- 172.21.15.23:0/3422523655 learned_addr learned my addr 172.21.15.23:0/3422523655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:05.535 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee73400700 1 -- 172.21.15.23:0/3422523655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee64009550 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee73400700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee64011670 tx=0x7fee64011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee64003b20 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee64003c80 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee64019770 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7419cb50 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.925+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7419cff0 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.926+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7fee6401a430 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.926+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7410a670 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.927+0000 7fee6be00700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 0x7fee5803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.927+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fee6404c9d0 con 0x7fee74106fb0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.927+0000 7fee72a00700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 0x7fee5803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.928+0000 7fee72a00700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 0x7fee5803ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:05.536 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:04.932+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fee6402d9a0 con 0x7fee74106fb0 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.163+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fee7410a8b0 con 0x7fee74106fb0 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.166+0000 7fee6be00700 1 -- 172.21.15.23:0/3422523655 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fee6402d350 con 0x7fee74106fb0 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 msgr2=0x7fee5803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 0x7fee5803ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fee60006fd0 tx=0x7fee60006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 msgr2=0x7fee7419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee64011670 tx=0x7fee64011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 shutdown_connections 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fee58038750 0x7fee5803ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 --2- 172.21.15.23:0/3422523655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fee74106fb0 0x7fee7419c410 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 >> 172.21.15.23:0/3422523655 conn(0x7fee741009c0 msgr2=0x7fee74101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.169+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 shutdown_connections 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:05.170+0000 7fee7a200700 1 -- 172.21.15.23:0/3422523655 wait complete. 2024-03-30T03:54:05.537 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-30T03:54:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:06 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3422523655' entity='client.admin' 2024-03-30T03:54:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1280462221' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-30T03:54:07.523 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setuser ceph since I am not root 2024-03-30T03:54:07.523 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setgroup ceph since I am not root 2024-03-30T03:54:07.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.246+0000 7f4158c00700 1 Processor -- start 2024-03-30T03:54:07.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.246+0000 7f4158c00700 1 -- start start 2024-03-30T03:54:07.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.247+0000 7f4158c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:07.584 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.247+0000 7f4158c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4154079e80 con 0x7f4154106f70 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.247+0000 7f4151e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.247+0000 7f4151e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55866/0 (socket says 172.21.15.23:55866) 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.247+0000 7f4151e00700 1 -- 172.21.15.23:0/3038250311 learned_addr learned my addr 172.21.15.23:0/3038250311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.248+0000 7f4151e00700 1 -- 172.21.15.23:0/3038250311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41541098a0 con 0x7f4154106f70 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.248+0000 7f4151e00700 1 --2- 172.21.15.23:0/3038250311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41440098a0 tx=0x7f4144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d2a79b52ac1b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.248+0000 7f4150a00700 1 -- 172.21.15.23:0/3038250311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4144004160 con 0x7f4154106f70 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.248+0000 7f4150a00700 1 -- 172.21.15.23:0/3038250311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41440042c0 con 0x7f4154106f70 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.248+0000 7f4150a00700 1 -- 172.21.15.23:0/3038250311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41440119b0 con 0x7f4154106f70 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 -- 172.21.15.23:0/3038250311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 msgr2=0x7f4154109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:07.585 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 --2- 172.21.15.23:0/3038250311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41440098a0 tx=0x7f4144009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 -- 172.21.15.23:0/3038250311 shutdown_connections 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 --2- 172.21.15.23:0/3038250311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f4154109360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 -- 172.21.15.23:0/3038250311 >> 172.21.15.23:0/3038250311 conn(0x7f41541009c0 msgr2=0x7f4154102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 -- 172.21.15.23:0/3038250311 shutdown_connections 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.249+0000 7f4158c00700 1 -- 172.21.15.23:0/3038250311 wait complete. 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.250+0000 7f4158c00700 1 Processor -- start 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.250+0000 7f4158c00700 1 -- start start 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.250+0000 7f4158c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.250+0000 7f4158c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415419a780 con 0x7f4154106f70 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4151e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4151e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55874/0 (socket says 172.21.15.23:55874) 2024-03-30T03:54:07.586 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4151e00700 1 -- 172.21.15.23:0/1280462221 learned_addr learned my addr 172.21.15.23:0/1280462221 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4151e00700 1 -- 172.21.15.23:0/1280462221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4144009550 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4151e00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4144011670 tx=0x7f4144011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4144003b20 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4144003c80 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.251+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415419a980 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.252+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4144019770 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.252+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415419ae20 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.253+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415410a680 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.253+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 45533+0+0 (secure 0 0 0) 0x7f414401a430 con 0x7f4154106f70 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.253+0000 7f414aa00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 0x7f413803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:07.587 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.253+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f414404c910 con 0x7f4154106f70 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.254+0000 7f4151400700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 0x7f413803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.254+0000 7f4151400700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 0x7f413803ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4140006fd0 tx=0x7f4140006e40 comp rx=0 tx=0).ready entity=mgr.14122 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.259+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f414402d960 con 0x7f4154106f70 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:06.533+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f415410a8c0 con 0x7f4154106f70 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.180+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f414402bd00 con 0x7f4154106f70 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.182+0000 7f414aa00700 1 -- 172.21.15.23:0/1280462221 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f4144021c80 con 0x7f4154106f70 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.185+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 msgr2=0x7f413803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.185+0000 7f4158c00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 0x7f413803ac10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4140006fd0 tx=0x7f4140006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.185+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 msgr2=0x7f415419a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4144011670 tx=0x7f4144011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 shutdown_connections 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7f4138038750 0x7f413803ac10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 --2- 172.21.15.23:0/1280462221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4154106f70 0x7f415419a240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:07.588 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 >> 172.21.15.23:0/1280462221 conn(0x7f41541009c0 msgr2=0x7f4154101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:07.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 shutdown_connections 2024-03-30T03:54:07.589 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.186+0000 7f4158c00700 1 -- 172.21.15.23:0/1280462221 wait complete. 2024-03-30T03:54:08.179 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:07.831+0000 7f45c6953200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T03:54:08.180 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:07.956+0000 7f45c6953200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T03:54:08.180 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:08.078+0000 7f45c6953200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T03:54:08.430 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:08 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1280462221' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-30T03:54:08.431 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:08 smithi023 ceph-mon[38254]: mgrmap e9: x(active, since 15s) 2024-03-30T03:54:08.433 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:08.194+0000 7f45c6953200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.961+0000 7fe962e00700 1 Processor -- start 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.961+0000 7fe962e00700 1 -- start start 2024-03-30T03:54:08.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.961+0000 7fe962e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.961+0000 7fe962e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c07a810 con 0x7fe95c07be20 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.962+0000 7fe95be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.962+0000 7fe95be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55902/0 (socket says 172.21.15.23:55902) 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.962+0000 7fe95be00700 1 -- 172.21.15.23:0/268336633 learned_addr learned my addr 172.21.15.23:0/268336633 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.962+0000 7fe95be00700 1 -- 172.21.15.23:0/268336633 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c07a950 con 0x7fe95c07be20 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.962+0000 7fe95be00700 1 --2- 172.21.15.23:0/268336633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0098a0 tx=0x7fe94c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e1d1fd81be231f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe95aa00700 1 -- 172.21.15.23:0/268336633 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe94c004160 con 0x7fe95c07be20 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe95aa00700 1 -- 172.21.15.23:0/268336633 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe94c0042c0 con 0x7fe95c07be20 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe95aa00700 1 -- 172.21.15.23:0/268336633 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe94c0119b0 con 0x7fe95c07be20 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 -- 172.21.15.23:0/268336633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 msgr2=0x7fe95c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 --2- 172.21.15.23:0/268336633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe94c0098a0 tx=0x7fe94c009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 -- 172.21.15.23:0/268336633 shutdown_connections 2024-03-30T03:54:08.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 --2- 172.21.15.23:0/268336633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c07a2d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 -- 172.21.15.23:0/268336633 >> 172.21.15.23:0/268336633 conn(0x7fe95c101ab0 msgr2=0x7fe95c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.963+0000 7fe962e00700 1 -- 172.21.15.23:0/268336633 shutdown_connections 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.964+0000 7fe962e00700 1 -- 172.21.15.23:0/268336633 wait complete. 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.965+0000 7fe962e00700 1 Processor -- start 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.965+0000 7fe962e00700 1 -- start start 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe962e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe962e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95c19c8f0 con 0x7fe95c07be20 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe95be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe95be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55908/0 (socket says 172.21.15.23:55908) 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe95be00700 1 -- 172.21.15.23:0/3723562789 learned_addr learned my addr 172.21.15.23:0/3723562789 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe95be00700 1 -- 172.21.15.23:0/3723562789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c009550 con 0x7fe95c07be20 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.966+0000 7fe95be00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe94c011670 tx=0x7fe94c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.967+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe94c003b20 con 0x7fe95c07be20 2024-03-30T03:54:08.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.967+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe94c003c80 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.967+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95c19caf0 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.967+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe94c019770 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.967+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95c19cf90 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.968+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95c10c850 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.968+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fe94c0199e0 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.969+0000 7fe958c00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 0x7fe94003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.969+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe94c04caf0 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.969+0000 7fe95b400700 1 -- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 msgr2=0x7fe94003ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.969+0000 7fe95b400700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 0x7fe94003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:07.974+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe94c018db0 con 0x7fe95c07be20 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.170+0000 7fe95b400700 1 -- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 msgr2=0x7fe94003ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.170+0000 7fe95b400700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 0x7fe94003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T03:54:08.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.249+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe95c19f880 con 0x7fe95c07be20 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.249+0000 7fe958c00700 1 -- 172.21.15.23:0/3723562789 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe94c018350 con 0x7fe95c07be20 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 msgr2=0x7fe94003ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 0x7fe94003ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 msgr2=0x7fe95c19c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe94c011670 tx=0x7fe94c011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 shutdown_connections 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fe9400387a0 0x7fe94003ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 --2- 172.21.15.23:0/3723562789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95c07be20 0x7fe95c19c3b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 >> 172.21.15.23:0/3723562789 conn(0x7fe95c101ab0 msgr2=0x7fe95c102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 shutdown_connections 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:08.252+0000 7fe962e00700 1 -- 172.21.15.23:0/3723562789 wait complete. 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-30T03:54:08.561 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-30T03:54:09.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3723562789' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-30T03:54:09.418 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:09.090+0000 7f45c6953200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:54:09.676 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:09.417+0000 7f45c6953200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T03:54:09.676 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:09.531+0000 7f45c6953200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T03:54:10.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:09.675+0000 7f45c6953200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T03:54:10.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:09.877+0000 7f45c6953200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T03:54:10.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:10.051+0000 7f45c6953200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T03:54:10.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:10.166+0000 7f45c6953200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T03:54:11.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:10.647+0000 7f45c6953200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T03:54:11.412 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:11.153+0000 7f45c6953200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T03:54:11.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:11.411+0000 7f45c6953200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T03:54:11.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:11.540+0000 7f45c6953200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T03:54:12.865 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:12.533+0000 7f45c6953200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T03:54:12.865 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:12.664+0000 7f45c6953200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T03:54:13.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:12.865+0000 7f45c6953200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T03:54:14.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:13.705+0000 7f45c6953200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T03:54:14.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:13.953+0000 7f45c6953200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T03:54:14.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:14.219+0000 7f45c6953200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T03:54:15.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:15.411+0000 7f45c6953200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T03:54:16.194 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:15.803+0000 7f45c6953200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:54:16.194 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:15.929+0000 7f45c6953200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T03:54:16.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:16.194+0000 7f45c6953200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T03:54:17.239 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: Active manager daemon x restarted 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: Activating manager daemon x 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: osdmap e3: 0 total, 0 up, 0 in 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: mgrmap e10: x(active, starting, since 0.00689195s) 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: Manager daemon x is now available 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T03:54:17.240 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T03:54:17.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-30T03:54:17.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-30T03:54:17.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-30T03:54:17.660 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.028+0000 7fc5e8200700 1 Processor -- start 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.028+0000 7fc5e8200700 1 -- start start 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.028+0000 7fc5e8200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.028+0000 7fc5e8200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e007a810 con 0x7fc5e007be20 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.029+0000 7fc5e5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.029+0000 7fc5e5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44304/0 (socket says 172.21.15.23:44304) 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.029+0000 7fc5e5800700 1 -- 172.21.15.23:0/3604434687 learned_addr learned my addr 172.21.15.23:0/3604434687 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.029+0000 7fc5e5800700 1 -- 172.21.15.23:0/3604434687 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e007a950 con 0x7fc5e007be20 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.029+0000 7fc5e5800700 1 --2- 172.21.15.23:0/3604434687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40098a0 tx=0x7fc5d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14573d57859bd84f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5dfe00700 1 -- 172.21.15.23:0/3604434687 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d4004160 con 0x7fc5e007be20 2024-03-30T03:54:17.661 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5dfe00700 1 -- 172.21.15.23:0/3604434687 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d40042c0 con 0x7fc5e007be20 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5dfe00700 1 -- 172.21.15.23:0/3604434687 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d40119b0 con 0x7fc5e007be20 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5e8200700 1 -- 172.21.15.23:0/3604434687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 msgr2=0x7fc5e007a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5e8200700 1 --2- 172.21.15.23:0/3604434687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40098a0 tx=0x7fc5d4009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5e8200700 1 -- 172.21.15.23:0/3604434687 shutdown_connections 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5e8200700 1 --2- 172.21.15.23:0/3604434687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e007a2d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.030+0000 7fc5e8200700 1 -- 172.21.15.23:0/3604434687 >> 172.21.15.23:0/3604434687 conn(0x7fc5e0101ae0 msgr2=0x7fc5e0103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.031+0000 7fc5e8200700 1 -- 172.21.15.23:0/3604434687 shutdown_connections 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.031+0000 7fc5e8200700 1 -- 172.21.15.23:0/3604434687 wait complete. 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.031+0000 7fc5e8200700 1 Processor -- start 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e8200700 1 -- start start 2024-03-30T03:54:17.662 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e8200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e8200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e019c8f0 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44320/0 (socket says 172.21.15.23:44320) 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.032+0000 7fc5e5800700 1 -- 172.21.15.23:0/1004929606 learned_addr learned my addr 172.21.15.23:0/1004929606 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5e5800700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4009550 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5e5800700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4011670 tx=0x7fc5d4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d4003b20 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d4003c80 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5d4019770 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e019caf0 con 0x7fc5e007be20 2024-03-30T03:54:17.663 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.033+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e019cf90 con 0x7fc5e007be20 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.034+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fc5d401a430 con 0x7fc5e007be20 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.035+0000 7fc5de000700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.035+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5c803b370 con 0x7fc5c80387a0 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.035+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc5d404ca30 con 0x7fc5e007be20 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.035+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.035+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.236+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.236+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.637+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:09.637+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T03:54:17.664 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:10.438+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:10.438+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:12.039+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:12.039+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:15.243+0000 7fc5e4e00700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/604121731 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:15.243+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:16.202+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) v1 ==== 45313+0+0 (secure 0 0 0) 0x7fc5d4022d90 con 0x7fc5e007be20 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:16.202+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 msgr2=0x7fc5c803ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:16.202+0000 7fc5de000700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.203+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7fc5d402bd00 con 0x7fc5e007be20 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.203+0000 7fc5de000700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 0x7fc5c803e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.203+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc5c803b370 con 0x7fc5c803bff0 2024-03-30T03:54:17.665 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.203+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 0x7fc5c803e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.204+0000 7fc5e4e00700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 0x7fc5c803e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc003a10 tx=0x7fc5cc009270 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.207+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fc5c803b370 con 0x7fc5c803bff0 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc5e010c860 con 0x7fc5c803bff0 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5de000700 1 -- 172.21.15.23:0/1004929606 <== mgr.14154 v2:172.21.15.23:6800/2266065890 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc5e010c860 con 0x7fc5c803bff0 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 msgr2=0x7fc5c803e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 0x7fc5c803e3e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc003a10 tx=0x7fc5cc009270 comp rx=0 tx=0).stop 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 msgr2=0x7fc5e019c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4011670 tx=0x7fc5d4011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 shutdown_connections 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc5c803bff0 0x7fc5c803e3e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.666 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5e007be20 0x7fc5e019c3b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 --2- 172.21.15.23:0/1004929606 >> [v2:172.21.15.23:6800/604121731,v1:172.21.15.23:6801/604121731] conn(0x7fc5c80387a0 0x7fc5c803ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.216+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 >> 172.21.15.23:0/1004929606 conn(0x7fc5e0101ae0 msgr2=0x7fc5e0102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.217+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 shutdown_connections 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:17.217+0000 7fc5e8200700 1 -- 172.21.15.23:0/1004929606 wait complete. 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-30T03:54:17.667 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: [30/Mar/2024:03:54:16] ENGINE Bus STARTING 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: [30/Mar/2024:03:54:17] ENGINE Serving on http://172.21.15.23:8765 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: [30/Mar/2024:03:54:17] ENGINE Serving on https://172.21.15.23:7150 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: [30/Mar/2024:03:54:17] ENGINE Bus STARTED 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: mgrmap e11: x(active, since 1.00873s) 2024-03-30T03:54:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: from='client.14158 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-30T03:54:18.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: from='client.14158 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-30T03:54:18.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-30T03:54:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.107+0000 7f7cf6000700 1 Processor -- start 2024-03-30T03:54:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.107+0000 7f7cf6000700 1 -- start start 2024-03-30T03:54:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.107+0000 7f7cf6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.107+0000 7f7cf6000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf007a8e0 con 0x7f7cf007be60 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.108+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.108+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46954/0 (socket says 172.21.15.23:46954) 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.108+0000 7f7cef400700 1 -- 172.21.15.23:0/230162607 learned_addr learned my addr 172.21.15.23:0/230162607 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.108+0000 7f7cef400700 1 -- 172.21.15.23:0/230162607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf007aa20 con 0x7f7cf007be60 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.108+0000 7f7cef400700 1 --2- 172.21.15.23:0/230162607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00098a0 tx=0x7f7ce0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ae40e15993959b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cee000700 1 -- 172.21.15.23:0/230162607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0004160 con 0x7f7cf007be60 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cee000700 1 -- 172.21.15.23:0/230162607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce00042c0 con 0x7f7cf007be60 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cee000700 1 -- 172.21.15.23:0/230162607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce00119b0 con 0x7f7cf007be60 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 -- 172.21.15.23:0/230162607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 msgr2=0x7f7cf007a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 --2- 172.21.15.23:0/230162607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00098a0 tx=0x7f7ce0009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 -- 172.21.15.23:0/230162607 shutdown_connections 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 --2- 172.21.15.23:0/230162607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf007a310 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 -- 172.21.15.23:0/230162607 >> 172.21.15.23:0/230162607 conn(0x7f7cf0101ab0 msgr2=0x7f7cf0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 -- 172.21.15.23:0/230162607 shutdown_connections 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.109+0000 7f7cf6000700 1 -- 172.21.15.23:0/230162607 wait complete. 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.110+0000 7f7cf6000700 1 Processor -- start 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.110+0000 7f7cf6000700 1 -- start start 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cf6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cf6000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf019c900 con 0x7f7cf007be60 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46958/0 (socket says 172.21.15.23:46958) 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cef400700 1 -- 172.21.15.23:0/565712893 learned_addr learned my addr 172.21.15.23:0/565712893 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.111+0000 7f7cef400700 1 -- 172.21.15.23:0/565712893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce0009550 con 0x7f7cf007be60 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7cef400700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0011670 tx=0x7f7ce0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:18.723 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0003b20 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce0003c80 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ce0019770 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf019cb00 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.112+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf019cfa0 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.113+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 45440+0+0 (secure 0 0 0) 0x7f7ce001ab60 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.113+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf010c870 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.114+0000 7f7ce7e00700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 0x7f7cd403ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.114+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ce004c8b0 con 0x7f7cf007be60 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.114+0000 7f7ceea00700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 0x7f7cd403ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.114+0000 7f7ceea00700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 0x7f7cd403ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc006fd0 tx=0x7f7cdc006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:18.724 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.119+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7ce001a430 con 0x7f7cf007be60 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.340+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cf019f830 con 0x7f7cd4038670 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.403+0000 7f7ce7e00700 1 -- 172.21.15.23:0/565712893 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f7cf019f830 con 0x7f7cd4038670 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 msgr2=0x7f7cd403ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 0x7f7cd403ab30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc006fd0 tx=0x7f7cdc006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 msgr2=0x7f7cf019c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0011670 tx=0x7f7ce0011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 shutdown_connections 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7cd4038670 0x7f7cd403ab30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 --2- 172.21.15.23:0/565712893 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf007be60 0x7f7cf019c3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 >> 172.21.15.23:0/565712893 conn(0x7f7cf0101ab0 msgr2=0x7f7cf0102660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.405+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 shutdown_connections 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:18.406+0000 7f7cf6000700 1 -- 172.21.15.23:0/565712893 wait complete. 2024-03-30T03:54:18.725 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-30T03:54:19.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:19 smithi023 ceph-mon[38254]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:19.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:19 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:19.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:19 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:19.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:19 smithi023 ceph-mon[38254]: mgrmap e12: x(active, since 2s) 2024-03-30T03:54:19.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:19 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$u1KVyM1JQDD58gkldlgAJu7OubRKXbwIQiIialhvPYdL5yPsgleYu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711770859, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.123+0000 7f266aa00700 1 Processor -- start 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.124+0000 7f266aa00700 1 -- start start 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.124+0000 7f266aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.124+0000 7f266aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266407a830 con 0x7f266407be40 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.124+0000 7f2663e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:20.041 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2663e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46970/0 (socket says 172.21.15.23:46970) 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2663e00700 1 -- 172.21.15.23:0/352636647 learned_addr learned my addr 172.21.15.23:0/352636647 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2663e00700 1 -- 172.21.15.23:0/352636647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266407a970 con 0x7f266407be40 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2663e00700 1 --2- 172.21.15.23:0/352636647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26540098a0 tx=0x7f2654009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d538a6ce1fbf0cbf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2662a00700 1 -- 172.21.15.23:0/352636647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2654004160 con 0x7f266407be40 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2662a00700 1 -- 172.21.15.23:0/352636647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26540042c0 con 0x7f266407be40 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.125+0000 7f2662a00700 1 -- 172.21.15.23:0/352636647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26540119b0 con 0x7f266407be40 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.126+0000 7f266aa00700 1 -- 172.21.15.23:0/352636647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 msgr2=0x7f266407a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.126+0000 7f266aa00700 1 --2- 172.21.15.23:0/352636647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26540098a0 tx=0x7f2654009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.127+0000 7f266aa00700 1 -- 172.21.15.23:0/352636647 shutdown_connections 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.127+0000 7f266aa00700 1 --2- 172.21.15.23:0/352636647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266407a2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.127+0000 7f266aa00700 1 -- 172.21.15.23:0/352636647 >> 172.21.15.23:0/352636647 conn(0x7f2664101ae0 msgr2=0x7f2664103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.127+0000 7f266aa00700 1 -- 172.21.15.23:0/352636647 shutdown_connections 2024-03-30T03:54:20.042 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.127+0000 7f266aa00700 1 -- 172.21.15.23:0/352636647 wait complete. 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.128+0000 7f266aa00700 1 Processor -- start 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f266aa00700 1 -- start start 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f266aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f266aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266419c8e0 con 0x7f266407be40 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f2663e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f2663e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46974/0 (socket says 172.21.15.23:46974) 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.129+0000 7f2663e00700 1 -- 172.21.15.23:0/3261181294 learned_addr learned my addr 172.21.15.23:0/3261181294 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f2663e00700 1 -- 172.21.15.23:0/3261181294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654009550 con 0x7f266407be40 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f2663e00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2654011670 tx=0x7f2654011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2654003bf0 con 0x7f266407be40 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f265401a8f0 con 0x7f266407be40 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266419cae0 con 0x7f266407be40 2024-03-30T03:54:20.043 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.130+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2654019840 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.131+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266419cf80 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.132+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266410c840 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.132+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f265401aa60 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.133+0000 7f2660c00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 0x7f264803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.133+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f265404cc20 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.133+0000 7f2663400700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 0x7f264803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.133+0000 7f2663400700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 0x7f264803ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.138+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f265402d950 con 0x7f266407be40 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.381+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f2664062200 con 0x7f26480387a0 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.629+0000 7f2660c00700 1 -- 172.21.15.23:0/3261181294 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f2664062200 con 0x7f26480387a0 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 msgr2=0x7f264803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:20.044 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 0x7f264803ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f264c006fd0 tx=0x7f264c006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 msgr2=0x7f266419c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2654011670 tx=0x7f2654011aa0 comp rx=0 tx=0).stop 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 shutdown_connections 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26480387a0 0x7f264803ac60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 --2- 172.21.15.23:0/3261181294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266407be40 0x7f266419c3a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 >> 172.21.15.23:0/3261181294 conn(0x7f2664101ae0 msgr2=0x7f2664102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 shutdown_connections 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:19.631+0000 7f266aa00700 1 -- 172.21.15.23:0/3261181294 wait complete. 2024-03-30T03:54:20.045 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-30T03:54:20.966 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:20 smithi023 ceph-mon[38254]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:20.967 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.561+0000 7f5645600700 1 Processor -- start 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f5645600700 1 -- start start 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f5645600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f5645600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564007a830 con 0x7f564007be40 2024-03-30T03:54:21.122 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f563ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f563ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46990/0 (socket says 172.21.15.23:46990) 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.562+0000 7f563ea00700 1 -- 172.21.15.23:0/468400109 learned_addr learned my addr 172.21.15.23:0/468400109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.563+0000 7f563ea00700 1 -- 172.21.15.23:0/468400109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564007a970 con 0x7f564007be40 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.563+0000 7f563ea00700 1 --2- 172.21.15.23:0/468400109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56300098a0 tx=0x7f5630009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fe6908594c78b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.563+0000 7f563d600700 1 -- 172.21.15.23:0/468400109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5630004160 con 0x7f564007be40 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.563+0000 7f563d600700 1 -- 172.21.15.23:0/468400109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56300042c0 con 0x7f564007be40 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.563+0000 7f563d600700 1 -- 172.21.15.23:0/468400109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56300119b0 con 0x7f564007be40 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 -- 172.21.15.23:0/468400109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 msgr2=0x7f564007a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 --2- 172.21.15.23:0/468400109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f56300098a0 tx=0x7f5630009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 -- 172.21.15.23:0/468400109 shutdown_connections 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 --2- 172.21.15.23:0/468400109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564007a2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 -- 172.21.15.23:0/468400109 >> 172.21.15.23:0/468400109 conn(0x7f5640101ae0 msgr2=0x7f5640103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:21.123 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 -- 172.21.15.23:0/468400109 shutdown_connections 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.564+0000 7f5645600700 1 -- 172.21.15.23:0/468400109 wait complete. 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.565+0000 7f5645600700 1 Processor -- start 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.565+0000 7f5645600700 1 -- start start 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.565+0000 7f5645600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.565+0000 7f5645600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564019c960 con 0x7f564007be40 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f563ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f563ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47000/0 (socket says 172.21.15.23:47000) 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f563ea00700 1 -- 172.21.15.23:0/3695992160 learned_addr learned my addr 172.21.15.23:0/3695992160 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f563ea00700 1 -- 172.21.15.23:0/3695992160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630009550 con 0x7f564007be40 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f563ea00700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5630011670 tx=0x7f5630011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5630003bf0 con 0x7f564007be40 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f563001a8f0 con 0x7f564007be40 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.566+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564019cb60 con 0x7f564007be40 2024-03-30T03:54:21.124 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.567+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5630019840 con 0x7f564007be40 2024-03-30T03:54:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.567+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564019d000 con 0x7f564007be40 2024-03-30T03:54:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.568+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7f563001a430 con 0x7f564007be40 2024-03-30T03:54:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.568+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564010c860 con 0x7f564007be40 2024-03-30T03:54:21.125 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.568+0000 7f5637400700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 0x7f562403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:21.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.568+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f563004cc20 con 0x7f564007be40 2024-03-30T03:54:21.126 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.569+0000 7f563e000700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 0x7f562403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.569+0000 7f563e000700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 0x7f562403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f562c006fd0 tx=0x7f562c006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.573+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f563000b3e0 con 0x7f564007be40 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.786+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f564010cad0 con 0x7f564007be40 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.787+0000 7f5637400700 1 -- 172.21.15.23:0/3695992160 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5630018350 con 0x7f564007be40 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 msgr2=0x7f562403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 0x7f562403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f562c006fd0 tx=0x7f562c006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 msgr2=0x7f564019c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:21.127 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5630011670 tx=0x7f5630011aa0 comp rx=0 tx=0).stop 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 shutdown_connections 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5624038750 0x7f562403ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 --2- 172.21.15.23:0/3695992160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f564007be40 0x7f564019c420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 >> 172.21.15.23:0/3695992160 conn(0x7f5640101ae0 msgr2=0x7f56401026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 shutdown_connections 2024-03-30T03:54:21.128 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:20.789+0000 7f5645600700 1 -- 172.21.15.23:0/3695992160 wait complete. 2024-03-30T03:54:21.138 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-30T03:54:21.139 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-30T03:54:21.154 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-30T03:54:21.154 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-30T03:54:21.154 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-30T03:54:21.154 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-30T03:54:21.156 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-30T03:54:21.156 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:21.156 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-30T03:54:21.156 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-30T03:54:21.157 INFO:teuthology.orchestra.run.smithi023.stdout: Password: 26ua5hdeiq 2024-03-30T03:54:21.157 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:21.157 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config directory 2024-03-30T03:54:21.157 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-30T03:54:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:21 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3695992160' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-30T03:54:22.271 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.704+0000 7fb143000700 1 Processor -- start 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.704+0000 7fb143000700 1 -- start start 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.705+0000 7fb143000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.705+0000 7fb143000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c07a900 con 0x7fb13c07be80 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.705+0000 7fb13be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.705+0000 7fb13be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47002/0 (socket says 172.21.15.23:47002) 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.705+0000 7fb13be00700 1 -- 172.21.15.23:0/1154163924 learned_addr learned my addr 172.21.15.23:0/1154163924 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.706+0000 7fb13be00700 1 -- 172.21.15.23:0/1154163924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c07aa40 con 0x7fb13c07be80 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.706+0000 7fb13be00700 1 --2- 172.21.15.23:0/1154163924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098a0 tx=0x7fb12c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef90f705ee53419b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.706+0000 7fb13aa00700 1 -- 172.21.15.23:0/1154163924 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb12c004160 con 0x7fb13c07be80 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.706+0000 7fb13aa00700 1 -- 172.21.15.23:0/1154163924 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12c0042c0 con 0x7fb13c07be80 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.706+0000 7fb13aa00700 1 -- 172.21.15.23:0/1154163924 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb12c0119b0 con 0x7fb13c07be80 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 -- 172.21.15.23:0/1154163924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 msgr2=0x7fb13c07a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:22.272 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 --2- 172.21.15.23:0/1154163924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098a0 tx=0x7fb12c009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 -- 172.21.15.23:0/1154163924 shutdown_connections 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 --2- 172.21.15.23:0/1154163924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c07a330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 -- 172.21.15.23:0/1154163924 >> 172.21.15.23:0/1154163924 conn(0x7fb13c101ae0 msgr2=0x7fb13c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 -- 172.21.15.23:0/1154163924 shutdown_connections 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.707+0000 7fb143000700 1 -- 172.21.15.23:0/1154163924 wait complete. 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.708+0000 7fb143000700 1 Processor -- start 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.708+0000 7fb143000700 1 -- start start 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.708+0000 7fb143000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.708+0000 7fb143000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c19c950 con 0x7fb13c07be80 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb13be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb13be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47016/0 (socket says 172.21.15.23:47016) 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb13be00700 1 -- 172.21.15.23:0/1845667586 learned_addr learned my addr 172.21.15.23:0/1845667586 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb13be00700 1 -- 172.21.15.23:0/1845667586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c009550 con 0x7fb13c07be80 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb13be00700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb12c011670 tx=0x7fb12c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:22.273 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb12c003b20 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c19cb50 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.709+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb12c003c80 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.710+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb12c0197e0 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.710+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c19cff0 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.711+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 45546+0+0 (secure 0 0 0) 0x7fb12c01ab60 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.711+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c10c880 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.711+0000 7fb138c00700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 0x7fb12003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.711+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb12c04ce60 con 0x7fb13c07be80 2024-03-30T03:54:22.274 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.711+0000 7fb13b400700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 0x7fb12003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.712+0000 7fb13b400700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 0x7fb12003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb128006fd0 tx=0x7fb128006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.717+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb12c01a430 con 0x7fb13c07be80 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.928+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb13c10cac0 con 0x7fb13c07be80 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.929+0000 7fb138c00700 1 -- 172.21.15.23:0/1845667586 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb12c00b3e0 con 0x7fb13c07be80 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 msgr2=0x7fb12003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 0x7fb12003ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb128006fd0 tx=0x7fb128006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 msgr2=0x7fb13c19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb12c011670 tx=0x7fb12c011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:22.275 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 shutdown_connections 2024-03-30T03:54:22.276 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb120038750 0x7fb12003ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:22.276 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 --2- 172.21.15.23:0/1845667586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c07be80 0x7fb13c19c410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:22.276 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 >> 172.21.15.23:0/1845667586 conn(0x7fb13c101ae0 msgr2=0x7fb13c102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:22.276 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 shutdown_connections 2024-03-30T03:54:22.276 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:21.931+0000 7fb143000700 1 -- 172.21.15.23:0/1845667586 wait complete. 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.677+0000 7fb3e2200700 1 Processor -- start 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.677+0000 7fb3e2200700 1 -- start start 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.677+0000 7fb3e2200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.677+0000 7fb3e2200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc106cf0 con 0x7fb3dc106300 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.678+0000 7fb3e0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.678+0000 7fb3e0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47032/0 (socket says 172.21.15.23:47032) 2024-03-30T03:54:23.363 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.678+0000 7fb3e0e00700 1 -- 172.21.15.23:0/3265488426 learned_addr learned my addr 172.21.15.23:0/3265488426 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.678+0000 7fb3e0e00700 1 -- 172.21.15.23:0/3265488426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc1074b0 con 0x7fb3dc106300 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.678+0000 7fb3e0e00700 1 --2- 172.21.15.23:0/3265488426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00098a0 tx=0x7fb3d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd07576d65df1d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3db400700 1 -- 172.21.15.23:0/3265488426 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0004160 con 0x7fb3dc106300 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3db400700 1 -- 172.21.15.23:0/3265488426 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d00042c0 con 0x7fb3dc106300 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3db400700 1 -- 172.21.15.23:0/3265488426 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d00119b0 con 0x7fb3dc106300 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 -- 172.21.15.23:0/3265488426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 msgr2=0x7fb3dc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 --2- 172.21.15.23:0/3265488426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb3d00098a0 tx=0x7fb3d0009bb0 comp rx=0 tx=0).stop 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 -- 172.21.15.23:0/3265488426 shutdown_connections 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 --2- 172.21.15.23:0/3265488426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc106720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 -- 172.21.15.23:0/3265488426 >> 172.21.15.23:0/3265488426 conn(0x7fb3dc1018c0 msgr2=0x7fb3dc103ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 -- 172.21.15.23:0/3265488426 shutdown_connections 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.679+0000 7fb3e2200700 1 -- 172.21.15.23:0/3265488426 wait complete. 2024-03-30T03:54:23.364 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e2200700 1 Processor -- start 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e2200700 1 -- start start 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e2200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e2200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc19caf0 con 0x7fb3dc106300 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47036/0 (socket says 172.21.15.23:47036) 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.681+0000 7fb3e0e00700 1 -- 172.21.15.23:0/2019114325 learned_addr learned my addr 172.21.15.23:0/2019114325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3e0e00700 1 -- 172.21.15.23:0/2019114325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0009550 con 0x7fb3dc106300 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3e0e00700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0003e60 tx=0x7fb3d0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d00038b0 con 0x7fb3dc106300 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb3d0003a10 con 0x7fb3dc106300 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc19ccf0 con 0x7fb3dc106300 2024-03-30T03:54:23.365 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3d0019530 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.682+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc19d190 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.683+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3dc196550 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.683+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fb3d001a980 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.684+0000 7fb3d9600700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 0x7fb3c403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.684+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3d004c990 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.684+0000 7fb3dbe00700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 0x7fb3c403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.685+0000 7fb3dbe00700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 0x7fb3c403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006fd0 tx=0x7fb3c8006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:22.689+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb3d0003b80 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.014+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb3dc0750b0 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.017+0000 7fb3d9600700 1 -- 172.21.15.23:0/2019114325 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb3d0023b40 con 0x7fb3dc106300 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 msgr2=0x7fb3c403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 0x7fb3c403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3c8006fd0 tx=0x7fb3c8006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:23.366 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 msgr2=0x7fb3dc19c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0003e60 tx=0x7fb3d0011a30 comp rx=0 tx=0).stop 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 shutdown_connections 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb3c40387f0 0x7fb3c403acb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 --2- 172.21.15.23:0/2019114325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3dc106300 0x7fb3dc19c5b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 >> 172.21.15.23:0/2019114325 conn(0x7fb3dc1018c0 msgr2=0x7fb3dc1027a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 shutdown_connections 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-30T03:54:23.019+0000 7fb3e2200700 1 -- 172.21.15.23:0/2019114325 wait complete. 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-30T03:54:23.367 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:23.368 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-30T03:54:23.422 INFO:tasks.cephadm:Fetching config... 2024-03-30T03:54:23.422 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:54:23.422 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-30T03:54:23.464 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-30T03:54:23.464 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:54:23.465 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-30T03:54:23.549 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-30T03:54:23.549 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:54:23.549 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/keyring of=/dev/stdout 2024-03-30T03:54:23.679 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-30T03:54:23.679 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:54:23.679 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-30T03:54:23.761 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-30T03:54:23.761 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T03:54:23.867 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:23 smithi023 ceph-mon[38254]: mgrmap e13: x(active, since 6s) 2024-03-30T03:54:23.868 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2019114325' entity='client.admin' 2024-03-30T03:54:23.894 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:54:23.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T03:54:23.969 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:54:23.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-30T03:54:24.027 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL0oTMhC1hZox3gfDE9PDDs9fVYqhaKzg87hKgMC8KGls2Icfx7gEQ9RWaiQ0eq7w+8dEkNUj2E22/atSdmtPwo0sPTrqODU7ARNaM4cjw0OuX4XaqGQHwHB2wSyfu/nox8c5haQsZuvAHY3Jq6sqsyvGZB/WcV8jYx8Eq9MdNlcVR7AeNfQUmTi5csY8k0PtNZ6jED3MbmiCEYOMZn8Jh/crkVFH7lwAsdzA9HQWRNcj+/i98BH78R4g4JcLS1nVZfsTqa4KHl6ZlfWFHP7ezX8bOJcjlaHGffNNl0ggNxqbHQCAfwn9+kDtDR7x7MHwa9xeb4w8UjZyaYNCsUokk9RraQKb/ALI3Uf24fBiXsZzhPicYdofIp+Ed+C8gRgsWf5Xtbs68LmmS/9ckPh9c+PLXwLw3uPlE0DF2NRFBpFkZNpeECCKwKCs/fZlHB2J9ee1PJ46DMQJ701+CMt+S1XtwBK2o43kL/grL1tMGIYW8Pa25BaYpbkKbzOuig5M= ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:54:24.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-30T03:54:24.402 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 -- 172.21.15.23:0/789357821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 msgr2=0x7f4998100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 --2- 172.21.15.23:0/789357821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998100b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f498c009910 tx=0x7f498c009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 -- 172.21.15.23:0/789357821 shutdown_connections 2024-03-30T03:54:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 --2- 172.21.15.23:0/789357821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998100b50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:25.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 -- 172.21.15.23:0/789357821 >> 172.21.15.23:0/789357821 conn(0x7f49980fbcd0 msgr2=0x7f49980fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 -- 172.21.15.23:0/789357821 shutdown_connections 2024-03-30T03:54:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.540+0000 7f49a0200700 1 -- 172.21.15.23:0/789357821 wait complete. 2024-03-30T03:54:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.541+0000 7f49a0200700 1 Processor -- start 2024-03-30T03:54:25.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.541+0000 7f49a0200700 1 -- start start 2024-03-30T03:54:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.541+0000 7f49a0200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f49a0200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4998198320 con 0x7f4998100730 2024-03-30T03:54:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f499d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:25.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f499d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47056/0 (socket says 172.21.15.23:47056) 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f499d800700 1 -- 172.21.15.23:0/327842846 learned_addr learned my addr 172.21.15.23:0/327842846 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f499d800700 1 -- 172.21.15.23:0/327842846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c0095f0 con 0x7f4998100730 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.542+0000 7f499d800700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f498c005950 tx=0x7f498c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.543+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f498c01d070 con 0x7f4998100730 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.543+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f498c00bc10 con 0x7f4998100730 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.543+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f498c003a10 con 0x7f4998100730 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.543+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4998198520 con 0x7f4998100730 2024-03-30T03:54:25.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.543+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49981989c0 con 0x7f4998100730 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.544+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f498c003c80 con 0x7f4998100730 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.544+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4998191850 con 0x7f4998100730 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.544+0000 7f4996000700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 0x7f498003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.544+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f498c04c520 con 0x7f4998100730 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.545+0000 7f499ce00700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 0x7f498003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:25.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.545+0000 7f499ce00700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 0x7f498003acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4984006fd0 tx=0x7f4984006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.549+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f498c00ba70 con 0x7f4998100730 2024-03-30T03:54:25.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.795+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4998106730 con 0x7f4998100730 2024-03-30T03:54:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.800+0000 7f4996000700 1 -- 172.21.15.23:0/327842846 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f498c026070 con 0x7f4998100730 2024-03-30T03:54:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.802+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 msgr2=0x7f498003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.802+0000 7f49a0200700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 0x7f498003acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4984006fd0 tx=0x7f4984006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.802+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 msgr2=0x7f4998197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.802+0000 7f49a0200700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f498c005950 tx=0x7f498c0057c0 comp rx=0 tx=0).stop 2024-03-30T03:54:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 shutdown_connections 2024-03-30T03:54:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49800387f0 0x7f498003acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 --2- 172.21.15.23:0/327842846 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4998100730 0x7f4998197de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 >> 172.21.15.23:0/327842846 conn(0x7f49980fbcd0 msgr2=0x7f49980fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 shutdown_connections 2024-03-30T03:54:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:25.803+0000 7f49a0200700 1 -- 172.21.15.23:0/327842846 wait complete. 2024-03-30T03:54:26.370 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-30T03:54:26.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-30T03:54:26.737 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:27.112 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:26 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/327842846' entity='client.admin' 2024-03-30T03:54:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 -- 172.21.15.23:0/1224222423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 msgr2=0x7fd2f4073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 --2- 172.21.15.23:0/1224222423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f4073240 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4009910 tx=0x7fd2e4009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 -- 172.21.15.23:0/1224222423 shutdown_connections 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 --2- 172.21.15.23:0/1224222423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f4073240 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 -- 172.21.15.23:0/1224222423 >> 172.21.15.23:0/1224222423 conn(0x7fd2f40fbcd0 msgr2=0x7fd2f40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 -- 172.21.15.23:0/1224222423 shutdown_connections 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.924+0000 7fd2fa000700 1 -- 172.21.15.23:0/1224222423 wait complete. 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2fa000700 1 Processor -- start 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2fa000700 1 -- start start 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2fa000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2fa000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f419c6e0 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47082/0 (socket says 172.21.15.23:47082) 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.926+0000 7fd2f3400700 1 -- 172.21.15.23:0/171931839 learned_addr learned my addr 172.21.15.23:0/171931839 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2f3400700 1 -- 172.21.15.23:0/171931839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e40095f0 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2f3400700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4005950 tx=0x7fd2e40057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2e401d070 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2e400bc10 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f419c8e0 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2e4003a10 con 0x7fd2f4074de0 2024-03-30T03:54:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.927+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f419cd80 con 0x7fd2f4074de0 2024-03-30T03:54:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.928+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fd2e4004390 con 0x7fd2f4074de0 2024-03-30T03:54:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.929+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f4195a90 con 0x7fd2f4074de0 2024-03-30T03:54:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.929+0000 7fd2ebe00700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 0x7fd2d803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.929+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd2e404c570 con 0x7fd2f4074de0 2024-03-30T03:54:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.929+0000 7fd2f2a00700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 0x7fd2d803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:27.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.930+0000 7fd2f2a00700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 0x7fd2d803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0006fd0 tx=0x7fd2e0006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:27.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:27.934+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd2e400ba70 con 0x7fd2f4074de0 2024-03-30T03:54:28.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.171+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f402cd20 con 0x7fd2d80387a0 2024-03-30T03:54:28.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.180+0000 7fd2ebe00700 1 -- 172.21.15.23:0/171931839 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd2f402cd20 con 0x7fd2d80387a0 2024-03-30T03:54:28.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 msgr2=0x7fd2d803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 0x7fd2d803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0006fd0 tx=0x7fd2e0006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 msgr2=0x7fd2f419c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd2e4005950 tx=0x7fd2e40057c0 comp rx=0 tx=0).stop 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 shutdown_connections 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2d80387a0 0x7fd2d803ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 --2- 172.21.15.23:0/171931839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2f4074de0 0x7fd2f419c1a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.182+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 >> 172.21.15.23:0/171931839 conn(0x7fd2f40fbcd0 msgr2=0x7fd2f40fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.183+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 shutdown_connections 2024-03-30T03:54:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:28.183+0000 7fd2fa000700 1 -- 172.21.15.23:0/171931839 wait complete. 2024-03-30T03:54:28.557 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-03-30T03:54:28.557 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:54:28.557 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-03-30T03:54:28.578 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:54:28.578 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:28.639 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-03-30T03:54:28.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi154 2024-03-30T03:54:29.016 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:29.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:29.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:29.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:29.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:29 smithi023 ceph-mon[38254]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 -- 172.21.15.23:0/2895774853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 msgr2=0x7fea280682d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 --2- 172.21.15.23:0/2895774853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea280682d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009910 tx=0x7fea1c009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 -- 172.21.15.23:0/2895774853 shutdown_connections 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 --2- 172.21.15.23:0/2895774853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea280682d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 -- 172.21.15.23:0/2895774853 >> 172.21.15.23:0/2895774853 conn(0x7fea28075110 msgr2=0x7fea28075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.063+0000 7fea2f800700 1 -- 172.21.15.23:0/2895774853 shutdown_connections 2024-03-30T03:54:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.064+0000 7fea2f800700 1 -- 172.21.15.23:0/2895774853 wait complete. 2024-03-30T03:54:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2f800700 1 Processor -- start 2024-03-30T03:54:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2f800700 1 -- start start 2024-03-30T03:54:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2f800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2f800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea281986d0 con 0x7fea28067ef0 2024-03-30T03:54:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35956/0 (socket says 172.21.15.23:35956) 2024-03-30T03:54:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.065+0000 7fea2ce00700 1 -- 172.21.15.23:0/3155451226 learned_addr learned my addr 172.21.15.23:0/3155451226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea2ce00700 1 -- 172.21.15.23:0/3155451226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c0095f0 con 0x7fea28067ef0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea2ce00700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea1c006010 tx=0x7fea1c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea1c01d070 con 0x7fea28067ef0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea1c00bc10 con 0x7fea28067ef0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea281988d0 con 0x7fea28067ef0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.066+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea1c0038a0 con 0x7fea28067ef0 2024-03-30T03:54:30.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.067+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea28198cf0 con 0x7fea28067ef0 2024-03-30T03:54:30.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.068+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fea1c003a00 con 0x7fea28067ef0 2024-03-30T03:54:30.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.068+0000 7fea25600700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 0x7fea1003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:30.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.068+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea280ffbf0 con 0x7fea28067ef0 2024-03-30T03:54:30.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.068+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fea1c04c400 con 0x7fea28067ef0 2024-03-30T03:54:30.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.068+0000 7fea27e00700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 0x7fea1003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:30.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.069+0000 7fea27e00700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 0x7fea1003acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea14006fd0 tx=0x7fea14006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:30.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.074+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fea1c026070 con 0x7fea28067ef0 2024-03-30T03:54:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:30 smithi023 ceph-mon[38254]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:30 smithi023 ceph-mon[38254]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:30.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:30.307+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}) v1 -- 0x7fea2810cbf0 con 0x7fea100387f0 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:31.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:31.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:32.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.514+0000 7fea25600700 1 -- 172.21.15.23:0/3155451226 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fea2810cbf0 con 0x7fea100387f0 2024-03-30T03:54:32.514 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-03-30T03:54:32.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 msgr2=0x7fea1003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 0x7fea1003acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fea14006fd0 tx=0x7fea14006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 msgr2=0x7fea28198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea1c006010 tx=0x7fea1c004ff0 comp rx=0 tx=0).stop 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 shutdown_connections 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fea100387f0 0x7fea1003acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 --2- 172.21.15.23:0/3155451226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea28067ef0 0x7fea28198190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.516+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 >> 172.21.15.23:0/3155451226 conn(0x7fea28075110 msgr2=0x7fea280fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.517+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 shutdown_connections 2024-03-30T03:54:32.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:32.517+0000 7fea2f800700 1 -- 172.21.15.23:0/3155451226 wait complete. 2024-03-30T03:54:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:32 smithi023 ceph-mon[38254]: Deploying cephadm binary to smithi154 2024-03-30T03:54:32.825 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T03:54:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:33.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:33 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:33.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:33 smithi023 ceph-mon[38254]: Added host smithi154 2024-03-30T03:54:33.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:33 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:33.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:33 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:33.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:33 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:34.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 -- 172.21.15.23:0/1646876377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 msgr2=0x7fc344106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:34.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1646876377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344106aa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc334009910 tx=0x7fc334009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:34.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 -- 172.21.15.23:0/1646876377 shutdown_connections 2024-03-30T03:54:34.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1646876377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344106aa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:34.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 -- 172.21.15.23:0/1646876377 >> 172.21.15.23:0/1646876377 conn(0x7fc3440fc1b0 msgr2=0x7fc3440fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 -- 172.21.15.23:0/1646876377 shutdown_connections 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.284+0000 7fc34ae00700 1 -- 172.21.15.23:0/1646876377 wait complete. 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.285+0000 7fc34ae00700 1 Processor -- start 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.285+0000 7fc34ae00700 1 -- start start 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc34ae00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:34.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc34ae00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3441986a0 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc343e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc343e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35980/0 (socket says 172.21.15.23:35980) 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc343e00700 1 -- 172.21.15.23:0/1875209558 learned_addr learned my addr 172.21.15.23:0/1875209558 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc343e00700 1 -- 172.21.15.23:0/1875209558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340095f0 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.286+0000 7fc343e00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc334004f40 tx=0x7fc334005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.287+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc33401d070 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.287+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc33400bbb0 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.287+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc344198900 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.287+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc334003810 con 0x7fc3441066c0 2024-03-30T03:54:34.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.287+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc344198d80 con 0x7fc3441066c0 2024-03-30T03:54:34.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.288+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fc33400bd20 con 0x7fc3441066c0 2024-03-30T03:54:34.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.288+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc344101e00 con 0x7fc3441066c0 2024-03-30T03:54:34.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.288+0000 7fc340c00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 0x7fc32803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:34.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.288+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc33404c360 con 0x7fc3441066c0 2024-03-30T03:54:34.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.289+0000 7fc343400700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 0x7fc32803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:34.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.289+0000 7fc343400700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 0x7fc32803acb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc330006fd0 tx=0x7fc330006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:34.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.294+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc33402ab60 con 0x7fc3441066c0 2024-03-30T03:54:34.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.510+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc34410a9c0 con 0x7fc3280387f0 2024-03-30T03:54:34.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.512+0000 7fc340c00700 1 -- 172.21.15.23:0/1875209558 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fc34410a9c0 con 0x7fc3280387f0 2024-03-30T03:54:34.511 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:34.511 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 msgr2=0x7fc32803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 0x7fc32803acb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc330006fd0 tx=0x7fc330006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 msgr2=0x7fc344198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc334004f40 tx=0x7fc334005020 comp rx=0 tx=0).stop 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 shutdown_connections 2024-03-30T03:54:34.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc3280387f0 0x7fc32803acb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 --2- 172.21.15.23:0/1875209558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3441066c0 0x7fc344198160 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 >> 172.21.15.23:0/1875209558 conn(0x7fc3440fc1b0 msgr2=0x7fc344108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 shutdown_connections 2024-03-30T03:54:34.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:34.514+0000 7fc34ae00700 1 -- 172.21.15.23:0/1875209558 wait complete. 2024-03-30T03:54:34.897 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-03-30T03:54:34.897 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:54:34.897 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-03-30T03:54:34.918 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:54:34.919 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:34.978 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-03-30T03:54:34.979 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch host add smithi193 2024-03-30T03:54:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:35 smithi023 ceph-mon[38254]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T03:54:36.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 -- 172.21.15.23:0/3689422262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 msgr2=0x7fbc0c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 --2- 172.21.15.23:0/3689422262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c106aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc00009910 tx=0x7fbc00009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 -- 172.21.15.23:0/3689422262 shutdown_connections 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 --2- 172.21.15.23:0/3689422262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c106aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 -- 172.21.15.23:0/3689422262 >> 172.21.15.23:0/3689422262 conn(0x7fbc0c0fc1b0 msgr2=0x7fbc0c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 -- 172.21.15.23:0/3689422262 shutdown_connections 2024-03-30T03:54:36.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.372+0000 7fbc14400700 1 -- 172.21.15.23:0/3689422262 wait complete. 2024-03-30T03:54:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.373+0000 7fbc14400700 1 Processor -- start 2024-03-30T03:54:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.373+0000 7fbc14400700 1 -- start start 2024-03-30T03:54:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc14400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:36.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc14400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c1986a0 con 0x7fbc0c1066c0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc11a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc11a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36000/0 (socket says 172.21.15.23:36000) 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc11a00700 1 -- 172.21.15.23:0/3131903818 learned_addr learned my addr 172.21.15.23:0/3131903818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.374+0000 7fbc11a00700 1 -- 172.21.15.23:0/3131903818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc000095f0 con 0x7fbc0c1066c0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc11a00700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc00006010 tx=0x7fbc00004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc0001d070 con 0x7fbc0c1066c0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc0000bbb0 con 0x7fbc0c1066c0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbc00003810 con 0x7fbc0c1066c0 2024-03-30T03:54:36.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c198900 con 0x7fbc0c1066c0 2024-03-30T03:54:36.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.375+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c198d80 con 0x7fbc0c1066c0 2024-03-30T03:54:36.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.376+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fbc0000bd20 con 0x7fbc0c1066c0 2024-03-30T03:54:36.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.376+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c101e00 con 0x7fbc0c1066c0 2024-03-30T03:54:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.377+0000 7fbc0a000700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 0x7fbbf403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.377+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbc0004c370 con 0x7fbc0c1066c0 2024-03-30T03:54:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.377+0000 7fbc11000700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 0x7fbbf403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:36.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.378+0000 7fbc11000700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 0x7fbbf403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8006fd0 tx=0x7fbbf8006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:36.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.382+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc0002aba0 con 0x7fbc0c1066c0 2024-03-30T03:54:36.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:36.640+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc0c10a9c0 con 0x7fbbf40387f0 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:54:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:37.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:37.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:37 smithi023 ceph-mon[38254]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-30T03:54:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T03:54:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: Deploying cephadm binary to smithi193 2024-03-30T03:54:38.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:38.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:38.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:38 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:38.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.892+0000 7fbc0a000700 1 -- 172.21.15.23:0/3131903818 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fbc0c10a9c0 con 0x7fbbf40387f0 2024-03-30T03:54:38.892 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 msgr2=0x7fbbf403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 0x7fbbf403acb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8006fd0 tx=0x7fbbf8006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 msgr2=0x7fbc0c198160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc00006010 tx=0x7fbc00004ff0 comp rx=0 tx=0).stop 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 shutdown_connections 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbbf40387f0 0x7fbbf403acb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.894+0000 7fbc14400700 1 --2- 172.21.15.23:0/3131903818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc0c1066c0 0x7fbc0c198160 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.895+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 >> 172.21.15.23:0/3131903818 conn(0x7fbc0c0fc1b0 msgr2=0x7fbc0c108240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.895+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 shutdown_connections 2024-03-30T03:54:38.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:38.895+0000 7fbc14400700 1 -- 172.21.15.23:0/3131903818 wait complete. 2024-03-30T03:54:39.267 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch host ls --format=json 2024-03-30T03:54:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:39 smithi023 ceph-mon[38254]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:39 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:39 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:39 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:39.622 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:40.505 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:40 smithi023 ceph-mon[38254]: Added host smithi193 2024-03-30T03:54:40.505 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:40.505 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 -- 172.21.15.23:0/535089077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 msgr2=0x7f8f3c106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 --2- 172.21.15.23:0/535089077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c106ab0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8f30009910 tx=0x7f8f30009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 -- 172.21.15.23:0/535089077 shutdown_connections 2024-03-30T03:54:40.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 --2- 172.21.15.23:0/535089077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c106ab0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 -- 172.21.15.23:0/535089077 >> 172.21.15.23:0/535089077 conn(0x7f8f3c075110 msgr2=0x7f8f3c075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 -- 172.21.15.23:0/535089077 shutdown_connections 2024-03-30T03:54:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.732+0000 7f8f44600700 1 -- 172.21.15.23:0/535089077 wait complete. 2024-03-30T03:54:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.733+0000 7f8f44600700 1 Processor -- start 2024-03-30T03:54:40.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f44600700 1 -- start start 2024-03-30T03:54:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f44600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:40.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f44600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c1964f0 con 0x7f8f3c1066d0 2024-03-30T03:54:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f41c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f41c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48516/0 (socket says 172.21.15.23:48516) 2024-03-30T03:54:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.734+0000 7f8f41c00700 1 -- 172.21.15.23:0/3498637513 learned_addr learned my addr 172.21.15.23:0/3498637513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f41c00700 1 -- 172.21.15.23:0/3498637513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300095f0 con 0x7f8f3c1066d0 2024-03-30T03:54:40.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f41c00700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006010 tx=0x7f8f30004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f3001d070 con 0x7f8f3c1066d0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f3000bbb0 con 0x7f8f3c1066d0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8f30003810 con 0x7f8f3c1066d0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3c196750 con 0x7f8f3c1066d0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.735+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c196bd0 con 0x7f8f3c1066d0 2024-03-30T03:54:40.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.737+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3c101e10 con 0x7f8f3c1066d0 2024-03-30T03:54:40.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.737+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f8f3000bd20 con 0x7f8f3c1066d0 2024-03-30T03:54:40.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.737+0000 7f8f3a000700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 0x7f8f2403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.737+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8f3004c430 con 0x7f8f3c1066d0 2024-03-30T03:54:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.738+0000 7f8f41200700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 0x7f8f2403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:40.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.738+0000 7f8f41200700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 0x7f8f2403acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006fd0 tx=0x7f8f28006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:40.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.742+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8f3002aba0 con 0x7f8f3c1066d0 2024-03-30T03:54:40.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.959+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f3c10a9b0 con 0x7f8f240387f0 2024-03-30T03:54:40.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.960+0000 7f8f3a000700 1 -- 172.21.15.23:0/3498637513 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f8f3c10a9b0 con 0x7f8f240387f0 2024-03-30T03:54:40.959 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:54:40.959 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-03-30T03:54:40.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 msgr2=0x7f8f2403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 0x7f8f2403acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8f28006fd0 tx=0x7f8f28006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 msgr2=0x7f8f3c195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8f30006010 tx=0x7f8f30004ff0 comp rx=0 tx=0).stop 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 shutdown_connections 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8f240387f0 0x7f8f2403acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 --2- 172.21.15.23:0/3498637513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f3c1066d0 0x7f8f3c195fb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.962+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 >> 172.21.15.23:0/3498637513 conn(0x7f8f3c075110 msgr2=0x7f8f3c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.963+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 shutdown_connections 2024-03-30T03:54:40.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:40.963+0000 7f8f44600700 1 -- 172.21.15.23:0/3498637513 wait complete. 2024-03-30T03:54:41.289 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-30T03:54:41.289 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd crush tunables default 2024-03-30T03:54:41.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:41 smithi023 ceph-mon[38254]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:54:42.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:42 smithi023 ceph-mon[38254]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.689+0000 7fe910c00700 1 -- 172.21.15.23:0/2837898691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 msgr2=0x7fe90c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.689+0000 7fe910c00700 1 --2- 172.21.15.23:0/2837898691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c106aa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc009910 tx=0x7fe8fc009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.689+0000 7fe910c00700 1 -- 172.21.15.23:0/2837898691 shutdown_connections 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.689+0000 7fe910c00700 1 --2- 172.21.15.23:0/2837898691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c106aa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.689+0000 7fe910c00700 1 -- 172.21.15.23:0/2837898691 >> 172.21.15.23:0/2837898691 conn(0x7fe90c0fc1b0 msgr2=0x7fe90c0fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.690+0000 7fe910c00700 1 -- 172.21.15.23:0/2837898691 shutdown_connections 2024-03-30T03:54:42.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.690+0000 7fe910c00700 1 -- 172.21.15.23:0/2837898691 wait complete. 2024-03-30T03:54:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.691+0000 7fe910c00700 1 Processor -- start 2024-03-30T03:54:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.691+0000 7fe910c00700 1 -- start start 2024-03-30T03:54:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe910c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:42.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe910c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe90c074f70 con 0x7fe90c1066c0 2024-03-30T03:54:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe909e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe909e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48546/0 (socket says 172.21.15.23:48546) 2024-03-30T03:54:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe909e00700 1 -- 172.21.15.23:0/935955428 learned_addr learned my addr 172.21.15.23:0/935955428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:54:42.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.692+0000 7fe909e00700 1 -- 172.21.15.23:0/935955428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc0095f0 con 0x7fe90c1066c0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe909e00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006010 tx=0x7fe8fc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8fc01d070 con 0x7fe90c1066c0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8fc00bbb0 con 0x7fe90c1066c0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90c0735f0 con 0x7fe90c1066c0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8fc003810 con 0x7fe90c1066c0 2024-03-30T03:54:42.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.693+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90c073a70 con 0x7fe90c1066c0 2024-03-30T03:54:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.694+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fe8fc00bd20 con 0x7fe90c1066c0 2024-03-30T03:54:42.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.694+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90c101e00 con 0x7fe90c1066c0 2024-03-30T03:54:42.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.695+0000 7fe902a00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 0x7fe8f003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:42.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.695+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe8fc04c450 con 0x7fe90c1066c0 2024-03-30T03:54:42.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.695+0000 7fe909400700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 0x7fe8f003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:42.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.695+0000 7fe909400700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 0x7fe8f003ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006fd0 tx=0x7fe8f8006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:42.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.700+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe8fc02ab60 con 0x7fe90c1066c0 2024-03-30T03:54:42.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:42.960+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe90c073cb0 con 0x7fe90c1066c0 2024-03-30T03:54:43.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.632+0000 7fe902a00700 1 -- 172.21.15.23:0/935955428 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe8fc026020 con 0x7fe90c1066c0 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 msgr2=0x7fe8f003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 0x7fe8f003ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8006fd0 tx=0x7fe8f8006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 msgr2=0x7fe90c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc006010 tx=0x7fe8fc004ff0 comp rx=0 tx=0).stop 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 shutdown_connections 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe8f00387a0 0x7fe8f003ac60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 --2- 172.21.15.23:0/935955428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe90c1066c0 0x7fe90c0730b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:43.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 >> 172.21.15.23:0/935955428 conn(0x7fe90c0fc1b0 msgr2=0x7fe90c1082f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:43.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 shutdown_connections 2024-03-30T03:54:43.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:54:43.635+0000 7fe910c00700 1 -- 172.21.15.23:0/935955428 wait complete. 2024-03-30T03:54:43.635 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-30T03:54:43.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:43.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:43.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:43.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:43.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:43.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:54:43.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:43.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:43.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/935955428' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T03:54:44.030 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-30T03:54:44.030 INFO:tasks.cephadm:Adding mon.b on smithi154 2024-03-30T03:54:44.030 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-03-30T03:54:44.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c' 2024-03-30T03:54:44.269 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/935955428' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T03:54:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:44 smithi023 ceph-mon[38254]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:54:45.290 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 -- 172.21.15.193:0/328495038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 msgr2=0x7f7048101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 --2- 172.21.15.193:0/328495038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048101c70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7038009910 tx=0x7f7038009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 -- 172.21.15.193:0/328495038 shutdown_connections 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 --2- 172.21.15.193:0/328495038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048101c70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 -- 172.21.15.193:0/328495038 >> 172.21.15.193:0/328495038 conn(0x7f70480fb0e0 msgr2=0x7f70480fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 -- 172.21.15.193:0/328495038 shutdown_connections 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 -- 172.21.15.193:0/328495038 wait complete. 2024-03-30T03:54:45.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.292+0000 7f704de77700 1 Processor -- start 2024-03-30T03:54:45.292 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f704de77700 1 -- start start 2024-03-30T03:54:45.292 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f704de77700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f704de77700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7048196530 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f70477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f70477fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.193:56654/0 (socket says 172.21.15.193:56654) 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.293+0000 7f70477fe700 1 -- 172.21.15.193:0/1060487020 learned_addr learned my addr 172.21.15.193:0/1060487020 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f70477fe700 1 -- 172.21.15.193:0/1060487020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380095f0 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f70477fe700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7038004f40 tx=0x7f7038005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f703801d070 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7048196730 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7038004370 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7038003970 con 0x7f7048101890 2024-03-30T03:54:45.293 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.294+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7048196b50 con 0x7f7048101890 2024-03-30T03:54:45.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.295+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7048069060 con 0x7f7048101890 2024-03-30T03:54:45.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.295+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f7038003c30 con 0x7f7048101890 2024-03-30T03:54:45.294 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.295+0000 7f7044ff9700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 0x7f703003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:45.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.295+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f703804c560 con 0x7f7048101890 2024-03-30T03:54:45.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.295+0000 7f7046ffd700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 0x7f703003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:45.295 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.296+0000 7f7046ffd700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 0x7f703003ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f703c006fd0 tx=0x7f703c006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:45.299 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.300+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f70380044e0 con 0x7f7048101890 2024-03-30T03:54:45.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.501+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f704810aaf0 con 0x7f70300387a0 2024-03-30T03:54:45.509 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.511+0000 7f7044ff9700 1 -- 172.21.15.193:0/1060487020 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f704810aaf0 con 0x7f70300387a0 2024-03-30T03:54:45.509 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 msgr2=0x7f703003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 0x7f703003ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f703c006fd0 tx=0x7f703c006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 msgr2=0x7f7048195ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7038004f40 tx=0x7f7038005020 comp rx=0 tx=0).stop 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 shutdown_connections 2024-03-30T03:54:45.511 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f70300387a0 0x7f703003ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:45.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 --2- 172.21.15.193:0/1060487020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7048101890 0x7f7048195ff0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:45.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 >> 172.21.15.193:0/1060487020 conn(0x7f70480fb0e0 msgr2=0x7f7048104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:45.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 shutdown_connections 2024-03-30T03:54:45.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:45.513+0000 7f704de77700 1 -- 172.21.15.193:0/1060487020 wait complete. 2024-03-30T03:54:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:45 smithi023 ceph-mon[38254]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T03:54:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:45 smithi023 ceph-mon[38254]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:45.869 DEBUG:teuthology.orchestra.run.smithi154:mon.b> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.b.service 2024-03-30T03:54:45.871 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.c.service 2024-03-30T03:54:45.874 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T03:54:45.874 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T03:54:45.903 INFO:journalctl@ceph.mon.c.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:54:45.909 INFO:journalctl@ceph.mon.b.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:54:46.308 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c;count:3 2024-03-30T03:54:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: Deploying daemon mon.c on smithi193 2024-03-30T03:54:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:46 smithi023 ceph-mon[38254]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 -- 172.21.15.193:0/3865496516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 msgr2=0x7fd118105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 --2- 172.21.15.193:0/3865496516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118105a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd114009910 tx=0x7fd114009c20 comp rx=0 tx=0).stop 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 -- 172.21.15.193:0/3865496516 shutdown_connections 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 --2- 172.21.15.193:0/3865496516 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118105a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 -- 172.21.15.193:0/3865496516 >> 172.21.15.193:0/3865496516 conn(0x7fd1180fb0e0 msgr2=0x7fd1180fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 -- 172.21.15.193:0/3865496516 shutdown_connections 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.754+0000 7fd1206a8700 1 -- 172.21.15.193:0/3865496516 wait complete. 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd1206a8700 1 Processor -- start 2024-03-30T03:54:47.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd1206a8700 1 -- start start 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd1206a8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd1206a8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd118195330 con 0x7fd118105660 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd11e444700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd11e444700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.193:56666/0 (socket says 172.21.15.193:56666) 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd11e444700 1 -- 172.21.15.193:0/2467764625 learned_addr learned my addr 172.21.15.193:0/2467764625 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.755+0000 7fd11e444700 1 -- 172.21.15.193:0/2467764625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1140095f0 con 0x7fd118105660 2024-03-30T03:54:47.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd11e444700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd114004f40 tx=0x7fd114005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:47.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd11401d070 con 0x7fd118105660 2024-03-30T03:54:47.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd11400b6f0 con 0x7fd118105660 2024-03-30T03:54:47.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd118195590 con 0x7fd118105660 2024-03-30T03:54:47.756 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd114003810 con 0x7fd118105660 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.756+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd118195a90 con 0x7fd118105660 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.757+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd118100d10 con 0x7fd118105660 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.757+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fd114004260 con 0x7fd118105660 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.757+0000 7fd10b7fe700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 0x7fd10403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.757+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd11404c290 con 0x7fd118105660 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.758+0000 7fd11dc43700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 0x7fd10403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:47.757 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.758+0000 7fd11dc43700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 0x7fd10403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd10c006fd0 tx=0x7fd10c006e40 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:47.761 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:47.763+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd11402ad00 con 0x7fd118105660 2024-03-30T03:54:48.059 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.059+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd118195a90 con 0x7fd118105660 2024-03-30T03:54:48.059 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:54:48.060 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","modified":"2024-03-30T03:53:11.933267Z","created":"2024-03-30T03:53:11.933267Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-30T03:54:48.060 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.060+0000 7fd10b7fe700 1 -- 172.21.15.193:0/2467764625 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fd118195a90 con 0x7fd118105660 2024-03-30T03:54:48.061 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 msgr2=0x7fd10403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 0x7fd10403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd10c006fd0 tx=0x7fd10c006e40 comp rx=0 tx=0).stop 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 msgr2=0x7fd118194df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd114004f40 tx=0x7fd114005020 comp rx=0 tx=0).stop 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 shutdown_connections 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1040387a0 0x7fd10403ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:48.062 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 --2- 172.21.15.193:0/2467764625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd118105660 0x7fd118194df0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:48.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 >> 172.21.15.193:0/2467764625 conn(0x7fd1180fb0e0 msgr2=0x7fd1180fca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:48.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.062+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 shutdown_connections 2024-03-30T03:54:48.063 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:48.063+0000 7fd1206a8700 1 -- 172.21.15.193:0/2467764625 wait complete. 2024-03-30T03:54:48.063 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-30T03:54:48.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:48 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/2467764625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T03:54:49.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:48 smithi193 bash[79544]: 1654bc87aa638e5f3fc9dcb693a2465b12b92f092754bf1a08b4f2d566de06ba 2024-03-30T03:54:49.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:48 smithi193 systemd[1]: Started Ceph mon.c for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:54:49.481 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T03:54:49.481 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T03:54:49.701 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:54:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:54:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: mon.a calling monitor election 2024-03-30T03:54:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: mon.c calling monitor election 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: fsmap 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: mgrmap e13: x(active, since 37s) 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: overall HEALTH_OK 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:54:54.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 -- 172.21.15.193:0/2376657305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 msgr2=0x7f549c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 --2- 172.21.15.193:0/2376657305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f549c005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f54b40ff620 tx=0x7f54a8014ec0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 -- 172.21.15.193:0/2376657305 shutdown_connections 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 --2- 172.21.15.193:0/2376657305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f549c005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 -- 172.21.15.193:0/2376657305 >> 172.21.15.193:0/2376657305 conn(0x7f54b4076390 msgr2=0x7f54b40fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 -- 172.21.15.193:0/2376657305 shutdown_connections 2024-03-30T03:54:54.281 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.280+0000 7f54b9fec700 1 -- 172.21.15.193:0/2376657305 wait complete. 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.282+0000 7f54b9fec700 1 Processor -- start 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.282+0000 7f54b9fec700 1 -- start start 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b9fec700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f54b4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b9fec700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b9fec700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b410a870 con 0x7f549c0035c0 2024-03-30T03:54:54.282 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b9fec700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b410a9e0 con 0x7f54b4108f40 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:34794/0 (socket says 172.21.15.193:34794) 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 -- 172.21.15.193:0/3059924522 learned_addr learned my addr 172.21.15.193:0/3059924522 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b37fe700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f54b4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 -- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 msgr2=0x7f54b41093c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 -- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 msgr2=0x7f54b41093c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b2ffd700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b37fe700 1 -- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 msgr2=0x7f54b41093c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:54:54.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b37fe700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.283+0000 7f54b37fe700 1 -- 172.21.15.193:0/3059924522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a8014910 con 0x7f549c0035c0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b37fe700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f54b4108a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54a8001eb0 tx=0x7f54a800fcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f54a8023070 con 0x7f549c0035c0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a8004b90 con 0x7f549c0035c0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b41a2340 con 0x7f549c0035c0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f54a8019040 con 0x7f549c0035c0 2024-03-30T03:54:54.285 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.284+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b41a2860 con 0x7f549c0035c0 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.285+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f54a8012450 con 0x7f549c0035c0 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.285+0000 7f54b0ff9700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 0x7f54a003aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.285+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f54a8000e40 con 0x7f549c0035c0 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.286+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b4100e30 con 0x7f549c0035c0 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.286+0000 7f54b2ffd700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 0x7f54a003aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:54:54.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.286+0000 7f54b2ffd700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 0x7f54a003aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009180 tx=0x7f54a4008040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:54:54.291 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.292+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f54a801d790 con 0x7f549c0035c0 2024-03-30T03:54:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:54:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: mon.a calling monitor election 2024-03-30T03:54:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: mon.c calling monitor election 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: fsmap 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: mgrmap e13: x(active, since 37s) 2024-03-30T03:54:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: overall HEALTH_OK 2024-03-30T03:54:54.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:54.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:54:54.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:54.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.569+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54b4107460 con 0x7f549c0035c0 2024-03-30T03:54:54.569 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.570+0000 7f54b0ff9700 1 -- 172.21.15.193:0/3059924522 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f54a8035030 con 0x7f549c0035c0 2024-03-30T03:54:54.569 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:54:54.569 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","modified":"2024-03-30T03:54:48.923824Z","created":"2024-03-30T03:53:11.933267Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T03:54:54.571 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 msgr2=0x7f54a003aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:54.571 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 0x7f54a003aaf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009180 tx=0x7f54a4008040 comp rx=0 tx=0).stop 2024-03-30T03:54:54.571 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 msgr2=0x7f54b4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f54b4108a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f54a8001eb0 tx=0x7f54a800fcd0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 shutdown_connections 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f549c0035c0 0x7f54b4108a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f54a0038630 0x7f54a003aaf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 --2- 172.21.15.193:0/3059924522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54b4108f40 0x7f54b41093c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 >> 172.21.15.193:0/3059924522 conn(0x7f54b4076390 msgr2=0x7f54b4103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 shutdown_connections 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:54:54.572+0000 7f54b9fec700 1 -- 172.21.15.193:0/3059924522 wait complete. 2024-03-30T03:54:54.572 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-30T03:54:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:55 smithi193 ceph-mon[79581]: Deploying daemon mon.b on smithi154 2024-03-30T03:54:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:55 smithi193 ceph-mon[79581]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/3059924522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T03:54:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:54:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:55 smithi023 ceph-mon[38254]: Deploying daemon mon.b on smithi154 2024-03-30T03:54:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:55 smithi023 ceph-mon[38254]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/3059924522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T03:54:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:54:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:55.931 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-30T03:54:55.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph mon dump -f json 2024-03-30T03:54:56.161 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:54:56.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 03:54:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T03:54:55.923+0000 7f4582a00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 systemd[1]: Started Ceph mon.b for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.merge_operator: 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_filter: None 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_filter_factory: None 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.sst_partitioner_factory: None 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-30T03:54:56.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555dd63e95e0) 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: cache_index_and_filter_blocks: 1 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: pin_top_level_index_and_filter: 1 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: index_type: 0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: data_block_index_type: 0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: index_shortening: 1 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: checksum: 4 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: no_block_cache: 0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_cache: 0x555dd57011f0 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_cache_name: BinnedLRUCache 2024-03-30T03:54:56.934 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_cache_options: 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: capacity : 536870912 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: num_shard_bits : 4 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: strict_capacity_limit : 0 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: high_pri_pool_ratio: 0.000 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_cache_compressed: (nil) 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: persistent_cache: (nil) 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_size: 4096 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_size_deviation: 10 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_restart_interval: 16 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: index_block_restart_interval: 1 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: metadata_block_size: 4096 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: partition_filters: 0 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: use_delta_encoding: 1 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: filter_policy: bloomfilter 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: whole_key_filtering: 1 2024-03-30T03:54:56.935 INFO:journalctl@ceph.mon.b.smithi154.stdout: verify_compression: 0 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: read_amp_bytes_per_bit: 0 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: format_version: 5 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: enable_index_compression: 1 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: block_align: 0 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: max_auto_readahead_size: 262144 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: prepopulate_block_cache: 0 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: initial_auto_readahead_size: 8192 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout: num_file_reads_for_auto_readahead: 2 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.write_buffer_size: 33554432 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_write_buffer_number: 2 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression: NoCompression 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression: Disabled 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.prefix_extractor: nullptr 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-30T03:54:56.936 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.num_levels: 7 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.level: 32767 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.strategy: 0 2024-03-30T03:54:56.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.enabled: false 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.target_file_size_base: 67108864 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-30T03:54:56.938 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.arena_block_size: 1048576 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.disable_auto_compactions: 0 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-30T03:54:56.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.table_properties_collectors: 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.inplace_update_support: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.bloom_locality: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.max_successive_merges: 0 2024-03-30T03:54:56.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.paranoid_file_checks: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.force_consistency_checks: 1 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.report_bg_io_stats: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.ttl: 2592000 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.enable_blob_files: false 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.min_blob_size: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_file_size: 268435456 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-30T03:54:56.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.blob_file_starting_level: 0 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1aef1a0c-53f8-4fee-a0b4-8b643874dfef 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770896610678, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770896611280, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711770896, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1aef1a0c-53f8-4fee-a0b4-8b643874dfef", "db_session_id": "LM7JRT1ZL80VJF7H08NJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711770896611368, "job": 1, "event": "recovery_finished"} 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555dd579e000 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: DB pointer 0x555dd578c000 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-30T03:54:56.942 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: ** DB Stats ** 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: ** Compaction Stats [default] ** 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:54:56.943 INFO:journalctl@ceph.mon.b.smithi154.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: ** Compaction Stats [default] ** 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-30T03:54:56.944 INFO:journalctl@ceph.mon.b.smithi154.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: Block cache BinnedLRUCache@0x555dd57011f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: using public_addr v2:172.21.15.154:0/0 -> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: starting mon.b rank -1 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(???) e0 preinit fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).mds e1 new map 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-30T03:54:56.945 INFO:journalctl@ceph.mon.b.smithi154.stdout: e1 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout: legacy client fscid: -1 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout: 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout: No filesystems configured 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:54:56.946 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Deploying cephadm binary to smithi193 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Added host smithi193 2024-03-30T03:54:56.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/935955428' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T03:54:56.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/935955428' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c;count:3 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:54:56.949 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Deploying daemon mon.c on smithi193 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/2467764625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.a calling monitor election 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.c calling monitor election 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.950 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: fsmap 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mgrmap e13: x(active, since 37s) 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: overall HEALTH_OK 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: Deploying daemon mon.b on smithi154 2024-03-30T03:54:56.951 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:54:56.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/3059924522' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-30T03:54:56.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:54:56.952 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:54:56 smithi154 ceph-mon[79161]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-30T03:55:01.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: mon.c calling monitor election 2024-03-30T03:55:01.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:55:01.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: mon.a calling monitor election 2024-03-30T03:55:01.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: fsmap 2024-03-30T03:55:01.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: mgrmap e13: x(active, since 45s) 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:01.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: mon.c calling monitor election 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: mon.a calling monitor election 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: fsmap 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: mgrmap e13: x(active, since 45s) 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:02.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:02.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:03.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 -- 172.21.15.193:0/756953361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 msgr2=0x7fd46c008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 --2- 172.21.15.193:0/756953361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd46c008710 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd4801007b0 tx=0x7fd46800fe90 comp rx=0 tx=0).stop 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 -- 172.21.15.193:0/756953361 shutdown_connections 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 --2- 172.21.15.193:0/756953361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd46c008710 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 -- 172.21.15.193:0/756953361 >> 172.21.15.193:0/756953361 conn(0x7fd480076390 msgr2=0x7fd4800fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 -- 172.21.15.193:0/756953361 shutdown_connections 2024-03-30T03:55:03.110 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.110+0000 7fd485a3b700 1 -- 172.21.15.193:0/756953361 wait complete. 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 Processor -- start 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 -- start start 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd4801012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:03.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4801a4e50 con 0x7fd46c0062a0 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4801a4f90 con 0x7fd480101800 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd485a3b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4801a5150 con 0x7fd4801a2490 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd47f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd47f7fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:36724/0 (socket says 172.21.15.193:36724) 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.111+0000 7fd47f7fe700 1 -- 172.21.15.193:0/2566073952 learned_addr learned my addr 172.21.15.193:0/2566073952 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd4801012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47e7fc700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47f7fe700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 msgr2=0x7fd4801a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47f7fe700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 msgr2=0x7fd4801a4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47f7fe700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47f7fe700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:55:03.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47e7fc700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 msgr2=0x7fd480101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47e7fc700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 msgr2=0x7fd480101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47e7fc700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47e7fc700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 msgr2=0x7fd4801a4880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 msgr2=0x7fd480101c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 -- 172.21.15.193:0/2566073952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46800f8e0 con 0x7fd46c0062a0 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd47effd700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd4801012c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4801007b0 tx=0x7fd468013f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:03.115 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd468011d60 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd468011ec0 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4801a53d0 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4680038e0 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.112+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4801a5870 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.113+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fd468003b00 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.113+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4800623e0 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.114+0000 7fd484a39700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 0x7fd46003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.114+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd46804e710 con 0x7fd46c0062a0 2024-03-30T03:55:03.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.114+0000 7fd47e7fc700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 0x7fd46003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:03.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.114+0000 7fd47e7fc700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 0x7fd46003b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd470009850 tx=0x7fd470008040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:03.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.119+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd468030c40 con 0x7fd46c0062a0 2024-03-30T03:55:03.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.398+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd480107400 con 0x7fd46c0062a0 2024-03-30T03:55:03.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.399+0000 7fd484a39700 1 -- 172.21.15.193:0/2566073952 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7fd468036020 con 0x7fd46c0062a0 2024-03-30T03:55:03.399 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-30T03:55:03.399 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","modified":"2024-03-30T03:54:56.645819Z","created":"2024-03-30T03:53:11.933267Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 msgr2=0x7fd46003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 0x7fd46003b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd470009850 tx=0x7fd470008040 comp rx=0 tx=0).stop 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 msgr2=0x7fd4801012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd4801012c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd4801007b0 tx=0x7fd468013f80 comp rx=0 tx=0).stop 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 shutdown_connections 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd460038c20 0x7fd46003b0e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4801a2490 0x7fd4801a4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.401 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd46c0062a0 0x7fd4801012c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 --2- 172.21.15.193:0/2566073952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd480101800 0x7fd480101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:03.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 >> 172.21.15.193:0/2566073952 conn(0x7fd480076390 msgr2=0x7fd4800683f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:03.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 shutdown_connections 2024-03-30T03:55:03.402 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:03.401+0000 7fd485a3b700 1 -- 172.21.15.193:0/2566073952 wait complete. 2024-03-30T03:55:03.402 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-03-30T03:55:03.762 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-30T03:55:03.762 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-03-30T03:55:03.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: mon.c calling monitor election 2024-03-30T03:55:03.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: mon.a calling monitor election 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:03.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: fsmap 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: mgrmap e13: x(active, since 45s) 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: mon.b (rank 2) addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] is down (out of quorum) 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:03.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mon.b calling monitor election 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mon.b calling monitor election 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mon.c calling monitor election 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mon.a calling monitor election 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: fsmap 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: mgrmap e13: x(active, since 47s) 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T03:55:04.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: overall HEALTH_OK 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:04.483 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mon.b calling monitor election 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mon.b calling monitor election 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mon.c calling monitor election 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mon.a calling monitor election 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: fsmap 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: mgrmap e13: x(active, since 47s) 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T03:55:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: overall HEALTH_OK 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:04.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mon.b calling monitor election 2024-03-30T03:55:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mon.b calling monitor election 2024-03-30T03:55:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mon.c calling monitor election 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mon.a calling monitor election 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: fsmap 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: osdmap e4: 0 total, 0 up, 0 in 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: mgrmap e13: x(active, since 47s) 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: overall HEALTH_OK 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:04.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 -- 172.21.15.23:0/1043055268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 msgr2=0x7fa6bc108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 --2- 172.21.15.23:0/1043055268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc108cb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009910 tx=0x7fa6ac009c20 comp rx=0 tx=0).stop 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 -- 172.21.15.23:0/1043055268 shutdown_connections 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 --2- 172.21.15.23:0/1043055268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc108cb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 -- 172.21.15.23:0/1043055268 >> 172.21.15.23:0/1043055268 conn(0x7fa6bc0fe400 msgr2=0x7fa6bc100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:05.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 -- 172.21.15.23:0/1043055268 shutdown_connections 2024-03-30T03:55:05.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.295+0000 7fa6c3000700 1 -- 172.21.15.23:0/1043055268 wait complete. 2024-03-30T03:55:05.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.296+0000 7fa6c3000700 1 Processor -- start 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.296+0000 7fa6c3000700 1 -- start start 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6bc1987f0 0x7fa6bc19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6bc19d0e0 0x7fa6bc19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc198ee0 con 0x7fa6bc1088d0 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc1a4e50 con 0x7fa6bc19d0e0 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c3000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc1a4f90 con 0x7fa6bc1987f0 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41892/0 (socket says 172.21.15.23:41892) 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 -- 172.21.15.23:0/2718482849 learned_addr learned my addr 172.21.15.23:0/2718482849 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6bbe00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6bc1987f0 0x7fa6bc19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:05.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6c1000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6bc19d0e0 0x7fa6bc19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:05.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 -- 172.21.15.23:0/2718482849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6bc1987f0 msgr2=0x7fa6bc19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6bc1987f0 0x7fa6bc19cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 -- 172.21.15.23:0/2718482849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6bc19d0e0 msgr2=0x7fa6bc19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6bc19d0e0 0x7fa6bc19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.297+0000 7fa6b3e00700 1 -- 172.21.15.23:0/2718482849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac0095f0 con 0x7fa6bc1088d0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6b3e00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac004cb0 tx=0x7fa6ac004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac01d070 con 0x7fa6bc1088d0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6ac005000 con 0x7fa6bc1088d0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc1a5270 con 0x7fa6bc1088d0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac022d50 con 0x7fa6bc1088d0 2024-03-30T03:55:05.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.298+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc1a5770 con 0x7fa6bc1088d0 2024-03-30T03:55:05.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.299+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6bc104010 con 0x7fa6bc1088d0 2024-03-30T03:55:05.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.299+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7fa6ac022470 con 0x7fa6bc1088d0 2024-03-30T03:55:05.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.300+0000 7fa6b9600700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 0x7fa6a003b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:05.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.300+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6ac04c810 con 0x7fa6bc1088d0 2024-03-30T03:55:05.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.300+0000 7fa6bbe00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 0x7fa6a003b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:05.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.301+0000 7fa6bbe00700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 0x7fa6a003b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ad0 tx=0x7fa6a4009210 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:05.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:05 smithi023 ceph-mon[38254]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T03:55:05.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:05 smithi023 ceph-mon[38254]: Reconfiguring daemon mon.a on smithi023 2024-03-30T03:55:05.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:05 smithi023 ceph-mon[38254]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:05.302 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:05.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.305+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa6ac00e5c0 con 0x7fa6bc1088d0 2024-03-30T03:55:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:05 smithi193 ceph-mon[79581]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T03:55:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:05 smithi193 ceph-mon[79581]: Reconfiguring daemon mon.a on smithi023 2024-03-30T03:55:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:05 smithi193 ceph-mon[79581]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:05.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.533+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa6bc0623e0 con 0x7fa6bc1088d0 2024-03-30T03:55:05.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.534+0000 7fa6b9600700 1 -- 172.21.15.23:0/2718482849 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fa6ac027020 con 0x7fa6bc1088d0 2024-03-30T03:55:05.533 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:55:05.533 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-30T03:55:05.533 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T03:55:05.533 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-03-30T03:55:05.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.536+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 msgr2=0x7fa6a003b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.536+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 0x7fa6a003b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ad0 tx=0x7fa6a4009210 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.536+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 msgr2=0x7fa6bc1982b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.536+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac004cb0 tx=0x7fa6ac004ce0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 shutdown_connections 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6bc19d0e0 0x7fa6bc19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6bc1088d0 0x7fa6bc1982b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6a0038c70 0x7fa6a003b130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 --2- 172.21.15.23:0/2718482849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6bc1987f0 0x7fa6bc19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 >> 172.21.15.23:0/2718482849 conn(0x7fa6bc0fe400 msgr2=0x7fa6bc1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 shutdown_connections 2024-03-30T03:55:05.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:05.537+0000 7fa6c3000700 1 -- 172.21.15.23:0/2718482849 wait complete. 2024-03-30T03:55:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:05 smithi154 ceph-mon[79161]: Reconfiguring mon.a (unknown last config time)... 2024-03-30T03:55:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:05 smithi154 ceph-mon[79161]: Reconfiguring daemon mon.a on smithi023 2024-03-30T03:55:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:05 smithi154 ceph-mon[79161]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:55:05.918 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-30T03:55:05.918 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:55:05.918 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T03:55:05.987 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:55:05.987 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:55:06.093 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:55:06.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T03:55:06.124 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:55:06.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:55:06.196 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:55:06.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-03-30T03:55:06.234 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:55:06.234 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-30T03:55:06.288 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2718482849' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:06.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:06.289 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.306 INFO:tasks.cephadm:Adding mgr.x on smithi023 2024-03-30T03:55:06.306 INFO:tasks.cephadm:Adding mgr.y on smithi154 2024-03-30T03:55:06.306 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-03-30T03:55:06.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch apply mgr '3;smithi023=x;smithi154=y;smithi193=z' 2024-03-30T03:55:06.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2718482849' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:06.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2718482849' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:06.562 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:55:07.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: Reconfiguring mon.b (monmap changed)... 2024-03-30T03:55:07.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: Reconfiguring daemon mon.b on smithi154 2024-03-30T03:55:07.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:07.288 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:07.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:07.289 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: Reconfiguring mon.b (monmap changed)... 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: Reconfiguring daemon mon.b on smithi154 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: Reconfiguring mon.b (monmap changed)... 2024-03-30T03:55:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: Reconfiguring daemon mon.b on smithi154 2024-03-30T03:55:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-30T03:55:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:07.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 -- 172.21.15.193:0/3949760014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 msgr2=0x7ff9680ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:07.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 --2- 172.21.15.193:0/3949760014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff9680ffb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff958009910 tx=0x7ff958009c20 comp rx=0 tx=0).stop 2024-03-30T03:55:07.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 -- 172.21.15.193:0/3949760014 shutdown_connections 2024-03-30T03:55:07.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 --2- 172.21.15.193:0/3949760014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff9680ffb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:07.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 -- 172.21.15.193:0/3949760014 >> 172.21.15.193:0/3949760014 conn(0x7ff9680fad20 msgr2=0x7ff9680fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:07.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 -- 172.21.15.193:0/3949760014 shutdown_connections 2024-03-30T03:55:07.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 -- 172.21.15.193:0/3949760014 wait complete. 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.949+0000 7ff96daf5700 1 Processor -- start 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 -- start start 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff968069790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 0x7ff96819d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96819dfe0 con 0x7ff9680ff760 2024-03-30T03:55:07.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96819e150 con 0x7ff96819b3d0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff96daf5700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96819e310 con 0x7ff968198aa0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:36740/0 (socket says 172.21.15.193:36740) 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 -- 172.21.15.193:0/1368911566 learned_addr learned my addr 172.21.15.193:0/1368911566 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff9677fe700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff968069790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff967fff700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 0x7ff96819d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 -- 172.21.15.193:0/1368911566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 msgr2=0x7ff96819d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 0x7ff96819d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 -- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 msgr2=0x7ff968069790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff968069790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 -- 172.21.15.193:0/1368911566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9580095f0 con 0x7ff968198aa0 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff967fff700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 0x7ff96819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:55:07.953 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff9677fe700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff968069790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.950+0000 7ff966ffd700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00b780 tx=0x7ff95c00ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.951+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c004230 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.951+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96819e5f0 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.951+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff95c004390 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.951+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c011ad0 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.951+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96819ebc0 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.952+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96818e430 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.952+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7ff95c0036a0 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.952+0000 7ff964ff9700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 0x7ff95003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.952+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff95c04acf0 con 0x7ff968198aa0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.952+0000 7ff9677fe700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 0x7ff95003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:07.954 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.953+0000 7ff9677fe700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 0x7ff95003b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff95800b5c0 tx=0x7ff958005200 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:07.956 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:07.957+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff95c004500 con 0x7ff968198aa0 2024-03-30T03:55:08.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.163+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi154=y;smithi193=z", "target": ["mon-mgr", ""]}) v1 -- 0x7ff96802cd20 con 0x7ff950038c20 2024-03-30T03:55:08.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.174+0000 7ff964ff9700 1 -- 172.21.15.193:0/1368911566 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff96802cd20 con 0x7ff950038c20 2024-03-30T03:55:08.174 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 msgr2=0x7ff95003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 0x7ff95003b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff95800b5c0 tx=0x7ff958005200 comp rx=0 tx=0).stop 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 msgr2=0x7ff96819ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff95c00b780 tx=0x7ff95c00ba90 comp rx=0 tx=0).stop 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 shutdown_connections 2024-03-30T03:55:08.176 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff96819b3d0 0x7ff96819d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9680ff760 0x7ff968069790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff950038c20 0x7ff95003b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 --2- 172.21.15.193:0/1368911566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff968198aa0 0x7ff96819ae90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 >> 172.21.15.193:0/1368911566 conn(0x7ff9680fad20 msgr2=0x7ff9680fb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 shutdown_connections 2024-03-30T03:55:08.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:08.176+0000 7ff96daf5700 1 -- 172.21.15.193:0/1368911566 wait complete. 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: Reconfiguring mon.c (monmap changed)... 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: Reconfiguring daemon mon.c on smithi193 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: Reconfiguring mon.c (monmap changed)... 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: Reconfiguring daemon mon.c on smithi193 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:08.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.578 DEBUG:teuthology.orchestra.run.smithi154:mgr.y> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.y.service 2024-03-30T03:55:08.581 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.z.service 2024-03-30T03:55:08.583 INFO:tasks.cephadm:Deploying OSDs... 2024-03-30T03:55:08.583 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:55:08.583 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T03:55:08.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: Reconfiguring mon.c (monmap changed)... 2024-03-30T03:55:08.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: Reconfiguring daemon mon.c on smithi193 2024-03-30T03:55:08.608 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:08.609 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:08.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:08.613 INFO:journalctl@ceph.mgr.z.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:55:08.614 INFO:journalctl@ceph.mgr.y.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:55:08.630 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T03:55:08.630 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-30T03:55:08.716 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T03:55:08.716 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:08.716 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 452 Links: 1 2024-03-30T03:55:08.716 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:08.717 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:08.717 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-30 03:54:27.050645877 +0000 2024-03-30T03:55:08.717 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-30 03:47:26.281731088 +0000 2024-03-30T03:55:08.717 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-30 03:47:26.281731088 +0000 2024-03-30T03:55:08.717 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-30 03:47:26.246731349 +0000 2024-03-30T03:55:08.717 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T03:55:08.821 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-30T03:55:08.821 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-30T03:55:08.821 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000342782 s, 1.5 MB/s 2024-03-30T03:55:08.824 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T03:55:08.867 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-30T03:55:08.949 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T03:55:08.949 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:08.949 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 458 Links: 1 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-30 03:54:27.143645225 +0000 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-30 03:47:26.771727437 +0000 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-30 03:47:26.771727437 +0000 2024-03-30T03:55:08.950 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-30 03:47:26.739727676 +0000 2024-03-30T03:55:08.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T03:55:09.052 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-30T03:55:09.052 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-30T03:55:09.052 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000364469 s, 1.4 MB/s 2024-03-30T03:55:09.054 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T03:55:09.097 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 464 Links: 1 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-30 03:54:27.228644629 +0000 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-30 03:47:27.244723913 +0000 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-30 03:47:27.244723913 +0000 2024-03-30T03:55:09.180 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-30 03:47:27.211724159 +0000 2024-03-30T03:55:09.181 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T03:55:09.284 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-30T03:55:09.284 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-30T03:55:09.285 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000516531 s, 991 kB/s 2024-03-30T03:55:09.287 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T03:55:09.330 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-30T03:55:09.412 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 470 Links: 1 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-30 03:54:27.311644047 +0000 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-30 03:47:27.731720285 +0000 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-30 03:47:27.731720285 +0000 2024-03-30T03:55:09.413 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-30 03:47:27.699720523 +0000 2024-03-30T03:55:09.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T03:55:09.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:09 smithi193 ceph-mon[79581]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi154=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:09.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:09 smithi193 ceph-mon[79581]: Saving service mgr spec with placement smithi023=x;smithi154=y;smithi193=z;count:3 2024-03-30T03:55:09.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:09 smithi193 ceph-mon[79581]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:09.450 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:09 smithi193 ceph-mon[79581]: Deploying daemon mgr.z on smithi193 2024-03-30T03:55:09.514 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-30T03:55:09.514 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-30T03:55:09.515 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.00033445 s, 1.5 MB/s 2024-03-30T03:55:09.517 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T03:55:09.560 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-03-30T03:55:09.560 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T03:55:09.581 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T03:55:09.581 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-03-30T03:55:09.642 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T03:55:09.642 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 211618 Links: 1 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-30 03:54:56.195364535 +0000 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-30 03:45:34.699694587 +0000 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-30 03:45:34.699694587 +0000 2024-03-30T03:55:09.643 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-03-30T03:55:09.643 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T03:55:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:09 smithi154 ceph-mon[79161]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi154=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:09.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:09 smithi154 ceph-mon[79161]: Saving service mgr spec with placement smithi023=x;smithi154=y;smithi193=z;count:3 2024-03-30T03:55:09.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:09 smithi154 ceph-mon[79161]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:09.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:09 smithi154 ceph-mon[79161]: Deploying daemon mgr.z on smithi193 2024-03-30T03:55:09.711 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-30T03:55:09.711 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-30T03:55:09.711 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000274661 s, 1.9 MB/s 2024-03-30T03:55:09.712 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T03:55:09.743 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:09 smithi193 systemd[1]: Starting Ceph mgr.z for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:55:09.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:09 smithi023 ceph-mon[38254]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi154=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:09.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:09 smithi023 ceph-mon[38254]: Saving service mgr spec with placement smithi023=x;smithi154=y;smithi193=z;count:3 2024-03-30T03:55:09.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:09 smithi023 ceph-mon[38254]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:09.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:09 smithi023 ceph-mon[38254]: Deploying daemon mgr.z on smithi193 2024-03-30T03:55:09.774 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-03-30T03:55:09.834 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 202419 Links: 1 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-30 03:54:56.195364535 +0000 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-30 03:45:35.042685535 +0000 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-30 03:45:35.042685535 +0000 2024-03-30T03:55:09.835 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-03-30T03:55:09.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T03:55:09.906 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-30T03:55:09.906 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-30T03:55:09.906 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000229758 s, 2.2 MB/s 2024-03-30T03:55:09.907 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T03:55:09.968 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 202460 Links: 1 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-30 03:54:56.195364535 +0000 2024-03-30T03:55:10.028 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-30 03:45:35.417675638 +0000 2024-03-30T03:55:10.029 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-30 03:45:35.417675638 +0000 2024-03-30T03:55:10.029 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-03-30T03:55:10.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T03:55:10.096 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-30T03:55:10.096 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-30T03:55:10.096 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000181504 s, 2.8 MB/s 2024-03-30T03:55:10.097 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T03:55:10.156 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-03-30T03:55:10.213 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 6h/6d Inode: 208830 Links: 1 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-03-30 03:54:56.195364535 +0000 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-03-30 03:45:35.770666323 +0000 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-03-30 03:45:35.770666323 +0000 2024-03-30T03:55:10.214 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-03-30T03:55:10.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T03:55:10.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:09 smithi193 ceph-mgr[81468]: -- 172.21.15.193:0/63805217 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560f9d4521a0 con 0x560f9d428c00 2024-03-30T03:55:10.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:09 smithi193 bash[81430]: c639521c91074b6b45a0907bfdf2e575f5a0134201bc19f2bfdcbb372dfcd517 2024-03-30T03:55:10.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:09 smithi193 systemd[1]: Started Ceph mgr.z for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:55:10.282 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-03-30T03:55:10.282 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-03-30T03:55:10.282 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000251511 s, 2.0 MB/s 2024-03-30T03:55:10.283 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T03:55:10.342 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-30T03:55:10.342 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T03:55:10.360 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T03:55:10.360 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-30T03:55:10.403 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:10.309+0000 7fd0d9652200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 207478 Links: 1 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 03:55:09.499747495 +0000 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 03:45:38.899061320 +0000 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 03:45:38.899061320 +0000 2024-03-30T03:55:10.419 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-03-30T03:55:10.419 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T03:55:10.488 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T03:55:10.489 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T03:55:10.489 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000205731 s, 2.5 MB/s 2024-03-30T03:55:10.490 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T03:55:10.549 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-30T03:55:10.585 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:10.403+0000 7fd0d9652200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T03:55:10.585 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:10.496+0000 7fd0d9652200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T03:55:10.585 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:10.585+0000 7fd0d9652200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T03:55:10.601 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 209448 Links: 1 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 03:55:09.499747495 +0000 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 03:45:39.247055810 +0000 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 03:45:39.247055810 +0000 2024-03-30T03:55:10.602 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-03-30T03:55:10.602 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T03:55:10.668 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T03:55:10.668 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T03:55:10.668 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000191331 s, 2.7 MB/s 2024-03-30T03:55:10.669 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T03:55:10.727 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-30T03:55:10.785 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T03:55:10.785 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.785 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 211505 Links: 1 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 03:55:09.499747495 +0000 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 03:45:39.601050205 +0000 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 03:45:39.601050205 +0000 2024-03-30T03:55:10.786 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-03-30T03:55:10.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T03:55:10.851 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T03:55:10.851 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T03:55:10.851 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00020572 s, 2.5 MB/s 2024-03-30T03:55:10.853 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T03:55:10.913 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-30T03:55:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T03:55:10.972 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T03:55:10.972 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 209484 Links: 1 2024-03-30T03:55:10.972 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T03:55:10.972 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-30T03:55:10.973 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-30 03:55:09.499747495 +0000 2024-03-30T03:55:10.973 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-30 03:45:39.958044553 +0000 2024-03-30T03:55:10.973 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-30 03:45:39.958044553 +0000 2024-03-30T03:55:10.973 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2024-03-30T03:55:10.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T03:55:11.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.027 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: Deploying daemon mgr.y on smithi154 2024-03-30T03:55:11.028 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:10 smithi154 ceph-mon[79161]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:11.048 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-30T03:55:11.048 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-30T03:55:11.048 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000291338 s, 1.8 MB/s 2024-03-30T03:55:11.050 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T03:55:11.110 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-30T03:55:11.110 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: Deploying daemon mgr.y on smithi154 2024-03-30T03:55:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:10 smithi023 ceph-mon[38254]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:11.432 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:11 smithi154 systemd[1]: Starting Ceph mgr.y for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:55:11.474 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.244+0000 7fd0d9652200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: Deploying daemon mgr.y on smithi154 2024-03-30T03:55:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:10 smithi193 ceph-mon[79581]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:11.844 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.493+0000 7fd0d9652200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T03:55:11.844 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.579+0000 7fd0d9652200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T03:55:11.844 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.685+0000 7fd0d9652200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T03:55:11.844 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.844+0000 7fd0d9652200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T03:55:11.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:11 smithi154 ceph-mgr[80658]: -- 172.21.15.154:0/2163842037 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1917ea1a0 con 0x55f1917c0c00 2024-03-30T03:55:11.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:11 smithi154 bash[80620]: 584510d83dcafac441a37c647fecf571556ca15631e7eb62fadc9986658c3956 2024-03-30T03:55:11.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:11 smithi154 systemd[1]: Started Ceph mgr.y for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:55:12.171 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:12.074+0000 7f92fc9ac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T03:55:12.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:11.970+0000 7fd0d9652200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T03:55:12.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:12.062+0000 7fd0d9652200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T03:55:12.378 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:12.171+0000 7f92fc9ac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T03:55:12.378 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:12.280+0000 7f92fc9ac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T03:55:12.379 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:12.378+0000 7f92fc9ac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T03:55:12.746 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:12.484+0000 7fd0d9652200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:13.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:12 smithi023 ceph-mon[38254]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:13.047 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:12 smithi154 ceph-mon[79161]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:12.893+0000 7fd0d9652200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.088 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:13.089 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:12 smithi193 ceph-mon[79581]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:13.225 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T03:55:13.225 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T03:55:13.225 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-30T03:55:13.226 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-30T03:55:13.226 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.049103 s, 214 MB/s 2024-03-30T03:55:13.226 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-30T03:55:13.297 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.046+0000 7f92fc9ac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:55:13.493 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:13.088+0000 7fd0d9652200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T03:55:13.493 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:13.181+0000 7fd0d9652200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T03:55:13.580 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-30T03:55:13.647 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.297+0000 7f92fc9ac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T03:55:13.647 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.382+0000 7f92fc9ac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T03:55:13.647 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.488+0000 7f92fc9ac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T03:55:13.648 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.647+0000 7f92fc9ac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T03:55:13.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.840 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:13.841 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:13 smithi023 ceph-mon[38254]: Reconfiguring daemon mgr.x on smithi023 2024-03-30T03:55:13.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.774+0000 7f92fc9ac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T03:55:13.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:13.859+0000 7f92fc9ac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:13.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:13.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:13 smithi154 ceph-mon[79161]: Reconfiguring daemon mgr.x on smithi023 2024-03-30T03:55:14.032 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:13.937+0000 7fd0d9652200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: Reconfiguring mgr.x (unknown last config time)... 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-30T03:55:14.033 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-30T03:55:14.034 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:14.034 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:13 smithi193 ceph-mon[79581]: Reconfiguring daemon mgr.x on smithi023 2024-03-30T03:55:14.198 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:55:14.493 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:14.033+0000 7fd0d9652200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T03:55:14.493 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:14.179+0000 7fd0d9652200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T03:55:14.618 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:14.230+0000 7f92fc9ac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T03:55:14.903 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:14.619+0000 7f92fc9ac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T03:55:14.903 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:14.811+0000 7f92fc9ac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T03:55:15.170 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:14.803+0000 7fd0d9652200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T03:55:15.170 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:14.981+0000 7fd0d9652200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T03:55:15.170 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:15.170+0000 7fd0d9652200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T03:55:15.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:14.903+0000 7f92fc9ac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T03:55:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 -- 172.21.15.23:0/1913521835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50074de0 msgr2=0x7f2a50073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 --2- 172.21.15.23:0/1913521835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50074de0 0x7f2a50073240 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a44009990 tx=0x7f2a44009ca0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 -- 172.21.15.23:0/1913521835 shutdown_connections 2024-03-30T03:55:15.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 --2- 172.21.15.23:0/1913521835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50108cd0 0x7f2a5010b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 --2- 172.21.15.23:0/1913521835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50074de0 0x7f2a50073240 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 --2- 172.21.15.23:0/1913521835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 0x7f2a50073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 -- 172.21.15.23:0/1913521835 >> 172.21.15.23:0/1913521835 conn(0x7f2a500fc3c0 msgr2=0x7f2a500fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 -- 172.21.15.23:0/1913521835 shutdown_connections 2024-03-30T03:55:15.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.258+0000 7f2a58600700 1 -- 172.21.15.23:0/1913521835 wait complete. 2024-03-30T03:55:15.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.259+0000 7f2a58600700 1 Processor -- start 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.259+0000 7f2a58600700 1 -- start start 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 0x7f2a50198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50108cd0 0x7f2a501a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a501a0a80 con 0x7f2a50108cd0 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a501a0bc0 con 0x7f2a50074de0 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a58600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a501a0d50 con 0x7f2a50073810 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 0x7f2a50198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52202/0 (socket says 172.21.15.23:52202) 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 -- 172.21.15.23:0/180241963 learned_addr learned my addr 172.21.15.23:0/180241963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:55:15.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a56600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50108cd0 0x7f2a501a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 -- 172.21.15.23:0/180241963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 msgr2=0x7f2a50198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 0x7f2a50198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 -- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50108cd0 msgr2=0x7f2a501a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50108cd0 0x7f2a501a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.260+0000 7f2a55200700 1 -- 172.21.15.23:0/180241963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c009520 con 0x7f2a50074de0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a55200700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00ea50 tx=0x7f2a3c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00cbd0 con 0x7f2a50074de0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a3c00cd30 con 0x7f2a50074de0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c018680 con 0x7f2a50074de0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a440095f0 con 0x7f2a50074de0 2024-03-30T03:55:15.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.261+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a501a12c0 con 0x7f2a50074de0 2024-03-30T03:55:15.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.262+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 13) v1 ==== 45589+0+0 (secure 0 0 0) 0x7f2a3c010430 con 0x7f2a50074de0 2024-03-30T03:55:15.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.262+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a50107a90 con 0x7f2a50074de0 2024-03-30T03:55:15.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.262+0000 7f2a4e000700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 0x7f2a3803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:15.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.262+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2a3c014070 con 0x7f2a50074de0 2024-03-30T03:55:15.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.263+0000 7f2a55c00700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 0x7f2a3803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:15.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.263+0000 7f2a55c00700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 0x7f2a3803ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400b680 tx=0x7f2a44005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:15.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.268+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2a3c01f310 con 0x7f2a50074de0 2024-03-30T03:55:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:15.491+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a5002cdc0 con 0x7f2a380389d0 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:15.664 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:15.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:15.664+0000 7f92fc9ac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T03:55:15.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:15.760+0000 7f92fc9ac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T03:55:15.932 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:15.908+0000 7f92fc9ac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T03:55:16.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:16.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:16.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:16.377 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:16.096+0000 7fd0d9652200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T03:55:16.377 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:16.377 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:16.377 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:16.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:16.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:16.546 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:16.656 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:16.377+0000 7fd0d9652200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:55:16.656 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:16.465+0000 7fd0d9652200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T03:55:16.657 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 03:55:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T03:55:16.657+0000 7fd0d9652200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T03:55:16.920 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:16.546+0000 7f92fc9ac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T03:55:16.921 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:16.726+0000 7f92fc9ac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T03:55:17.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:16.920+0000 7f92fc9ac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T03:55:17.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:17.280+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 14) v1 ==== 90699+0+0 (secure 0 0 0) 0x7f2a3c018d00 con 0x7f2a50074de0 2024-03-30T03:55:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: Standby manager daemon z started 2024-03-30T03:55:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T03:55:17.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:17.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T03:55:17.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:17 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: Standby manager daemon z started 2024-03-30T03:55:17.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T03:55:17.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:17.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T03:55:17.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: Standby manager daemon z started 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T03:55:17.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:17 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/3566555318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:18.132 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:17.849+0000 7f92fc9ac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T03:55:18.412 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:18.132+0000 7f92fc9ac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T03:55:18.412 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:18.221+0000 7f92fc9ac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T03:55:18.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T03:55:18.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T03:55:18.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/856685315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]': finished 2024-03-30T03:55:18.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T03:55:18.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:18.682 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 03:55:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T03:55:18.412+0000 7f92fc9ac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/856685315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]': finished 2024-03-30T03:55:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T03:55:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: mgrmap e14: x(active, since 61s), standbys: z 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/856685315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]: dispatch 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60b581f8-6bf1-4e71-8ad6-974da9b9db58"}]': finished 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: osdmap e5: 1 total, 0 up, 1 in 2024-03-30T03:55:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:19.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:19.295+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mon.2 v2:172.21.15.154:3300/0 8 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7f2a3c0187e0 con 0x7f2a50074de0 2024-03-30T03:55:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: Standby manager daemon y started 2024-03-30T03:55:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T03:55:19.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:19.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T03:55:19.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:19.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:19 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2273738126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: Standby manager daemon y started 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:19.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:19 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2273738126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:19.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: Standby manager daemon y started 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/3262972978' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T03:55:19.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:19 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2273738126' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:20 smithi154 ceph-mon[79161]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-30T03:55:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T03:55:20.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:20 smithi193 ceph-mon[79581]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-30T03:55:20.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T03:55:20.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:20 smithi023 ceph-mon[38254]: mgrmap e15: x(active, since 63s), standbys: z, y 2024-03-30T03:55:20.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T03:55:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:21 smithi154 ceph-mon[79161]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:21 smithi193 ceph-mon[79581]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:21 smithi023 ceph-mon[38254]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:23 smithi154 ceph-mon[79161]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:23 smithi193 ceph-mon[79581]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:23.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:23 smithi023 ceph-mon[38254]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:24.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T03:55:24.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T03:55:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:24.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-30T03:55:24.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:25.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:25 smithi023 ceph-mon[38254]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:25.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:25 smithi023 ceph-mon[38254]: Deploying daemon osd.0 on smithi023 2024-03-30T03:55:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:25 smithi154 ceph-mon[79161]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:25 smithi154 ceph-mon[79161]: Deploying daemon osd.0 on smithi023 2024-03-30T03:55:25.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:25 smithi193 ceph-mon[79581]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:25.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:25 smithi193 ceph-mon[79581]: Deploying daemon osd.0 on smithi023 2024-03-30T03:55:27.004 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:26 smithi023 ceph-mon[38254]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:27.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:26 smithi154 ceph-mon[79161]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:26 smithi193 ceph-mon[79581]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:29 smithi023 ceph-mon[38254]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:29 smithi154 ceph-mon[79161]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:29.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:29 smithi193 ceph-mon[79581]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:29.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:29.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:31 smithi154 ceph-mon[79161]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:31.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:31 smithi193 ceph-mon[79581]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:31.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:31.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:31.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:31 smithi023 ceph-mon[38254]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:31.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:31.767 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.142 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-30T03:55:32.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.143+0000 7f2a4e000700 1 -- 172.21.15.23:0/180241963 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2a5002cdc0 con 0x7f2a380389d0 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 msgr2=0x7f2a3803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 0x7f2a3803ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400b680 tx=0x7f2a44005fd0 comp rx=0 tx=0).stop 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 msgr2=0x7f2a501994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00ea50 tx=0x7f2a3c00ed60 comp rx=0 tx=0).stop 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 shutdown_connections 2024-03-30T03:55:32.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a50108cd0 0x7f2a501a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2a380389d0 0x7f2a3803ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a50073810 0x7f2a50198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 --2- 172.21.15.23:0/180241963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a50074de0 0x7f2a501994d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 >> 172.21.15.23:0/180241963 conn(0x7f2a500fc3c0 msgr2=0x7f2a50104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 shutdown_connections 2024-03-30T03:55:32.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:55:32.146+0000 7f2a58600700 1 -- 172.21.15.23:0/180241963 wait complete. 2024-03-30T03:55:32.369 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:32.370 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:32.370 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.370 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.370 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.370 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:32.555 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.0.service 2024-03-30T03:55:32.560 INFO:tasks.cephadm:Deploying osd.1 on smithi154 with /dev/vg_nvme/lv_4... 2024-03-30T03:55:32.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T03:55:32.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:32.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:32.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:32.636 INFO:journalctl@ceph.osd.0.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:32.788 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:55:33.481 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:33 smithi023 ceph-mon[38254]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:33.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:33 smithi154 ceph-mon[79161]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:33 smithi193 ceph-mon[79581]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:33.772 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:55:33 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T03:55:33.480+0000 7fbdac8be640 -1 osd.0 0 log_to_monitors true 2024-03-30T03:55:34.110 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T03:55:34.110 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T03:55:34.111 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-30T03:55:34.111 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-30T03:55:34.111 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112537 s, 932 MB/s 2024-03-30T03:55:34.111 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-30T03:55:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:34 smithi154 ceph-mon[79161]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T03:55:34.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-03-30T03:55:34.738 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:55:34.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:34 smithi193 ceph-mon[79581]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T03:55:34.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:34 smithi023 ceph-mon[38254]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-30T03:55:35.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:35 smithi154 ceph-mon[79161]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:35.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:35 smithi154 ceph-mon[79161]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T03:55:35.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:35 smithi154 ceph-mon[79161]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T03:55:35.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:35.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:35 smithi154 ceph-mon[79161]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:55:35.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-mon[38254]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:35.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-mon[38254]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T03:55:35.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-mon[38254]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T03:55:35.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:35.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-mon[38254]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:55:35.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T03:55:35.306+0000 7fbda0e00700 -1 osd.0 0 waiting for initial osdmap 2024-03-30T03:55:35.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:55:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T03:55:35.310+0000 7fbd9bc00700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 -- 172.21.15.154:0/2379102707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904106ae0 msgr2=0x7fd904108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 --2- 172.21.15.154:0/2379102707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904106ae0 0x7fd904108f50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b0a0 tx=0x7fd8fc00b3b0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 -- 172.21.15.154:0/2379102707 shutdown_connections 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 --2- 172.21.15.154:0/2379102707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904106ae0 0x7fd904108f50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 --2- 172.21.15.154:0/2379102707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904101880 0x7fd904103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 --2- 172.21.15.154:0/2379102707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 0x7fd9041065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 -- 172.21.15.154:0/2379102707 >> 172.21.15.154:0/2379102707 conn(0x7fd9040fb2b0 msgr2=0x7fd9040fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:35.699 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 -- 172.21.15.154:0/2379102707 shutdown_connections 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.697+0000 7fd909a89700 1 -- 172.21.15.154:0/2379102707 wait complete. 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.698+0000 7fd909a89700 1 Processor -- start 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.698+0000 7fd909a89700 1 -- start start 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 0x7fd9040ff830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904106ae0 0x7fd9040ffd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904100510 con 0x7fd904106ae0 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904100680 con 0x7fd904101880 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9041a0f10 con 0x7fd9041041b0 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:38618/0 (socket says 172.21.15.154:38618) 2024-03-30T03:55:35.700 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 -- 172.21.15.154:0/2461343097 learned_addr learned my addr 172.21.15.154:0/2461343097 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd902ffd700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 0x7fd9040ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd903fff700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904106ae0 0x7fd9040ffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 -- 172.21.15.154:0/2461343097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 msgr2=0x7fd9040ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 0x7fd9040ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 -- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904106ae0 msgr2=0x7fd9040ffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904106ae0 0x7fd9040ffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 -- 172.21.15.154:0/2461343097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009d00 con 0x7fd904101880 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd9037fe700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d950 tx=0x7fd8f400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4009940 con 0x7fd904101880 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8f4010460 con 0x7fd904101880 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9041a11a0 con 0x7fd904101880 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400f620 con 0x7fd904101880 2024-03-30T03:55:35.701 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.699+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9041a16a0 con 0x7fd904101880 2024-03-30T03:55:35.702 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.700+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9040fce30 con 0x7fd904101880 2024-03-30T03:55:35.703 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.701+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7fd8f40108d0 con 0x7fd904101880 2024-03-30T03:55:35.703 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.702+0000 7fd900ff9700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 0x7fd8ec0a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:35.703 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.702+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1487+0+0 (secure 0 0 0) 0x7fd8f40cabb0 con 0x7fd904101880 2024-03-30T03:55:35.703 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.702+0000 7fd902ffd700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 0x7fd8ec0a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:35.703 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.703+0000 7fd902ffd700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 0x7fd8ec0a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009a10 tx=0x7fd8f8009190 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:35.705 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.706+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd8f4095ed0 con 0x7fd904101880 2024-03-30T03:55:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:35 smithi193 ceph-mon[79581]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:35 smithi193 ceph-mon[79581]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-30T03:55:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:35 smithi193 ceph-mon[79581]: osdmap e6: 1 total, 0 up, 1 in 2024-03-30T03:55:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:35 smithi193 ceph-mon[79581]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:55:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:35.904+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9040611f0 con 0x7fd8ec0a1090 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi023 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:36.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:36.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:36.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:36.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:36.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:55:36.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T03:55:36.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi023 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:36.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: osdmap e7: 1 total, 0 up, 1 in 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi023 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:36.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:55:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:55:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 17270M 2024-03-30T03:55:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:37.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] boot 2024-03-30T03:55:37.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T03:55:37.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 17270M 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] boot 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T03:55:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 17270M 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: osd.0 [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] boot 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: osdmap e8: 1 total, 1 up, 1 in 2024-03-30T03:55:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/1801251306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]': finished 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T03:55:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:38 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/1801251306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]': finished 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T03:55:38.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:38 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: osdmap e9: 1 total, 1 up, 1 in 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/1801251306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]: dispatch 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4e38bd8a-9f96-4585-ad8f-a364a12543df"}]': finished 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: osdmap e10: 2 total, 1 up, 2 in 2024-03-30T03:55:38.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:38 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:39 smithi154 ceph-mon[79161]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:39 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/2641547214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:39.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:39 smithi193 ceph-mon[79581]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:39.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:39 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/2641547214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:39.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:39 smithi023 ceph-mon[38254]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:39.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/2641547214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:41.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:41 smithi154 ceph-mon[79161]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:41 smithi193 ceph-mon[79581]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:41 smithi023 ceph-mon[38254]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:43.603 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:43 smithi154 ceph-mon[79161]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:43 smithi193 ceph-mon[79581]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:43 smithi023 ceph-mon[38254]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T03:55:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T03:55:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:44.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-30T03:55:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:45.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:45 smithi154 ceph-mon[79161]: Deploying daemon osd.1 on smithi154 2024-03-30T03:55:45.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:45 smithi154 ceph-mon[79161]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:45 smithi193 ceph-mon[79581]: Deploying daemon osd.1 on smithi154 2024-03-30T03:55:45.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:45 smithi193 ceph-mon[79581]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:45.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:45 smithi023 ceph-mon[38254]: Deploying daemon osd.1 on smithi154 2024-03-30T03:55:45.793 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:45 smithi023 ceph-mon[38254]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:46.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:46 smithi154 ceph-mon[79161]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:47.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:46 smithi193 ceph-mon[79581]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:47.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:46 smithi023 ceph-mon[38254]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:48.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.635 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:49.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:49.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:49.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:49.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.612+0000 7fd900ff9700 1 -- 172.21.15.154:0/2461343097 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd9040611f0 con 0x7fd8ec0a1090 2024-03-30T03:55:49.612 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 1 on host 'smithi154' 2024-03-30T03:55:49.614 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 msgr2=0x7fd8ec0a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:49.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 0x7fd8ec0a3550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8009a10 tx=0x7fd8f8009190 comp rx=0 tx=0).stop 2024-03-30T03:55:49.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 msgr2=0x7fd904101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:49.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400d950 tx=0x7fd8f400dc60 comp rx=0 tx=0).stop 2024-03-30T03:55:49.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 shutdown_connections 2024-03-30T03:55:49.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd904106ae0 0x7fd9040ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd904101880 0x7fd904101190 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd8ec0a1090 0x7fd8ec0a3550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 --2- 172.21.15.154:0/2461343097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9041041b0 0x7fd9040ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 >> 172.21.15.154:0/2461343097 conn(0x7fd9040fb2b0 msgr2=0x7fd9041022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 shutdown_connections 2024-03-30T03:55:49.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:55:49.614+0000 7fd909a89700 1 -- 172.21.15.154:0/2461343097 wait complete. 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:49.997 DEBUG:teuthology.orchestra.run.smithi154:osd.1> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.1.service 2024-03-30T03:55:49.999 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-03-30T03:55:49.999 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-30T03:55:50.031 INFO:journalctl@ceph.osd.1.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:55:50.224 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:55:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:50 smithi154 ceph-mon[79161]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:50.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:50.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:50.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:50.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:50 smithi193 ceph-mon[79581]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:55:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:50 smithi023 ceph-mon[38254]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011216 s, 935 MB/s 2024-03-30T03:55:51.543 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T03:55:51.682 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:55:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T03:55:51.324+0000 7f086dd8d640 -1 osd.1 0 log_to_monitors true 2024-03-30T03:55:51.964 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-03-30T03:55:52.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:51 smithi154 ceph-mon[79161]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:51 smithi154 ceph-mon[79161]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.186 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:55:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:51 smithi193 ceph-mon[79581]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:51 smithi193 ceph-mon[79581]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:51 smithi023 ceph-mon[38254]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:51 smithi023 ceph-mon[38254]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi154 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 17114M 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:52.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:52 smithi193 ceph-mon[79581]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:53.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T03:55:53.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T03:55:53.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:53.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:53.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi154 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 17114M 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:52 smithi154 ceph-mon[79161]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/398992349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec100ef0 msgr2=0x7fe6ec101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/398992349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec100ef0 0x7fe6ec101350 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009990 tx=0x7fe6d8009ca0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/398992349 shutdown_connections 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/398992349 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6ec101890 0x7fe6ec108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/398992349 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 0x7fe6ec100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.203 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/398992349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec100ef0 0x7fe6ec101350 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/398992349 >> 172.21.15.193:0/398992349 conn(0x7fe6ec0fb2b0 msgr2=0x7fe6ec0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:55:53.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/398992349 shutdown_connections 2024-03-30T03:55:53.204 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/398992349 wait complete. 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 Processor -- start 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- start start 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 0x7fe6ec196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6ec100ef0 0x7fe6ec197220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec19e810 con 0x7fe6ec100ef0 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec19e950 con 0x7fe6ec0ffcf0 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.205+0000 7fe6f2d1e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6ec19eb10 con 0x7fe6ec101890 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:53.206 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:55468/0 (socket says 172.21.15.193:55468) 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 -- 172.21.15.193:0/3134870011 learned_addr learned my addr 172.21.15.193:0/3134870011 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f0aba700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 0x7fe6ec196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 -- 172.21.15.193:0/3134870011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 msgr2=0x7fe6ec196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 0x7fe6ec196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 -- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6ec100ef0 msgr2=0x7fe6ec197220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6ec100ef0 0x7fe6ec197220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 -- 172.21.15.193:0/3134870011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80095f0 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f12bb700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400ada0 tx=0x7fe6e400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e400f070 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6e4007b30 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6ec102230 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e40053e0 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.206+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6ec102730 con 0x7fe6ec101890 2024-03-30T03:55:53.207 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.207+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6ec04ea80 con 0x7fe6ec101890 2024-03-30T03:55:53.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.208+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7fe6e400c980 con 0x7fe6ec101890 2024-03-30T03:55:53.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.209+0000 7fe6e9ffb700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 0x7fe6dc0a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:55:53.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.209+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2029+0+0 (secure 0 0 0) 0x7fe6e40ca4e0 con 0x7fe6ec101890 2024-03-30T03:55:53.208 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.209+0000 7fe6f0aba700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 0x7fe6dc0a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:55:53.209 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.210+0000 7fe6f0aba700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 0x7fe6dc0a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0fdaa0 tx=0x7fe6e000a380 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:55:53.212 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.214+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe6e4095670 con 0x7fe6ec101890 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: osdmap e11: 2 total, 1 up, 2 in 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi154 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:55:53.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 17114M 2024-03-30T03:55:53.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:53.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:55:53.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:55:53.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:52 smithi023 ceph-mon[38254]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:53.412 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:55:53.414+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6ec068e90 con 0x7fe6dc0a1110 2024-03-30T03:55:53.876 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T03:55:53.529+0000 7f08633f1700 -1 osd.1 0 waiting for initial osdmap 2024-03-30T03:55:53.877 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T03:55:53.533+0000 7f085b1de700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:55:54.049 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:55:54.049 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T03:55:54.049 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.049 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.049 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:54.050 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:54.050 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:54.050 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:53 smithi193 ceph-mon[79581]: from='osd.1 ' entity='osd.1' 2024-03-30T03:55:54.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:55:54.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T03:55:54.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:53 smithi154 ceph-mon[79161]: from='osd.1 ' entity='osd.1' 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: osdmap e12: 2 total, 1 up, 2 in 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:55:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:53 smithi023 ceph-mon[38254]: from='osd.1 ' entity='osd.1' 2024-03-30T03:55:55.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] boot 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T03:55:55.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:54 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] boot 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T03:55:55.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:54 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: osd.1 [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] boot 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: osdmap e13: 2 total, 2 up, 2 in 2024-03-30T03:55:55.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:54 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T03:55:56.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/316432841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]': finished 2024-03-30T03:55:56.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T03:55:56.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:56.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:55 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/105722724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/316432841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]': finished 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:56.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/105722724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/316432841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]: dispatch 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e2244e3c-4dd3-4c06-a796-07187e9a77a0"}]': finished 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: osdmap e14: 3 total, 2 up, 3 in 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:56.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/105722724' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:55:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:57 smithi193 ceph-mon[79581]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T03:55:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:57 smithi193 ceph-mon[79581]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:55:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:57 smithi023 ceph-mon[38254]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T03:55:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:57 smithi023 ceph-mon[38254]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:55:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:57 smithi154 ceph-mon[79161]: osdmap e15: 3 total, 2 up, 3 in 2024-03-30T03:55:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:55:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:57 smithi154 ceph-mon[79161]: pgmap v55: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:55:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:55:59 smithi193 ceph-mon[79581]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:55:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:55:59 smithi154 ceph-mon[79161]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:55:59.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:55:59 smithi023 ceph-mon[38254]: pgmap v56: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:01.481 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:01 smithi193 ceph-mon[79581]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:01.481 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T03:56:01.481 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:01 smithi154 ceph-mon[79161]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T03:56:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:01 smithi023 ceph-mon[38254]: pgmap v57: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-30T03:56:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:02.634 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:02 smithi193 ceph-mon[79581]: Deploying daemon osd.2 on smithi193 2024-03-30T03:56:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:02 smithi154 ceph-mon[79161]: Deploying daemon osd.2 on smithi193 2024-03-30T03:56:02.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:02 smithi023 ceph-mon[38254]: Deploying daemon osd.2 on smithi193 2024-03-30T03:56:03.616 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:03 smithi193 ceph-mon[79581]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:03 smithi154 ceph-mon[79161]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:03.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:03 smithi023 ceph-mon[38254]: pgmap v58: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:05.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:05 smithi193 ceph-mon[79581]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:05.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:05 smithi154 ceph-mon[79161]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:05.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:05 smithi023 ceph-mon[38254]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:05.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:05.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:06.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.069 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.070+0000 7fe6e9ffb700 1 -- 172.21.15.193:0/3134870011 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6ec068e90 con 0x7fe6dc0a1110 2024-03-30T03:56:07.070 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 msgr2=0x7fe6dc0a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 0x7fe6dc0a35d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec0fdaa0 tx=0x7fe6e000a380 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 msgr2=0x7fe6ec19e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400ada0 tx=0x7fe6e400c330 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 shutdown_connections 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6dc0a1110 0x7fe6dc0a35d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6ec101890 0x7fe6ec19e240 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6ec0ffcf0 0x7fe6ec196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 --2- 172.21.15.193:0/3134870011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6ec100ef0 0x7fe6ec197220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:07.072 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 >> 172.21.15.193:0/3134870011 conn(0x7fe6ec0fb2b0 msgr2=0x7fe6ec068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:07.073 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.072+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 shutdown_connections 2024-03-30T03:56:07.073 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:07.073+0000 7fe6f2d1e700 1 -- 172.21.15.193:0/3134870011 wait complete. 2024-03-30T03:56:07.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:07.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:07.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:07.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:07.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:07.449 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.2.service 2024-03-30T03:56:07.451 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-30T03:56:07.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T03:56:07.485 INFO:journalctl@ceph.osd.2.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:56:07.822 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:56:08.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:08.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.298 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:09.224 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T03:56:08.917+0000 7f0d1a62e640 -1 osd.2 0 log_to_monitors true 2024-03-30T03:56:09.224 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:09 smithi193 ceph-mon[79581]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:09.224 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:09 smithi193 ceph-mon[79581]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.224 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:09 smithi193 ceph-mon[79581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:09 smithi154 ceph-mon[79161]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:09 smithi154 ceph-mon[79161]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:09 smithi154 ceph-mon[79161]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:09 smithi023 ceph-mon[38254]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:09 smithi023 ceph-mon[38254]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:09 smithi023 ceph-mon[38254]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0298127 s, 352 MB/s 2024-03-30T03:56:09.530 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-30T03:56:09.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:10.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:10.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:10 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.289 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:56:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T03:56:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T03:56:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:10.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:10 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T03:56:10.076+0000 7f0d0fc92700 -1 osd.2 0 waiting for initial osdmap 2024-03-30T03:56:10.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T03:56:10.081+0000 7f0d0ba87700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:56:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-30T03:56:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: osdmap e16: 3 total, 2 up, 3 in 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:10.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:10 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 -- 172.21.15.23:0/592093170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0073810 msgr2=0x7fadb0073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 --2- 172.21.15.23:0/592093170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0073810 0x7fadb0073c90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fad9c009990 tx=0x7fad9c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 -- 172.21.15.23:0/592093170 shutdown_connections 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 --2- 172.21.15.23:0/592093170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0108cd0 0x7fadb010b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 --2- 172.21.15.23:0/592093170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0074de0 0x7fadb0073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 --2- 172.21.15.23:0/592093170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0073810 0x7fadb0073c90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 -- 172.21.15.23:0/592093170 >> 172.21.15.23:0/592093170 conn(0x7fadb00fc3c0 msgr2=0x7fadb00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:11.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.389+0000 7fadb6c00700 1 -- 172.21.15.23:0/592093170 shutdown_connections 2024-03-30T03:56:11.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.390+0000 7fadb6c00700 1 -- 172.21.15.23:0/592093170 wait complete. 2024-03-30T03:56:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 Processor -- start 2024-03-30T03:56:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 -- start start 2024-03-30T03:56:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0073810 0x7fadb0198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0074de0 0x7fadb01994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:11.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb01a0a90 con 0x7fadb0108cd0 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb01a0bd0 con 0x7fadb0074de0 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb6c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb01a0d60 con 0x7fadb0073810 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.391+0000 7fadb4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadafe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0073810 0x7fadb0198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59944/0 (socket says 172.21.15.23:59944) 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 -- 172.21.15.23:0/1426915691 learned_addr learned my addr 172.21.15.23:0/1426915691 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadaf400700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0074de0 0x7fadb01994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 -- 172.21.15.23:0/1426915691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0073810 msgr2=0x7fadb0198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:11.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0073810 0x7fadb0198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 -- 172.21.15.23:0/1426915691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0074de0 msgr2=0x7fadb01994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0074de0 0x7fadb01994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 -- 172.21.15.23:0/1426915691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c0095f0 con 0x7fadb0108cd0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadb4c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fada800ef60 tx=0x7fada800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8018070 con 0x7fadb0108cd0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.392+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fada800f040 con 0x7fadb0108cd0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.393+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb01a0fe0 con 0x7fadb0108cd0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.393+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada800ccc0 con 0x7fadb0108cd0 2024-03-30T03:56:11.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.393+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb01a1400 con 0x7fadb0108cd0 2024-03-30T03:56:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.394+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb0107a90 con 0x7fadb0108cd0 2024-03-30T03:56:11.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.396+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 135809+0+0 (secure 0 0 0) 0x7fada8003a00 con 0x7fadb0108cd0 2024-03-30T03:56:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.397+0000 7fadacc00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 0x7fad940a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.397+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fada80cadd0 con 0x7fadb0108cd0 2024-03-30T03:56:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.397+0000 7fadafe00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 0x7fad940a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:11.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.398+0000 7fadafe00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 0x7fad940a3550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fadb00751c0 tx=0x7fada0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:11.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.402+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fada8095b80 con 0x7fadb0108cd0 2024-03-30T03:56:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi193 2024-03-30T03:56:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 17114M 2024-03-30T03:56:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:56:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T03:56:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:11 smithi154 ceph-mon[79161]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi193 2024-03-30T03:56:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 17114M 2024-03-30T03:56:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:56:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T03:56:11.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:11 smithi193 ceph-mon[79581]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi193 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 17114M 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: osdmap e17: 3 total, 2 up, 3 in 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:11 smithi023 ceph-mon[38254]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-30T03:56:11.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:11.625+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fadb002cda0 con 0x7fad940a1090 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] boot 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.337 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:12.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:12.338 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:56:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:56:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] boot 2024-03-30T03:56:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T03:56:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: osd.2 [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] boot 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: osdmap e18: 3 total, 3 up, 3 in 2024-03-30T03:56:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T03:56:12.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:12.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:12.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:13 smithi023 ceph-mon[38254]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:13 smithi023 ceph-mon[38254]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T03:56:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:13 smithi023 ceph-mon[38254]: pgmap v67: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:13 smithi154 ceph-mon[79161]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:13 smithi154 ceph-mon[79161]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T03:56:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:13 smithi154 ceph-mon[79161]: pgmap v67: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:13.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:13.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:13 smithi193 ceph-mon[79581]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:13.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:13 smithi193 ceph-mon[79581]: osdmap e19: 3 total, 3 up, 3 in 2024-03-30T03:56:13.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:13 smithi193 ceph-mon[79581]: pgmap v67: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:13.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2311008929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]': finished 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T03:56:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2311008929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]': finished 2024-03-30T03:56:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T03:56:14.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: osdmap e20: 3 total, 3 up, 3 in 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2311008929' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]: dispatch 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f3b063ec-ce2c-4142-a8c8-5bca5f8252bf"}]': finished 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: osdmap e21: 4 total, 3 up, 4 in 2024-03-30T03:56:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:15.022 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:14 smithi023 sudo[62856]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:15.022 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:14 smithi023 sudo[62856]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:15.022 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:14 smithi023 sudo[62856]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:15.022 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:14 smithi023 sudo[62856]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:15.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:15.122+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fada8003cb0 con 0x7fadb0108cd0 2024-03-30T03:56:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:15 smithi154 ceph-mon[79161]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:15 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/89132508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:15 smithi154 ceph-mon[79161]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T03:56:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:15 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:15 smithi193 ceph-mon[79581]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:15 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/89132508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:15 smithi193 ceph-mon[79581]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T03:56:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:15 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:15 smithi023 ceph-mon[38254]: pgmap v70: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:15 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/89132508' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:15 smithi023 ceph-mon[38254]: osdmap e22: 4 total, 3 up, 4 in 2024-03-30T03:56:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:15 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:15.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:15 smithi023 sudo[62905]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:15.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:15 smithi023 sudo[62905]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:15.522 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:15 smithi023 sudo[62905]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:15.523 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 03:56:15 smithi023 sudo[62905]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:15.931 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89605]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:15.931 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89605]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:15.931 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89605]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:15.931 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89605]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:15.932 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89608]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:15.932 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89608]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:15.932 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89608]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:15.932 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 03:56:15 smithi154 sudo[89608]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:16 smithi154 ceph-mon[79161]: mgrmap e16: x(active, since 118s), standbys: z, y 2024-03-30T03:56:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:16 smithi193 ceph-mon[79581]: mgrmap e16: x(active, since 118s), standbys: z, y 2024-03-30T03:56:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:16.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90150]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:16.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:16.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90150]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:16.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90150]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:16.493 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90153]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:16.494 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90153]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:16.494 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90153]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:16.494 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 03:56:16 smithi193 sudo[90153]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 ceph-mon[38254]: mgrmap e16: x(active, since 118s), standbys: z, y 2024-03-30T03:56:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 sudo[63660]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 sudo[63660]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 sudo[63660]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:16 smithi023 sudo[63660]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:17 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90156]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:17.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90156]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 sudo[64042]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 sudo[64042]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 sudo[64042]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 sudo[64042]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:17.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:17 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90159]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:17 smithi193 sudo[90159]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:18.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89611]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-30T03:56:18.416 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89611]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89611]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89611]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89614]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89614]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89614]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-30T03:56:18.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 sudo[89614]: pam_unix(sudo:session): session closed for user root 2024-03-30T03:56:18.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: pgmap v73: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:18 smithi154 ceph-mon[79161]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: pgmap v73: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:18 smithi193 ceph-mon[79581]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: pgmap v72: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:18.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-30T03:56:18.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T03:56:18.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T03:56:18.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T03:56:18.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: pgmap v73: 1 pgs: 1 active+clean; 449 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:18.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:18 smithi023 ceph-mon[38254]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-30T03:56:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:21 smithi023 ceph-mon[38254]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 351 KiB/s wr, 24 op/s 2024-03-30T03:56:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T03:56:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:21 smithi154 ceph-mon[79161]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 351 KiB/s wr, 24 op/s 2024-03-30T03:56:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T03:56:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:21 smithi193 ceph-mon[79581]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 7.7 KiB/s rd, 351 KiB/s wr, 24 op/s 2024-03-30T03:56:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-30T03:56:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:22.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:22 smithi023 ceph-mon[38254]: Deploying daemon osd.3 on smithi023 2024-03-30T03:56:22.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:22 smithi154 ceph-mon[79161]: Deploying daemon osd.3 on smithi023 2024-03-30T03:56:22.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:22 smithi193 ceph-mon[79581]: Deploying daemon osd.3 on smithi023 2024-03-30T03:56:23.366 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:23 smithi023 ceph-mon[38254]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 253 KiB/s wr, 17 op/s 2024-03-30T03:56:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:23 smithi154 ceph-mon[79161]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 253 KiB/s wr, 17 op/s 2024-03-30T03:56:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:23 smithi193 ceph-mon[79581]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 253 KiB/s wr, 17 op/s 2024-03-30T03:56:25.555 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:25 smithi023 ceph-mon[38254]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-03-30T03:56:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:25 smithi154 ceph-mon[79161]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-03-30T03:56:25.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:25 smithi193 ceph-mon[79581]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 4.3 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-03-30T03:56:26.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:26.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:26 smithi023 ceph-mon[38254]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-30T03:56:26.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:26.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:26 smithi154 ceph-mon[79161]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-30T03:56:26.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:26.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:26.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:26 smithi193 ceph-mon[79581]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-03-30T03:56:28.426 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-30T03:56:28.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.426+0000 7fadacc00700 1 -- 172.21.15.23:0/1426915691 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fadb002cda0 con 0x7fad940a1090 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 msgr2=0x7fad940a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 0x7fad940a3550 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fadb00751c0 tx=0x7fada0009260 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 msgr2=0x7fadb01a04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fada800ef60 tx=0x7fada800ef90 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 shutdown_connections 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fadb0108cd0 0x7fadb01a04c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fad940a1090 0x7fad940a3550 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fadb0073810 0x7fadb0198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 --2- 172.21.15.23:0/1426915691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadb0074de0 0x7fadb01994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:28.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 >> 172.21.15.23:0/1426915691 conn(0x7fadb00fc3c0 msgr2=0x7fadb0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:28.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 shutdown_connections 2024-03-30T03:56:28.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:56:28.429+0000 7fadb6c00700 1 -- 172.21.15.23:0/1426915691 wait complete. 2024-03-30T03:56:28.765 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.3.service 2024-03-30T03:56:28.770 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_3... 2024-03-30T03:56:28.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T03:56:28.845 INFO:journalctl@ceph.osd.3.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:56:28.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:28.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:28.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-30T03:56:28.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:28.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:28.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:28.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-30T03:56:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:28.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:29.003 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-03-30T03:56:29.010 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:29.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:29.011 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:29.685 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 03:56:29 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T03:56:29.648+0000 7fcddfbd2640 -1 osd.3 0 log_to_monitors true 2024-03-30T03:56:29.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:29 smithi154 ceph-mon[79161]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:29.792 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:29 smithi154 ceph-mon[79161]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:29.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:29 smithi193 ceph-mon[79581]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:29.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:29 smithi193 ceph-mon[79581]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:30.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:29 smithi023 ceph-mon[38254]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:30.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:29 smithi023 ceph-mon[38254]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111546 s, 940 MB/s 2024-03-30T03:56:30.314 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-30T03:56:30.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-03-30T03:56:30.906 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:56:30.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T03:56:30.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T03:56:30.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:30.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:30.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:30 smithi193 ceph-mon[79581]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T03:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T03:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:31.017 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:31.018 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:30 smithi154 ceph-mon[79161]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:31.022 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T03:56:30.709+0000 7fcdd4200700 -1 osd.3 0 waiting for initial osdmap 2024-03-30T03:56:31.022 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T03:56:30.735+0000 7fcdcf000700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: osdmap e23: 4 total, 3 up, 4 in 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: from='osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:56:31.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:30 smithi023 ceph-mon[38254]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-30T03:56:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 -- 172.21.15.154:0/2706972396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc0ffd10 msgr2=0x7fe6bc100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 --2- 172.21.15.154:0/2706972396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc100130 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe6b00099c0 tx=0x7fe6b0009cd0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 -- 172.21.15.154:0/2706972396 shutdown_connections 2024-03-30T03:56:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 --2- 172.21.15.154:0/2706972396 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.849 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 --2- 172.21.15.154:0/2706972396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc100130 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 --2- 172.21.15.154:0/2706972396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 0x7fe6bc101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.847+0000 7fe6c4497700 1 -- 172.21.15.154:0/2706972396 >> 172.21.15.154:0/2706972396 conn(0x7fe6bc0fb2d0 msgr2=0x7fe6bc0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- 172.21.15.154:0/2706972396 shutdown_connections 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- 172.21.15.154:0/2706972396 wait complete. 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 Processor -- start 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- start start 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 0x7fe6bc1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:31.850 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc19e8d0 con 0x7fe6bc0ffd10 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc19ea40 con 0x7fe6bc1018b0 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c4497700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc19ec00 con 0x7fe6bc100f10 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c2a34700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c2a34700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:35048/0 (socket says 172.21.15.154:35048) 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c2a34700 1 -- 172.21.15.154:0/4072891409 learned_addr learned my addr 172.21.15.154:0/4072891409 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c1a32700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 0x7fe6bc1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.848+0000 7fe6c2233700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 -- 172.21.15.154:0/4072891409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 msgr2=0x7fe6bc1973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 0x7fe6bc1973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 -- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 msgr2=0x7fe6bc196e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc196e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:31.851 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 -- 172.21.15.154:0/4072891409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b00095f0 con 0x7fe6bc1018b0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2233700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c2a34700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400b970 tx=0x7fe6b400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b400c930 con 0x7fe6bc1018b0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6bc19eee0 con 0x7fe6bc1018b0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6b4014440 con 0x7fe6bc1018b0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40075d0 con 0x7fe6bc1018b0 2024-03-30T03:56:31.852 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.849+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6bc19f350 con 0x7fe6bc1018b0 2024-03-30T03:56:31.853 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.850+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6bc04ea80 con 0x7fe6bc1018b0 2024-03-30T03:56:31.853 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.851+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe6b40145b0 con 0x7fe6bc1018b0 2024-03-30T03:56:31.853 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.852+0000 7fe6af7fe700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 0x7fe6a80a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.852+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7fe6b4006e40 con 0x7fe6bc1018b0 2024-03-30T03:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.852+0000 7fe6c2233700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 0x7fe6a80a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:31.854 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.852+0000 7fe6c2233700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 0x7fe6a80a3610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc197d70 tx=0x7fe6b0005e80 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:31.855 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:31.855+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe6b4095f00 con 0x7fe6bc1018b0 2024-03-30T03:56:31.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:56:31.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T03:56:31.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] boot 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:31.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] boot 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T03:56:31.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:31.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: osdmap e24: 4 total, 3 up, 4 in 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: osd.3 [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] boot 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: osdmap e25: 4 total, 4 up, 4 in 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:32.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:32.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:32.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:32.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:32.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:32.057 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:32.058+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6bc069af0 con 0x7fe6a80a1150 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi023 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 8635M 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:33.107 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:32 smithi154 ceph-mon[79161]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi023 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 8635M 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T03:56:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:33.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:33.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:33.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:33.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:32 smithi193 ceph-mon[79581]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi023 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 8635M 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: osdmap e26: 4 total, 4 up, 4 in 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:33.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:33.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:32 smithi023 ceph-mon[38254]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:34.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:33 smithi154 ceph-mon[79161]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T03:56:34.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:33 smithi193 ceph-mon[79581]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T03:56:34.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:33 smithi023 ceph-mon[38254]: osdmap e27: 4 total, 4 up, 4 in 2024-03-30T03:56:35.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/3809682964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T03:56:35.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]': finished 2024-03-30T03:56:35.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T03:56:35.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:35.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:35.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:34 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/997655923' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/3809682964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]': finished 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:35.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:34 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/997655923' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/3809682964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: osdmap e28: 4 total, 4 up, 4 in 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]: dispatch 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c90bc547-0492-4582-be42-0b7cdd2f2fb1"}]': finished 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: osdmap e29: 5 total, 4 up, 5 in 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:35.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:34 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/997655923' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:36 smithi154 ceph-mon[79161]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-30T03:56:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:36 smithi193 ceph-mon[79581]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-30T03:56:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:36 smithi023 ceph-mon[38254]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-30T03:56:39.581 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:39 smithi154 ceph-mon[79161]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:56:39.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:39 smithi193 ceph-mon[79581]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:56:39.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:39 smithi023 ceph-mon[38254]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:56:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T03:56:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:40.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T03:56:40.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:40.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-30T03:56:40.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:41.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:41 smithi154 ceph-mon[79161]: Deploying daemon osd.4 on smithi154 2024-03-30T03:56:41.451 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:41 smithi154 ceph-mon[79161]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-30T03:56:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:41 smithi193 ceph-mon[79581]: Deploying daemon osd.4 on smithi154 2024-03-30T03:56:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:41 smithi193 ceph-mon[79581]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-30T03:56:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:41 smithi023 ceph-mon[38254]: Deploying daemon osd.4 on smithi154 2024-03-30T03:56:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:41 smithi023 ceph-mon[38254]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-30T03:56:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:43 smithi154 ceph-mon[79161]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:56:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:43 smithi193 ceph-mon[79581]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:56:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:43 smithi023 ceph-mon[38254]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:56:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:44 smithi154 ceph-mon[79161]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-30T03:56:44.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:44.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:44 smithi193 ceph-mon[79581]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-30T03:56:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:44 smithi023 ceph-mon[38254]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-03-30T03:56:45.613 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2024-03-30T03:56:45.613 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.613+0000 7fe6af7fe700 1 -- 172.21.15.154:0/4072891409 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6bc069af0 con 0x7fe6a80a1150 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 msgr2=0x7fe6a80a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 0x7fe6a80a3610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc197d70 tx=0x7fe6b0005e80 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 msgr2=0x7fe6bc19e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe6b400b970 tx=0x7fe6b400bc80 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 shutdown_connections 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6bc1018b0 0x7fe6bc19e300 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6bc0ffd10 0x7fe6bc196e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6a80a1150 0x7fe6a80a3610 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 --2- 172.21.15.154:0/4072891409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6bc100f10 0x7fe6bc1973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:45.615 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 >> 172.21.15.154:0/4072891409 conn(0x7fe6bc0fb2d0 msgr2=0x7fe6bc0693e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:45.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 shutdown_connections 2024-03-30T03:56:45.616 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:56:45.616+0000 7fe6c4497700 1 -- 172.21.15.154:0/4072891409 wait complete. 2024-03-30T03:56:45.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:45 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:45.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:45 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:45.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:45 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:45.796 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:45 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:45.797 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:45 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:45.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:45 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:45.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:45 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:45.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:45 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:45.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:45 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:45.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:45 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.000 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.4.service 2024-03-30T03:56:46.002 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-03-30T03:56:46.002 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-30T03:56:46.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:45 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:45 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:45 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:46.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:45 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:46.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:45 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.034 INFO:journalctl@ceph.osd.4.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:56:46.234 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:56:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:46 smithi193 ceph-mon[79581]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:56:46.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.798 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:46.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:46.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:46 smithi154 ceph-mon[79161]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:56:47.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:47.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:47.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:56:47.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:46 smithi023 ceph-mon[38254]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111629 s, 939 MB/s 2024-03-30T03:56:47.653 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T03:56:47.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 03:56:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T03:56:47.337+0000 7fa37e3ed640 -1 osd.4 0 log_to_monitors true 2024-03-30T03:56:47.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:47 smithi154 ceph-mon[79161]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:47.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:47 smithi154 ceph-mon[79161]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:47.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:47 smithi193 ceph-mon[79581]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:47.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:47 smithi193 ceph-mon[79581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:47.996 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-03-30T03:56:48.213 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:56:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:47 smithi023 ceph-mon[38254]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:48.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:47 smithi023 ceph-mon[38254]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-30T03:56:48.990 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi154 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 8557M 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:48.991 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:48.992 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:48.992 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:48 smithi193 ceph-mon[79581]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.160+0000 7fed1cb94700 1 -- 172.21.15.193:0/845149809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed180ffcf0 msgr2=0x7fed18100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.160+0000 7fed1cb94700 1 --2- 172.21.15.193:0/845149809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed180ffcf0 0x7fed18100110 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fed08009990 tx=0x7fed08009ca0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 -- 172.21.15.193:0/845149809 shutdown_connections 2024-03-30T03:56:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 --2- 172.21.15.193:0/845149809 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed18101890 0x7fed18108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 --2- 172.21.15.193:0/845149809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed180ffcf0 0x7fed18100110 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 --2- 172.21.15.193:0/845149809 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18100ef0 0x7fed18101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 -- 172.21.15.193:0/845149809 >> 172.21.15.193:0/845149809 conn(0x7fed180fb2b0 msgr2=0x7fed180fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:56:49.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 -- 172.21.15.193:0/845149809 shutdown_connections 2024-03-30T03:56:49.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 -- 172.21.15.193:0/845149809 wait complete. 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 Processor -- start 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.161+0000 7fed1cb94700 1 -- start start 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed180ffcf0 0x7fed18196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18101890 0x7fed1819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819e8e0 con 0x7fed18101890 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819ea20 con 0x7fed180ffcf0 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819ebe0 con 0x7fed18100ef0 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:49.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed180ffcf0 0x7fed18196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:43510/0 (socket says 172.21.15.193:43510) 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 -- 172.21.15.193:0/30877347 learned_addr learned my addr 172.21.15.193:0/30877347 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed16d9d700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18101890 0x7fed1819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 -- 172.21.15.193:0/30877347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed180ffcf0 msgr2=0x7fed18196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed180ffcf0 0x7fed18196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 -- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18101890 msgr2=0x7fed1819e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18101890 0x7fed1819e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 -- 172.21.15.193:0/30877347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed080095f0 con 0x7fed18100ef0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed15d9b700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b530 tx=0x7fed0c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c011830 con 0x7fed18100ef0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1819eec0 con 0x7fed18100ef0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed0c011e50 con 0x7fed18100ef0 2024-03-30T03:56:49.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00f2e0 con 0x7fed18100ef0 2024-03-30T03:56:49.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.162+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1819f2d0 con 0x7fed18100ef0 2024-03-30T03:56:49.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.163+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1804eaf0 con 0x7fed18100ef0 2024-03-30T03:56:49.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.164+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fed0c010380 con 0x7fed18100ef0 2024-03-30T03:56:49.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.165+0000 7fed077fe700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 0x7fed000a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:56:49.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.165+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3821+0+0 (secure 0 0 0) 0x7fed0c0cb2d0 con 0x7fed18100ef0 2024-03-30T03:56:49.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.165+0000 7fed1659c700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 0x7fed000a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:56:49.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.166+0000 7fed1659c700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 0x7fed000a35e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed18198270 tx=0x7fed08005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:56:49.169 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.170+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed1804eaf0 con 0x7fed18100ef0 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi154 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 8557M 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:48 smithi154 ceph-mon[79161]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: osdmap e30: 5 total, 4 up, 5 in 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi154 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:49.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:56:49.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 8557M 2024-03-30T03:56:49.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:49.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:56:49.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:56:49.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:48 smithi023 ceph-mon[38254]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:49.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:56:49.374+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fed1804ea80 con 0x7fed000a1120 2024-03-30T03:56:49.886 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T03:56:49.596+0000 7fa373a51700 -1 osd.4 0 waiting for initial osdmap 2024-03-30T03:56:49.887 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T03:56:49.602+0000 7fa36b83e700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:49.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:49 smithi193 ceph-mon[79581]: from='osd.4 ' entity='osd.4' 2024-03-30T03:56:50.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:56:50.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:50.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:49 smithi154 ceph-mon[79161]: from='osd.4 ' entity='osd.4' 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: osdmap e31: 5 total, 4 up, 5 in 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:49 smithi023 ceph-mon[38254]: from='osd.4 ' entity='osd.4' 2024-03-30T03:56:51.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:56:51.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:56:51.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:51.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:51.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:51.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] boot 2024-03-30T03:56:51.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T03:56:51.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:51.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] boot 2024-03-30T03:56:51.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T03:56:51.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:51.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:56:51.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: osd.4 [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] boot 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: osdmap e32: 5 total, 5 up, 5 in 2024-03-30T03:56:51.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T03:56:52.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/4213804053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]': finished 2024-03-30T03:56:52.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T03:56:52.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:52.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/990907747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/4213804053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]': finished 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:52.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/990907747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:52.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/4213804053' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]: dispatch 2024-03-30T03:56:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "93e66232-688d-4fc5-a935-1c4e147cd6d9"}]': finished 2024-03-30T03:56:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: osdmap e33: 6 total, 5 up, 6 in 2024-03-30T03:56:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:52.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/990907747' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:56:53.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:53 smithi154 ceph-mon[79161]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T03:56:53.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:53 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:53.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:53 smithi154 ceph-mon[79161]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:53.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:53 smithi193 ceph-mon[79581]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T03:56:53.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:53 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:53.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:53 smithi193 ceph-mon[79581]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:53.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:53 smithi023 ceph-mon[38254]: osdmap e34: 6 total, 5 up, 6 in 2024-03-30T03:56:53.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:53 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:56:53.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:53 smithi023 ceph-mon[38254]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:55.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:55 smithi193 ceph-mon[79581]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:55 smithi154 ceph-mon[79161]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:55 smithi023 ceph-mon[38254]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:56.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:56 smithi193 ceph-mon[79581]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:56 smithi154 ceph-mon[79161]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:56 smithi023 ceph-mon[38254]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:58.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T03:56:58.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:58.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:57 smithi154 ceph-mon[79161]: Deploying daemon osd.5 on smithi193 2024-03-30T03:56:58.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T03:56:58.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:58.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:57 smithi193 ceph-mon[79581]: Deploying daemon osd.5 on smithi193 2024-03-30T03:56:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-30T03:56:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:56:58.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:57 smithi023 ceph-mon[38254]: Deploying daemon osd.5 on smithi193 2024-03-30T03:56:58.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:56:58 smithi193 ceph-mon[79581]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:59.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:56:58 smithi154 ceph-mon[79161]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:56:59.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:56:58 smithi023 ceph-mon[38254]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:01 smithi193 ceph-mon[79581]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:01 smithi154 ceph-mon[79161]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:01 smithi023 ceph-mon[38254]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:03.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:03 smithi193 ceph-mon[79581]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:03.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.442 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.495 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-03-30T03:57:03.495 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.495+0000 7fed077fe700 1 -- 172.21.15.193:0/30877347 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fed1804ea80 con 0x7fed000a1120 2024-03-30T03:57:03.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 msgr2=0x7fed000a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:03.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 0x7fed000a35e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fed18198270 tx=0x7fed08005fd0 comp rx=0 tx=0).stop 2024-03-30T03:57:03.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 msgr2=0x7fed18197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:03.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00b530 tx=0x7fed0c00b840 comp rx=0 tx=0).stop 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 shutdown_connections 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed18101890 0x7fed1819e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed180ffcf0 0x7fed18196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed000a1120 0x7fed000a35e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 --2- 172.21.15.193:0/30877347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed18100ef0 0x7fed18197320 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 >> 172.21.15.193:0/30877347 conn(0x7fed180fb2b0 msgr2=0x7fed18068780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 shutdown_connections 2024-03-30T03:57:03.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:03.498+0000 7fed1cb94700 1 -- 172.21.15.193:0/30877347 wait complete. 2024-03-30T03:57:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:03 smithi154 ceph-mon[79161]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:03 smithi023 ceph-mon[38254]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:03.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:03.915 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.5.service 2024-03-30T03:57:03.917 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-30T03:57:03.917 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T03:57:03.952 INFO:journalctl@ceph.osd.5.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:57:04.283 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:57:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:04.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:04.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:04.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:05.377 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:05 smithi023 ceph-mon[38254]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:05.377 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:05 smithi023 ceph-mon[38254]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:05.377 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:05 smithi023 ceph-mon[38254]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:05 smithi193 ceph-mon[79581]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:05 smithi193 ceph-mon[79581]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:05 smithi193 ceph-mon[79581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:05.493 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 03:57:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T03:57:05.118+0000 7ffa7f566640 -1 osd.5 0 log_to_monitors true 2024-03-30T03:57:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:05 smithi154 ceph-mon[79161]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:05 smithi154 ceph-mon[79161]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:05 smithi154 ceph-mon[79161]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0355575 s, 295 MB/s 2024-03-30T03:57:06.171 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-30T03:57:06.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi193 2024-03-30T03:57:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 8557M 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:06 smithi193 ceph-mon[79581]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi193 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 8557M 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:06 smithi023 ceph-mon[38254]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:06.916 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:57:06.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-30T03:57:06.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: osdmap e35: 6 total, 5 up, 6 in 2024-03-30T03:57:06.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi193 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 8557M 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:06.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:06 smithi154 ceph-mon[79161]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:07.493 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T03:57:07.203+0000 7ffa74bca700 -1 osd.5 0 waiting for initial osdmap 2024-03-30T03:57:07.493 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T03:57:07.211+0000 7ffa6c9b7700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:57:07.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:07 smithi023 ceph-mon[38254]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:57:07.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:07 smithi023 ceph-mon[38254]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T03:57:07.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:07 smithi023 ceph-mon[38254]: from='osd.5 ' entity='osd.5' 2024-03-30T03:57:07.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:07 smithi154 ceph-mon[79161]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:57:07.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:07 smithi154 ceph-mon[79161]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T03:57:07.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:07 smithi154 ceph-mon[79161]: from='osd.5 ' entity='osd.5' 2024-03-30T03:57:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-mon[79581]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:57:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-mon[79581]: osdmap e36: 6 total, 5 up, 6 in 2024-03-30T03:57:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:07 smithi193 ceph-mon[79581]: from='osd.5 ' entity='osd.5' 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 -- 172.21.15.23:0/1244117048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0074de0 msgr2=0x7f23d0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 --2- 172.21.15.23:0/1244117048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0074de0 0x7f23d0073240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009990 tx=0x7f23c0009ca0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 -- 172.21.15.23:0/1244117048 shutdown_connections 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 --2- 172.21.15.23:0/1244117048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0108cd0 0x7f23d010b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 --2- 172.21.15.23:0/1244117048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0074de0 0x7f23d0073240 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 --2- 172.21.15.23:0/1244117048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0073810 0x7f23d0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 -- 172.21.15.23:0/1244117048 >> 172.21.15.23:0/1244117048 conn(0x7f23d00fc3c0 msgr2=0x7f23d00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:08.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 -- 172.21.15.23:0/1244117048 shutdown_connections 2024-03-30T03:57:08.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.005+0000 7f23d6800700 1 -- 172.21.15.23:0/1244117048 wait complete. 2024-03-30T03:57:08.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 Processor -- start 2024-03-30T03:57:08.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 -- start start 2024-03-30T03:57:08.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:08.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 0x7f23d0199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:08.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0108cd0 0x7f23d01a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:08.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01a0ac0 con 0x7f23d0074de0 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01a0c00 con 0x7f23d0108cd0 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23d6800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d01a0d90 con 0x7f23d0073810 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.007+0000 7f23cf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:42578/0 (socket says 172.21.15.23:42578) 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 -- 172.21.15.23:0/4048947266 learned_addr learned my addr 172.21.15.23:0/4048947266 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cea00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 0x7f23d0199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cfe00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0108cd0 0x7f23d01a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 -- 172.21.15.23:0/4048947266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0108cd0 msgr2=0x7f23d01a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0108cd0 0x7f23d01a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 -- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 msgr2=0x7f23d0199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 0x7f23d0199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 -- 172.21.15.23:0/4048947266 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c00095f0 con 0x7f23d0073810 2024-03-30T03:57:08.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cea00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 0x7f23d0199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23cf400700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23c0005740 tx=0x7f23c0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.008+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c001d070 con 0x7f23d0073810 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.009+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23c0021410 con 0x7f23d0073810 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.009+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d01a1010 con 0x7f23d0073810 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.009+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c0017650 con 0x7f23d0073810 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.009+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d01a1420 con 0x7f23d0073810 2024-03-30T03:57:08.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.010+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d0107a90 con 0x7f23d0073810 2024-03-30T03:57:08.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.012+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f23c0021580 con 0x7f23d0073810 2024-03-30T03:57:08.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.013+0000 7f23c7e00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 0x7f23b40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:08.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.013+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4566+0+0 (secure 0 0 0) 0x7f23c00ccfe0 con 0x7f23d0073810 2024-03-30T03:57:08.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.013+0000 7f23cea00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 0x7f23b40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:08.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.014+0000 7f23cea00700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 0x7f23b40a35e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23d0074b10 tx=0x7f23bc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:08.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.015+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f23c0097620 con 0x7f23d0073810 2024-03-30T03:57:08.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:08.237+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f23d00611f0 con 0x7f23b40a1120 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] boot 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:08.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:08.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:08.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:57:08.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] boot 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:08.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: osd.5 [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] boot 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: osdmap e37: 6 total, 6 up, 6 in 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-30T03:57:08.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:08.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:08.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:08.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:09.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:09 smithi023 ceph-mon[38254]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T03:57:09.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:09 smithi154 ceph-mon[79161]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T03:57:09.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:09 smithi193 ceph-mon[79581]: osdmap e38: 6 total, 6 up, 6 in 2024-03-30T03:57:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:10 smithi154 ceph-mon[79161]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T03:57:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:10 smithi154 ceph-mon[79161]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:10 smithi193 ceph-mon[79581]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T03:57:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:10 smithi193 ceph-mon[79581]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:10 smithi023 ceph-mon[38254]: osdmap e39: 6 total, 6 up, 6 in 2024-03-30T03:57:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:10 smithi023 ceph-mon[38254]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/843184480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]': finished 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:11 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/207575122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:11.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T03:57:11.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/843184480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]': finished 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:11.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:11 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/207575122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:12.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-30T03:57:12.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: osdmap e40: 6 total, 6 up, 6 in 2024-03-30T03:57:12.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/843184480' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:12.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]: dispatch 2024-03-30T03:57:12.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c236d02-e504-41f1-be6d-acc36f957d54"}]': finished 2024-03-30T03:57:12.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: osdmap e41: 7 total, 6 up, 7 in 2024-03-30T03:57:12.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:12.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:11 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/207575122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:12.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:12 smithi154 ceph-mon[79161]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:12.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:12 smithi193 ceph-mon[79581]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:13.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:12 smithi023 ceph-mon[38254]: pgmap v119: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:15.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:15 smithi023 ceph-mon[38254]: pgmap v120: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:15 smithi154 ceph-mon[79161]: pgmap v120: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:15 smithi193 ceph-mon[79581]: pgmap v120: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-30T03:57:17.130 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:16 smithi023 ceph-mon[38254]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-30T03:57:17.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:16 smithi154 ceph-mon[79161]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-30T03:57:17.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:16 smithi193 ceph-mon[79581]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-30T03:57:18.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:17 smithi154 ceph-mon[79161]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T03:57:18.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:17 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T03:57:18.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T03:57:18.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:18.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:17 smithi154 ceph-mon[79161]: Deploying daemon osd.6 on smithi023 2024-03-30T03:57:18.224 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:17 smithi023 ceph-mon[38254]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T03:57:18.224 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:17 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T03:57:18.224 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T03:57:18.224 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:18.224 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:17 smithi023 ceph-mon[38254]: Deploying daemon osd.6 on smithi023 2024-03-30T03:57:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:17 smithi193 ceph-mon[79581]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-30T03:57:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:17 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T03:57:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-30T03:57:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:18.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:17 smithi193 ceph-mon[79581]: Deploying daemon osd.6 on smithi023 2024-03-30T03:57:19.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:18 smithi023 ceph-mon[38254]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:57:19.163 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:18 smithi193 ceph-mon[79581]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:57:19.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:18 smithi154 ceph-mon[79161]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:57:21.583 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:21 smithi023 ceph-mon[38254]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T03:57:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:21 smithi154 ceph-mon[79161]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T03:57:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:21 smithi193 ceph-mon[79581]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-30T03:57:23.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:23 smithi023 ceph-mon[38254]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-30T03:57:23.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:23 smithi154 ceph-mon[79161]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-30T03:57:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:23 smithi193 ceph-mon[79581]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-03-30T03:57:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:25.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.339+0000 7f23c7e00700 1 -- 172.21.15.23:0/4048947266 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f23d00611f0 con 0x7f23b40a1120 2024-03-30T03:57:25.339 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 msgr2=0x7f23b40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 0x7f23b40a35e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f23d0074b10 tx=0x7f23bc009260 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 msgr2=0x7f23d0198fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f23c0005740 tx=0x7f23c0009580 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 shutdown_connections 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23d0108cd0 0x7f23d01a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f23b40a1120 0x7f23b40a35e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f23d0073810 0x7f23d0198fd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 --2- 172.21.15.23:0/4048947266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23d0074de0 0x7f23d0199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 >> 172.21.15.23:0/4048947266 conn(0x7f23d00fc3c0 msgr2=0x7f23d0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 shutdown_connections 2024-03-30T03:57:25.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:57:25.342+0000 7f23d6800700 1 -- 172.21.15.23:0/4048947266 wait complete. 2024-03-30T03:57:25.363 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.363 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.364 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:25.364 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:25.364 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.364 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:25 smithi023 ceph-mon[38254]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:25 smithi154 ceph-mon[79161]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:25 smithi193 ceph-mon[79581]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:25.829 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.6.service 2024-03-30T03:57:25.833 INFO:tasks.cephadm:Deploying osd.7 on smithi154 with /dev/vg_nvme/lv_2... 2024-03-30T03:57:25.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T03:57:25.934 INFO:journalctl@ceph.osd.6.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:57:26.076 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:57:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:26 smithi154 ceph-mon[79161]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T03:57:26.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:26.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:26 smithi023 ceph-mon[38254]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T03:57:26.515 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 03:57:26 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T03:57:26.262+0000 7fb00d4d2640 -1 osd.6 0 log_to_monitors true 2024-03-30T03:57:26.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:26.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:26.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:26 smithi193 ceph-mon[79581]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-30T03:57:27.349 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T03:57:27.350 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T03:57:27.350 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-30T03:57:27.350 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-30T03:57:27.350 INFO:teuthology.orchestra.run.smithi154.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0159088 s, 659 MB/s 2024-03-30T03:57:27.350 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-30T03:57:27.650 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:27 smithi154 ceph-mon[79161]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:27.651 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:27 smithi154 ceph-mon[79161]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T03:57:27.651 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:27 smithi154 ceph-mon[79161]: osdmap e42: 7 total, 6 up, 7 in 2024-03-30T03:57:27.651 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:27.651 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:27 smithi154 ceph-mon[79161]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:57:27.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:27 smithi193 ceph-mon[79581]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:27.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:27 smithi193 ceph-mon[79581]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T03:57:27.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:27 smithi193 ceph-mon[79581]: osdmap e42: 7 total, 6 up, 7 in 2024-03-30T03:57:27.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:27.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:27 smithi193 ceph-mon[79581]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:57:27.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-03-30T03:57:27.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-mon[38254]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-30T03:57:27.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-mon[38254]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-30T03:57:27.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-mon[38254]: osdmap e42: 7 total, 6 up, 7 in 2024-03-30T03:57:27.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:27.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-mon[38254]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:57:27.969 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:57:28.272 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T03:57:27.901+0000 7fb001a00700 -1 osd.6 0 waiting for initial osdmap 2024-03-30T03:57:28.272 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 03:57:27 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T03:57:27.919+0000 7faffc800700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 -- 172.21.15.154:0/239475899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc106ae0 msgr2=0x7feabc108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 --2- 172.21.15.154:0/239475899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc106ae0 0x7feabc108f50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7feab400b0a0 tx=0x7feab400b3b0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 -- 172.21.15.154:0/239475899 shutdown_connections 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 --2- 172.21.15.154:0/239475899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc106ae0 0x7feabc108f50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 --2- 172.21.15.154:0/239475899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc101880 0x7feabc103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 --2- 172.21.15.154:0/239475899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1041b0 0x7feabc1065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 -- 172.21.15.154:0/239475899 >> 172.21.15.154:0/239475899 conn(0x7feabc0fb2b0 msgr2=0x7feabc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 -- 172.21.15.154:0/239475899 shutdown_connections 2024-03-30T03:57:28.958 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.959+0000 7feac195d700 1 -- 172.21.15.154:0/239475899 wait complete. 2024-03-30T03:57:28.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 Processor -- start 2024-03-30T03:57:28.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 -- start start 2024-03-30T03:57:28.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:28.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1041b0 0x7feabc197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:28.960 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc106ae0 0x7feabc19e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19e8f0 con 0x7feabc1041b0 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19ea30 con 0x7feabc101880 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feac195d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19ebf0 con 0x7feabc106ae0 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.960+0000 7feabaffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47900/0 (socket says 172.21.15.154:47900) 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 -- 172.21.15.154:0/3445118734 learned_addr learned my addr 172.21.15.154:0/3445118734 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabb7fe700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc106ae0 0x7feabc19e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 -- 172.21.15.154:0/3445118734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc106ae0 msgr2=0x7feabc19e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc106ae0 0x7feabc19e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 -- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1041b0 msgr2=0x7feabc197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1041b0 0x7feabc197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 -- 172.21.15.154:0/3445118734 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4009d00 con 0x7feabc101880 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feabaffd700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e9a0 tx=0x7feaac00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:28.961 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac009790 con 0x7feabc101880 2024-03-30T03:57:28.962 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc0ff810 con 0x7feabc101880 2024-03-30T03:57:28.962 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feaac00cbd0 con 0x7feabc101880 2024-03-30T03:57:28.962 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac018680 con 0x7feabc101880 2024-03-30T03:57:28.962 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.961+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0ffd60 con 0x7feabc101880 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.962+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc04ea80 con 0x7feabc101880 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.963+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7feaac00cd40 con 0x7feabc101880 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.964+0000 7feac095b700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 0x7feaa40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.964+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4965+0+0 (secure 0 0 0) 0x7feaac014070 con 0x7feabc101880 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.964+0000 7feaba7fc700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 0x7feaa40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:28.963 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.965+0000 7feaba7fc700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 0x7feaa40a3660 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feab0005fd0 tx=0x7feab0005f60 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:28.967 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:28.968+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feaac0529f0 con 0x7feabc101880 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:29.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.167 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:29.168+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7feabc02cd20 con 0x7feaa40a11a0 2024-03-30T03:57:29.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:57:29.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T03:57:29.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:29.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: osdmap e43: 7 total, 6 up, 7 in 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:29.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi023 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 5756M 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] boot 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: osdmap e44: 7 total, 7 up, 7 in 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:30.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:30.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:30.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:30.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi023 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 5756M 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] boot 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: osdmap e44: 7 total, 7 up, 7 in 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:30.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:30.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:30.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:30.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi023 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 5756M 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: osd.6 [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] boot 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: osdmap e44: 7 total, 7 up, 7 in 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T03:57:30.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:30.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:30.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:30.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:31.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:30 smithi154 ceph-mon[79161]: osdmap e45: 7 total, 7 up, 7 in 2024-03-30T03:57:31.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:30 smithi154 ceph-mon[79161]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:31.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:30 smithi193 ceph-mon[79581]: osdmap e45: 7 total, 7 up, 7 in 2024-03-30T03:57:31.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:30 smithi193 ceph-mon[79581]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:31.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:30 smithi023 ceph-mon[38254]: osdmap e45: 7 total, 7 up, 7 in 2024-03-30T03:57:31.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:30 smithi023 ceph-mon[38254]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:32.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/4241371997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]': finished 2024-03-30T03:57:32.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: osdmap e46: 8 total, 7 up, 8 in 2024-03-30T03:57:32.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:32.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:31 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/1667897105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:32.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/4241371997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]': finished 2024-03-30T03:57:32.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: osdmap e46: 8 total, 7 up, 8 in 2024-03-30T03:57:32.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:32.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:31 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/1667897105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:32.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/4241371997' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]: dispatch 2024-03-30T03:57:32.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dca94a7d-3a01-47d2-aac6-b633e302111d"}]': finished 2024-03-30T03:57:32.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: osdmap e46: 8 total, 7 up, 8 in 2024-03-30T03:57:32.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:32.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:31 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/1667897105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:33.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:32 smithi154 ceph-mon[79161]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:33.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:32 smithi193 ceph-mon[79581]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:33.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:32 smithi023 ceph-mon[38254]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:35.612 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:35 smithi154 ceph-mon[79161]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:35 smithi193 ceph-mon[79581]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:35.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:35 smithi023 ceph-mon[38254]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:36 smithi154 ceph-mon[79161]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:36 smithi193 ceph-mon[79581]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:37.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:36 smithi023 ceph-mon[38254]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T03:57:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:37 smithi154 ceph-mon[79161]: Deploying daemon osd.7 on smithi154 2024-03-30T03:57:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T03:57:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:37 smithi193 ceph-mon[79581]: Deploying daemon osd.7 on smithi154 2024-03-30T03:57:38.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-30T03:57:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:37 smithi023 ceph-mon[38254]: Deploying daemon osd.7 on smithi154 2024-03-30T03:57:39.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:38 smithi154 ceph-mon[79161]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:38 smithi193 ceph-mon[79581]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:39.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:38 smithi023 ceph-mon[38254]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:41.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:41 smithi154 ceph-mon[79161]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:41.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:41 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:41 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:41 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:41 smithi193 ceph-mon[79581]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:41 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:41 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:41 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:41 smithi023 ceph-mon[38254]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:41 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:41 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:41.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:41 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:42.925 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 7 on host 'smithi154' 2024-03-30T03:57:42.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.925+0000 7feac095b700 1 -- 172.21.15.154:0/3445118734 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7feabc02cd20 con 0x7feaa40a11a0 2024-03-30T03:57:42.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 msgr2=0x7feaa40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:42.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 0x7feaa40a3660 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7feab0005fd0 tx=0x7feab0005f60 comp rx=0 tx=0).stop 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 msgr2=0x7feabc196df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e9a0 tx=0x7feaac00ecb0 comp rx=0 tx=0).stop 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 shutdown_connections 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa40a11a0 0x7feaa40a3660 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc106ae0 0x7feabc19e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:42.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101880 0x7feabc196df0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:42.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 --2- 172.21.15.154:0/3445118734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc1041b0 0x7feabc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:42.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 >> 172.21.15.154:0/3445118734 conn(0x7feabc0fb2b0 msgr2=0x7feabc1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:42.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 shutdown_connections 2024-03-30T03:57:42.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:57:42.928+0000 7feac195d700 1 -- 172.21.15.154:0/3445118734 wait complete. 2024-03-30T03:57:43.316 DEBUG:teuthology.orchestra.run.smithi154:osd.7> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.7.service 2024-03-30T03:57:43.318 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-03-30T03:57:43.319 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-30T03:57:43.348 INFO:journalctl@ceph.osd.7.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:57:43.564 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:57:43.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:43.573 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.574 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.682 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.683 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112012 s, 936 MB/s 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 2024-03-30T03:57:44.879 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T03:57:44.931 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 03:57:44 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T03:57:44.555+0000 7f1c3cbb0640 -1 osd.7 0 log_to_monitors true 2024-03-30T03:57:45.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-03-30T03:57:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:45 smithi154 ceph-mon[79161]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:45 smithi154 ceph-mon[79161]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:45 smithi154 ceph-mon[79161]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:45.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:45 smithi193 ceph-mon[79581]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:45.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:45 smithi193 ceph-mon[79581]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:45.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:45 smithi193 ceph-mon[79581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:45.497 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:57:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:45 smithi023 ceph-mon[38254]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:45 smithi023 ceph-mon[38254]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:45 smithi023 ceph-mon[38254]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-30T03:57:46.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- 172.21.15.193:0/728122459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0ff830 msgr2=0x7f61bc1093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 --2- 172.21.15.193:0/728122459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0ff830 0x7f61bc1093e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f61b400b0a0 tx=0x7f61b400b3b0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- 172.21.15.193:0/728122459 shutdown_connections 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 --2- 172.21.15.193:0/728122459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0ff830 0x7f61bc1093e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 --2- 172.21.15.193:0/728122459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc1023f0 0x7f61bc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 --2- 172.21.15.193:0/728122459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61bc0fee70 0x7f61bc0ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- 172.21.15.193:0/728122459 >> 172.21.15.193:0/728122459 conn(0x7f61bc0faa10 msgr2=0x7f61bc0fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- 172.21.15.193:0/728122459 shutdown_connections 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- 172.21.15.193:0/728122459 wait complete. 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 Processor -- start 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.525+0000 7f61c22e6700 1 -- start start 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 0x7f61bc197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61bc1023f0 0x7f61bc19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:46.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc19e9c0 con 0x7f61bc1023f0 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc19eb00 con 0x7f61bc0ff830 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc19ec90 con 0x7f61bc0fee70 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:44176/0 (socket says 172.21.15.193:44176) 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 -- 172.21.15.193:0/1399429981 learned_addr learned my addr 172.21.15.193:0/1399429981 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bb7fe700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 0x7f61bc197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 -- 172.21.15.193:0/1399429981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 msgr2=0x7f61bc197410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 0x7f61bc197410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 -- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61bc1023f0 msgr2=0x7f61bc19e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61bc1023f0 0x7f61bc19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 -- 172.21.15.193:0/1399429981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b4009d00 con 0x7f61bc0fee70 2024-03-30T03:57:46.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bb7fe700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 0x7f61bc197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61bbfff700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c9a0 tx=0x7f61b000cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0004220 con 0x7f61bc0fee70 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61b0007d20 con 0x7f61bc0fee70 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61bc19ef70 con 0x7f61bc0fee70 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b0003d00 con 0x7f61bc0fee70 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.526+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61bc19f380 con 0x7f61bc0fee70 2024-03-30T03:57:46.529 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.527+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61bc108350 con 0x7f61bc0fee70 2024-03-30T03:57:46.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.529+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f61b0004380 con 0x7f61bc0fee70 2024-03-30T03:57:46.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.530+0000 7f61b97fa700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 0x7f61a40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:57:46.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.530+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5117+0+0 (secure 0 0 0) 0x7f61b00cbcc0 con 0x7f61bc0fee70 2024-03-30T03:57:46.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.530+0000 7f61bb7fe700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 0x7f61a40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:57:46.530 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.531+0000 7f61bb7fe700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 0x7f61a40a35e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0686c0 tx=0x7f61ac008040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:57:46.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.533+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61b0096030 con 0x7f61bc0fee70 2024-03-30T03:57:46.681 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T03:57:46.314+0000 7f1c32a15700 -1 osd.7 0 waiting for initial osdmap 2024-03-30T03:57:46.681 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T03:57:46.325+0000 7f1c2e009700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: osdmap e47: 8 total, 7 up, 8 in 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:46.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:46.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.729 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:57:46.730+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f61bc02ce60 con 0x7f61a40a1120 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: osdmap e47: 8 total, 7 up, 8 in 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:46.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:46.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: osdmap e47: 8 total, 7 up, 8 in 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:57:46.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:57:46.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi154 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 5704M 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: osdmap e48: 8 total, 7 up, 8 in 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.439 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:47.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:47.440 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi154 2024-03-30T03:57:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 5704M 2024-03-30T03:57:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: osdmap e48: 8 total, 7 up, 8 in 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:47.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi154 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 5704M 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: osdmap e48: 8 total, 7 up, 8 in 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:57:47.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:57:47.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] boot 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: osdmap e49: 8 total, 8 up, 8 in 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:48.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:48 smithi154 ceph-mon[79161]: osdmap e50: 8 total, 8 up, 8 in 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] boot 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: osdmap e49: 8 total, 8 up, 8 in 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:48 smithi193 ceph-mon[79581]: osdmap e50: 8 total, 8 up, 8 in 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: osd.7 [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] boot 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: osdmap e49: 8 total, 8 up, 8 in 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T03:57:48.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:48 smithi023 ceph-mon[38254]: osdmap e50: 8 total, 8 up, 8 in 2024-03-30T03:57:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/3906786097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]': finished 2024-03-30T03:57:49.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T03:57:49.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:49.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:49 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/2601055990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/3906786097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]': finished 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:49 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/2601055990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/3906786097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]: dispatch 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81abadd3-48ab-4d42-abd2-f687f5122279"}]': finished 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: osdmap e51: 9 total, 8 up, 9 in 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:49 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/2601055990' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:57:50.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:50 smithi154 ceph-mon[79161]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T03:57:50.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:50 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:50.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:50 smithi154 ceph-mon[79161]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:50.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:50 smithi193 ceph-mon[79581]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T03:57:50.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:50 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:50.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:50 smithi193 ceph-mon[79581]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:50 smithi023 ceph-mon[38254]: osdmap e52: 9 total, 8 up, 9 in 2024-03-30T03:57:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:50 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:51.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:50 smithi023 ceph-mon[38254]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:51.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:51 smithi154 ceph-mon[79161]: osdmap e53: 9 total, 8 up, 9 in 2024-03-30T03:57:51.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:51.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:51 smithi193 ceph-mon[79581]: osdmap e53: 9 total, 8 up, 9 in 2024-03-30T03:57:51.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:51 smithi023 ceph-mon[38254]: osdmap e53: 9 total, 8 up, 9 in 2024-03-30T03:57:52.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:57:52.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:52 smithi154 ceph-mon[79161]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:52.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:52 smithi193 ceph-mon[79581]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:53.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:52 smithi023 ceph-mon[38254]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:55.488 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:55 smithi193 ceph-mon[79581]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:55.488 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T03:57:55.488 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:55 smithi154 ceph-mon[79161]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T03:57:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:55.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:55 smithi023 ceph-mon[38254]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-30T03:57:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:57:56.487 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:56 smithi193 ceph-mon[79581]: Deploying daemon osd.8 on smithi193 2024-03-30T03:57:56.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:56 smithi154 ceph-mon[79161]: Deploying daemon osd.8 on smithi193 2024-03-30T03:57:56.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:56 smithi023 ceph-mon[38254]: Deploying daemon osd.8 on smithi193 2024-03-30T03:57:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:57 smithi154 ceph-mon[79161]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:57.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:57 smithi193 ceph-mon[79581]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:57.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:57 smithi023 ceph-mon[38254]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:59.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.240 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:59.240 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.240 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:57:59 smithi193 ceph-mon[79581]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:57:59 smithi154 ceph-mon[79161]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:57:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:57:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:57:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:57:59 smithi023 ceph-mon[38254]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:00.239 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-03-30T03:58:00.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.240+0000 7f61b97fa700 1 -- 172.21.15.193:0/1399429981 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f61bc02ce60 con 0x7f61a40a1120 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.242+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 msgr2=0x7f61a40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.242+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 0x7f61a40a35e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f61bc0686c0 tx=0x7f61ac008040 comp rx=0 tx=0).stop 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 msgr2=0x7f61bc196ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c9a0 tx=0x7f61b000cd60 comp rx=0 tx=0).stop 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 shutdown_connections 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f61bc1023f0 0x7f61bc19e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:00.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61a40a1120 0x7f61a40a35e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:00.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61bc0fee70 0x7f61bc196ed0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:00.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 --2- 172.21.15.193:0/1399429981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f61bc0ff830 0x7f61bc197410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:00.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 >> 172.21.15.193:0/1399429981 conn(0x7f61bc0faa10 msgr2=0x7f61bc105490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:00.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 shutdown_connections 2024-03-30T03:58:00.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:00.243+0000 7f61c22e6700 1 -- 172.21.15.193:0/1399429981 wait complete. 2024-03-30T03:58:00.620 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.8.service 2024-03-30T03:58:00.623 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-30T03:58:00.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T03:58:00.656 INFO:journalctl@ceph.osd.8.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:58:00.997 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:58:01.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:01.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:00 smithi023 ceph-mon[38254]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:01.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:01.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:01.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:01.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:00 smithi154 ceph-mon[79161]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:01.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:00 smithi193 ceph-mon[79581]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:02.493 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 03:58:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T03:58:02.075+0000 7f734d8ed640 -1 osd.8 0 log_to_monitors true 2024-03-30T03:58:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:02 smithi193 ceph-mon[79581]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:02 smithi193 ceph-mon[79581]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:02 smithi023 ceph-mon[38254]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:02 smithi023 ceph-mon[38254]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:02 smithi154 ceph-mon[79161]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:02 smithi154 ceph-mon[79161]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-30T03:58:02.716 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T03:58:02.716 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T03:58:02.717 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-30T03:58:02.717 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-30T03:58:02.717 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0297866 s, 352 MB/s 2024-03-30T03:58:02.717 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-30T03:58:03.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-30T03:58:03.478 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: osdmap e54: 9 total, 8 up, 9 in 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: osdmap e54: 9 total, 8 up, 9 in 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:03.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:03.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:03 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-30T03:58:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: osdmap e54: 9 total, 8 up, 9 in 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:03.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:03 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:04.230 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T03:58:03.896+0000 7f7343752700 -1 osd.8 0 waiting for initial osdmap 2024-03-30T03:58:04.230 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 03:58:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T03:58:03.907+0000 7f733ad3e700 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi193 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 5704M 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: osdmap e55: 9 total, 8 up, 9 in 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:04 smithi193 ceph-mon[79581]: from='osd.8 ' entity='osd.8' 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi193 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 5704M 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: osdmap e55: 9 total, 8 up, 9 in 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:04 smithi023 ceph-mon[38254]: from='osd.8 ' entity='osd.8' 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 -- 172.21.15.23:0/1336554348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 msgr2=0x7f1b60109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 --2- 172.21.15.23:0/1336554348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b60109240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800b0a0 tx=0x7f1b5800b3b0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 -- 172.21.15.23:0/1336554348 shutdown_connections 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 --2- 172.21.15.23:0/1336554348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b60109240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 --2- 172.21.15.23:0/1336554348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60100e00 0x7f1b60101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 --2- 172.21.15.23:0/1336554348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60101f60 0x7f1b601023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 -- 172.21.15.23:0/1336554348 >> 172.21.15.23:0/1336554348 conn(0x7f1b600fc3c0 msgr2=0x7f1b600fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 -- 172.21.15.23:0/1336554348 shutdown_connections 2024-03-30T03:58:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.591+0000 7f1b68400700 1 -- 172.21.15.23:0/1336554348 wait complete. 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 Processor -- start 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 -- start start 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 0x7f1b60196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60101f60 0x7f1b601973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b6019e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6019e960 con 0x7f1b60102920 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6019eaa0 con 0x7f1b60100e00 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.592+0000 7f1b68400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6019ec30 con 0x7f1b60101f60 2024-03-30T03:58:04.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b65a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 0x7f1b60196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b6019e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b65a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 0x7f1b60196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35520/0 (socket says 172.21.15.23:35520) 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b65a00700 1 -- 172.21.15.23:0/3140641823 learned_addr learned my addr 172.21.15.23:0/3140641823 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b65000700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60101f60 0x7f1b601973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 -- 172.21.15.23:0/3140641823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60101f60 msgr2=0x7f1b601973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60101f60 0x7f1b601973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 -- 172.21.15.23:0/3140641823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 msgr2=0x7f1b60196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 0x7f1b60196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 -- 172.21.15.23:0/3140641823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b58009d00 con 0x7f1b60102920 2024-03-30T03:58:04.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.593+0000 7f1b66400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b6019e390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800b0a0 tx=0x7f1b5800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:04.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.594+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800e040 con 0x7f1b60102920 2024-03-30T03:58:04.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.594+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b58003c00 con 0x7f1b60102920 2024-03-30T03:58:04.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.594+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b60072fe0 con 0x7f1b60102920 2024-03-30T03:58:04.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.594+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5801b6b0 con 0x7f1b60102920 2024-03-30T03:58:04.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.594+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b60073530 con 0x7f1b60102920 2024-03-30T03:58:04.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.595+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b601081b0 con 0x7f1b60102920 2024-03-30T03:58:04.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.597+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f1b5801d480 con 0x7f1b60102920 2024-03-30T03:58:04.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.598+0000 7f1b5e000700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 0x7f1b480a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:04.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.598+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5862+0+0 (secure 0 0 0) 0x7f1b580cd450 con 0x7f1b60102920 2024-03-30T03:58:04.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.599+0000 7f1b65a00700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 0x7f1b480a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:04.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.599+0000 7f1b65a00700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 0x7f1b480a3660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d20 tx=0x7f1b54009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:04.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.603+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b58097560 con 0x7f1b60102920 2024-03-30T03:58:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi193 2024-03-30T03:58:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 5704M 2024-03-30T03:58:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:58:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: osdmap e55: 9 total, 8 up, 9 in 2024-03-30T03:58:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:04.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:04 smithi154 ceph-mon[79161]: from='osd.8 ' entity='osd.8' 2024-03-30T03:58:04.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:04.825+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b600611f0 con 0x7f1b480a11a0 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] boot 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: osdmap e56: 9 total, 9 up, 9 in 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:05.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:05 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] boot 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: osdmap e56: 9 total, 9 up, 9 in 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:05 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:58:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:58:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: osd.8 [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] boot 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: osdmap e56: 9 total, 9 up, 9 in 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:05.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:05 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:06 smithi023 ceph-mon[38254]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:06 smithi023 ceph-mon[38254]: osdmap e57: 9 total, 9 up, 9 in 2024-03-30T03:58:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:06 smithi154 ceph-mon[79161]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:06 smithi154 ceph-mon[79161]: osdmap e57: 9 total, 9 up, 9 in 2024-03-30T03:58:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:06 smithi193 ceph-mon[79581]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:06 smithi193 ceph-mon[79581]: osdmap e57: 9 total, 9 up, 9 in 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: osdmap e58: 9 total, 9 up, 9 in 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]: dispatch 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]': finished 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T03:58:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:07 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: osdmap e58: 9 total, 9 up, 9 in 2024-03-30T03:58:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]: dispatch 2024-03-30T03:58:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]': finished 2024-03-30T03:58:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T03:58:07.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:07 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: osdmap e58: 9 total, 9 up, 9 in 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]: dispatch 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3023822557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89bd59bf-4452-4a10-b466-f00fab24b496"}]': finished 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: osdmap e59: 10 total, 9 up, 10 in 2024-03-30T03:58:07.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:07 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:08 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4135739353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:08 smithi023 ceph-mon[38254]: osdmap e60: 10 total, 9 up, 10 in 2024-03-30T03:58:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:08 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4135739353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:08 smithi154 ceph-mon[79161]: osdmap e60: 10 total, 9 up, 10 in 2024-03-30T03:58:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:08.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:08 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4135739353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:08.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:08 smithi193 ceph-mon[79581]: osdmap e60: 10 total, 9 up, 10 in 2024-03-30T03:58:08.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:09 smithi023 ceph-mon[38254]: pgmap v166: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:09 smithi154 ceph-mon[79161]: pgmap v166: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:09.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:09 smithi193 ceph-mon[79581]: pgmap v166: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:11 smithi023 ceph-mon[38254]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:11.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:11 smithi154 ceph-mon[79161]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:11 smithi193 ceph-mon[79581]: pgmap v167: 1 pgs: 1 peering; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:13.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:13 smithi023 ceph-mon[38254]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:58:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:13 smithi154 ceph-mon[79161]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:58:13.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:13 smithi193 ceph-mon[79581]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-30T03:58:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T03:58:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:14 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T03:58:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:14 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:14.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-30T03:58:14.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:14 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:15.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:15 smithi023 ceph-mon[38254]: Deploying daemon osd.9 on smithi023 2024-03-30T03:58:15.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:15 smithi023 ceph-mon[38254]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:58:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:15 smithi154 ceph-mon[79161]: Deploying daemon osd.9 on smithi023 2024-03-30T03:58:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:15 smithi154 ceph-mon[79161]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:58:15.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:15 smithi193 ceph-mon[79581]: Deploying daemon osd.9 on smithi023 2024-03-30T03:58:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:15 smithi193 ceph-mon[79581]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-30T03:58:16.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:16 smithi023 ceph-mon[38254]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T03:58:17.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:16 smithi154 ceph-mon[79161]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T03:58:17.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:16 smithi193 ceph-mon[79581]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-30T03:58:19.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:19 smithi023 ceph-mon[38254]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:19 smithi154 ceph-mon[79161]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:19.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:19 smithi193 ceph-mon[79581]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:20.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:20.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:20 smithi023 ceph-mon[38254]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:20 smithi154 ceph-mon[79161]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:20 smithi193 ceph-mon[79581]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:22.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.686+0000 7f1b5e000700 1 -- 172.21.15.23:0/3140641823 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1b600611f0 con 0x7f1b480a11a0 2024-03-30T03:58:22.685 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 msgr2=0x7f1b480a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 0x7f1b480a3660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d20 tx=0x7f1b54009260 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 msgr2=0x7f1b6019e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b6019e390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800b0a0 tx=0x7f1b5800ba50 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 shutdown_connections 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b60102920 0x7f1b6019e390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b60100e00 0x7f1b60196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b480a11a0 0x7f1b480a3660 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 --2- 172.21.15.23:0/3140641823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b60101f60 0x7f1b601973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 >> 172.21.15.23:0/3140641823 conn(0x7f1b600fc3c0 msgr2=0x7f1b601052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 shutdown_connections 2024-03-30T03:58:22.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:22.689+0000 7f1b68400700 1 -- 172.21.15.23:0/3140641823 wait complete. 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:22.898 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:23.075 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.9.service 2024-03-30T03:58:23.079 INFO:tasks.cephadm:Deploying osd.10 on smithi154 with /dev/vg_nvme/lv_1... 2024-03-30T03:58:23.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T03:58:23.159 INFO:journalctl@ceph.osd.9.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:58:23.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:23.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:23.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:23.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:23.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:23.313 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:58:24.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:23 smithi023 ceph-mon[38254]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T03:58:24.022 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 03:58:23 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T03:58:23.715+0000 7f3bec3bc640 -1 osd.9 0 log_to_monitors true 2024-03-30T03:58:24.123 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:23 smithi154 ceph-mon[79161]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T03:58:24.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:23 smithi193 ceph-mon[79581]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-30T03:58:24.650 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T03:58:24.651 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T03:58:24.651 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10+0 records in 2024-03-30T03:58:24.651 INFO:teuthology.orchestra.run.smithi154.stderr:10+0 records out 2024-03-30T03:58:24.651 INFO:teuthology.orchestra.run.smithi154.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149957 s, 699 MB/s 2024-03-30T03:58:24.651 INFO:teuthology.orchestra.run.smithi154.stderr:--> Zapping successful for: 2024-03-30T03:58:24.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:24 smithi154 ceph-mon[79161]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T03:58:24.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:24 smithi154 ceph-mon[79161]: osdmap e61: 10 total, 9 up, 10 in 2024-03-30T03:58:24.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:24.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:24 smithi154 ceph-mon[79161]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:58:24.941 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:24 smithi154 ceph-mon[79161]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:24.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-03-30T03:58:25.200 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T03:58:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:24 smithi193 ceph-mon[79581]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T03:58:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:24 smithi193 ceph-mon[79581]: osdmap e61: 10 total, 9 up, 10 in 2024-03-30T03:58:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:24 smithi193 ceph-mon[79581]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:58:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:24 smithi193 ceph-mon[79581]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:24 smithi023 ceph-mon[38254]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-30T03:58:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:24 smithi023 ceph-mon[38254]: osdmap e61: 10 total, 9 up, 10 in 2024-03-30T03:58:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:24 smithi023 ceph-mon[38254]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-30T03:58:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:24 smithi023 ceph-mon[38254]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:26.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:25 smithi154 ceph-mon[79161]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:58:26.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:25 smithi154 ceph-mon[79161]: osdmap e62: 10 total, 9 up, 10 in 2024-03-30T03:58:26.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:25 smithi154 ceph-mon[79161]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 -- 172.21.15.154:0/1663109208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 msgr2=0x7f56a0108f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 --2- 172.21.15.154:0/1663109208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a0108f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f569800b0a0 tx=0x7f569800b3b0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 -- 172.21.15.154:0/1663109208 shutdown_connections 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 --2- 172.21.15.154:0/1663109208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a0108f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 --2- 172.21.15.154:0/1663109208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 0x7f56a0103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 --2- 172.21.15.154:0/1663109208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0106590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.192 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 -- 172.21.15.154:0/1663109208 >> 172.21.15.154:0/1663109208 conn(0x7f56a00fb2a0 msgr2=0x7f56a00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:26.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 -- 172.21.15.154:0/1663109208 shutdown_connections 2024-03-30T03:58:26.193 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.192+0000 7f56a80b1700 1 -- 172.21.15.154:0/1663109208 wait complete. 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 Processor -- start 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 -- start start 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 0x7f56a0196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019ea20 con 0x7f56a0101870 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019eb60 con 0x7f56a0106ad0 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a80b1700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a019ecf0 con 0x7f56a01041a0 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:26.195 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:49640/0 (socket says 172.21.15.154:49640) 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 -- 172.21.15.154:0/3580132863 learned_addr learned my addr 172.21.15.154:0/3580132863 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a5e4d700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 0x7f56a0196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a564c700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 -- 172.21.15.154:0/3580132863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 msgr2=0x7f56a0197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 -- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 msgr2=0x7f56a0196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 0x7f56a0196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.193+0000 7f56a664e700 1 -- 172.21.15.154:0/3580132863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5698009d00 con 0x7f56a0106ad0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f56a564c700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f56a664e700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5698004d00 tx=0x7f5698004de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569800e040 con 0x7f56a0106ad0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f569801db50 con 0x7f56a0106ad0 2024-03-30T03:58:26.196 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a019ef70 con 0x7f56a0106ad0 2024-03-30T03:58:26.197 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569801bc00 con 0x7f56a0106ad0 2024-03-30T03:58:26.197 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.194+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a019f400 con 0x7f56a0106ad0 2024-03-30T03:58:26.197 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.195+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a00fce20 con 0x7f56a0106ad0 2024-03-30T03:58:26.197 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.196+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f569801dcc0 con 0x7f56a0106ad0 2024-03-30T03:58:26.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.196+0000 7f5696ffd700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 0x7f568c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:26.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.197+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5981+0+0 (secure 0 0 0) 0x7f56980ce2b0 con 0x7f56a0106ad0 2024-03-30T03:58:26.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.197+0000 7f56a5e4d700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 0x7f568c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:26.198 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.197+0000 7f56a5e4d700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 0x7f568c0a3660 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f569c006fd0 tx=0x7f569c008040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:26.200 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.201+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5698098350 con 0x7f56a0106ad0 2024-03-30T03:58:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:25 smithi193 ceph-mon[79581]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:58:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:25 smithi193 ceph-mon[79581]: osdmap e62: 10 total, 9 up, 10 in 2024-03-30T03:58:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:25 smithi193 ceph-mon[79581]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' 2024-03-30T03:58:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-mon[38254]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-30T03:58:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-mon[38254]: osdmap e62: 10 total, 9 up, 10 in 2024-03-30T03:58:26.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:26.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-mon[38254]: from='osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643]' entity='osd.9' 2024-03-30T03:58:26.272 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T03:58:25.866+0000 7f3be0a00700 -1 osd.9 0 waiting for initial osdmap 2024-03-30T03:58:26.272 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 03:58:25 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T03:58:25.894+0000 7f3bd6800700 -1 osd.9 62 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:58:26.400 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:26.401+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a002cdc0 con 0x7f568c0a11a0 2024-03-30T03:58:27.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:58:27.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:58:27.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:27.062 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi023 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 4317M 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.063 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] boot 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: osdmap e63: 10 total, 10 up, 10 in 2024-03-30T03:58:27.064 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:26 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi023 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 4317M 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] boot 2024-03-30T03:58:27.245 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: osdmap e63: 10 total, 10 up, 10 in 2024-03-30T03:58:27.245 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:26 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi023 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 4317M 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: osd.9 [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] boot 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: osdmap e63: 10 total, 10 up, 10 in 2024-03-30T03:58:27.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:26 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T03:58:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:28 smithi154 ceph-mon[79161]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:28 smithi193 ceph-mon[79581]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:28.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:28 smithi023 ceph-mon[38254]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: osdmap e64: 10 total, 10 up, 10 in 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/3625598216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]': finished 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: osdmap e65: 11 total, 10 up, 11 in 2024-03-30T03:58:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:29.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:29.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.154:0/1468582903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: osdmap e64: 10 total, 10 up, 10 in 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/3625598216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]': finished 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: osdmap e65: 11 total, 10 up, 11 in 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.154:0/1468582903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: osdmap e64: 10 total, 10 up, 10 in 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/3625598216' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]: dispatch 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0803d869-d43c-46f0-9725-be71809628b6"}]': finished 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: osdmap e65: 11 total, 10 up, 11 in 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.154:0/1468582903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:31 smithi154 ceph-mon[79161]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:31.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:31 smithi193 ceph-mon[79581]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:31.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:31 smithi023 ceph-mon[38254]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:33 smithi154 ceph-mon[79161]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:33 smithi193 ceph-mon[79581]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:33.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:33 smithi023 ceph-mon[38254]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:34.652 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T03:58:34.653 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T03:58:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:34.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-30T03:58:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:35.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:35 smithi154 ceph-mon[79161]: Deploying daemon osd.10 on smithi154 2024-03-30T03:58:35.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:35 smithi154 ceph-mon[79161]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:35 smithi193 ceph-mon[79581]: Deploying daemon osd.10 on smithi154 2024-03-30T03:58:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:35 smithi193 ceph-mon[79581]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:35.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:35 smithi023 ceph-mon[38254]: Deploying daemon osd.10 on smithi154 2024-03-30T03:58:35.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:35 smithi023 ceph-mon[38254]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:37.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:36 smithi154 ceph-mon[79161]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:36 smithi193 ceph-mon[79581]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:36 smithi023 ceph-mon[38254]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:39.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:38 smithi154 ceph-mon[79161]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:38 smithi193 ceph-mon[79581]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:39.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:38 smithi023 ceph-mon[38254]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:39.806 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 10 on host 'smithi154' 2024-03-30T03:58:39.806 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.806+0000 7f5696ffd700 1 -- 172.21.15.154:0/3580132863 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f56a002cdc0 con 0x7f568c0a11a0 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 msgr2=0x7f568c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 0x7f568c0a3660 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f569c006fd0 tx=0x7f569c008040 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 msgr2=0x7f56a019e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5698004d00 tx=0x7f5698004de0 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 shutdown_connections 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a0106ad0 0x7f56a019e450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a0101870 0x7f56a0196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f568c0a11a0 0x7f568c0a3660 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 --2- 172.21.15.154:0/3580132863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a01041a0 0x7f56a0197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 >> 172.21.15.154:0/3580132863 conn(0x7f56a00fb2a0 msgr2=0x7f56a01022d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:39.808 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 shutdown_connections 2024-03-30T03:58:39.809 INFO:teuthology.orchestra.run.smithi154.stderr:2024-03-30T03:58:39.809+0000 7f56a80b1700 1 -- 172.21.15.154:0/3580132863 wait complete. 2024-03-30T03:58:40.290 DEBUG:teuthology.orchestra.run.smithi154:osd.10> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.10.service 2024-03-30T03:58:40.292 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-03-30T03:58:40.293 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-30T03:58:40.325 INFO:journalctl@ceph.osd.10.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:40.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:40.523 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:58:41.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:41 smithi154 ceph-mon[79161]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:41.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:41 smithi023 ceph-mon[38254]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:41 smithi193 ceph-mon[79581]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112372 s, 933 MB/s 2024-03-30T03:58:41.844 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-30T03:58:41.931 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 03:58:41 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T03:58:41.593+0000 7f08bcac0640 -1 osd.10 0 log_to_monitors true 2024-03-30T03:58:42.233 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-03-30T03:58:42.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:42 smithi154 ceph-mon[79161]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:42.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:42 smithi154 ceph-mon[79161]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:42.453 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:58:42.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:42 smithi193 ceph-mon[79581]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:42.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:42 smithi193 ceph-mon[79581]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:42 smithi023 ceph-mon[38254]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:42 smithi023 ceph-mon[38254]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 -- 172.21.15.193:0/3372769806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa208106b20 msgr2=0x7fa208108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 --2- 172.21.15.193:0/3372769806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa208106b20 0x7fa208108f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa20000b0a0 tx=0x7fa20000b3b0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 -- 172.21.15.193:0/3372769806 shutdown_connections 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 --2- 172.21.15.193:0/3372769806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa208106b20 0x7fa208108f90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 --2- 172.21.15.193:0/3372769806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2081018c0 0x7fa208103cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 --2- 172.21.15.193:0/3372769806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2081041f0 0x7fa2081065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 -- 172.21.15.193:0/3372769806 >> 172.21.15.193:0/3372769806 conn(0x7fa2080fb2d0 msgr2=0x7fa2080fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 -- 172.21.15.193:0/3372769806 shutdown_connections 2024-03-30T03:58:43.368 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.368+0000 7fa20caee700 1 -- 172.21.15.193:0/3372769806 wait complete. 2024-03-30T03:58:43.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 Processor -- start 2024-03-30T03:58:43.369 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 -- start start 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2081018c0 0x7fa208196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208106b20 0x7fa20819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20819ea60 con 0x7fa2081018c0 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20819eba0 con 0x7fa208106b20 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa20caee700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20819ed30 con 0x7fa2081041f0 2024-03-30T03:58:43.370 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50412/0 (socket says 172.21.15.193:50412) 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 -- 172.21.15.193:0/1423937272 learned_addr learned my addr 172.21.15.193:0/1423937272 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa206d9d700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208106b20 0x7fa20819e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 -- 172.21.15.193:0/1423937272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208106b20 msgr2=0x7fa20819e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208106b20 0x7fa20819e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 -- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2081018c0 msgr2=0x7fa208196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2081018c0 0x7fa208196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.369+0000 7fa205d9b700 1 -- 172.21.15.193:0/1423937272 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa200009d00 con 0x7fa2081041f0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa205d9b700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800b560 tx=0x7fa1f800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8011830 con 0x7fa2081041f0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa1f8011e50 con 0x7fa2081041f0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa20819efb0 con 0x7fa2081041f0 2024-03-30T03:58:43.371 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f800f2e0 con 0x7fa2081041f0 2024-03-30T03:58:43.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.370+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa20819f470 con 0x7fa2081041f0 2024-03-30T03:58:43.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.371+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2080fce50 con 0x7fa2081041f0 2024-03-30T03:58:43.372 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.372+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa1f8010580 con 0x7fa2081041f0 2024-03-30T03:58:43.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.372+0000 7fa1f77fe700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 0x7fa1f00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:43.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.373+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6414+0+0 (secure 0 0 0) 0x7fa1f80cbc00 con 0x7fa2081041f0 2024-03-30T03:58:43.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.373+0000 7fa20659c700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 0x7fa1f00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:43.373 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.374+0000 7fa20659c700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 0x7fa1f00a35e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc005950 tx=0x7fa1fc0058e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:43.376 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.377+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa1f80d2080 con 0x7fa2081041f0 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: osdmap e66: 11 total, 10 up, 11 in 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:43.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:43 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: osdmap e66: 11 total, 10 up, 11 in 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:43.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:43 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.579 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:43.579+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa20802ce70 con 0x7fa1f00a1120 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: osdmap e66: 11 total, 10 up, 11 in 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-03-30T03:58:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:43.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:44.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T03:58:43.927+0000 7f08b2124700 -1 osd.10 0 waiting for initial osdmap 2024-03-30T03:58:44.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 03:58:43 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T03:58:43.940+0000 7f08a9f11700 -1 osd.10 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi154 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 4278M 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: osdmap e67: 11 total, 10 up, 11 in 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:44.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='osd.10 ' entity='osd.10' 2024-03-30T03:58:44.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] boot 2024-03-30T03:58:44.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: osdmap e68: 11 total, 11 up, 11 in 2024-03-30T03:58:44.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:44 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi154 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 4278M 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: osdmap e67: 11 total, 10 up, 11 in 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:44.629 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='osd.10 ' entity='osd.10' 2024-03-30T03:58:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] boot 2024-03-30T03:58:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: osdmap e68: 11 total, 11 up, 11 in 2024-03-30T03:58:44.630 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:44 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi154 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 4278M 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: osdmap e67: 11 total, 10 up, 11 in 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:44.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='osd.10 ' entity='osd.10' 2024-03-30T03:58:44.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: osd.10 [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] boot 2024-03-30T03:58:44.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: osdmap e68: 11 total, 11 up, 11 in 2024-03-30T03:58:44.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:44 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T03:58:45.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:45 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:58:45.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:45 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:58:45.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:45 smithi023 ceph-mon[38254]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:45.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:45 smithi023 ceph-mon[38254]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:45.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:45 smithi023 ceph-mon[38254]: osdmap e69: 11 total, 11 up, 11 in 2024-03-30T03:58:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:45 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:58:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:45 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:58:45.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:45 smithi154 ceph-mon[79161]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:45.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:45 smithi154 ceph-mon[79161]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:45.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:45 smithi154 ceph-mon[79161]: osdmap e69: 11 total, 11 up, 11 in 2024-03-30T03:58:45.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:45 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:58:45.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:45 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:58:45.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:45 smithi193 ceph-mon[79581]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:58:45.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:45 smithi193 ceph-mon[79581]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-03-30T03:58:45.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:45 smithi193 ceph-mon[79581]: osdmap e69: 11 total, 11 up, 11 in 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/2900806739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]': finished 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T03:58:46.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.193:0/2510820503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/2900806739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]': finished 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T03:58:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.193:0/2510820503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/2900806739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]: dispatch 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7746428c-6c3a-4801-922d-7dd9000dfe7d"}]': finished 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: osdmap e70: 12 total, 11 up, 12 in 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: osdmap e71: 12 total, 11 up, 12 in 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.193:0/2510820503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-30T03:58:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:47 smithi023 ceph-mon[38254]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:58:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:47 smithi023 ceph-mon[38254]: osdmap e72: 12 total, 11 up, 12 in 2024-03-30T03:58:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:47 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:47 smithi154 ceph-mon[79161]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:58:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:47 smithi154 ceph-mon[79161]: osdmap e72: 12 total, 11 up, 12 in 2024-03-30T03:58:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:47 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:47.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:47 smithi193 ceph-mon[79581]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:58:47.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:47 smithi193 ceph-mon[79581]: osdmap e72: 12 total, 11 up, 12 in 2024-03-30T03:58:47.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:47 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:58:49.415 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:49 smithi193 ceph-mon[79581]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:49 smithi154 ceph-mon[79161]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:49.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:49 smithi023 ceph-mon[38254]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 719 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-30T03:58:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:51 smithi193 ceph-mon[79581]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T03:58:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:51.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:51 smithi154 ceph-mon[79161]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:51.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T03:58:51.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:51.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:51 smithi023 ceph-mon[38254]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-30T03:58:51.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-30T03:58:51.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:52.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:52 smithi154 ceph-mon[79161]: Deploying daemon osd.11 on smithi193 2024-03-30T03:58:52.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:52 smithi193 ceph-mon[79581]: Deploying daemon osd.11 on smithi193 2024-03-30T03:58:52.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:52 smithi023 ceph-mon[38254]: Deploying daemon osd.11 on smithi193 2024-03-30T03:58:53.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:53 smithi193 ceph-mon[79581]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-30T03:58:53.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:53 smithi154 ceph-mon[79161]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-30T03:58:53.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:53 smithi023 ceph-mon[38254]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-30T03:58:55.663 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:55 smithi193 ceph-mon[79581]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:58:55.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.664 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:55 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:55 smithi154 ceph-mon[79161]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:58:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:55 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:55.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:55 smithi023 ceph-mon[38254]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-30T03:58:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:55 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:57.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:56 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:57.239 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:56 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-30T03:58:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:58:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:58:57.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:56 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:57.286 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.287+0000 7fa1f77fe700 1 -- 172.21.15.193:0/1423937272 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fa20802ce70 con 0x7fa1f00a1120 2024-03-30T03:58:57.286 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 msgr2=0x7fa1f00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 0x7fa1f00a35e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc005950 tx=0x7fa1fc0058e0 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 msgr2=0x7fa208197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1f800b560 tx=0x7fa1f800b870 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 shutdown_connections 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa208106b20 0x7fa20819e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2081018c0 0x7fa208196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa1f00a1120 0x7fa1f00a35e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 --2- 172.21.15.193:0/1423937272 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa2081041f0 0x7fa208197460 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 >> 172.21.15.193:0/1423937272 conn(0x7fa2080fb2d0 msgr2=0x7fa208102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 shutdown_connections 2024-03-30T03:58:57.289 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:58:57.289+0000 7fa20caee700 1 -- 172.21.15.193:0/1423937272 wait complete. 2024-03-30T03:58:57.725 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.11.service 2024-03-30T03:58:57.727 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-30T03:58:57.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T03:58:57.757 INFO:journalctl@ceph.osd.11.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:58:58.106 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:58:58.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:58.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:58.135 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:57 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:58.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:58.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:58.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:57 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:58.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:58:58.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:58.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:57 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:58:59.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:58 smithi154 ceph-mon[79161]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T03:58:59.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:58 smithi193 ceph-mon[79581]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T03:58:59.244 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T03:58:59.004+0000 7f2bd85fc640 -1 osd.11 0 log_to_monitors true 2024-03-30T03:58:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 -- 172.21.15.23:0/190432296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018102920 msgr2=0x7f6018109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:59.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 --2- 172.21.15.23:0/190432296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018102920 0x7f6018109240 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f601000b0a0 tx=0x7f601000b3b0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 -- 172.21.15.23:0/190432296 shutdown_connections 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 --2- 172.21.15.23:0/190432296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018102920 0x7f6018109240 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 --2- 172.21.15.23:0/190432296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018100e00 0x7f6018101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 --2- 172.21.15.23:0/190432296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018101f60 0x7f60181023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 -- 172.21.15.23:0/190432296 >> 172.21.15.23:0/190432296 conn(0x7f60180fc3c0 msgr2=0x7f60180fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 -- 172.21.15.23:0/190432296 shutdown_connections 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.270+0000 7f601d000700 1 -- 172.21.15.23:0/190432296 wait complete. 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.271+0000 7f601d000700 1 Processor -- start 2024-03-30T03:58:59.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.271+0000 7f601d000700 1 -- start start 2024-03-30T03:58:59.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:58 smithi023 ceph-mon[38254]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 0x7f6018196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018102920 0x7f601819e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601819e740 con 0x7f6018101f60 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601819e880 con 0x7f6018102920 2024-03-30T03:58:59.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f601d000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601819ea10 con 0x7f6018100e00 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f6015400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f6015400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56442/0 (socket says 172.21.15.23:56442) 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f6015400700 1 -- 172.21.15.23:0/3907060271 learned_addr learned my addr 172.21.15.23:0/3907060271 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f6015e00700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 0x7f6018196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.272+0000 7f6016800700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018102920 0x7f601819e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 -- 172.21.15.23:0/3907060271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018102920 msgr2=0x7f601819e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018102920 0x7f601819e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 -- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 msgr2=0x7f6018197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 -- 172.21.15.23:0/3907060271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6008009520 con 0x7f6018100e00 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015400700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f6015e00700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 0x7f6018196c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f600800ea20 tx=0x7f600800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:59.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008004220 con 0x7f6018100e00 2024-03-30T03:58:59.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f600800eec0 con 0x7f6018100e00 2024-03-30T03:58:59.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008003cf0 con 0x7f6018100e00 2024-03-30T03:58:59.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6010009d00 con 0x7f6018100e00 2024-03-30T03:58:59.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.273+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601819f060 con 0x7f6018100e00 2024-03-30T03:58:59.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.275+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018108280 con 0x7f6018100e00 2024-03-30T03:58:59.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.277+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f600800c970 con 0x7f6018100e00 2024-03-30T03:58:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.278+0000 7f600ea00700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 0x7f5ffc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:58:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.278+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6809+0+0 (secure 0 0 0) 0x7f6008014070 con 0x7f6018100e00 2024-03-30T03:58:59.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.278+0000 7f6015400700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 0x7f5ffc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:58:59.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.279+0000 7f6015400700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 0x7f5ffc0a3660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6004011550 tx=0x7f6004009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:58:59.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.282+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f60080962e0 con 0x7f6018100e00 2024-03-30T03:58:59.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.497+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f601804ea80 con 0x7f6018100e00 2024-03-30T03:58:59.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.498+0000 7f600ea00700 1 -- 172.21.15.23:0/3907060271 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6008099900 con 0x7f6018100e00 2024-03-30T03:58:59.497 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 msgr2=0x7f5ffc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 0x7f5ffc0a3660 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6004011550 tx=0x7f6004009260 comp rx=0 tx=0).stop 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 msgr2=0x7f6018196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 0x7f6018196c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f600800ea20 tx=0x7f600800ed30 comp rx=0 tx=0).stop 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 shutdown_connections 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6018102920 0x7f601819e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5ffc0a11a0 0x7f5ffc0a3660 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6018100e00 0x7f6018196c00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 --2- 172.21.15.23:0/3907060271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6018101f60 0x7f6018197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:58:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 >> 172.21.15.23:0/3907060271 conn(0x7f60180fc3c0 msgr2=0x7f60181053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:58:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 shutdown_connections 2024-03-30T03:58:59.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:58:59.500+0000 7f601d000700 1 -- 172.21.15.23:0/3907060271 wait complete. 2024-03-30T03:58:59.915 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":72,"num_osds":12,"num_up_osds":11,"osd_up_since":1711771124,"num_in_osds":12,"osd_in_since":1711771125,"num_remapped_pgs":0} 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3907060271' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:00.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:58:59 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3907060271' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:00.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:00.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:58:59 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3907060271' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:00.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:58:59 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:00.916 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd stat -f json 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: Detected new or changed devices on smithi193 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 4278M 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: osdmap e73: 12 total, 11 up, 12 in 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:00 smithi193 ceph-mon[79581]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:01.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: Detected new or changed devices on smithi193 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 4278M 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: osdmap e73: 12 total, 11 up, 12 in 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:00 smithi023 ceph-mon[38254]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:01.293 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:59:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: Detected new or changed devices on smithi193 2024-03-30T03:59:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 4278M 2024-03-30T03:59:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-30T03:59:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: osdmap e73: 12 total, 11 up, 12 in 2024-03-30T03:59:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: from='osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-30T03:59:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:00 smithi154 ceph-mon[79161]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:01.938 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T03:59:01.610+0000 7f2bcdc60700 -1 osd.11 0 waiting for initial osdmap 2024-03-30T03:59:01.938 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T03:59:01.626+0000 7f2bc5a4d700 -1 osd.11 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T03:59:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:01 smithi023 ceph-mon[38254]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:59:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:01 smithi023 ceph-mon[38254]: osdmap e74: 12 total, 11 up, 12 in 2024-03-30T03:59:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:01 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:01 smithi023 ceph-mon[38254]: from='osd.11 ' entity='osd.11' 2024-03-30T03:59:02.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-mon[79581]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:59:02.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-mon[79581]: osdmap e74: 12 total, 11 up, 12 in 2024-03-30T03:59:02.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:01 smithi193 ceph-mon[79581]: from='osd.11 ' entity='osd.11' 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.418+0000 7f6545c00700 1 -- 172.21.15.23:0/2871605164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540101f60 msgr2=0x7f65401023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.418+0000 7f6545c00700 1 --2- 172.21.15.23:0/2871605164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540101f60 0x7f65401023e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f652c009990 tx=0x7f652c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 -- 172.21.15.23:0/2871605164 shutdown_connections 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 --2- 172.21.15.23:0/2871605164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 0x7f6540109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 --2- 172.21.15.23:0/2871605164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540100e00 0x7f6540101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 --2- 172.21.15.23:0/2871605164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540101f60 0x7f65401023e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 -- 172.21.15.23:0/2871605164 >> 172.21.15.23:0/2871605164 conn(0x7f65400fc3c0 msgr2=0x7f65400fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 -- 172.21.15.23:0/2871605164 shutdown_connections 2024-03-30T03:59:02.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.419+0000 7f6545c00700 1 -- 172.21.15.23:0/2871605164 wait complete. 2024-03-30T03:59:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 Processor -- start 2024-03-30T03:59:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 -- start start 2024-03-30T03:59:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 0x7f6540199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:02.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 0x7f65401a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401a0a90 con 0x7f6540100e00 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401a0bd0 con 0x7f6540101f60 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f6545c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401a0d60 con 0x7f6540102920 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.420+0000 7f653ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56452/0 (socket says 172.21.15.23:56452) 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 -- 172.21.15.23:0/1824250187 learned_addr learned my addr 172.21.15.23:0/1824250187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653f400700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 0x7f65401a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653e000700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 0x7f6540199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 -- 172.21.15.23:0/1824250187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 msgr2=0x7f65401a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 0x7f65401a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 -- 172.21.15.23:0/1824250187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 msgr2=0x7f6540199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 0x7f6540199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 -- 172.21.15.23:0/1824250187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f652c0095f0 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653e000700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 0x7f6540199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f653ea00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9d0 tx=0x7f653000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.421+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530009790 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.422+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f653000ee70 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.422+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65401a0fe0 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.422+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530010ea0 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.422+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65401a14a0 con 0x7f6540100e00 2024-03-30T03:59:02.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.423+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654004ea80 con 0x7f6540100e00 2024-03-30T03:59:02.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.425+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f653000cbd0 con 0x7f6540100e00 2024-03-30T03:59:02.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.426+0000 7f6537400700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 0x7f65240a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:02.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.426+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f6530014070 con 0x7f6540100e00 2024-03-30T03:59:02.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.426+0000 7f653e000700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 0x7f65240a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:02.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.427+0000 7f653e000700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 0x7f65240a3660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b2e0 tx=0x7f652c00b6f0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.429+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f65300965f0 con 0x7f6540100e00 2024-03-30T03:59:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:01 smithi154 ceph-mon[79161]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-30T03:59:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:01 smithi154 ceph-mon[79161]: osdmap e74: 12 total, 11 up, 12 in 2024-03-30T03:59:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:01 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:01 smithi154 ceph-mon[79161]: from='osd.11 ' entity='osd.11' 2024-03-30T03:59:02.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.645+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f654010c610 con 0x7f6540100e00 2024-03-30T03:59:02.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.645+0000 7f6537400700 1 -- 172.21.15.23:0/1824250187 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7f6530099c10 con 0x7f6540100e00 2024-03-30T03:59:02.645 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:02.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.647+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 msgr2=0x7f65240a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:02.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.647+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 0x7f65240a3660 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f652c00b2e0 tx=0x7f652c00b6f0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.647+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 msgr2=0x7f6540198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.647+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9d0 tx=0x7f653000ece0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 shutdown_connections 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65240a11a0 0x7f65240a3660 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6540102920 0x7f65401a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540100e00 0x7f6540198f50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 --2- 172.21.15.23:0/1824250187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6540101f60 0x7f6540199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 >> 172.21.15.23:0/1824250187 conn(0x7f65400fc3c0 msgr2=0x7f6540072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 shutdown_connections 2024-03-30T03:59:02.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:02.648+0000 7f6545c00700 1 -- 172.21.15.23:0/1824250187 wait complete. 2024-03-30T03:59:03.075 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":75,"num_osds":12,"num_up_osds":12,"osd_up_since":1711771142,"num_in_osds":12,"osd_in_since":1711771125,"num_remapped_pgs":0} 2024-03-30T03:59:03.076 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: purged_snaps scrub starts 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: purged_snaps scrub ok 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] boot 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1824250187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: purged_snaps scrub starts 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: purged_snaps scrub ok 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] boot 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1824250187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: purged_snaps scrub starts 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: purged_snaps scrub ok 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: osd.11 [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] boot 2024-03-30T03:59:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: osdmap e75: 12 total, 12 up, 12 in 2024-03-30T03:59:03.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T03:59:03.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1824250187' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-30T03:59:03.438 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:59:04.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 -- 172.21.15.23:0/3926049243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4100e60 msgr2=0x7faad41012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:04.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 --2- 172.21.15.23:0/3926049243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4100e60 0x7faad41012e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faabc0099c0 tx=0x7faabc009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 -- 172.21.15.23:0/3926049243 shutdown_connections 2024-03-30T03:59:04.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 --2- 172.21.15.23:0/3926049243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 --2- 172.21.15.23:0/3926049243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4106ef0 0x7faad41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 --2- 172.21.15.23:0/3926049243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4100e60 0x7faad41012e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 -- 172.21.15.23:0/3926049243 >> 172.21.15.23:0/3926049243 conn(0x7faad4077fa0 msgr2=0x7faad40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 -- 172.21.15.23:0/3926049243 shutdown_connections 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.603+0000 7faad9e00700 1 -- 172.21.15.23:0/3926049243 wait complete. 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.604+0000 7faad9e00700 1 Processor -- start 2024-03-30T03:59:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 -- start start 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad419e320 con 0x7faad4106ef0 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad419e490 con 0x7faad4100e60 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad9e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad419e650 con 0x7faad4101820 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faacbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56474/0 (socket says 172.21.15.23:56474) 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faacbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34666/0 (socket says 172.21.15.23:34666) 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faad2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:44374/0 (socket says 172.21.15.23:44374) 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.605+0000 7faacbe00700 1 -- 172.21.15.23:0/4174204270 learned_addr learned my addr 172.21.15.23:0/4174204270 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 -- 172.21.15.23:0/4174204270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 msgr2=0x7faad4199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 -- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 msgr2=0x7faad419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 -- 172.21.15.23:0/4174204270 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0095f0 con 0x7faad4101820 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad2a00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad3400700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faacbe00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faabc005b40 tx=0x7faabc00b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:04.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc01d070 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad419e8d0 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faabc00be00 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.606+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc00fb00 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.607+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad41a6190 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.608+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad41024f0 con 0x7faad4101820 2024-03-30T03:59:04.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.610+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7faabc003680 con 0x7faad4101820 2024-03-30T03:59:04.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.611+0000 7faad8a00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 0x7faab80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.611+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faabc0cd660 con 0x7faad4101820 2024-03-30T03:59:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.611+0000 7faad2a00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 0x7faab80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.612+0000 7faad2a00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 0x7faab80a35e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faac4011490 tx=0x7faac4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.613+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faabc097280 con 0x7faad4101820 2024-03-30T03:59:04.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.827+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad419a730 con 0x7faad4101820 2024-03-30T03:59:04.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.828+0000 7faad8a00700 1 -- 172.21.15.23:0/4174204270 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18420 (secure 0 0 0) 0x7faabc09a8a0 con 0x7faad4101820 2024-03-30T03:59:04.828 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:04.828 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:03.946304+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:04.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 msgr2=0x7faab80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:04.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 0x7faab80a35e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7faac4011490 tx=0x7faac4009260 comp rx=0 tx=0).stop 2024-03-30T03:59:04.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 msgr2=0x7faad4199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faabc005b40 tx=0x7faabc00b9c0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 shutdown_connections 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faad4106ef0 0x7faad419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad4100e60 0x7faad4199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faab80a1120 0x7faab80a35e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 --2- 172.21.15.23:0/4174204270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faad4101820 0x7faad4199960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.830+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 >> 172.21.15.23:0/4174204270 conn(0x7faad4077fa0 msgr2=0x7faad410c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.831+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 shutdown_connections 2024-03-30T03:59:04.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:04.831+0000 7faad9e00700 1 -- 172.21.15.23:0/4174204270 wait complete. 2024-03-30T03:59:05.203 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:04 smithi023 ceph-mon[38254]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T03:59:05.203 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:04 smithi023 ceph-mon[38254]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:05.203 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:04 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4174204270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:05.226 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T03:59:05.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-03-30T03:59:05.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:04 smithi193 ceph-mon[79581]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T03:59:05.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:04 smithi193 ceph-mon[79581]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:05.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:04 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4174204270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:05.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:04 smithi154 ceph-mon[79161]: osdmap e76: 12 total, 12 up, 12 in 2024-03-30T03:59:05.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:04 smithi154 ceph-mon[79161]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:05.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:04 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4174204270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:05.586 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:59:06.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 -- 172.21.15.23:0/3292366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40101820 msgr2=0x7f6b4010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 --2- 172.21.15.23:0/3292366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40101820 0x7f6b4010dc10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6b3800b0d0 tx=0x7f6b3800b3e0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 -- 172.21.15.23:0/3292366942 shutdown_connections 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 --2- 172.21.15.23:0/3292366942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40101820 0x7f6b4010dc10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 --2- 172.21.15.23:0/3292366942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 0x7f6b401072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 --2- 172.21.15.23:0/3292366942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40100e60 0x7f6b401012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 -- 172.21.15.23:0/3292366942 >> 172.21.15.23:0/3292366942 conn(0x7f6b40077fa0 msgr2=0x7f6b400783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 -- 172.21.15.23:0/3292366942 shutdown_connections 2024-03-30T03:59:06.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.729+0000 7f6b48200700 1 -- 172.21.15.23:0/3292366942 wait complete. 2024-03-30T03:59:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.730+0000 7f6b48200700 1 Processor -- start 2024-03-30T03:59:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.730+0000 7f6b48200700 1 -- start start 2024-03-30T03:59:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 0x7f6b401999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 0x7f6b4019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4019e3a0 con 0x7f6b40100e60 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4019e510 con 0x7f6b40101820 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b48200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4019e6d0 con 0x7f6b40106ef0 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56506/0 (socket says 172.21.15.23:56506) 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 -- 172.21.15.23:0/3184665916 learned_addr learned my addr 172.21.15.23:0/3184665916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b44e00700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 0x7f6b401999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b46200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 0x7f6b4019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 -- 172.21.15.23:0/3184665916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 msgr2=0x7f6b4019dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 0x7f6b4019dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 -- 172.21.15.23:0/3184665916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 msgr2=0x7f6b401999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.731+0000 7f6b45800700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 0x7f6b401999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b45800700 1 -- 172.21.15.23:0/3184665916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b38009d00 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b44e00700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 0x7f6b401999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b45800700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400e9a0 tx=0x7f6b3400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34009790 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b34010430 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4019e9b0 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b34018680 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.732+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b401a6210 con 0x7f6b40100e60 2024-03-30T03:59:06.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.733+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b401024f0 con 0x7f6b40100e60 2024-03-30T03:59:06.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.735+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6b340105a0 con 0x7f6b40100e60 2024-03-30T03:59:06.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.736+0000 7f6b3e000700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 0x7f6b280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:06.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.737+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6b34014070 con 0x7f6b40100e60 2024-03-30T03:59:06.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.737+0000 7f6b44e00700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 0x7f6b280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:06.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.737+0000 7f6b44e00700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 0x7f6b280a35e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b4019aa00 tx=0x7f6b2c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:06.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.741+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6b340529f0 con 0x7f6b40100e60 2024-03-30T03:59:06.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.950+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b4019a7b0 con 0x7f6b40100e60 2024-03-30T03:59:06.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.951+0000 7f6b3e000700 1 -- 172.21.15.23:0/3184665916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b34096220 con 0x7f6b40100e60 2024-03-30T03:59:06.951 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 msgr2=0x7f6b280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 0x7f6b280a35e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6b4019aa00 tx=0x7f6b2c009260 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 msgr2=0x7f6b401994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b3400e9a0 tx=0x7f6b3400ecb0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 shutdown_connections 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b280a1120 0x7f6b280a35e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b40106ef0 0x7f6b4019dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b40100e60 0x7f6b401994a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 --2- 172.21.15.23:0/3184665916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b40101820 0x7f6b401999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 >> 172.21.15.23:0/3184665916 conn(0x7f6b40077fa0 msgr2=0x7f6b400ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 shutdown_connections 2024-03-30T03:59:06.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:06.953+0000 7f6b48200700 1 -- 172.21.15.23:0/3184665916 wait complete. 2024-03-30T03:59:07.179 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:06 smithi023 ceph-mon[38254]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:07.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:06 smithi154 ceph-mon[79161]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:07.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:06 smithi193 ceph-mon[79581]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:07.366 INFO:tasks.cephadm:Adding mds.a on smithi023 2024-03-30T03:59:07.366 INFO:tasks.cephadm:Adding mds.d on smithi023 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.g on smithi023 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.j on smithi023 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.b on smithi154 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.e on smithi154 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.h on smithi154 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.k on smithi154 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-03-30T03:59:07.367 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-03-30T03:59:07.367 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-03-30T03:59:07.599 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T03:59:07.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3184665916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:08.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3184665916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:08.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3184665916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:08.606 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 -- 172.21.15.193:0/952970917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de8106b40 msgr2=0x7f8de8108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:08.606 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 --2- 172.21.15.193:0/952970917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de8106b40 0x7f8de8108f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8de000b0a0 tx=0x7f8de000b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.606 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 -- 172.21.15.193:0/952970917 shutdown_connections 2024-03-30T03:59:08.606 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 --2- 172.21.15.193:0/952970917 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de8106b40 0x7f8de8108f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.606 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 --2- 172.21.15.193:0/952970917 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de81018e0 0x7f8de8103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 --2- 172.21.15.193:0/952970917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8104210 0x7f8de8106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 -- 172.21.15.193:0/952970917 >> 172.21.15.193:0/952970917 conn(0x7f8de80fb2b0 msgr2=0x7f8de80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 -- 172.21.15.193:0/952970917 shutdown_connections 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.606+0000 7f8deedb4700 1 -- 172.21.15.193:0/952970917 wait complete. 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 Processor -- start 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 -- start start 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 0x7f8de80ff890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8106b40 0x7f8de80ffdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de8100480 con 0x7f8de8104210 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81005f0 con 0x7f8de8106b40 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8deedb4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de81a0f00 con 0x7f8de81018e0 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:08.607 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:40120/0 (socket says 172.21.15.193:40120) 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 -- 172.21.15.193:0/485821030 learned_addr learned my addr 172.21.15.193:0/485821030 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8ded351700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8106b40 0x7f8de80ffdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 -- 172.21.15.193:0/485821030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8106b40 msgr2=0x7f8de80ffdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8106b40 0x7f8de80ffdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 -- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 msgr2=0x7f8de80ff890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8de7fff700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 0x7f8de80ff890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 0x7f8de80ff890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8decb50700 1 -- 172.21.15.193:0/485821030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de0009d00 con 0x7f8de81018e0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.607+0000 7f8de7fff700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 0x7f8de80ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8decb50700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00d950 tx=0x7f8ddc00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc009940 con 0x7f8de81018e0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ddc010460 con 0x7f8de81018e0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de81a1100 con 0x7f8de81018e0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00f620 con 0x7f8de81018e0 2024-03-30T03:59:08.608 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.608+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de81a1600 con 0x7f8de81018e0 2024-03-30T03:59:08.609 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.609+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de80fce30 con 0x7f8de81018e0 2024-03-30T03:59:08.609 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.610+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8ddc0108d0 con 0x7f8de81018e0 2024-03-30T03:59:08.609 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.610+0000 7f8de5ffb700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 0x7f8dd80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:08.610 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.611+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8ddc0cc2d0 con 0x7f8de81018e0 2024-03-30T03:59:08.610 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.611+0000 7f8de7fff700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 0x7f8dd80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:08.610 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.611+0000 7f8de7fff700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 0x7f8dd80a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005fd0 tx=0x7f8dd4005dc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:08.613 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.615+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8ddc095f00 con 0x7f8de81018e0 2024-03-30T03:59:08.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.815+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de80612f0 con 0x7f8dd80a1120 2024-03-30T03:59:08.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.825+0000 7f8de5ffb700 1 -- 172.21.15.193:0/485821030 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f8de80612f0 con 0x7f8dd80a1120 2024-03-30T03:59:08.823 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-03-30T03:59:08.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 msgr2=0x7f8dd80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:08.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 0x7f8dd80a35e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4005fd0 tx=0x7f8dd4005dc0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 msgr2=0x7f8de8101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:08.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00d950 tx=0x7f8ddc00dc60 comp rx=0 tx=0).stop 2024-03-30T03:59:08.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 shutdown_connections 2024-03-30T03:59:08.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8de8106b40 0x7f8de80ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8dd80a1120 0x7f8dd80a35e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81018e0 0x7f8de8101150 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 --2- 172.21.15.193:0/485821030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8de8104210 0x7f8de80ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:08.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 >> 172.21.15.193:0/485821030 conn(0x7f8de80fb2b0 msgr2=0x7f8de8102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:08.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 shutdown_connections 2024-03-30T03:59:08.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-30T03:59:08.827+0000 7f8deedb4700 1 -- 172.21.15.193:0/485821030 wait complete. 2024-03-30T03:59:09.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:08 smithi193 ceph-mon[79581]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:09.136 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:09.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:08 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:09.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:08 smithi154 ceph-mon[79161]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:09.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:09.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:08 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:09.203 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.a.service 2024-03-30T03:59:09.208 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.d.service 2024-03-30T03:59:09.252 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.g.service 2024-03-30T03:59:09.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:08 smithi023 ceph-mon[38254]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:09.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:09.264 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:08 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:09.267 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.j.service 2024-03-30T03:59:09.281 INFO:journalctl@ceph.mds.a.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:59:09.283 DEBUG:teuthology.orchestra.run.smithi154:mds.b> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.b.service 2024-03-30T03:59:09.286 DEBUG:teuthology.orchestra.run.smithi154:mds.e> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.e.service 2024-03-30T03:59:09.325 INFO:journalctl@ceph.mds.b.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:59:09.326 DEBUG:teuthology.orchestra.run.smithi154:mds.h> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.h.service 2024-03-30T03:59:09.329 INFO:journalctl@ceph.mds.d.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:59:09.339 INFO:journalctl@ceph.mds.g.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:59:09.356 INFO:journalctl@ceph.mds.e.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:59:09.357 INFO:journalctl@ceph.mds.j.smithi023.stdout:-- Logs begin at Sat 2024-03-30 03:36:18 UTC. -- 2024-03-30T03:59:09.357 DEBUG:teuthology.orchestra.run.smithi154:mds.k> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.k.service 2024-03-30T03:59:09.385 INFO:journalctl@ceph.mds.h.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:59:09.386 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.c.service 2024-03-30T03:59:09.388 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.f.service 2024-03-30T03:59:09.413 INFO:journalctl@ceph.mds.k.smithi154.stdout:-- Logs begin at Sat 2024-03-30 03:28:34 UTC. -- 2024-03-30T03:59:09.419 INFO:journalctl@ceph.mds.c.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:59:09.420 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.i.service 2024-03-30T03:59:09.446 INFO:journalctl@ceph.mds.f.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:59:09.447 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.l.service 2024-03-30T03:59:09.472 INFO:journalctl@ceph.mds.i.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:59:09.474 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-30T03:59:09.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:09.502 INFO:journalctl@ceph.mds.l.smithi193.stdout:-- Logs begin at Sat 2024-03-30 03:28:12 UTC. -- 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 -- 172.21.15.23:0/829992500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04074d90 msgr2=0x7f4a040731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 --2- 172.21.15.23:0/829992500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04074d90 0x7f4a040731e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f49f4009990 tx=0x7f49f4009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 -- 172.21.15.23:0/829992500 shutdown_connections 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 --2- 172.21.15.23:0/829992500 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a04108b50 0x7f4a0410afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 --2- 172.21.15.23:0/829992500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04074d90 0x7f4a040731e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 --2- 172.21.15.23:0/829992500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a040737b0 0x7f4a04073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 -- 172.21.15.23:0/829992500 >> 172.21.15.23:0/829992500 conn(0x7f4a040fc1f0 msgr2=0x7f4a040fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 -- 172.21.15.23:0/829992500 shutdown_connections 2024-03-30T03:59:09.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.793+0000 7f4a09200700 1 -- 172.21.15.23:0/829992500 wait complete. 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 Processor -- start 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 -- start start 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a040737b0 0x7f4a04198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 0x7f4a04199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a041a08e0 con 0x7f4a040737b0 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a041a0a50 con 0x7f4a04074d90 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.794+0000 7f4a09200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a041a0c10 con 0x7f4a04108b50 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:38756/0 (socket says 172.21.15.23:38756) 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 -- 172.21.15.23:0/1869922405 learned_addr learned my addr 172.21.15.23:0/1869922405 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:09.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a01e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a040737b0 0x7f4a04198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a01400700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 0x7f4a04199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 -- 172.21.15.23:0/1869922405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 msgr2=0x7f4a04199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 0x7f4a04199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 -- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a040737b0 msgr2=0x7f4a04198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a040737b0 0x7f4a04198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 -- 172.21.15.23:0/1869922405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc009c40 con 0x7f4a04108b50 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a01400700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 0x7f4a04199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f4a02800700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c370 tx=0x7f49fc00c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc017070 con 0x7f4a04108b50 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.795+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49fc00f040 con 0x7f4a04108b50 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.796+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f40095f0 con 0x7f4a04108b50 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.796+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00e050 con 0x7f4a04108b50 2024-03-30T03:59:09.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.796+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a041a11b0 con 0x7f4a04108b50 2024-03-30T03:59:09.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.797+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a04107960 con 0x7f4a04108b50 2024-03-30T03:59:09.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.798+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f49fc004bd0 con 0x7f4a04108b50 2024-03-30T03:59:09.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.800+0000 7f49faa00700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 0x7f49e80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:09.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.800+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f49fc0cbdb0 con 0x7f4a04108b50 2024-03-30T03:59:09.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.800+0000 7f4a01e00700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 0x7f49e80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:09.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.801+0000 7f4a01e00700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 0x7f49e80a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b680 tx=0x7f49f4005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:09.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:09.802+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f49fc095940 con 0x7f4a04108b50 2024-03-30T03:59:10.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.063+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a04067a10 con 0x7f4a04108b50 2024-03-30T03:59:10.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.065+0000 7f49faa00700 1 -- 172.21.15.23:0/1869922405 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18420 (secure 0 0 0) 0x7f49fc098f60 con 0x7f4a04108b50 2024-03-30T03:59:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:10.065 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:03.946304+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 msgr2=0x7f49e80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 0x7f49e80a3660 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f49f400b680 tx=0x7f49f4005fd0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 msgr2=0x7f4a041a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c370 tx=0x7f49fc00c680 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 shutdown_connections 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f49e80a11a0 0x7f49e80a3660 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a04108b50 0x7f4a041a0310 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a040737b0 0x7f4a04198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 --2- 172.21.15.23:0/1869922405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a04074d90 0x7f4a04199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 >> 172.21.15.23:0/1869922405 conn(0x7f4a040fc1f0 msgr2=0x7f4a04104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 shutdown_connections 2024-03-30T03:59:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.067+0000 7f4a09200700 1 -- 172.21.15.23:0/1869922405 wait complete. 2024-03-30T03:59:10.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T03:59:10.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T03:59:10.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:59:10.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:10.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:09 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:59:10.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:10.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:09 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:09 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:10.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 -- 172.21.15.23:0/2181609152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc1041a0 msgr2=0x7efebc106590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 --2- 172.21.15.23:0/2181609152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc1041a0 0x7efebc106590 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efea8009990 tx=0x7efea8009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 -- 172.21.15.23:0/2181609152 shutdown_connections 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 --2- 172.21.15.23:0/2181609152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 0x7efebc108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 --2- 172.21.15.23:0/2181609152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc101870 0x7efebc103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 --2- 172.21.15.23:0/2181609152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc1041a0 0x7efebc106590 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 -- 172.21.15.23:0/2181609152 >> 172.21.15.23:0/2181609152 conn(0x7efebc0fb200 msgr2=0x7efebc0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 -- 172.21.15.23:0/2181609152 shutdown_connections 2024-03-30T03:59:10.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.291+0000 7efec4600700 1 -- 172.21.15.23:0/2181609152 wait complete. 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.292+0000 7efec4600700 1 Processor -- start 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.292+0000 7efec4600700 1 -- start start 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc1041a0 0x7efebc1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 0x7efebc19e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc19e880 con 0x7efebc101870 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc19e9f0 con 0x7efebc106ad0 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec4600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc19ebb0 con 0x7efebc1041a0 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33402/0 (socket says 172.21.15.23:33402) 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 -- 172.21.15.23:0/4056456333 learned_addr learned my addr 172.21.15.23:0/4056456333 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1200700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc1041a0 0x7efebc1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec2600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 0x7efebc19e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 -- 172.21.15.23:0/4056456333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc1041a0 msgr2=0x7efebc1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc1041a0 0x7efebc1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 -- 172.21.15.23:0/4056456333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 msgr2=0x7efebc19e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 0x7efebc19e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.293+0000 7efec1c00700 1 -- 172.21.15.23:0/4056456333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009520 con 0x7efebc101870 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efec1c00700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efeb000eb30 tx=0x7efeb000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0009880 con 0x7efebc101870 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efeb00044e0 con 0x7efebc101870 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea80095f0 con 0x7efebc101870 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0003e90 con 0x7efebc101870 2024-03-30T03:59:10.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.294+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc19f1b0 con 0x7efebc101870 2024-03-30T03:59:10.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.295+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc0fcd80 con 0x7efebc101870 2024-03-30T03:59:10.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.297+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7efeb0004000 con 0x7efebc101870 2024-03-30T03:59:10.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.298+0000 7efeba000700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 0x7efea40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.298+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efeb0014070 con 0x7efebc101870 2024-03-30T03:59:10.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.298+0000 7efec1200700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 0x7efea40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.299+0000 7efec1200700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 0x7efea40a3660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efea800b2e0 tx=0x7efea8005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:10.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.302+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efeb0096470 con 0x7efebc101870 2024-03-30T03:59:10.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.516+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efebc10c780 con 0x7efebc101870 2024-03-30T03:59:10.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.516+0000 7efeba000700 1 -- 172.21.15.23:0/4056456333 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7efeb0099a90 con 0x7efebc101870 2024-03-30T03:59:10.516 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T03:59:10.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.518+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 msgr2=0x7efea40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.518+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 0x7efea40a3660 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efea800b2e0 tx=0x7efea8005fd0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.518+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 msgr2=0x7efebc196d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.518+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efeb000eb30 tx=0x7efeb000ee40 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 shutdown_connections 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efebc106ad0 0x7efebc19e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efebc101870 0x7efebc196d90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efea40a11a0 0x7efea40a3660 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 --2- 172.21.15.23:0/4056456333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc1041a0 0x7efebc1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 >> 172.21.15.23:0/4056456333 conn(0x7efebc0fb200 msgr2=0x7efebc102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 shutdown_connections 2024-03-30T03:59:10.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.519+0000 7efec4600700 1 -- 172.21.15.23:0/4056456333 wait complete. 2024-03-30T03:59:10.538 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:10.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.736+0000 7f3803400700 1 -- 172.21.15.23:0/3392922828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 msgr2=0x7f37fc1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.736+0000 7f3803400700 1 --2- 172.21.15.23:0/3392922828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc1010a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f37f00099c0 tx=0x7f37f0009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 -- 172.21.15.23:0/3392922828 shutdown_connections 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 --2- 172.21.15.23:0/3392922828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 --2- 172.21.15.23:0/3392922828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc1010a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 --2- 172.21.15.23:0/3392922828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 -- 172.21.15.23:0/3392922828 >> 172.21.15.23:0/3392922828 conn(0x7f37fc0fc1f0 msgr2=0x7f37fc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 -- 172.21.15.23:0/3392922828 shutdown_connections 2024-03-30T03:59:10.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.737+0000 7f3803400700 1 -- 172.21.15.23:0/3392922828 wait complete. 2024-03-30T03:59:10.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 Processor -- start 2024-03-30T03:59:10.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 -- start start 2024-03-30T03:59:10.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc073e80 con 0x7f37fc100c80 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc073ff0 con 0x7f37fc101e70 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3803400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc1a0c80 con 0x7f37fc102810 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3801400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3800a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3801400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:38798/0 (socket says 172.21.15.23:38798) 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f3801400700 1 -- 172.21.15.23:0/3412310137 learned_addr learned my addr 172.21.15.23:0/3412310137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.738+0000 7f37fbe00700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 -- 172.21.15.23:0/3412310137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 msgr2=0x7f37fc0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 -- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 msgr2=0x7f37fc074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:10.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 -- 172.21.15.23:0/3412310137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4009c40 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f37fbe00700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3801400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c370 tx=0x7f37f400c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4017070 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f00095f0 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37f400f040 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f400e050 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.739+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37fc1a11d0 con 0x7f37fc102810 2024-03-30T03:59:10.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.740+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37fc108100 con 0x7f37fc102810 2024-03-30T03:59:10.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.743+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f37f4004bd0 con 0x7f37fc102810 2024-03-30T03:59:10.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.744+0000 7f37f9600700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 0x7f37e40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:10.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.744+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37f40cbe40 con 0x7f37fc102810 2024-03-30T03:59:10.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.744+0000 7f3800a00700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 0x7f37e40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:10.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.745+0000 7f3800a00700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 0x7f37e40a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37f0003730 tx=0x7f37f0000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:10.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:10.745+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f37f40959d0 con 0x7f37fc102810 2024-03-30T03:59:10.930 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 03:59:10 smithi193 systemd[1]: Starting Ceph mds.l for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:11.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.016+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37fc067a10 con 0x7f37fc102810 2024-03-30T03:59:11.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.017+0000 7f37f9600700 1 -- 172.21.15.23:0/3412310137 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f37f4098ff0 con 0x7f37fc102810 2024-03-30T03:59:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:11.017 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-30T03:59:11.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.019+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 msgr2=0x7f37e40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.019+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 0x7f37e40a35e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f37f0003730 tx=0x7f37f0000bc0 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.019+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 msgr2=0x7f37fc0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.019+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c370 tx=0x7f37f400c680 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 shutdown_connections 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f37e40a1120 0x7f37e40a35e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37fc102810 0x7f37fc0736e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37fc100c80 0x7f37fc074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 --2- 172.21.15.23:0/3412310137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37fc101e70 0x7f37fc0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 >> 172.21.15.23:0/3412310137 conn(0x7f37fc0fc1f0 msgr2=0x7f37fc105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 shutdown_connections 2024-03-30T03:59:11.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:11.020+0000 7f3803400700 1 -- 172.21.15.23:0/3412310137 wait complete. 2024-03-30T03:59:11.021 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2024-03-30T03:59:11.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:10 smithi154 ceph-mon[79161]: Deploying daemon mds.l on smithi193 2024-03-30T03:59:11.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:10 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1869922405' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:11.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:10 smithi154 ceph-mon[79161]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:11.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:10 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4056456333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:11.196 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mds[115577]: -- 172.21.15.193:0/1638097011 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561aed05c1a0 con 0x561aed032c00 2024-03-30T03:59:11.196 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: starting mds.l at 2024-03-30T03:59:11.196 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:10 smithi193 ceph-mon[79581]: Deploying daemon mds.l on smithi193 2024-03-30T03:59:11.197 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:10 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1869922405' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:11.197 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:10 smithi193 ceph-mon[79581]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:11.197 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:10 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4056456333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:10 smithi023 ceph-mon[38254]: Deploying daemon mds.l on smithi193 2024-03-30T03:59:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:10 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1869922405' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:10 smithi023 ceph-mon[38254]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:10 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4056456333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:11.493 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 03:59:11 smithi193 bash[115540]: d316f730517ca87534760aef612862f9ff7e0a55e6bacb1c28eb85cbdf2f26dd 2024-03-30T03:59:11.493 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 03:59:11 smithi193 systemd[1]: Started Ceph mds.l for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:12.221 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3412310137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:12.221 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6832/2731954066,v1:172.21.15.193:6833/2731954066] up:boot 2024-03-30T03:59:12.221 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: fsmap 1 up:standby 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:12.222 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:11 smithi154 ceph-mon[79161]: Deploying daemon mds.k on smithi154 2024-03-30T03:59:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3412310137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6832/2731954066,v1:172.21.15.193:6833/2731954066] up:boot 2024-03-30T03:59:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: fsmap 1 up:standby 2024-03-30T03:59:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T03:59:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:12.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:11 smithi193 ceph-mon[79581]: Deploying daemon mds.k on smithi154 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3412310137' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6832/2731954066,v1:172.21.15.193:6833/2731954066] up:boot 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: fsmap 1 up:standby 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:12.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:11 smithi023 ceph-mon[38254]: Deploying daemon mds.k on smithi154 2024-03-30T03:59:12.681 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 03:59:12 smithi154 systemd[1]: Starting Ceph mds.k for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:13.040 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:13.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:12 smithi154 ceph-mon[79161]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:13.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:12 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.182 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 03:59:12 smithi154 ceph-mds[115260]: -- 172.21.15.154:0/3357290859 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564de4b3c1a0 con 0x564de4b12c00 2024-03-30T03:59:13.182 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 03:59:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: starting mds.k at 2024-03-30T03:59:13.182 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 03:59:12 smithi154 bash[115222]: 2c26fe29f57e37ab8c3ee7d60769863afa061ed117aa4e0e8a803f2df65e38ee 2024-03-30T03:59:13.182 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 03:59:12 smithi154 systemd[1]: Started Ceph mds.k for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:12 smithi193 ceph-mon[79581]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:12 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 -- 172.21.15.23:0/3413961159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 msgr2=0x7f6e5c073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 --2- 172.21.15.23:0/3413961159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c073c90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6e48009990 tx=0x7f6e48009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 -- 172.21.15.23:0/3413961159 shutdown_connections 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 --2- 172.21.15.23:0/3413961159 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c10d250 0x7f6e5c10f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 --2- 172.21.15.23:0/3413961159 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c074df0 0x7f6e5c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 --2- 172.21.15.23:0/3413961159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c073c90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.247+0000 7f6e62200700 1 -- 172.21.15.23:0/3413961159 >> 172.21.15.23:0/3413961159 conn(0x7f6e5c0fc1f0 msgr2=0x7f6e5c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.248+0000 7f6e62200700 1 -- 172.21.15.23:0/3413961159 shutdown_connections 2024-03-30T03:59:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.248+0000 7f6e62200700 1 -- 172.21.15.23:0/3413961159 wait complete. 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 Processor -- start 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 -- start start 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 0x7f6e5c19d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10d250 0x7f6e5c1a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1a4d50 con 0x7f6e5c074df0 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1a4e90 con 0x7f6e5c073810 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e62200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1a5020 con 0x7f6e5c10d250 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e5aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 0x7f6e5c19d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e5b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e5aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 0x7f6e5c19d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33438/0 (socket says 172.21.15.23:33438) 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e5aa00700 1 -- 172.21.15.23:0/3092935221 learned_addr learned my addr 172.21.15.23:0/3092935221 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.249+0000 7f6e5be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10d250 0x7f6e5c1a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 -- 172.21.15.23:0/3092935221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10d250 msgr2=0x7f6e5c1a4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10d250 0x7f6e5c1a4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 -- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 msgr2=0x7f6e5c19d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 0x7f6e5c19d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 -- 172.21.15.23:0/3092935221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480095f0 con 0x7f6e5c073810 2024-03-30T03:59:13.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e5b400700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c19d220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00e9f0 tx=0x7f6e4c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c00c9c0 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4c010430 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1a5300 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4c018680 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.250+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c1a5810 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.251+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c1034b0 con 0x7f6e5c073810 2024-03-30T03:59:13.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.253+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6e4c010860 con 0x7f6e5c073810 2024-03-30T03:59:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.254+0000 7f6e53e00700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 0x7f6e400a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.255+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6e4c014070 con 0x7f6e5c073810 2024-03-30T03:59:13.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.255+0000 7f6e5aa00700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 0x7f6e400a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:13.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.255+0000 7f6e5aa00700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 0x7f6e400a35e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800b2e0 tx=0x7f6e48005fb0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:13.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.257+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6e4c096130 con 0x7f6e5c073810 2024-03-30T03:59:13.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:12 smithi023 ceph-mon[38254]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:12 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:13.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.542+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e5c112d10 con 0x7f6e5c073810 2024-03-30T03:59:13.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.543+0000 7f6e53e00700 1 -- 172.21.15.23:0/3092935221 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f6e4c099750 con 0x7f6e5c073810 2024-03-30T03:59:13.542 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:13.542 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 msgr2=0x7f6e400a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 0x7f6e400a35e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6e4800b2e0 tx=0x7f6e48005fb0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 msgr2=0x7f6e5c19d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c19d220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c00e9f0 tx=0x7f6e4c00ed00 comp rx=0 tx=0).stop 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 shutdown_connections 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6e400a1120 0x7f6e400a35e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c10d250 0x7f6e5c1a4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e5c073810 0x7f6e5c19d220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 --2- 172.21.15.23:0/3092935221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6e5c074df0 0x7f6e5c19d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 >> 172.21.15.23:0/3092935221 conn(0x7f6e5c0fc1f0 msgr2=0x7f6e5c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 shutdown_connections 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:13.545+0000 7f6e62200700 1 -- 172.21.15.23:0/3092935221 wait complete. 2024-03-30T03:59:13.546 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:boot 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: fsmap 2 up:standby 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: Deploying daemon mds.j on smithi023 2024-03-30T03:59:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:13 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3092935221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:boot 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: fsmap 2 up:standby 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: Deploying daemon mds.j on smithi023 2024-03-30T03:59:14.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:13 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3092935221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:boot 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: fsmap 2 up:standby 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: Deploying daemon mds.j on smithi023 2024-03-30T03:59:14.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:13 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3092935221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:14.835 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 03:59:14 smithi023 systemd[1]: Starting Ceph mds.j for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:15.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:14 smithi023 ceph-mon[38254]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:15.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:14 smithi193 ceph-mon[79581]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:14 smithi154 ceph-mon[79161]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:15.567 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:15.629 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 03:59:15 smithi023 ceph-mds[87356]: -- 172.21.15.23:0/990791557 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555bb48f41a0 con 0x555bb48cc800 2024-03-30T03:59:15.629 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 03:59:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-j[87333]: starting mds.j at 2024-03-30T03:59:15.629 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 03:59:15 smithi023 bash[87317]: db07187900b018cdd3f16ed3a4d671ddca97a39fca77660f7f4d1e6fa57ddd9f 2024-03-30T03:59:15.629 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 03:59:15 smithi023 systemd[1]: Started Ceph mds.j for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:15.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.774+0000 7f5337200700 1 -- 172.21.15.23:0/1431287860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 msgr2=0x7f533010f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.774+0000 7f5337200700 1 --2- 172.21.15.23:0/1431287860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f533010f640 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f532800b0a0 tx=0x7f532800b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 -- 172.21.15.23:0/1431287860 shutdown_connections 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 --2- 172.21.15.23:0/1431287860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f533010f640 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 --2- 172.21.15.23:0/1431287860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f5330073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 --2- 172.21.15.23:0/1431287860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f5330073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 -- 172.21.15.23:0/1431287860 >> 172.21.15.23:0/1431287860 conn(0x7f53300fc1f0 msgr2=0x7f53300fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 -- 172.21.15.23:0/1431287860 shutdown_connections 2024-03-30T03:59:15.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.775+0000 7f5337200700 1 -- 172.21.15.23:0/1431287860 wait complete. 2024-03-30T03:59:15.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 Processor -- start 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 -- start start 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f533019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f53301a4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a4e40 con 0x7f533010d250 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a4f80 con 0x7f5330074df0 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5337200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53301a5110 con 0x7f5330073810 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f532fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f532fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:38852/0 (socket says 172.21.15.23:38852) 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f532fe00700 1 -- 172.21.15.23:0/1106971793 learned_addr learned my addr 172.21.15.23:0/1106971793 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.776+0000 7f5335200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f53301a4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:15.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532f400700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f533019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 -- 172.21.15.23:0/1106971793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 msgr2=0x7f533019d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f533019d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 -- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 msgr2=0x7f53301a4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f53301a4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 -- 172.21.15.23:0/1106971793 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5328009d00 con 0x7f5330073810 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532f400700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f533019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532fe00700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f532000e9a0 tx=0x7f532000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320009790 con 0x7f5330073810 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53301a53f0 con 0x7f5330073810 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f532000cbd0 con 0x7f5330073810 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.777+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320018680 con 0x7f5330073810 2024-03-30T03:59:15.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.778+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53301a5950 con 0x7f5330073810 2024-03-30T03:59:15.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.779+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53301034b0 con 0x7f5330073810 2024-03-30T03:59:15.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.781+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f532000cd40 con 0x7f5330073810 2024-03-30T03:59:15.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.782+0000 7f532cc00700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 0x7f53140a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:15.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.782+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5320014070 con 0x7f5330073810 2024-03-30T03:59:15.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.782+0000 7f532f400700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 0x7f53140a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:15.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.783+0000 7f532f400700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 0x7f53140a35e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5330074b10 tx=0x7f531c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:15.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:15.784+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f53200965e0 con 0x7f5330073810 2024-03-30T03:59:16.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.057+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5330112d10 con 0x7f5330073810 2024-03-30T03:59:16.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.058+0000 7f532cc00700 1 -- 172.21.15.23:0/1106971793 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7f5320099c00 con 0x7f5330073810 2024-03-30T03:59:16.058 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:16.058 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T03:59:16.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 msgr2=0x7f53140a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:16.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 0x7f53140a35e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5330074b10 tx=0x7f531c009260 comp rx=0 tx=0).stop 2024-03-30T03:59:16.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 msgr2=0x7f533019d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f532000e9a0 tx=0x7f532000ecb0 comp rx=0 tx=0).stop 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 shutdown_connections 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f533010d250 0x7f53301a4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f53140a1120 0x7f53140a35e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5330073810 0x7f533019d310 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 --2- 172.21.15.23:0/1106971793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5330074df0 0x7f533019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.060+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 >> 172.21.15.23:0/1106971793 conn(0x7f53300fc1f0 msgr2=0x7f53301005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.061+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 shutdown_connections 2024-03-30T03:59:16.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:16.061+0000 7f5337200700 1 -- 172.21.15.23:0/1106971793 wait complete. 2024-03-30T03:59:16.062 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: Deploying daemon mds.d on smithi023 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6834/1931657906,v1:172.21.15.23:6835/1931657906] up:boot 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: fsmap 3 up:standby 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1106971793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:16.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:16 smithi193 ceph-mon[79581]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:16.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:16.750 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: Deploying daemon mds.d on smithi023 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6834/1931657906,v1:172.21.15.23:6835/1931657906] up:boot 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: fsmap 3 up:standby 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1106971793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:16.751 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:16 smithi023 ceph-mon[38254]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:16.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:16.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: Deploying daemon mds.d on smithi023 2024-03-30T03:59:16.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6834/1931657906,v1:172.21.15.23:6835/1931657906] up:boot 2024-03-30T03:59:16.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: fsmap 3 up:standby 2024-03-30T03:59:16.933 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T03:59:16.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1106971793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:16.934 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:16 smithi154 ceph-mon[79161]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:17.322 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 03:59:17 smithi023 systemd[1]: Starting Ceph mds.d for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:18.081 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:18.142 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 03:59:17 smithi023 ceph-mds[87744]: -- 172.21.15.23:0/3571323139 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5562044441a0 con 0x55620441cc00 2024-03-30T03:59:18.142 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 03:59:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-d[87720]: starting mds.d at 2024-03-30T03:59:18.142 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 03:59:17 smithi023 bash[87705]: 207612e9ad0dfa3742bdc5f924024d390a2f7731718b9833080952b47ff86309 2024-03-30T03:59:18.142 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 03:59:17 smithi023 systemd[1]: Started Ceph mds.d for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 -- 172.21.15.23:0/974981167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 msgr2=0x7fed801010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 --2- 172.21.15.23:0/974981167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed801010a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed740099c0 tx=0x7fed74009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 -- 172.21.15.23:0/974981167 shutdown_connections 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 --2- 172.21.15.23:0/974981167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80102810 0x7fed801090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 --2- 172.21.15.23:0/974981167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed801010a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 --2- 172.21.15.23:0/974981167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80101e70 0x7fed801022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 -- 172.21.15.23:0/974981167 >> 172.21.15.23:0/974981167 conn(0x7fed800fc1f0 msgr2=0x7fed800fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 -- 172.21.15.23:0/974981167 shutdown_connections 2024-03-30T03:59:18.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.286+0000 7fed88600700 1 -- 172.21.15.23:0/974981167 wait complete. 2024-03-30T03:59:18.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.287+0000 7fed88600700 1 Processor -- start 2024-03-30T03:59:18.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.287+0000 7fed88600700 1 -- start start 2024-03-30T03:59:18.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:18.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80101e70 0x7fed801993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:18.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80102810 0x7fed801a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed801a09d0 con 0x7fed80100c80 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed801a0b10 con 0x7fed80102810 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed88600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed801a0ca0 con 0x7fed80101e70 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed85c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed85c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:52788/0 (socket says 172.21.15.23:52788) 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed85c00700 1 -- 172.21.15.23:0/3201994049 learned_addr learned my addr 172.21.15.23:0/3201994049 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed85200700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80101e70 0x7fed801993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.288+0000 7fed86600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80102810 0x7fed801a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 -- 172.21.15.23:0/3201994049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80101e70 msgr2=0x7fed801993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80101e70 0x7fed801993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 -- 172.21.15.23:0/3201994049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80102810 msgr2=0x7fed801a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:18.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80102810 0x7fed801a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 -- 172.21.15.23:0/3201994049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740095f0 con 0x7fed80100c80 2024-03-30T03:59:18.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed85c00700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed74006010 tx=0x7fed74005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:18.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7401d070 con 0x7fed80100c80 2024-03-30T03:59:18.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed801a0f20 con 0x7fed80100c80 2024-03-30T03:59:18.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed74003f90 con 0x7fed80100c80 2024-03-30T03:59:18.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74022cb0 con 0x7fed80100c80 2024-03-30T03:59:18.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.289+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed801a12e0 con 0x7fed80100c80 2024-03-30T03:59:18.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.290+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed80067a10 con 0x7fed80100c80 2024-03-30T03:59:18.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.292+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fed74022470 con 0x7fed80100c80 2024-03-30T03:59:18.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.293+0000 7fed7e000700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 0x7fed680a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:18.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.294+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed740cdf70 con 0x7fed80100c80 2024-03-30T03:59:18.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.294+0000 7fed85200700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 0x7fed680a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:18.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.294+0000 7fed85200700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 0x7fed680a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009a10 tx=0x7fed6c009190 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:18.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.296+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fed74097b90 con 0x7fed80100c80 2024-03-30T03:59:18.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.562+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed8010c7b0 con 0x7fed80100c80 2024-03-30T03:59:18.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.563+0000 7fed7e000700 1 -- 172.21.15.23:0/3201994049 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3531 (secure 0 0 0) 0x7fed74027de0 con 0x7fed80100c80 2024-03-30T03:59:18.563 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:18.564 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-30T03:59:18.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.565+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 msgr2=0x7fed680a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:18.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 0x7fed680a35e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009a10 tx=0x7fed6c009190 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 msgr2=0x7fed80198ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed74006010 tx=0x7fed74005740 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 shutdown_connections 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80102810 0x7fed801a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed80100c80 0x7fed80198ea0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fed680a1120 0x7fed680a35e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 --2- 172.21.15.23:0/3201994049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed80101e70 0x7fed801993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 >> 172.21.15.23:0/3201994049 conn(0x7fed800fc1f0 msgr2=0x7fed80072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:18.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 shutdown_connections 2024-03-30T03:59:18.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:18.566+0000 7fed88600700 1 -- 172.21.15.23:0/3201994049 wait complete. 2024-03-30T03:59:18.567 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: Deploying daemon mds.e on smithi154 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6836/180927964,v1:172.21.15.23:6837/180927964] up:boot 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: fsmap 4 up:standby 2024-03-30T03:59:19.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T03:59:19.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:19.183 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:18 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3201994049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:19.183 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 03:59:19 smithi154 systemd[1]: Starting Ceph mds.e for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: Deploying daemon mds.e on smithi154 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6836/180927964,v1:172.21.15.23:6837/180927964] up:boot 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: fsmap 4 up:standby 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:19.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:18 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3201994049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: Deploying daemon mds.e on smithi154 2024-03-30T03:59:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6836/180927964,v1:172.21.15.23:6837/180927964] up:boot 2024-03-30T03:59:19.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: fsmap 4 up:standby 2024-03-30T03:59:19.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T03:59:19.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:19.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:18 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3201994049' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:19.681 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 03:59:19 smithi154 ceph-mds[115602]: -- 172.21.15.154:0/2767135329 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe72b6a1a0 con 0x55fe72b40800 2024-03-30T03:59:19.681 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 03:59:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: starting mds.e at 2024-03-30T03:59:19.681 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 03:59:19 smithi154 bash[115564]: 6551cb6a691fea2196f99bda9c99de422f0685eedd469d2486078ca3841ddbba 2024-03-30T03:59:19.681 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 03:59:19 smithi154 systemd[1]: Started Ceph mds.e for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:20.584 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 -- 172.21.15.23:0/3767599258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 msgr2=0x7fbca41022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 --2- 172.21.15.23:0/3767599258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca41022d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbc900099c0 tx=0x7fbc90009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 -- 172.21.15.23:0/3767599258 shutdown_connections 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 --2- 172.21.15.23:0/3767599258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4102810 0x7fbca41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 --2- 172.21.15.23:0/3767599258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4100c80 0x7fbca41010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 --2- 172.21.15.23:0/3767599258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca41022d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 -- 172.21.15.23:0/3767599258 >> 172.21.15.23:0/3767599258 conn(0x7fbca40fc1f0 msgr2=0x7fbca40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:20.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 -- 172.21.15.23:0/3767599258 shutdown_connections 2024-03-30T03:59:20.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.795+0000 7fbcac400700 1 -- 172.21.15.23:0/3767599258 wait complete. 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 Processor -- start 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 -- start start 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4102810 0x7fbca40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca4073e80 con 0x7fbca4101e70 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca4073ff0 con 0x7fbca4102810 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.796+0000 7fbcac400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca41a0c60 con 0x7fbca4100c80 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44934/0 (socket says 172.21.15.23:44934) 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 -- 172.21.15.23:0/1474902293 learned_addr learned my addr 172.21.15.23:0/1474902293 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:20.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9000700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbcaa400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4102810 0x7fbca40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 -- 172.21.15.23:0/1474902293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4102810 msgr2=0x7fbca40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4102810 0x7fbca40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 -- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 msgr2=0x7fbca40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca40731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 -- 172.21.15.23:0/1474902293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc900095f0 con 0x7fbca4100c80 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9000700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca9a00700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800e970 tx=0x7fbc9800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.797+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98009790 con 0x7fbca4100c80 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.798+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca41a0e90 con 0x7fbca4100c80 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.798+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc9800c7e0 con 0x7fbca4100c80 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.798+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc98003c70 con 0x7fbca4100c80 2024-03-30T03:59:20.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.798+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca41a12b0 con 0x7fbca4100c80 2024-03-30T03:59:20.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.799+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca4067a80 con 0x7fbca4100c80 2024-03-30T03:59:20.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.801+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fbc9800cc50 con 0x7fbca4100c80 2024-03-30T03:59:20.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.802+0000 7fbca2000700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 0x7fbc8c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:20.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.802+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbc98014070 con 0x7fbca4100c80 2024-03-30T03:59:20.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.802+0000 7fbca9000700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 0x7fbc8c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:20.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.803+0000 7fbca9000700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 0x7fbc8c0a35e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc90006010 tx=0x7fbc9000b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:20.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:20.808+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbc98096160 con 0x7fbca4100c80 2024-03-30T03:59:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:20.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: Deploying daemon mds.f on smithi193 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:boot 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: fsmap 5 up:standby 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T03:59:20.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:20 smithi154 ceph-mon[79161]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 03:59:20 smithi193 systemd[1]: Starting Ceph mds.f for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:20.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:20.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: Deploying daemon mds.f on smithi193 2024-03-30T03:59:20.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:boot 2024-03-30T03:59:20.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: fsmap 5 up:standby 2024-03-30T03:59:20.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T03:59:20.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:20 smithi193 ceph-mon[79581]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: Deploying daemon mds.f on smithi193 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:boot 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: fsmap 5 up:standby 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T03:59:21.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:20 smithi023 ceph-mon[38254]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:21.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.073+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbca410c7c0 con 0x7fbca4100c80 2024-03-30T03:59:21.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.075+0000 7fbca2000700 1 -- 172.21.15.23:0/1474902293 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4291 (secure 0 0 0) 0x7fbca410c7c0 con 0x7fbca4100c80 2024-03-30T03:59:21.073 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:21.074 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2024-03-30T03:59:21.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.076+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 msgr2=0x7fbc8c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:21.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.076+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 0x7fbc8c0a35e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc90006010 tx=0x7fbc9000b540 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.076+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 msgr2=0x7fbca4074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.076+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc9800e970 tx=0x7fbc9800ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 shutdown_connections 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbca4102810 0x7fbca40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbc8c0a1120 0x7fbc8c0a35e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbca4100c80 0x7fbca4074b10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 --2- 172.21.15.23:0/1474902293 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbca4101e70 0x7fbca40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 >> 172.21.15.23:0/1474902293 conn(0x7fbca40fc1f0 msgr2=0x7fbca4105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 shutdown_connections 2024-03-30T03:59:21.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:21.077+0000 7fbcac400700 1 -- 172.21.15.23:0/1474902293 wait complete. 2024-03-30T03:59:21.078 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2024-03-30T03:59:21.493 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mds[115919]: -- 172.21.15.193:0/350405474 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5601450021a0 con 0x560144fd8800 2024-03-30T03:59:21.493 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: starting mds.f at 2024-03-30T03:59:21.493 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 03:59:21 smithi193 bash[115881]: d97e8fe86f0986741a5634a3462bb94c33da018125447e613e0a8d0bf2fa04ec 2024-03-30T03:59:21.493 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 03:59:21 smithi193 systemd[1]: Started Ceph mds.f for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:22.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1474902293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:22.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:22.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:22.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:22.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:21 smithi193 ceph-mon[79581]: Deploying daemon mds.c on smithi193 2024-03-30T03:59:22.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1474902293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:22.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:21 smithi023 ceph-mon[38254]: Deploying daemon mds.c on smithi193 2024-03-30T03:59:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1474902293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:22.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:22.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:22.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:22.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:21 smithi154 ceph-mon[79161]: Deploying daemon mds.c on smithi193 2024-03-30T03:59:22.545 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 03:59:22 smithi193 systemd[1]: Starting Ceph mds.c for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:23.097 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:boot 2024-03-30T03:59:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-mon[79581]: fsmap 6 up:standby 2024-03-30T03:59:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T03:59:23.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-mon[79581]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:23.243 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-mds[116261]: -- 172.21.15.193:0/1201857918 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fc35ee1a0 con 0x558fc35c4800 2024-03-30T03:59:23.244 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 03:59:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: starting mds.c at 2024-03-30T03:59:23.244 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 03:59:22 smithi193 bash[116223]: a0df9def68eb8a8ea1c75db791820876597fdcad8fc9fb650364a4fadb7914b7 2024-03-30T03:59:23.244 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 03:59:22 smithi193 systemd[1]: Started Ceph mds.c for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:22 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:boot 2024-03-30T03:59:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:22 smithi023 ceph-mon[38254]: fsmap 6 up:standby 2024-03-30T03:59:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:22 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T03:59:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:22 smithi023 ceph-mon[38254]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:23.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 -- 172.21.15.23:0/4055935476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c101e70 msgr2=0x7f0a0c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:23.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 --2- 172.21.15.23:0/4055935476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c101e70 0x7f0a0c1022d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09f80099c0 tx=0x7f09f8009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 -- 172.21.15.23:0/4055935476 shutdown_connections 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 --2- 172.21.15.23:0/4055935476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c102810 0x7f0a0c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 --2- 172.21.15.23:0/4055935476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c100c80 0x7f0a0c1010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 --2- 172.21.15.23:0/4055935476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c101e70 0x7f0a0c1022d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 -- 172.21.15.23:0/4055935476 >> 172.21.15.23:0/4055935476 conn(0x7f0a0c0fc1f0 msgr2=0x7f0a0c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 -- 172.21.15.23:0/4055935476 shutdown_connections 2024-03-30T03:59:23.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.304+0000 7f0a11200700 1 -- 172.21.15.23:0/4055935476 wait complete. 2024-03-30T03:59:23.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.305+0000 7f0a11200700 1 Processor -- start 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 -- start start 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c101e70 0x7f0a0c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 0x7f0a0c1a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c1a0950 con 0x7f0a0c102810 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c1a0a90 con 0x7f0a0c100c80 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a11200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c1a0c20 con 0x7f0a0c101e70 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a09e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a09e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58586/0 (socket says 172.21.15.23:58586) 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a09e00700 1 -- 172.21.15.23:0/3889448571 learned_addr learned my addr 172.21.15.23:0/3889448571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a0a800700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 0x7f0a0c1a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:23.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.306+0000 7f0a09400700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c101e70 0x7f0a0c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 -- 172.21.15.23:0/3889448571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c101e70 msgr2=0x7f0a0c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c101e70 0x7f0a0c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 -- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 msgr2=0x7f0a0c1a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 0x7f0a0c1a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 -- 172.21.15.23:0/3889448571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a04009c40 con 0x7f0a0c100c80 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a0a800700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 0x7f0a0c1a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a09e00700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00e970 tx=0x7f09fc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc009790 con 0x7f0a0c100c80 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09fc00c7c0 con 0x7f0a0c100c80 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f80095f0 con 0x7f0a0c100c80 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc003c20 con 0x7f0a0c100c80 2024-03-30T03:59:23.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.307+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c1a11d0 con 0x7f0a0c100c80 2024-03-30T03:59:23.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.308+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0c067a80 con 0x7f0a0c100c80 2024-03-30T03:59:23.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.310+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f09fc00c930 con 0x7f0a0c100c80 2024-03-30T03:59:23.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.311+0000 7f0a02a00700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 0x7f09f00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:23.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.312+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09fc014070 con 0x7f0a0c100c80 2024-03-30T03:59:23.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.312+0000 7f0a09400700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 0x7f09f00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:23.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.312+0000 7f0a09400700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 0x7f09f00a35e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f09f8003730 tx=0x7f09f800b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:23.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.317+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f09fc096750 con 0x7f0a0c100c80 2024-03-30T03:59:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:22 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:boot 2024-03-30T03:59:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:22 smithi154 ceph-mon[79161]: fsmap 6 up:standby 2024-03-30T03:59:23.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:22 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T03:59:23.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:22 smithi154 ceph-mon[79161]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.582+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a0c10c770 con 0x7f0a0c100c80 2024-03-30T03:59:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.583+0000 7f0a02a00700 1 -- 172.21.15.23:0/3889448571 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7f09fc099d70 con 0x7f0a0c100c80 2024-03-30T03:59:23.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:23.583 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 msgr2=0x7f09f00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 0x7f09f00a35e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f09f8003730 tx=0x7f09f800b540 comp rx=0 tx=0).stop 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 msgr2=0x7f0a0c198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00e970 tx=0x7f09fc00ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 shutdown_connections 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a0c102810 0x7f0a0c1a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a0c100c80 0x7f0a0c198e20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09f00a1120 0x7f09f00a35e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 --2- 172.21.15.23:0/3889448571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a0c101e70 0x7f0a0c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 >> 172.21.15.23:0/3889448571 conn(0x7f0a0c0fc1f0 msgr2=0x7f0a0c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 shutdown_connections 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:23.585+0000 7f0a11200700 1 -- 172.21.15.23:0/3889448571 wait complete. 2024-03-30T03:59:23.586 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-30T03:59:24.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6836/1829293538,v1:172.21.15.193:6837/1829293538] up:boot 2024-03-30T03:59:24.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: fsmap 7 up:standby 2024-03-30T03:59:24.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T03:59:24.045 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: Deploying daemon mds.h on smithi154 2024-03-30T03:59:24.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:23 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3889448571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:24.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6836/1829293538,v1:172.21.15.193:6837/1829293538] up:boot 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: fsmap 7 up:standby 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: Deploying daemon mds.h on smithi154 2024-03-30T03:59:24.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:23 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3889448571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6836/1829293538,v1:172.21.15.193:6837/1829293538] up:boot 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: fsmap 7 up:standby 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: Deploying daemon mds.h on smithi154 2024-03-30T03:59:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3889448571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:24.295 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 03:59:24 smithi154 systemd[1]: Starting Ceph mds.h for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:24.931 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mds[115943]: -- 172.21.15.154:0/4248053021 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5590a65e01a0 con 0x5590a65b6c00 2024-03-30T03:59:24.931 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: starting mds.h at 2024-03-30T03:59:24.931 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 03:59:24 smithi154 bash[115905]: bf0cc1692e78d3ae86b517df99d6a168d27520c7ab407df2cbeb8f05166e63d9 2024-03-30T03:59:24.931 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 03:59:24 smithi154 systemd[1]: Started Ceph mds.h for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:25.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:24 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:24 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:25.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:24 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:25.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 -- 172.21.15.23:0/400933604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944108b50 msgr2=0x7fc94410afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 --2- 172.21.15.23:0/400933604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944108b50 0x7fc94410afc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00b0a0 tx=0x7fc93c00b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 -- 172.21.15.23:0/400933604 shutdown_connections 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 --2- 172.21.15.23:0/400933604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944108b50 0x7fc94410afc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 --2- 172.21.15.23:0/400933604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944074d90 0x7fc9440731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 --2- 172.21.15.23:0/400933604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 0x7fc944073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 -- 172.21.15.23:0/400933604 >> 172.21.15.23:0/400933604 conn(0x7fc9440fc210 msgr2=0x7fc9440fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 -- 172.21.15.23:0/400933604 shutdown_connections 2024-03-30T03:59:25.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.816+0000 7fc94ac00700 1 -- 172.21.15.23:0/400933604 wait complete. 2024-03-30T03:59:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 Processor -- start 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 -- start start 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 0x7fc944198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944108b50 0x7fc9441a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9441a0a70 con 0x7fc944108b50 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9441a0bb0 con 0x7fc9440737b0 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.817+0000 7fc94ac00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9441a0d40 con 0x7fc944074d90 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44972/0 (socket says 172.21.15.23:44972) 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 -- 172.21.15.23:0/1112450699 learned_addr learned my addr 172.21.15.23:0/1112450699 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc948c00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944108b50 0x7fc9441a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943e00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 0x7fc944198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 -- 172.21.15.23:0/1112450699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 msgr2=0x7fc944198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 0x7fc944198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 -- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944108b50 msgr2=0x7fc9441a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944108b50 0x7fc9441a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 -- 172.21.15.23:0/1112450699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c009d00 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc943400700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc93000e9a0 tx=0x7fc93000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.818+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930009790 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.819+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc930010430 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.819+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc930018680 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.819+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9441a0fc0 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.819+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9441a1450 con 0x7fc944074d90 2024-03-30T03:59:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.820+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944107960 con 0x7fc944074d90 2024-03-30T03:59:25.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.822+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc9300105a0 con 0x7fc944074d90 2024-03-30T03:59:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.823+0000 7fc940c00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 0x7fc9280a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.823+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc930014070 con 0x7fc944074d90 2024-03-30T03:59:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.823+0000 7fc943e00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 0x7fc9280a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:25.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.824+0000 7fc943e00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 0x7fc9280a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc934009c60 tx=0x7fc934009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:25.825+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc9300961d0 con 0x7fc944074d90 2024-03-30T03:59:26.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.092+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9440671d0 con 0x7fc944074d90 2024-03-30T03:59:26.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.092+0000 7fc940c00700 1 -- 172.21.15.23:0/1112450699 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7fc9300997f0 con 0x7fc944074d90 2024-03-30T03:59:26.091 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:26.091 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T03:59:26.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 msgr2=0x7fc9280a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:26.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 0x7fc9280a3660 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc934009c60 tx=0x7fc934009260 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 msgr2=0x7fc944199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc93000e9a0 tx=0x7fc93000ecb0 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 shutdown_connections 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc944108b50 0x7fc9441a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9440737b0 0x7fc944198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9280a11a0 0x7fc9280a3660 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 --2- 172.21.15.23:0/1112450699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc944074d90 0x7fc944199480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 >> 172.21.15.23:0/1112450699 conn(0x7fc9440fc210 msgr2=0x7fc944104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 shutdown_connections 2024-03-30T03:59:26.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:26.095+0000 7fc94ac00700 1 -- 172.21.15.23:0/1112450699 wait complete. 2024-03-30T03:59:26.095 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 9 2024-03-30T03:59:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:25 smithi193 ceph-mon[79581]: Deploying daemon mds.i on smithi193 2024-03-30T03:59:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:25 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:boot 2024-03-30T03:59:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:25 smithi193 ceph-mon[79581]: fsmap 8 up:standby 2024-03-30T03:59:26.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:25 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T03:59:26.244 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 03:59:26 smithi193 systemd[1]: Starting Ceph mds.i for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:25 smithi023 ceph-mon[38254]: Deploying daemon mds.i on smithi193 2024-03-30T03:59:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:25 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:boot 2024-03-30T03:59:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:25 smithi023 ceph-mon[38254]: fsmap 8 up:standby 2024-03-30T03:59:26.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:25 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T03:59:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:25 smithi154 ceph-mon[79161]: Deploying daemon mds.i on smithi193 2024-03-30T03:59:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:25 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:boot 2024-03-30T03:59:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:25 smithi154 ceph-mon[79161]: fsmap 8 up:standby 2024-03-30T03:59:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:25 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T03:59:26.742 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 03:59:26 smithi193 ceph-mds[116602]: -- 172.21.15.193:0/1107829785 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562bf4d061a0 con 0x562bf4cdcc00 2024-03-30T03:59:26.743 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 03:59:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: starting mds.i at 2024-03-30T03:59:26.743 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 03:59:26 smithi193 bash[116564]: 7de756b6354fdff829e1949947a15bf1d4d70075fee585a12546e047195e06d8 2024-03-30T03:59:26.743 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 03:59:26 smithi193 systemd[1]: Started Ceph mds.i for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1112450699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:27 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1112450699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:27.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:27.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:27 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1112450699' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:27 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:28.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:28.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:28 smithi023 ceph-mon[38254]: Deploying daemon mds.g on smithi023 2024-03-30T03:59:28.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:28 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6838/606583058,v1:172.21.15.193:6839/606583058] up:boot 2024-03-30T03:59:28.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:28 smithi023 ceph-mon[38254]: fsmap 9 up:standby 2024-03-30T03:59:28.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:28 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T03:59:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:28 smithi154 ceph-mon[79161]: Deploying daemon mds.g on smithi023 2024-03-30T03:59:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:28 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6838/606583058,v1:172.21.15.193:6839/606583058] up:boot 2024-03-30T03:59:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:28 smithi154 ceph-mon[79161]: fsmap 9 up:standby 2024-03-30T03:59:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:28 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 -- 172.21.15.23:0/2326025447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 msgr2=0x7f3f2c10f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 --2- 172.21.15.23:0/2326025447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c10f640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3f2400b0a0 tx=0x7f3f2400b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 -- 172.21.15.23:0/2326025447 shutdown_connections 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 --2- 172.21.15.23:0/2326025447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c10f640 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 --2- 172.21.15.23:0/2326025447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c074df0 0x7f3f2c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 --2- 172.21.15.23:0/2326025447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c073810 0x7f3f2c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 -- 172.21.15.23:0/2326025447 >> 172.21.15.23:0/2326025447 conn(0x7f3f2c0fc1f0 msgr2=0x7f3f2c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 -- 172.21.15.23:0/2326025447 shutdown_connections 2024-03-30T03:59:28.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.454+0000 7f3f34400700 1 -- 172.21.15.23:0/2326025447 wait complete. 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 Processor -- start 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 -- start start 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 0x7f3f2c19d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c074df0 0x7f3f2c19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1a4e30 con 0x7f3f2c10d250 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1a4f70 con 0x7f3f2c074df0 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f34400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c1a5100 con 0x7f3f2c073810 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f32400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:28.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f31a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 0x7f3f2c19d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f31000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c074df0 0x7f3f2c19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f31a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 0x7f3f2c19d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45644/0 (socket says 172.21.15.23:45644) 2024-03-30T03:59:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f31a00700 1 -- 172.21.15.23:0/1580132807 learned_addr learned my addr 172.21.15.23:0/1580132807 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:28.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.456+0000 7f3f32400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48106/0 (socket says 172.21.15.23:48106) 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 -- 172.21.15.23:0/1580132807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 msgr2=0x7f3f2c19d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 0x7f3f2c19d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 -- 172.21.15.23:0/1580132807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c074df0 msgr2=0x7f3f2c19d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c074df0 0x7f3f2c19d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 -- 172.21.15.23:0/1580132807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24009d00 con 0x7f3f2c10d250 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f32400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f24007b90 tx=0x7f3f24007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2400e040 con 0x7f3f2c10d250 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f2400be40 con 0x7f3f2c10d250 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2401b460 con 0x7f3f2c10d250 2024-03-30T03:59:28.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.457+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c1a5380 con 0x7f3f2c10d250 2024-03-30T03:59:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.458+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c1a5750 con 0x7f3f2c10d250 2024-03-30T03:59:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.459+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c1034b0 con 0x7f3f2c10d250 2024-03-30T03:59:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.461+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f3f2401d470 con 0x7f3f2c10d250 2024-03-30T03:59:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.463+0000 7f3f2a000700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 0x7f3f140a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.463+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f240cdda0 con 0x7f3f2c10d250 2024-03-30T03:59:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.463+0000 7f3f31a00700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 0x7f3f140a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.464+0000 7f3f31a00700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 0x7f3f140a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009d20 tx=0x7f3f20009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:28.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.466+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3f24097ae0 con 0x7f3f2c10d250 2024-03-30T03:59:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:28 smithi193 ceph-mon[79581]: Deploying daemon mds.g on smithi023 2024-03-30T03:59:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:28 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6838/606583058,v1:172.21.15.193:6839/606583058] up:boot 2024-03-30T03:59:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:28 smithi193 ceph-mon[79581]: fsmap 9 up:standby 2024-03-30T03:59:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:28 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T03:59:28.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.737+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f2c067a10 con 0x7f3f2c10d250 2024-03-30T03:59:28.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.738+0000 7f3f2a000700 1 -- 172.21.15.23:0/1580132807 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f3f2409b100 con 0x7f3f2c10d250 2024-03-30T03:59:28.736 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:28.736 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T03:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 msgr2=0x7f3f140a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 0x7f3f140a35e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009d20 tx=0x7f3f20009260 comp rx=0 tx=0).stop 2024-03-30T03:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 msgr2=0x7f3f2c1a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f24007b90 tx=0x7f3f24007930 comp rx=0 tx=0).stop 2024-03-30T03:59:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 shutdown_connections 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f2c10d250 0x7f3f2c1a4860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3f140a1120 0x7f3f140a35e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f2c073810 0x7f3f2c19d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 --2- 172.21.15.23:0/1580132807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f2c074df0 0x7f3f2c19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.740+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 >> 172.21.15.23:0/1580132807 conn(0x7f3f2c0fc1f0 msgr2=0x7f3f2c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.741+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 shutdown_connections 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:28.741+0000 7f3f34400700 1 -- 172.21.15.23:0/1580132807 wait complete. 2024-03-30T03:59:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-30T03:59:29.014 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 03:59:28 smithi023 systemd[1]: Starting Ceph mds.g for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:29.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:29 smithi023 ceph-mon[38254]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:29.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1580132807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:29.272 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 03:59:29 smithi023 ceph-mds[88311]: -- 172.21.15.23:0/1259549700 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c1e453a1a0 con 0x55c1e4512800 2024-03-30T03:59:29.272 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 03:59:29 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: starting mds.g at 2024-03-30T03:59:29.272 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 03:59:29 smithi023 bash[88272]: 6a98f76e7b4da7cbde06d354a3914fc72fd6bf87db20e7f204723fce0c167d65 2024-03-30T03:59:29.272 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 03:59:29 smithi023 systemd[1]: Started Ceph mds.g for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:29 smithi154 ceph-mon[79161]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1580132807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:29 smithi193 ceph-mon[79581]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1580132807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:30.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:30.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: Deploying daemon mds.a on smithi023 2024-03-30T03:59:30.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6838/667186519,v1:172.21.15.23:6839/667186519] up:boot 2024-03-30T03:59:30.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: fsmap 10 up:standby 2024-03-30T03:59:30.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:30 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T03:59:30.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: Deploying daemon mds.a on smithi023 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6838/667186519,v1:172.21.15.23:6839/667186519] up:boot 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: fsmap 10 up:standby 2024-03-30T03:59:30.510 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:30 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: Deploying daemon mds.a on smithi023 2024-03-30T03:59:30.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6838/667186519,v1:172.21.15.23:6839/667186519] up:boot 2024-03-30T03:59:30.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: fsmap 10 up:standby 2024-03-30T03:59:30.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:30 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T03:59:30.761 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 -- 172.21.15.23:0/3015364952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 msgr2=0x7f1b70104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 --2- 172.21.15.23:0/3015364952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70104460 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c009990 tx=0x7f1b5c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 -- 172.21.15.23:0/3015364952 shutdown_connections 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 --2- 172.21.15.23:0/3015364952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b7010b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 --2- 172.21.15.23:0/3015364952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 0x7f1b701032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 --2- 172.21.15.23:0/3015364952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70104460 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 -- 172.21.15.23:0/3015364952 >> 172.21.15.23:0/3015364952 conn(0x7f1b700fe420 msgr2=0x7f1b70100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:30.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 -- 172.21.15.23:0/3015364952 shutdown_connections 2024-03-30T03:59:30.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.991+0000 7f1b77200700 1 -- 172.21.15.23:0/3015364952 wait complete. 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.992+0000 7f1b77200700 1 Processor -- start 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.992+0000 7f1b77200700 1 -- start start 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 0x7f1b70198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b701a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701a0950 con 0x7f1b70103fe0 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701a0a90 con 0x7f1b701049a0 2024-03-30T03:59:30.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b77200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701a0c20 con 0x7f1b70102e90 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48136/0 (socket says 172.21.15.23:48136) 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b75200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b701a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 -- 172.21.15.23:0/3300999315 learned_addr learned my addr 172.21.15.23:0/3300999315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6fe00700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 0x7f1b70198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 -- 172.21.15.23:0/3300999315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 msgr2=0x7f1b70198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 0x7f1b70198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 -- 172.21.15.23:0/3300999315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 msgr2=0x7f1b701a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b701a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b6f400700 1 -- 172.21.15.23:0/3300999315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c0095f0 con 0x7f1b70103fe0 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.993+0000 7f1b75200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b701a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:30.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b6f400700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00b620 tx=0x7f1b5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c01d070 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b5c022470 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c017650 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701a0ea0 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.994+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701a1210 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.995+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7010a2d0 con 0x7f1b70103fe0 2024-03-30T03:59:30.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.997+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f1b5c0225e0 con 0x7f1b70103fe0 2024-03-30T03:59:30.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.998+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 0x7f1b540a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:30.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.998+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1b5c0cd750 con 0x7f1b70103fe0 2024-03-30T03:59:30.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.999+0000 7f1b6fe00700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 0x7f1b540a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:30.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:30.999+0000 7f1b6fe00700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 0x7f1b540a35e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009c60 tx=0x7f1b60009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:31.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.003+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1b5c097370 con 0x7f1b70103fe0 2024-03-30T03:59:31.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.285+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b70067e60 con 0x7f1b70103fe0 2024-03-30T03:59:31.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.285+0000 7f1b6cc00700 1 -- 172.21.15.23:0/3300999315 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8085 (secure 0 0 0) 0x7f1b5c09a990 con 0x7f1b70103fe0 2024-03-30T03:59:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:31.284 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14544,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T03:59:31.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 msgr2=0x7f1b540a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:31.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 0x7f1b540a35e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009c60 tx=0x7f1b60009260 comp rx=0 tx=0).stop 2024-03-30T03:59:31.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 msgr2=0x7f1b70199360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:31.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c00b620 tx=0x7f1b5c004730 comp rx=0 tx=0).stop 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 shutdown_connections 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b701049a0 0x7f1b701a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1b540a1120 0x7f1b540a35e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b70102e90 0x7f1b70198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 --2- 172.21.15.23:0/3300999315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b70103fe0 0x7f1b70199360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 >> 172.21.15.23:0/3300999315 conn(0x7f1b700fe420 msgr2=0x7f1b70107410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 shutdown_connections 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:31.288+0000 7f1b77200700 1 -- 172.21.15.23:0/3300999315 wait complete. 2024-03-30T03:59:31.288 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-30T03:59:31.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:31 smithi023 ceph-mon[38254]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:31.484 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 03:59:31 smithi023 systemd[1]: Starting Ceph mds.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T03:59:31.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:31 smithi193 ceph-mon[79581]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:31 smithi154 ceph-mon[79161]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:32.022 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 03:59:31 smithi023 ceph-mds[88697]: -- 172.21.15.23:0/3777964974 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56319f2a81a0 con 0x56319f281000 2024-03-30T03:59:32.022 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 03:59:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: starting mds.a at 2024-03-30T03:59:32.022 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 03:59:31 smithi023 bash[88657]: 99643adfed19394108ce34d4ddb340fbd1eaa7fb4bdc0afdf907e74816e8ab18 2024-03-30T03:59:32.022 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 03:59:31 smithi023 systemd[1]: Started Ceph mds.a for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3300999315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:32.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:32.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:32 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3300999315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:32 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3300999315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-30T03:59:32.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:32 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:33.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:33.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:33 smithi023 ceph-mon[38254]: Deploying daemon mds.b on smithi154 2024-03-30T03:59:33.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:33 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6840/2954889723,v1:172.21.15.23:6841/2954889723] up:boot 2024-03-30T03:59:33.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:33 smithi023 ceph-mon[38254]: fsmap 11 up:standby 2024-03-30T03:59:33.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:33 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T03:59:33.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:33 smithi023 ceph-mon[38254]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 -- 172.21.15.23:0/3484350827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc074df0 msgr2=0x7f43cc073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 --2- 172.21.15.23:0/3484350827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc074df0 0x7f43cc073240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43c00098f0 tx=0x7f43c0009c00 comp rx=0 tx=0).stop 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 -- 172.21.15.23:0/3484350827 shutdown_connections 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 --2- 172.21.15.23:0/3484350827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc10afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 --2- 172.21.15.23:0/3484350827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc074df0 0x7f43cc073240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 --2- 172.21.15.23:0/3484350827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc073810 0x7f43cc073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 -- 172.21.15.23:0/3484350827 >> 172.21.15.23:0/3484350827 conn(0x7f43cc0fc1f0 msgr2=0x7f43cc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 -- 172.21.15.23:0/3484350827 shutdown_connections 2024-03-30T03:59:33.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.531+0000 7f43d4600700 1 -- 172.21.15.23:0/3484350827 wait complete. 2024-03-30T03:59:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.532+0000 7f43d4600700 1 Processor -- start 2024-03-30T03:59:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.532+0000 7f43d4600700 1 -- start start 2024-03-30T03:59:33.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 0x7f43cc198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 0x7f43cc1993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc1a09a0 con 0x7f43cc108b40 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc1a0ae0 con 0x7f43cc074df0 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d4600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43cc1a0c70 con 0x7f43cc073810 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 0x7f43cc198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 0x7f43cc1993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48176/0 (socket says 172.21.15.23:48176) 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d2600700 1 -- 172.21.15.23:0/1642083762 learned_addr learned my addr 172.21.15.23:0/1642083762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1200700 1 -- 172.21.15.23:0/1642083762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 msgr2=0x7f43cc198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:33.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1200700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 0x7f43cc198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1200700 1 -- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 msgr2=0x7f43cc1a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.533+0000 7f43d1200700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d1200700 1 -- 172.21.15.23:0/1642083762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b80095f0 con 0x7f43cc074df0 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d1c00700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 0x7f43cc198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d2600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d1200700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 0x7f43cc1993b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43b8005fd0 tx=0x7f43b800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b800cbd0 con 0x7f43cc074df0 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43b800cd30 con 0x7f43cc074df0 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43b8018710 con 0x7f43cc074df0 2024-03-30T03:59:33.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c0009520 con 0x7f43cc074df0 2024-03-30T03:59:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.534+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43cc1a1270 con 0x7f43cc074df0 2024-03-30T03:59:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.535+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43cc107900 con 0x7f43cc074df0 2024-03-30T03:59:33.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.537+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f43b8010980 con 0x7f43cc074df0 2024-03-30T03:59:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.538+0000 7f43ca000700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 0x7f43b40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.538+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f43b8014070 con 0x7f43cc074df0 2024-03-30T03:59:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.538+0000 7f43d1c00700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 0x7f43b40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:33.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.539+0000 7f43d1c00700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 0x7f43b40a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f43c000fa40 tx=0x7f43c0019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:33.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.542+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f43b8096480 con 0x7f43cc074df0 2024-03-30T03:59:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:33 smithi154 ceph-mon[79161]: Deploying daemon mds.b on smithi154 2024-03-30T03:59:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:33 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6840/2954889723,v1:172.21.15.23:6841/2954889723] up:boot 2024-03-30T03:59:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:33 smithi154 ceph-mon[79161]: fsmap 11 up:standby 2024-03-30T03:59:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:33 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T03:59:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:33 smithi154 ceph-mon[79161]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:33.682 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 03:59:33 smithi154 bash[116250]: f303fea401f7e38bd2cafc04b1fb0f669cd0aa37b8800dd1c48697a7704d7e04 2024-03-30T03:59:33.682 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 03:59:33 smithi154 systemd[1]: Started Ceph mds.b for eac7e298-ee48-11ee-b647-cb9ed24678a4. 2024-03-30T03:59:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:33 smithi193 ceph-mon[79581]: Deploying daemon mds.b on smithi154 2024-03-30T03:59:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:33 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6840/2954889723,v1:172.21.15.23:6841/2954889723] up:boot 2024-03-30T03:59:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:33 smithi193 ceph-mon[79581]: fsmap 11 up:standby 2024-03-30T03:59:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:33 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T03:59:33.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:33 smithi193 ceph-mon[79581]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:33.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.819+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43cc067a10 con 0x7f43cc074df0 2024-03-30T03:59:33.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.821+0000 7f43ca000700 1 -- 172.21.15.23:0/1642083762 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8843 (secure 0 0 0) 0x7f43b8099aa0 con 0x7f43cc074df0 2024-03-30T03:59:33.819 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:33.819 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14544,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2024-03-30T03:59:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 msgr2=0x7f43b40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 0x7f43b40a3660 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f43c000fa40 tx=0x7f43c0019040 comp rx=0 tx=0).stop 2024-03-30T03:59:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 msgr2=0x7f43cc1993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 0x7f43cc1993b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f43b8005fd0 tx=0x7f43b800ee70 comp rx=0 tx=0).stop 2024-03-30T03:59:33.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 shutdown_connections 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43cc108b40 0x7f43cc1a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f43b40a11a0 0x7f43b40a3660 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f43cc073810 0x7f43cc198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 --2- 172.21.15.23:0/1642083762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43cc074df0 0x7f43cc1993b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 >> 172.21.15.23:0/1642083762 conn(0x7f43cc0fc1f0 msgr2=0x7f43cc104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 shutdown_connections 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:33.823+0000 7f43d4600700 1 -- 172.21.15.23:0/1642083762 wait complete. 2024-03-30T03:59:33.823 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1642083762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:boot 2024-03-30T03:59:34.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: fsmap 12 up:standby 2024-03-30T03:59:34.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:34 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1642083762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:boot 2024-03-30T03:59:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: fsmap 12 up:standby 2024-03-30T03:59:34.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:34 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T03:59:34.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1642083762' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:boot 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: fsmap 12 up:standby 2024-03-30T03:59:34.932 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:34 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T03:59:35.843 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:36.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:36.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:35 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 -- 172.21.15.23:0/3373009714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184073810 msgr2=0x7f8184073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 --2- 172.21.15.23:0/3373009714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184073810 0x7f8184073c90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8170009990 tx=0x7f8170009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 -- 172.21.15.23:0/3373009714 shutdown_connections 2024-03-30T03:59:36.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 --2- 172.21.15.23:0/3373009714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184108b40 0x7f818410afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 --2- 172.21.15.23:0/3373009714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 --2- 172.21.15.23:0/3373009714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184073810 0x7f8184073c90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 -- 172.21.15.23:0/3373009714 >> 172.21.15.23:0/3373009714 conn(0x7f81840fc1f0 msgr2=0x7f81840fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:36.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 -- 172.21.15.23:0/3373009714 shutdown_connections 2024-03-30T03:59:36.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.051+0000 7f818bc00700 1 -- 172.21.15.23:0/3373009714 wait complete. 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.052+0000 7f818bc00700 1 Processor -- start 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 -- start start 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184108b40 0x7f81841a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841a08f0 con 0x7f8184073810 2024-03-30T03:59:36.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841a0a60 con 0x7f8184074df0 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f818bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81841a0c20 con 0x7f8184108b40 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48192/0 (socket says 172.21.15.23:48192) 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8183e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 -- 172.21.15.23:0/3107689979 learned_addr learned my addr 172.21.15.23:0/3107689979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189c00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184108b40 0x7f81841a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 -- 172.21.15.23:0/3107689979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184108b40 msgr2=0x7f81841a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184108b40 0x7f81841a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 -- 172.21.15.23:0/3107689979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 msgr2=0x7f8184199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.053+0000 7f8189200700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8189200700 1 -- 172.21.15.23:0/3107689979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700095f0 con 0x7f8184073810 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8183e00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:36.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8189200700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f817800e9a0 tx=0x7f817800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178009790 con 0x7f8184073810 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8178010430 con 0x7f8184073810 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81841a0f00 con 0x7f8184073810 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8178018680 con 0x7f8184073810 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.054+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81841a1330 con 0x7f8184073810 2024-03-30T03:59:36.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.055+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8184107900 con 0x7f8184073810 2024-03-30T03:59:36.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.057+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f81780105a0 con 0x7f8184073810 2024-03-30T03:59:36.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.058+0000 7f8181600700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 0x7f816c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.059+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8178014070 con 0x7f8184073810 2024-03-30T03:59:36.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.059+0000 7f8183e00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 0x7f816c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.059+0000 7f8183e00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 0x7f816c0a3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f817000b680 tx=0x7f8170005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:36.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.063+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f81780529f0 con 0x7f8184073810 2024-03-30T03:59:36.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:36.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:36.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:35 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:36.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:35 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:36.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.341+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8184067a10 con 0x7f8184073810 2024-03-30T03:59:36.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.342+0000 7f8181600700 1 -- 172.21.15.23:0/3107689979 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9604 (secure 0 0 0) 0x7f81780961a0 con 0x7f8184073810 2024-03-30T03:59:36.341 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:36.341 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14544,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2024-03-30T03:59:36.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 msgr2=0x7f816c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 0x7f816c0a3660 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f817000b680 tx=0x7f8170005fd0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 msgr2=0x7f8184198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f817800e9a0 tx=0x7f817800ecb0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 shutdown_connections 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f816c0a11a0 0x7f816c0a3660 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8184108b40 0x7f81841a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8184073810 0x7f8184198e00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 --2- 172.21.15.23:0/3107689979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8184074df0 0x7f8184199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 >> 172.21.15.23:0/3107689979 conn(0x7f81840fc1f0 msgr2=0x7f8184104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 shutdown_connections 2024-03-30T03:59:36.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.345+0000 7f818bc00700 1 -- 172.21.15.23:0/3107689979 wait complete. 2024-03-30T03:59:36.345 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2024-03-30T03:59:36.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:36.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.572+0000 7f6b42800700 1 -- 172.21.15.23:0/3607386598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 msgr2=0x7f6b3c073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.572+0000 7f6b42800700 1 --2- 172.21.15.23:0/3607386598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c073240 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009990 tx=0x7f6b2c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 -- 172.21.15.23:0/3607386598 shutdown_connections 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 --2- 172.21.15.23:0/3607386598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c10d250 0x7f6b3c10f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 --2- 172.21.15.23:0/3607386598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c073240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 --2- 172.21.15.23:0/3607386598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c073810 0x7f6b3c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 -- 172.21.15.23:0/3607386598 >> 172.21.15.23:0/3607386598 conn(0x7f6b3c0fc1f0 msgr2=0x7f6b3c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 -- 172.21.15.23:0/3607386598 shutdown_connections 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.573+0000 7f6b42800700 1 -- 172.21.15.23:0/3607386598 wait complete. 2024-03-30T03:59:36.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.574+0000 7f6b42800700 1 Processor -- start 2024-03-30T03:59:36.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.574+0000 7f6b42800700 1 -- start start 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c19d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 0x7f6b3c1a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c1a4d40 con 0x7f6b3c073810 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c1a4e80 con 0x7f6b3c074df0 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b42800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c1a5010 con 0x7f6b3c10d250 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48214/0 (socket says 172.21.15.23:48214) 2024-03-30T03:59:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b3b400700 1 -- 172.21.15.23:0/327757904 learned_addr learned my addr 172.21.15.23:0/327757904 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b3aa00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c19d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.575+0000 7f6b3be00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 0x7f6b3c1a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 -- 172.21.15.23:0/327757904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 msgr2=0x7f6b3c1a4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 0x7f6b3c1a4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 -- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 msgr2=0x7f6b3c19d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 -- 172.21.15.23:0/327757904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0095f0 con 0x7f6b3c074df0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3b400700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3be00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 0x7f6b3c1a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b3aa00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c19d750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800e9c0 tx=0x7f6b2800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28004220 con 0x7f6b3c074df0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b28004380 con 0x7f6b3c074df0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c1a52f0 con 0x7f6b3c074df0 2024-03-30T03:59:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.576+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b28003be0 con 0x7f6b3c074df0 2024-03-30T03:59:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.577+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c1a5800 con 0x7f6b3c074df0 2024-03-30T03:59:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.577+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c1034b0 con 0x7f6b3c074df0 2024-03-30T03:59:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.581+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6b28003d40 con 0x7f6b3c074df0 2024-03-30T03:59:36.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.582+0000 7f6b33e00700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 0x7f6b200a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:36.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.582+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6b28014070 con 0x7f6b3c074df0 2024-03-30T03:59:36.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.583+0000 7f6b3b400700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 0x7f6b200a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:36.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.583+0000 7f6b3b400700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 0x7f6b200a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00b2e0 tx=0x7f6b2c00b750 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:36.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.587+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6b28096310 con 0x7f6b3c074df0 2024-03-30T03:59:36.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.851+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b3c067a10 con 0x7f6b3c074df0 2024-03-30T03:59:36.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.853+0000 7f6b33e00700 1 -- 172.21.15.23:0/327757904 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+18420 (secure 0 0 0) 0x7f6b28099930 con 0x7f6b3c074df0 2024-03-30T03:59:36.852 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:36.852 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":76,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:03.946304+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 msgr2=0x7f6b200a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 0x7f6b200a35e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00b2e0 tx=0x7f6b2c00b750 comp rx=0 tx=0).stop 2024-03-30T03:59:36.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 msgr2=0x7f6b3c19d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c19d750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6b2800e9c0 tx=0x7f6b2800ecd0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 shutdown_connections 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6b200a1120 0x7f6b200a35e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b3c10d250 0x7f6b3c1a4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b3c073810 0x7f6b3c19d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 --2- 172.21.15.23:0/327757904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c074df0 0x7f6b3c19d750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 >> 172.21.15.23:0/327757904 conn(0x7f6b3c0fc1f0 msgr2=0x7f6b3c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.855+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 shutdown_connections 2024-03-30T03:59:36.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:36.856+0000 7f6b42800700 1 -- 172.21.15.23:0/327757904 wait complete. 2024-03-30T03:59:36.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-30T03:59:36.873 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T03:59:37.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.047 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:36 smithi023 ceph-mon[38254]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:37.047 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:36 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:37.047 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:36 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3107689979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 -- 172.21.15.23:0/2680390465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c102910 msgr2=0x7f7d3c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 --2- 172.21.15.23:0/2680390465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c102910 0x7f7d3c1091c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400b0a0 tx=0x7f7d3400b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 -- 172.21.15.23:0/2680390465 shutdown_connections 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 --2- 172.21.15.23:0/2680390465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c102910 0x7f7d3c1091c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 --2- 172.21.15.23:0/2680390465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c100d80 0x7f7d3c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 --2- 172.21.15.23:0/2680390465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c101f70 0x7f7d3c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 -- 172.21.15.23:0/2680390465 >> 172.21.15.23:0/2680390465 conn(0x7f7d3c0fc2f0 msgr2=0x7f7d3c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 -- 172.21.15.23:0/2680390465 shutdown_connections 2024-03-30T03:59:37.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.097+0000 7f7d42000700 1 -- 172.21.15.23:0/2680390465 wait complete. 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.098+0000 7f7d42000700 1 Processor -- start 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 -- start start 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c101f70 0x7f7d3c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c102910 0x7f7d3c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c073e80 con 0x7f7d3c102910 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c073ff0 con 0x7f7d3c101f70 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d42000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c1a0e50 con 0x7f7d3c100d80 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45756/0 (socket says 172.21.15.23:45756) 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3b400700 1 -- 172.21.15.23:0/862387985 learned_addr learned my addr 172.21.15.23:0/862387985 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3be00700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c102910 0x7f7d3c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3aa00700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c101f70 0x7f7d3c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3b400700 1 -- 172.21.15.23:0/862387985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c101f70 msgr2=0x7f7d3c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.099+0000 7f7d3b400700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c101f70 0x7f7d3c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d3b400700 1 -- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c102910 msgr2=0x7f7d3c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d3b400700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c102910 0x7f7d3c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d3b400700 1 -- 172.21.15.23:0/862387985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d34009d00 con 0x7f7d3c100d80 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d3b400700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e970 tx=0x7f7d2c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c009790 con 0x7f7d3c100d80 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d2c00c7e0 con 0x7f7d3c100d80 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c003c70 con 0x7f7d3c100d80 2024-03-30T03:59:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c1a1080 con 0x7f7d3c100d80 2024-03-30T03:59:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.100+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c1a1590 con 0x7f7d3c100d80 2024-03-30T03:59:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.101+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c108130 con 0x7f7d3c100d80 2024-03-30T03:59:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.103+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7d2c00cc50 con 0x7f7d3c100d80 2024-03-30T03:59:37.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.104+0000 7f7d33e00700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 0x7f7d200a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.105+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7d2c014070 con 0x7f7d3c100d80 2024-03-30T03:59:37.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.105+0000 7f7d3aa00700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 0x7f7d200a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.105+0000 7f7d3aa00700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 0x7f7d200a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009c60 tx=0x7f7d28009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:37.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.109+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d2c096200 con 0x7f7d3c100d80 2024-03-30T03:59:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:36 smithi154 ceph-mon[79161]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:36 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:37.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:36 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3107689979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:36 smithi193 ceph-mon[79581]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:36 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:36 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3107689979' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.344+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d3c107b10 con 0x7f7d3c100d80 2024-03-30T03:59:37.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.345+0000 7f7d33e00700 1 -- 172.21.15.23:0/862387985 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d2c099820 con 0x7f7d3c100d80 2024-03-30T03:59:37.343 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T03:59:37.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.347+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 msgr2=0x7f7d200a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.347+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 0x7f7d200a3660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009c60 tx=0x7f7d28009260 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.347+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 msgr2=0x7f7d3c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.347+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c00e970 tx=0x7f7d2c00ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 shutdown_connections 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d3c102910 0x7f7d3c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7d200a11a0 0x7f7d200a3660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d3c100d80 0x7f7d3c074b10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 --2- 172.21.15.23:0/862387985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d3c101f70 0x7f7d3c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 >> 172.21.15.23:0/862387985 conn(0x7f7d3c0fc2f0 msgr2=0x7f7d3c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 shutdown_connections 2024-03-30T03:59:37.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.348+0000 7f7d42000700 1 -- 172.21.15.23:0/862387985 wait complete. 2024-03-30T03:59:37.365 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 -- 172.21.15.23:0/1968097771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 msgr2=0x7fef381010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 --2- 172.21.15.23:0/1968097771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef381010a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fef28009920 tx=0x7fef28009c30 comp rx=0 tx=0).stop 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 -- 172.21.15.23:0/1968097771 shutdown_connections 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 --2- 172.21.15.23:0/1968097771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 0x7fef381090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 --2- 172.21.15.23:0/1968097771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef381010a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 --2- 172.21.15.23:0/1968097771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef381022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 -- 172.21.15.23:0/1968097771 >> 172.21.15.23:0/1968097771 conn(0x7fef380fc1f0 msgr2=0x7fef380fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 -- 172.21.15.23:0/1968097771 shutdown_connections 2024-03-30T03:59:37.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.587+0000 7fef3e600700 1 -- 172.21.15.23:0/1968097771 wait complete. 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.588+0000 7fef3e600700 1 Processor -- start 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.588+0000 7fef3e600700 1 -- start start 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef38199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 0x7fef381a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381a0a60 con 0x7fef38101e70 2024-03-30T03:59:37.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381a0ba0 con 0x7fef38100c80 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef3e600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef381a0d30 con 0x7fef38102810 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef36a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef38199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34704/0 (socket says 172.21.15.23:34704) 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 -- 172.21.15.23:0/3938806679 learned_addr learned my addr 172.21.15.23:0/3938806679 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37e00700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 0x7fef381a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 -- 172.21.15.23:0/3938806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 msgr2=0x7fef381a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 0x7fef381a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 -- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 msgr2=0x7fef38199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.589+0000 7fef37400700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef38199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef37400700 1 -- 172.21.15.23:0/3938806679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef28009520 con 0x7fef38100c80 2024-03-30T03:59:37.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef36a00700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef38199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:37.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef37400700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28009230 tx=0x7fef2800bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:37.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2801c070 con 0x7fef38100c80 2024-03-30T03:59:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef2800bba0 con 0x7fef38100c80 2024-03-30T03:59:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef381a0fb0 con 0x7fef38100c80 2024-03-30T03:59:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef28004930 con 0x7fef38100c80 2024-03-30T03:59:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.590+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef381a1320 con 0x7fef38100c80 2024-03-30T03:59:37.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.591+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef38067a80 con 0x7fef38100c80 2024-03-30T03:59:37.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.594+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fef28020070 con 0x7fef38100c80 2024-03-30T03:59:37.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.595+0000 7fef2fe00700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 0x7fef1c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:37.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.596+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fef280cd160 con 0x7fef38100c80 2024-03-30T03:59:37.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.596+0000 7fef36a00700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 0x7fef1c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:37.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.596+0000 7fef36a00700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 0x7fef1c0a35e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fef24005fd0 tx=0x7fef24009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:37.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.600+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef38067a80 con 0x7fef38100c80 2024-03-30T03:59:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.875+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fef3810c4e0 con 0x7fef38100c80 2024-03-30T03:59:37.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.876+0000 7fef2fe00700 1 -- 172.21.15.23:0/3938806679 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v76) v1 ==== 84+0+65 (secure 0 0 0) 0x7fef28096d80 con 0x7fef38100c80 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T03:59:37.875 INFO:teuthology.orchestra.run.smithi023.stdout:] 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 msgr2=0x7fef1c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 0x7fef1c0a35e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fef24005fd0 tx=0x7fef24009310 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 msgr2=0x7fef38198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fef28009230 tx=0x7fef2800bef0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 shutdown_connections 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef1c0a1120 0x7fef1c0a35e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef38102810 0x7fef381a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef38100c80 0x7fef38198f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 --2- 172.21.15.23:0/3938806679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef38101e70 0x7fef38199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 >> 172.21.15.23:0/3938806679 conn(0x7fef380fc1f0 msgr2=0x7fef381076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 shutdown_connections 2024-03-30T03:59:37.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:37.879+0000 7fef3e600700 1 -- 172.21.15.23:0/3938806679 wait complete. 2024-03-30T03:59:37.896 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-30T03:59:37.897 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 -- 172.21.15.23:0/2189593037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102750 msgr2=0x7f476c109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 --2- 172.21.15.23:0/2189593037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102750 0x7f476c109000 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4764009fc0 tx=0x7f476400b330 comp rx=0 tx=0).stop 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 -- 172.21.15.23:0/2189593037 shutdown_connections 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 --2- 172.21.15.23:0/2189593037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c102750 0x7f476c109000 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 --2- 172.21.15.23:0/2189593037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 --2- 172.21.15.23:0/2189593037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c101db0 0x7f476c102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 -- 172.21.15.23:0/2189593037 >> 172.21.15.23:0/2189593037 conn(0x7f476c0fc110 msgr2=0x7f476c0fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:38.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 -- 172.21.15.23:0/2189593037 shutdown_connections 2024-03-30T03:59:38.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.101+0000 7f4771400700 1 -- 172.21.15.23:0/2189593037 wait complete. 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.102+0000 7f4771400700 1 Processor -- start 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.102+0000 7f4771400700 1 -- start start 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 0x7f476c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c102750 0x7f476c1a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1a08f0 con 0x7f476c100bc0 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1a0a60 con 0x7f476c101db0 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4771400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476c1a0c20 con 0x7f476c102750 2024-03-30T03:59:38.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32896/0 (socket says 172.21.15.23:32896) 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 -- 172.21.15.23:0/1427549161 learned_addr learned my addr 172.21.15.23:0/1427549161 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476b400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 0x7f476c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f4770a00700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c102750 0x7f476c1a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 -- 172.21.15.23:0/1427549161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c102750 msgr2=0x7f476c1a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c102750 0x7f476c1a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 -- 172.21.15.23:0/1427549161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 msgr2=0x7f476c199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 0x7f476c199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.103+0000 7f476be00700 1 -- 172.21.15.23:0/1427549161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f475c0095f0 con 0x7f476c100bc0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f476b400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 0x7f476c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f476be00700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f475c00ebf0 tx=0x7f475c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c009a10 con 0x7f476c100bc0 2024-03-30T03:59:38.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4764009c40 con 0x7f476c100bc0 2024-03-30T03:59:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f475c00cd40 con 0x7f476c100bc0 2024-03-30T03:59:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475c018820 con 0x7f476c100bc0 2024-03-30T03:59:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.104+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f476c1a1210 con 0x7f476c100bc0 2024-03-30T03:59:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.105+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476c0679a0 con 0x7f476c100bc0 2024-03-30T03:59:38.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.107+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f475c018980 con 0x7f476c100bc0 2024-03-30T03:59:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.108+0000 7f4768c00700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 0x7f47500a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.108+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f475c014070 con 0x7f476c100bc0 2024-03-30T03:59:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.109+0000 7f476b400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 0x7f47500a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:38.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.109+0000 7f476b400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 0x7f47500a3660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4758009cb0 tx=0x7f4758009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.112+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f475c096690 con 0x7f476c100bc0 2024-03-30T03:59:38.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/327757904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:38.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/862387985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:38.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:37 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/327757904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/862387985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:38.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:38.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:37 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/327757904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/862387985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T03:59:38.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:37 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T03:59:38.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.333+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f476c10c3e0 con 0x7f476c100bc0 2024-03-30T03:59:38.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.905+0000 7f4768c00700 1 -- 172.21.15.23:0/1427549161 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f475c099cb0 con 0x7f476c100bc0 2024-03-30T03:59:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.911+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 msgr2=0x7f47500a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.911+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 0x7f47500a3660 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4758009cb0 tx=0x7f4758009260 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.911+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 msgr2=0x7f476c198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.911+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f475c00ebf0 tx=0x7f475c00ef00 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 shutdown_connections 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f47500a11a0 0x7f47500a3660 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f476c102750 0x7f476c1a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f476c100bc0 0x7f476c198e00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 --2- 172.21.15.23:0/1427549161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f476c101db0 0x7f476c199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 >> 172.21.15.23:0/1427549161 conn(0x7f476c0fc110 msgr2=0x7f476c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 shutdown_connections 2024-03-30T03:59:38.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:38.912+0000 7f4771400700 1 -- 172.21.15.23:0/1427549161 wait complete. 2024-03-30T03:59:38.912 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2024-03-30T03:59:38.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-30T03:59:39.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 -- 172.21.15.23:0/1732920580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68101f70 msgr2=0x7efd681023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:39.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1732920580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68101f70 0x7efd681023d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efd540099c0 tx=0x7efd54009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 -- 172.21.15.23:0/1732920580 shutdown_connections 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1732920580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68102910 0x7efd681091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1732920580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd681011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1732920580 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68101f70 0x7efd681023d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 -- 172.21.15.23:0/1732920580 >> 172.21.15.23:0/1732920580 conn(0x7efd680fc2f0 msgr2=0x7efd680fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.137+0000 7efd6ea00700 1 -- 172.21.15.23:0/1732920580 shutdown_connections 2024-03-30T03:59:39.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.138+0000 7efd6ea00700 1 -- 172.21.15.23:0/1732920580 wait complete. 2024-03-30T03:59:39.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 Processor -- start 2024-03-30T03:59:39.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 -- start start 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 0x7efd68199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 0x7efd681a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681a0a00 con 0x7efd68102910 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681a0b40 con 0x7efd68101f70 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681a0cd0 con 0x7efd68100d80 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd67e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd67e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55000/0 (socket says 172.21.15.23:55000) 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd67400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 0x7efd68199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd67e00700 1 -- 172.21.15.23:0/1311746029 learned_addr learned my addr 172.21.15.23:0/1311746029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.139+0000 7efd6ca00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 0x7efd681a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 -- 172.21.15.23:0/1311746029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 msgr2=0x7efd68199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 0x7efd68199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 -- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 msgr2=0x7efd681a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 0x7efd681a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 -- 172.21.15.23:0/1311746029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd58009520 con 0x7efd68100d80 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67400700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 0x7efd68199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd6ca00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 0x7efd681a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd67e00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efd5800ea50 tx=0x7efd5800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5800cbd0 con 0x7efd68100d80 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd540095f0 con 0x7efd68100d80 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd5800cd30 con 0x7efd68100d80 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.140+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd58018680 con 0x7efd68100d80 2024-03-30T03:59:39.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.141+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd681a1250 con 0x7efd68100d80 2024-03-30T03:59:39.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.142+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd68067af0 con 0x7efd68100d80 2024-03-30T03:59:39.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.144+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7efd58010430 con 0x7efd68100d80 2024-03-30T03:59:39.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.145+0000 7efd64c00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 0x7efd4c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:39.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.145+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7497+0+0 (secure 0 0 0) 0x7efd58014070 con 0x7efd68100d80 2024-03-30T03:59:39.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.145+0000 7efd67400700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 0x7efd4c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:39.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.146+0000 7efd67400700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 0x7efd4c0a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efd54009fd0 tx=0x7efd5400b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:39.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.150+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efd58096660 con 0x7efd68100d80 2024-03-30T03:59:39.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:38 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3938806679' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T03:59:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:38 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3938806679' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T03:59:39.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:38 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3938806679' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-30T03:59:39.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.369+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7efd681a1610 con 0x7efd68100d80 2024-03-30T03:59:39.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.915+0000 7efd64c00700 1 -- 172.21.15.23:0/1311746029 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v78) v1 ==== 171+0+0 (secure 0 0 0) 0x7efd58099c80 con 0x7efd68100d80 2024-03-30T03:59:39.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.922+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 msgr2=0x7efd4c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:39.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.922+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 0x7efd4c0a35e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efd54009fd0 tx=0x7efd5400b540 comp rx=0 tx=0).stop 2024-03-30T03:59:39.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.922+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 msgr2=0x7efd68198ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:39.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.922+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efd5800ea50 tx=0x7efd5800ed60 comp rx=0 tx=0).stop 2024-03-30T03:59:39.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 shutdown_connections 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efd68102910 0x7efd681a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efd4c0a1120 0x7efd4c0a35e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efd68100d80 0x7efd68198ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 --2- 172.21.15.23:0/1311746029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd68101f70 0x7efd68199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 >> 172.21.15.23:0/1311746029 conn(0x7efd680fc2f0 msgr2=0x7efd68107760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 shutdown_connections 2024-03-30T03:59:39.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:39.923+0000 7efd6ea00700 1 -- 172.21.15.23:0/1311746029 wait complete. 2024-03-30T03:59:39.927 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2024-03-30T03:59:39.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-30T03:59:40.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 -- 172.21.15.23:0/2191043690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 msgr2=0x7fb9481011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 --2- 172.21.15.23:0/2191043690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb9481011a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0099c0 tx=0x7fb93c009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 -- 172.21.15.23:0/2191043690 shutdown_connections 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 --2- 172.21.15.23:0/2191043690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 0x7fb9481091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 --2- 172.21.15.23:0/2191043690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb9481011a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 --2- 172.21.15.23:0/2191043690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 0x7fb9481023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 -- 172.21.15.23:0/2191043690 >> 172.21.15.23:0/2191043690 conn(0x7fb9480fc2f0 msgr2=0x7fb9480fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 -- 172.21.15.23:0/2191043690 shutdown_connections 2024-03-30T03:59:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.154+0000 7fb94f800700 1 -- 172.21.15.23:0/2191043690 wait complete. 2024-03-30T03:59:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 Processor -- start 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 -- start start 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 0x7fb9480731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 0x7fb9480736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb948073de0 con 0x7fb948100d80 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb948073f50 con 0x7fb948102910 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.155+0000 7fb94f800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9481a0dc0 con 0x7fb948101f70 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32928/0 (socket says 172.21.15.23:32928) 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 -- 172.21.15.23:0/88679562 learned_addr learned my addr 172.21.15.23:0/88679562 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94d800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 0x7fb9480736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb947e00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 0x7fb9480731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 -- 172.21.15.23:0/88679562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 msgr2=0x7fb9480731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 0x7fb9480731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 -- 172.21.15.23:0/88679562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 msgr2=0x7fb9480736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 0x7fb9480736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 -- 172.21.15.23:0/88679562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93c0095f0 con 0x7fb948100d80 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.156+0000 7fb94ce00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb93c006010 tx=0x7fb93c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.157+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c01d070 con 0x7fb948100d80 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.157+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb93c005020 con 0x7fb948100d80 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.157+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9481a0fc0 con 0x7fb948100d80 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.157+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb93c004b10 con 0x7fb948100d80 2024-03-30T03:59:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.157+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9481a1330 con 0x7fb948100d80 2024-03-30T03:59:40.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.158+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb948108250 con 0x7fb948100d80 2024-03-30T03:59:40.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.160+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb93c005190 con 0x7fb948100d80 2024-03-30T03:59:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.161+0000 7fb945600700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 0x7fb9300a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.161+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fb93c0cd290 con 0x7fb948100d80 2024-03-30T03:59:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.161+0000 7fb947e00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 0x7fb9300a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.162+0000 7fb947e00700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 0x7fb9300a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb934009c60 tx=0x7fb934009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:40.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.164+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb93c096bc0 con 0x7fb948100d80 2024-03-30T03:59:40.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:39 smithi154 ceph-mon[79161]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:40.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:39 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T03:59:40.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:39 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T03:59:40.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:39 smithi154 ceph-mon[79161]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T03:59:40.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:39 smithi193 ceph-mon[79581]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:40.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:39 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T03:59:40.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:39 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T03:59:40.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:39 smithi193 ceph-mon[79581]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T03:59:40.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:39 smithi023 ceph-mon[38254]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:40.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-30T03:59:40.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1427549161' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-30T03:59:40.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:39 smithi023 ceph-mon[38254]: osdmap e77: 12 total, 12 up, 12 in 2024-03-30T03:59:40.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.431+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb948067af0 con 0x7fb948100d80 2024-03-30T03:59:40.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.440+0000 7fb945600700 1 -- 172.21.15.23:0/88679562 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb93c09a1e0 con 0x7fb948100d80 2024-03-30T03:59:40.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 msgr2=0x7fb9300a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 0x7fb9300a3660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb934009c60 tx=0x7fb934009260 comp rx=0 tx=0).stop 2024-03-30T03:59:40.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 msgr2=0x7fb948074ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb93c006010 tx=0x7fb93c00f690 comp rx=0 tx=0).stop 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 shutdown_connections 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb948102910 0x7fb9480736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb948100d80 0x7fb948074ac0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9300a11a0 0x7fb9300a3660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 --2- 172.21.15.23:0/88679562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb948101f70 0x7fb9480731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.445+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 >> 172.21.15.23:0/88679562 conn(0x7fb9480fc2f0 msgr2=0x7fb948105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.446+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 shutdown_connections 2024-03-30T03:59:40.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.446+0000 7fb94f800700 1 -- 172.21.15.23:0/88679562 wait complete. 2024-03-30T03:59:40.450 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2024-03-30T03:59:40.480 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-30T03:59:40.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-30T03:59:40.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.684+0000 7ffb14a00700 1 -- 172.21.15.23:0/1845132725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10101f70 msgr2=0x7ffb101023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.684+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1845132725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10101f70 0x7ffb101023d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffafc0099c0 tx=0x7ffafc009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 -- 172.21.15.23:0/1845132725 shutdown_connections 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1845132725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10102910 0x7ffb101091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1845132725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1845132725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10101f70 0x7ffb101023d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 -- 172.21.15.23:0/1845132725 >> 172.21.15.23:0/1845132725 conn(0x7ffb100fc2f0 msgr2=0x7ffb100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 -- 172.21.15.23:0/1845132725 shutdown_connections 2024-03-30T03:59:40.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.685+0000 7ffb14a00700 1 -- 172.21.15.23:0/1845132725 wait complete. 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 Processor -- start 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 -- start start 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 0x7ffb101972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 0x7ffb1019e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1019e860 con 0x7ffb10102910 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1019e9d0 con 0x7ffb10101f70 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb14a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1019eb90 con 0x7ffb10100d80 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb0de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb0e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 0x7ffb1019e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb0de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55030/0 (socket says 172.21.15.23:55030) 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb0de00700 1 -- 172.21.15.23:0/1883316082 learned_addr learned my addr 172.21.15.23:0/1883316082 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.686+0000 7ffb0d400700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 0x7ffb101972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 -- 172.21.15.23:0/1883316082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 msgr2=0x7ffb101972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 0x7ffb101972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 -- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 msgr2=0x7ffb1019e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 0x7ffb1019e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 -- 172.21.15.23:0/1883316082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc0095f0 con 0x7ffb10100d80 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0d400700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 0x7ffb101972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:40.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0e800700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 0x7ffb1019e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb0de00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e970 tx=0x7ffb0000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00009790 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1019ee70 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb0000c7e0 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.687+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb00003c70 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.688+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1019f2a0 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.688+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb10108250 con 0x7ffb10100d80 2024-03-30T03:59:40.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.691+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ffb0000cc50 con 0x7ffb10100d80 2024-03-30T03:59:40.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.692+0000 7ffb06a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 0x7ffaf40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:40.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.692+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ffb00014070 con 0x7ffb10100d80 2024-03-30T03:59:40.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.692+0000 7ffb0d400700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 0x7ffaf40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:40.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.693+0000 7ffb0d400700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 0x7ffaf40a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffafc003730 tx=0x7ffafc01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:40.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.694+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb00096300 con 0x7ffb10100d80 2024-03-30T03:59:40.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:40.926+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7ffb10067af0 con 0x7ffb10100d80 2024-03-30T03:59:41.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1311746029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: pgmap v230: 97 pgs: 8 creating+peering, 7 active+clean, 82 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T03:59:41.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:40 smithi154 ceph-mon[79161]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1311746029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: pgmap v230: 97 pgs: 8 creating+peering, 7 active+clean, 82 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T03:59:41.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T03:59:41.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T03:59:41.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T03:59:41.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:40 smithi193 ceph-mon[79581]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1311746029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: osdmap e78: 12 total, 12 up, 12 in 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: pgmap v230: 97 pgs: 8 creating+peering, 7 active+clean, 82 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: osdmap e79: 12 total, 12 up, 12 in 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/88679562' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-30T03:59:41.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-30T03:59:41.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:40 smithi023 ceph-mon[38254]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-30T03:59:41.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.440+0000 7ffb06a00700 1 -- 172.21.15.23:0/1883316082 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v80) v1 ==== 157+0+0 (secure 0 0 0) 0x7ffb00003dd0 con 0x7ffb10100d80 2024-03-30T03:59:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 msgr2=0x7ffaf40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 0x7ffaf40a3660 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ffafc003730 tx=0x7ffafc01a040 comp rx=0 tx=0).stop 2024-03-30T03:59:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 msgr2=0x7ffb10196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb0000e970 tx=0x7ffb0000ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:41.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 shutdown_connections 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb10102910 0x7ffb1019e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffaf40a11a0 0x7ffaf40a3660 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb10100d80 0x7ffb10196d70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 --2- 172.21.15.23:0/1883316082 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb10101f70 0x7ffb101972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 >> 172.21.15.23:0/1883316082 conn(0x7ffb100fc2f0 msgr2=0x7ffb10105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 shutdown_connections 2024-03-30T03:59:41.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.443+0000 7ffb14a00700 1 -- 172.21.15.23:0/1883316082 wait complete. 2024-03-30T03:59:41.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-30T03:59:41.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 -- 172.21.15.23:0/483492133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 msgr2=0x7f3534103ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:41.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 --2- 172.21.15.23:0/483492133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534103ae0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3524009990 tx=0x7f3524009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 -- 172.21.15.23:0/483492133 shutdown_connections 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 --2- 172.21.15.23:0/483492133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534106950 0x7f3534108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 --2- 172.21.15.23:0/483492133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534103ae0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 --2- 172.21.15.23:0/483492133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534104020 0x7f3534106410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 -- 172.21.15.23:0/483492133 >> 172.21.15.23:0/483492133 conn(0x7f35340fb0e0 msgr2=0x7f35340fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 -- 172.21.15.23:0/483492133 shutdown_connections 2024-03-30T03:59:41.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.670+0000 7f3538c00700 1 -- 172.21.15.23:0/483492133 wait complete. 2024-03-30T03:59:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.671+0000 7f3538c00700 1 Processor -- start 2024-03-30T03:59:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.671+0000 7f3538c00700 1 -- start start 2024-03-30T03:59:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:41.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534104020 0x7f3534199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534106950 0x7f35341a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35341a0940 con 0x7f35341016f0 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35341a0a80 con 0x7f3534104020 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3538c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35341a0c10 con 0x7f3534106950 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32960/0 (socket says 172.21.15.23:32960) 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 -- 172.21.15.23:0/239143076 learned_addr learned my addr 172.21.15.23:0/239143076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531400700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534104020 0x7f3534199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3532800700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534106950 0x7f35341a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 -- 172.21.15.23:0/239143076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534106950 msgr2=0x7f35341a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534106950 0x7f35341a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 -- 172.21.15.23:0/239143076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534104020 msgr2=0x7f3534199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:41.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534104020 0x7f3534199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.672+0000 7f3531e00700 1 -- 172.21.15.23:0/239143076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35240095f0 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f3531e00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3524006010 tx=0x7f3524004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352401d070 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3524005030 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3524004b40 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35341a0e90 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.673+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35341a1200 con 0x7f35341016f0 2024-03-30T03:59:41.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.674+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3534067a80 con 0x7f35341016f0 2024-03-30T03:59:41.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.676+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f3524004d40 con 0x7f35341016f0 2024-03-30T03:59:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.677+0000 7f352aa00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 0x7f35180a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.677+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f35240cd930 con 0x7f35341016f0 2024-03-30T03:59:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.678+0000 7f3531400700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 0x7f35180a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:41.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.678+0000 7f3531400700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 0x7f35180a3650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3520009c60 tx=0x7f3520009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:41.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.681+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3534067a80 con 0x7f35341016f0 2024-03-30T03:59:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:41.903+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3534067a10 con 0x7f35341016f0 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:active 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1883316082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T03:59:42.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:41 smithi154 ceph-mon[79161]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:active 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1883316082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T03:59:42.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:41 smithi193 ceph-mon[79581]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] up:active 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1883316082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-30T03:59:42.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:41 smithi023 ceph-mon[38254]: osdmap e80: 12 total, 12 up, 12 in 2024-03-30T03:59:43.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:42 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T03:59:43.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:42 smithi193 ceph-mon[79581]: pgmap v233: 97 pgs: 8 creating+peering, 34 active+clean, 55 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-30T03:59:43.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:42 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T03:59:43.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:42 smithi023 ceph-mon[38254]: pgmap v233: 97 pgs: 8 creating+peering, 34 active+clean, 55 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-30T03:59:43.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:42 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-30T03:59:43.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:42 smithi154 ceph-mon[79161]: pgmap v233: 97 pgs: 8 creating+peering, 34 active+clean, 55 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 2 op/s 2024-03-30T03:59:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.445+0000 7f352aa00700 1 -- 172.21.15.23:0/239143076 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v82) v1 ==== 228+0+0 (secure 0 0 0) 0x7f3524097110 con 0x7f35341016f0 2024-03-30T03:59:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 msgr2=0x7f35180a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 0x7f35180a3650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3520009c60 tx=0x7f3520009260 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 msgr2=0x7f3534198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3524006010 tx=0x7f3524004730 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 shutdown_connections 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35180a1190 0x7f35180a3650 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3534106950 0x7f35341a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f35341016f0 0x7f3534198e10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 --2- 172.21.15.23:0/239143076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3534104020 0x7f3534199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.452+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 >> 172.21.15.23:0/239143076 conn(0x7f35340fb0e0 msgr2=0x7f3534100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:43.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.453+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 shutdown_connections 2024-03-30T03:59:43.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.453+0000 7f3538c00700 1 -- 172.21.15.23:0/239143076 wait complete. 2024-03-30T03:59:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2024-03-30T03:59:43.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 -- 172.21.15.23:0/2700919551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 msgr2=0x7f2930103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 --2- 172.21.15.23:0/2700919551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930103bd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2920009990 tx=0x7f2920009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 -- 172.21.15.23:0/2700919551 shutdown_connections 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 --2- 172.21.15.23:0/2700919551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930106a40 0x7f2930108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 --2- 172.21.15.23:0/2700919551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930103bd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 --2- 172.21.15.23:0/2700919551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930104110 0x7f2930106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 -- 172.21.15.23:0/2700919551 >> 172.21.15.23:0/2700919551 conn(0x7f29300fb1d0 msgr2=0x7f29300fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 -- 172.21.15.23:0/2700919551 shutdown_connections 2024-03-30T03:59:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.679+0000 7f2936400700 1 -- 172.21.15.23:0/2700919551 wait complete. 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 Processor -- start 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 -- start start 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930104110 0x7f29300ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930106a40 0x7f29300ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293019aaa0 con 0x7f29301017e0 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930100210 con 0x7f2930104110 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.680+0000 7f2936400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29301003d0 con 0x7f2930106a40 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32968/0 (socket says 172.21.15.23:32968) 2024-03-30T03:59:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 -- 172.21.15.23:0/477041266 learned_addr learned my addr 172.21.15.23:0/477041266 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930104110 0x7f29300ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292fe00700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930106a40 0x7f29300ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 -- 172.21.15.23:0/477041266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930106a40 msgr2=0x7f29300ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930106a40 0x7f29300ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 -- 172.21.15.23:0/477041266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930104110 msgr2=0x7f29300ff790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930104110 0x7f29300ff790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 -- 172.21.15.23:0/477041266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29200095f0 con 0x7f29301017e0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.681+0000 7f292f400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f29200098e0 tx=0x7f2920005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.682+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292001d070 con 0x7f29301017e0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.682+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2920022470 con 0x7f29301017e0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.682+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29301005f0 con 0x7f29301017e0 2024-03-30T03:59:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.682+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920017650 con 0x7f29301017e0 2024-03-30T03:59:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.682+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29301a5740 con 0x7f29301017e0 2024-03-30T03:59:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.683+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29300fcd50 con 0x7f29301017e0 2024-03-30T03:59:43.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.686+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f29200225e0 con 0x7f29301017e0 2024-03-30T03:59:43.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.688+0000 7f2927e00700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 0x7f29140a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:43.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.688+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f29200ce5c0 con 0x7f29301017e0 2024-03-30T03:59:43.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.688+0000 7f292ea00700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 0x7f29140a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:43.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.688+0000 7f292ea00700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 0x7f29140a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f291c007c60 tx=0x7f291c0073d0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:43.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.690+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2920097c40 con 0x7f29301017e0 2024-03-30T03:59:43.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:43.914+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f2930067af0 con 0x7f29301017e0 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T03:59:44.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:43 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T03:59:44.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T03:59:44.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-30T03:59:44.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T03:59:44.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T03:59:44.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T03:59:44.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: osdmap e81: 12 total, 12 up, 12 in 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/239143076' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: osdmap e82: 12 total, 12 up, 12 in 2024-03-30T03:59:44.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:43 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-30T03:59:44.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.449+0000 7f2927e00700 1 -- 172.21.15.23:0/477041266 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v83) v1 ==== 169+0+0 (secure 0 0 0) 0x7f29200b9e60 con 0x7f29301017e0 2024-03-30T03:59:44.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 msgr2=0x7f29140a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 0x7f29140a3660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f291c007c60 tx=0x7f291c0073d0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 msgr2=0x7f2930101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f29200098e0 tx=0x7f2920005720 comp rx=0 tx=0).stop 2024-03-30T03:59:44.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 shutdown_connections 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f29140a11a0 0x7f29140a3660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2930106a40 0x7f29300ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29301017e0 0x7f2930101100 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 --2- 172.21.15.23:0/477041266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2930104110 0x7f29300ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.456+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 >> 172.21.15.23:0/477041266 conn(0x7f29300fb1d0 msgr2=0x7f2930103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.457+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 shutdown_connections 2024-03-30T03:59:44.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.457+0000 7f2936400700 1 -- 172.21.15.23:0/477041266 wait complete. 2024-03-30T03:59:44.463 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2024-03-30T03:59:44.488 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-30T03:59:44.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T03:59:44.279+0000 7f7e1571a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-30T03:59:44.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 -- 172.21.15.23:0/203542146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 msgr2=0x7fefa41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 --2- 172.21.15.23:0/203542146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa41011a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fef980099c0 tx=0x7fef98009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 -- 172.21.15.23:0/203542146 shutdown_connections 2024-03-30T03:59:44.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 --2- 172.21.15.23:0/203542146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 0x7fefa41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 --2- 172.21.15.23:0/203542146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa41011a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 --2- 172.21.15.23:0/203542146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 0x7fefa41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 -- 172.21.15.23:0/203542146 >> 172.21.15.23:0/203542146 conn(0x7fefa40fc2f0 msgr2=0x7fefa40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:44.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 -- 172.21.15.23:0/203542146 shutdown_connections 2024-03-30T03:59:44.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.693+0000 7fefab600700 1 -- 172.21.15.23:0/203542146 wait complete. 2024-03-30T03:59:44.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.694+0000 7fefab600700 1 Processor -- start 2024-03-30T03:59:44.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 -- start start 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 0x7fefa41994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 0x7fefa41a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa41a0ab0 con 0x7fefa4100d80 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa41a0bf0 con 0x7fefa4102910 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefab600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa41a0d80 con 0x7fefa4101f70 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefa8c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefa8c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32990/0 (socket says 172.21.15.23:32990) 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefa8c00700 1 -- 172.21.15.23:0/65652627 learned_addr learned my addr 172.21.15.23:0/65652627 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefa3e00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 0x7fefa41994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.695+0000 7fefa9600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 0x7fefa41a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 -- 172.21.15.23:0/65652627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 msgr2=0x7fefa41994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 0x7fefa41994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 -- 172.21.15.23:0/65652627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 msgr2=0x7fefa41a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 0x7fefa41a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 -- 172.21.15.23:0/65652627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980095f0 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa8c00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef98003730 tx=0x7fef9800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9801d070 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef98005050 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa41a1000 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef98004a50 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.696+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa41a1480 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.697+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa4067af0 con 0x7fefa4100d80 2024-03-30T03:59:44.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.699+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fef980051c0 con 0x7fefa4100d80 2024-03-30T03:59:44.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.700+0000 7fefa1600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 0x7fef8c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:44.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.701+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fef980cd7b0 con 0x7fefa4100d80 2024-03-30T03:59:44.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.701+0000 7fefa3e00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 0x7fef8c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:44.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.701+0000 7fefa3e00700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 0x7fef8c0a3660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef90009c60 tx=0x7fef90009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:44.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.704+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef98096da0 con 0x7fefa4100d80 2024-03-30T03:59:44.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.967+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fefa410c5e0 con 0x7fefa4100d80 2024-03-30T03:59:44.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.975+0000 7fefa1600700 1 -- 172.21.15.23:0/65652627 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7fef9809a3c0 con 0x7fefa4100d80 2024-03-30T03:59:44.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 msgr2=0x7fef8c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 0x7fef8c0a3660 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef90009c60 tx=0x7fef90009260 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 msgr2=0x7fefa4198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef98003730 tx=0x7fef9800f690 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 shutdown_connections 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefa4102910 0x7fefa41a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fefa4100d80 0x7fefa4198f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef8c0a11a0 0x7fef8c0a3660 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 --2- 172.21.15.23:0/65652627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefa4101f70 0x7fefa41994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 >> 172.21.15.23:0/65652627 conn(0x7fefa40fc2f0 msgr2=0x7fefa4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 shutdown_connections 2024-03-30T03:59:44.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:44.978+0000 7fefab600700 1 -- 172.21.15.23:0/65652627 wait complete. 2024-03-30T03:59:44.978 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 4 to fsmap 2024-03-30T03:59:44.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:45.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 -- 172.21.15.23:0/2686191387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 msgr2=0x7f8b501022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 --2- 172.21.15.23:0/2686191387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b501022d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c0099c0 tx=0x7f8b3c009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 -- 172.21.15.23:0/2686191387 shutdown_connections 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 --2- 172.21.15.23:0/2686191387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50102810 0x7f8b501090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 --2- 172.21.15.23:0/2686191387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50100c80 0x7f8b501010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.198+0000 7f8b57600700 1 --2- 172.21.15.23:0/2686191387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b501022d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.199+0000 7f8b57600700 1 -- 172.21.15.23:0/2686191387 >> 172.21.15.23:0/2686191387 conn(0x7f8b500fc1f0 msgr2=0x7f8b500fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.199+0000 7f8b57600700 1 -- 172.21.15.23:0/2686191387 shutdown_connections 2024-03-30T03:59:45.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.199+0000 7f8b57600700 1 -- 172.21.15.23:0/2686191387 wait complete. 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 Processor -- start 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 -- start start 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50100c80 0x7f8b50198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 0x7f8b501a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b501a0900 con 0x7f8b50101e70 2024-03-30T03:59:45.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b501a0a70 con 0x7f8b50100c80 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b57600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b501a0c30 con 0x7f8b50102810 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b4fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33016/0 (socket says 172.21.15.23:33016) 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b4fe00700 1 -- 172.21.15.23:0/2448130597 learned_addr learned my addr 172.21.15.23:0/2448130597 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b54c00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50100c80 0x7f8b50198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.200+0000 7f8b55600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 0x7f8b501a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 -- 172.21.15.23:0/2448130597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 msgr2=0x7f8b501a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 0x7f8b501a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 -- 172.21.15.23:0/2448130597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50100c80 msgr2=0x7f8b50198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50100c80 0x7f8b50198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 -- 172.21.15.23:0/2448130597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c0095f0 con 0x7f8b50101e70 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b55600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 0x7f8b501a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4fe00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c006010 tx=0x7f8b3c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:45.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c01d070 con 0x7f8b50101e70 2024-03-30T03:59:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b3c005030 con 0x7f8b50101e70 2024-03-30T03:59:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b501a0eb0 con 0x7f8b50101e70 2024-03-30T03:59:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c004bb0 con 0x7f8b50101e70 2024-03-30T03:59:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.201+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b501a1270 con 0x7f8b50101e70 2024-03-30T03:59:45.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.202+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b50067a10 con 0x7f8b50101e70 2024-03-30T03:59:45.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.204+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8b3c0051a0 con 0x7f8b50101e70 2024-03-30T03:59:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.206+0000 7f8b4d600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 0x7f8b380a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.206+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b3c0cdc70 con 0x7f8b50101e70 2024-03-30T03:59:45.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.206+0000 7f8b54c00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 0x7f8b380a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.206+0000 7f8b54c00700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 0x7f8b380a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8b5019a290 tx=0x7f8b44009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:45.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.208+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8b3c097240 con 0x7f8b50101e70 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: pgmap v236: 129 pgs: 15 creating+peering, 79 active+clean, 35 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T03:59:45.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:44 smithi193 ceph-mon[79581]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:45.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: pgmap v236: 129 pgs: 15 creating+peering, 79 active+clean, 35 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T03:59:45.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:44 smithi023 ceph-mon[38254]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: pgmap v236: 129 pgs: 15 creating+peering, 79 active+clean, 35 unknown; 577 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2024-03-30T03:59:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/477041266' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-30T03:59:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: osdmap e83: 12 total, 12 up, 12 in 2024-03-30T03:59:45.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-30T03:59:45.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: osdmap e84: 12 total, 12 up, 12 in 2024-03-30T03:59:45.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/65652627' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-30T03:59:45.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:44 smithi154 ceph-mon[79161]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T03:59:45.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.474+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b5010c4e0 con 0x7f8b50101e70 2024-03-30T03:59:45.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.474+0000 7f8b4d600700 1 -- 172.21.15.23:0/2448130597 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f8b3c09a860 con 0x7f8b50101e70 2024-03-30T03:59:45.473 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:45.473 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14544,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T03:59:44.968704+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 msgr2=0x7f8b380a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 0x7f8b380a35e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8b5019a290 tx=0x7f8b44009260 comp rx=0 tx=0).stop 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 msgr2=0x7f8b50199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c006010 tx=0x7f8b3c00f690 comp rx=0 tx=0).stop 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 shutdown_connections 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8b380a1120 0x7f8b380a35e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b50102810 0x7f8b501a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b50100c80 0x7f8b50198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 --2- 172.21.15.23:0/2448130597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b50101e70 0x7f8b50199350 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 >> 172.21.15.23:0/2448130597 conn(0x7f8b500fc1f0 msgr2=0x7f8b50072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 shutdown_connections 2024-03-30T03:59:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.477+0000 7f8b57600700 1 -- 172.21.15.23:0/2448130597 wait complete. 2024-03-30T03:59:45.477 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 16 2024-03-30T03:59:45.496 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 -- 172.21.15.23:0/2278669502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 msgr2=0x7fd630108db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 --2- 172.21.15.23:0/2278669502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd630108db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd62800b0a0 tx=0x7fd62800b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 -- 172.21.15.23:0/2278669502 shutdown_connections 2024-03-30T03:59:45.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 --2- 172.21.15.23:0/2278669502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd630108db0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 --2- 172.21.15.23:0/2278669502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630101720 0x7fd630103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 --2- 172.21.15.23:0/2278669502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630104050 0x7fd630106440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 -- 172.21.15.23:0/2278669502 >> 172.21.15.23:0/2278669502 conn(0x7fd6300fb0d0 msgr2=0x7fd6300fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:45.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 -- 172.21.15.23:0/2278669502 shutdown_connections 2024-03-30T03:59:45.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.694+0000 7fd636400700 1 -- 172.21.15.23:0/2278669502 wait complete. 2024-03-30T03:59:45.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.695+0000 7fd636400700 1 Processor -- start 2024-03-30T03:59:45.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.695+0000 7fd636400700 1 -- start start 2024-03-30T03:59:45.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.695+0000 7fd636400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 0x7fd630196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.695+0000 7fd636400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630104050 0x7fd630197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd636400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd636400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63019e7e0 con 0x7fd630106980 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd636400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63019e950 con 0x7fd630101720 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd636400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63019eb10 con 0x7fd630104050 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 0x7fd630196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33040/0 (socket says 172.21.15.23:33040) 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 0x7fd630196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:44150/0 (socket says 172.21.15.23:44150) 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 -- 172.21.15.23:0/1505544916 learned_addr learned my addr 172.21.15.23:0/1505544916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62ea00700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630104050 0x7fd630197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 -- 172.21.15.23:0/1505544916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630104050 msgr2=0x7fd630197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630104050 0x7fd630197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 -- 172.21.15.23:0/1505544916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 msgr2=0x7fd630196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 0x7fd630196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 -- 172.21.15.23:0/1505544916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628009d00 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.696+0000 7fd62fe00700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd628007a60 tx=0x7fd62800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.697+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800e040 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.697+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6280094c0 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.697+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6300ff510 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.697+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280052a0 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.697+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6300ff9d0 con 0x7fd630106980 2024-03-30T03:59:45.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.698+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6300fcc50 con 0x7fd630106980 2024-03-30T03:59:45.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.700+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd62801d470 con 0x7fd630106980 2024-03-30T03:59:45.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.702+0000 7fd627e00700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 0x7fd6140a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:45.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.702+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd6280ce620 con 0x7fd630106980 2024-03-30T03:59:45.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.702+0000 7fd62f400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 0x7fd6140a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:45.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.703+0000 7fd62f400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 0x7fd6140a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd630102d70 tx=0x7fd620012040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:45.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.704+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd628097bf0 con 0x7fd630106980 2024-03-30T03:59:45.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.955+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd630067a10 con 0x7fd630106980 2024-03-30T03:59:45.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.957+0000 7fd627e00700 1 -- 172.21.15.23:0/1505544916 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7fd62801ae60 con 0x7fd630106980 2024-03-30T03:59:45.956 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:45.956 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:45.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 msgr2=0x7fd6140a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 0x7fd6140a35e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd630102d70 tx=0x7fd620012040 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 msgr2=0x7fd63019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd628007a60 tx=0x7fd62800bce0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 shutdown_connections 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd630106980 0x7fd63019e210 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd630101720 0x7fd630196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6140a1120 0x7fd6140a35e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 --2- 172.21.15.23:0/1505544916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630104050 0x7fd630197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 >> 172.21.15.23:0/1505544916 conn(0x7fd6300fb0d0 msgr2=0x7fd630102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 shutdown_connections 2024-03-30T03:59:45.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:45.960+0000 7fd636400700 1 -- 172.21.15.23:0/1505544916 wait complete. 2024-03-30T03:59:45.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 -- 172.21.15.23:0/1048501136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 msgr2=0x7fc87c1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 --2- 172.21.15.23:0/1048501136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c1010a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0099c0 tx=0x7fc86c009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 -- 172.21.15.23:0/1048501136 shutdown_connections 2024-03-30T03:59:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 --2- 172.21.15.23:0/1048501136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 0x7fc87c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 --2- 172.21.15.23:0/1048501136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c1010a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 --2- 172.21.15.23:0/1048501136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 0x7fc87c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 -- 172.21.15.23:0/1048501136 >> 172.21.15.23:0/1048501136 conn(0x7fc87c0fc1f0 msgr2=0x7fc87c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:46.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 -- 172.21.15.23:0/1048501136 shutdown_connections 2024-03-30T03:59:46.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.178+0000 7fc882c00700 1 -- 172.21.15.23:0/1048501136 wait complete. 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.179+0000 7fc882c00700 1 Processor -- start 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.179+0000 7fc882c00700 1 -- start start 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 0x7fc87c199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 0x7fc87c1a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1a0950 con 0x7fc87c100c80 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1a0a90 con 0x7fc87c102810 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc882c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c1a0c20 con 0x7fc87c101e70 2024-03-30T03:59:46.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33062/0 (socket says 172.21.15.23:33062) 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 -- 172.21.15.23:0/333947502 learned_addr learned my addr 172.21.15.23:0/333947502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87b400700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 0x7fc87c199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc880c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 0x7fc87c1a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 -- 172.21.15.23:0/333947502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 msgr2=0x7fc87c199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 0x7fc87c199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 -- 172.21.15.23:0/333947502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 msgr2=0x7fc87c1a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 0x7fc87c1a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.180+0000 7fc87be00700 1 -- 172.21.15.23:0/333947502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0095f0 con 0x7fc87c100c80 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc87be00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc86c003730 tx=0x7fc86c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c01d070 con 0x7fc87c100c80 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc86c00fe70 con 0x7fc87c100c80 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc87c1a0ea0 con 0x7fc87c100c80 2024-03-30T03:59:46.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86c004b30 con 0x7fc87c100c80 2024-03-30T03:59:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.181+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc87c1a1260 con 0x7fc87c100c80 2024-03-30T03:59:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.182+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc87c067a10 con 0x7fc87c100c80 2024-03-30T03:59:46.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.184+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc86c004cf0 con 0x7fc87c100c80 2024-03-30T03:59:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.185+0000 7fc878c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 0x7fc8600a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.185+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc86c0cd900 con 0x7fc87c100c80 2024-03-30T03:59:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.186+0000 7fc87b400700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 0x7fc8600a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.186+0000 7fc87b400700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 0x7fc8600a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc868009c60 tx=0x7fc868009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:46.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.189+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc86c096ed0 con 0x7fc87c100c80 2024-03-30T03:59:46.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:45 smithi193 ceph-mon[79581]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:46.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2448130597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1505544916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:45 smithi023 ceph-mon[38254]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:46.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2448130597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1505544916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.429+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc87c10c4e0 con 0x7fc87c100c80 2024-03-30T03:59:46.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.431+0000 7fc878c00700 1 -- 172.21.15.23:0/333947502 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7fc86c09a4f0 con 0x7fc87c100c80 2024-03-30T03:59:46.429 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:46.430 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:45 smithi154 ceph-mon[79161]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-30T03:59:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2448130597' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1505544916' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 msgr2=0x7fc8600a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 0x7fc8600a3660 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc868009c60 tx=0x7fc868009260 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 msgr2=0x7fc87c198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc86c003730 tx=0x7fc86c00fa00 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 shutdown_connections 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c102810 0x7fc87c1a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc87c100c80 0x7fc87c198e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc8600a11a0 0x7fc8600a3660 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 --2- 172.21.15.23:0/333947502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc87c101e70 0x7fc87c199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 >> 172.21.15.23:0/333947502 conn(0x7fc87c0fc1f0 msgr2=0x7fc87c107690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:46.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 shutdown_connections 2024-03-30T03:59:46.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.433+0000 7fc882c00700 1 -- 172.21.15.23:0/333947502 wait complete. 2024-03-30T03:59:46.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 -- 172.21.15.23:0/3372722206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 msgr2=0x7f15281090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 --2- 172.21.15.23:0/3372722206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281090c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f152000b0a0 tx=0x7f152000b3b0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 -- 172.21.15.23:0/3372722206 shutdown_connections 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 --2- 172.21.15.23:0/3372722206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281090c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 --2- 172.21.15.23:0/3372722206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528100c80 0x7f15281010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 --2- 172.21.15.23:0/3372722206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528101e70 0x7f15281022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 -- 172.21.15.23:0/3372722206 >> 172.21.15.23:0/3372722206 conn(0x7f15280fc1f0 msgr2=0x7f15280fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 -- 172.21.15.23:0/3372722206 shutdown_connections 2024-03-30T03:59:46.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.664+0000 7f152e800700 1 -- 172.21.15.23:0/3372722206 wait complete. 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.665+0000 7f152e800700 1 Processor -- start 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.665+0000 7f152e800700 1 -- start start 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528100c80 0x7f1528198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15281a0aa0 con 0x7f1528101e70 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15281a0be0 con 0x7f1528100c80 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f152e800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15281a0d70 con 0x7f1528102810 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1526a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1526a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33080/0 (socket says 172.21.15.23:33080) 2024-03-30T03:59:46.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1526a00700 1 -- 172.21.15.23:0/153994692 learned_addr learned my addr 172.21.15.23:0/153994692 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527400700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528100c80 0x7f1528198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 -- 172.21.15.23:0/153994692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528100c80 msgr2=0x7f1528198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528100c80 0x7f1528198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 -- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 msgr2=0x7f15281994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1527e00700 1 -- 172.21.15.23:0/153994692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1520009d00 con 0x7f1528102810 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.666+0000 7f1526a00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f1527e00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281a04d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1520007bc0 tx=0x7f1520007930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152000e040 con 0x7f1528102810 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f152001d470 con 0x7f1528102810 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152001b6e0 con 0x7f1528102810 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15281a0ff0 con 0x7f1528102810 2024-03-30T03:59:46.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.667+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15281a1360 con 0x7f1528102810 2024-03-30T03:59:46.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.668+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1528067a10 con 0x7f1528102810 2024-03-30T03:59:46.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.670+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f152001d5e0 con 0x7f1528102810 2024-03-30T03:59:46.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.671+0000 7f151fe00700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 0x7f150c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:46.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.671+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f15200cdf40 con 0x7f1528102810 2024-03-30T03:59:46.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.672+0000 7f1527400700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 0x7f150c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:46.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.672+0000 7f1527400700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 0x7f150c0a3660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1528101cd0 tx=0x7f1518009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:46.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.674+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1520097510 con 0x7f1528102810 2024-03-30T03:59:46.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.940+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f152810c760 con 0x7f1528102810 2024-03-30T03:59:46.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.941+0000 7f151fe00700 1 -- 172.21.15.23:0/153994692 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f152009ab30 con 0x7f1528102810 2024-03-30T03:59:46.940 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:46.940 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14544,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14556,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24476,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24508,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24509,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24514,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T03:59:44.968704+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24569},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 msgr2=0x7f150c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 0x7f150c0a3660 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1528101cd0 tx=0x7f1518009260 comp rx=0 tx=0).stop 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 msgr2=0x7f15281a04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281a04d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1520007bc0 tx=0x7f1520007930 comp rx=0 tx=0).stop 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 shutdown_connections 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f150c0a11a0 0x7f150c0a3660 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1528102810 0x7f15281a04d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1528100c80 0x7f1528198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 --2- 172.21.15.23:0/153994692 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1528101e70 0x7f15281994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 >> 172.21.15.23:0/153994692 conn(0x7f15280fc1f0 msgr2=0x7f1528072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 shutdown_connections 2024-03-30T03:59:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:46.944+0000 7f152e800700 1 -- 172.21.15.23:0/153994692 wait complete. 2024-03-30T03:59:46.944 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 16 2024-03-30T03:59:46.962 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 -- 172.21.15.23:0/4267154577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550101e50 msgr2=0x7f95501022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 --2- 172.21.15.23:0/4267154577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550101e50 0x7f95501022b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f953c0099c0 tx=0x7f953c009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 -- 172.21.15.23:0/4267154577 shutdown_connections 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 --2- 172.21.15.23:0/4267154577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 0x7f95501090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 --2- 172.21.15.23:0/4267154577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550100c60 0x7f9550101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 --2- 172.21.15.23:0/4267154577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550101e50 0x7f95501022b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 -- 172.21.15.23:0/4267154577 >> 172.21.15.23:0/4267154577 conn(0x7f95500fc1f0 msgr2=0x7f95500fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 -- 172.21.15.23:0/4267154577 shutdown_connections 2024-03-30T03:59:47.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.164+0000 7f954fe00700 1 -- 172.21.15.23:0/4267154577 wait complete. 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.165+0000 7f954fe00700 1 Processor -- start 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.165+0000 7f954fe00700 1 -- start start 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550101e50 0x7f9550199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 0x7f95501a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501a0960 con 0x7f9550100c60 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501a0aa0 con 0x7f95501027f0 2024-03-30T03:59:47.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501a0c30 con 0x7f9550101e50 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33088/0 (socket says 172.21.15.23:33088) 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 -- 172.21.15.23:0/1596181135 learned_addr learned my addr 172.21.15.23:0/1596181135 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954f400700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 0x7f95501a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954e000700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550101e50 0x7f9550199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 -- 172.21.15.23:0/1596181135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550101e50 msgr2=0x7f9550199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550101e50 0x7f9550199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 -- 172.21.15.23:0/1596181135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 msgr2=0x7f95501a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 0x7f95501a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.166+0000 7f954ea00700 1 -- 172.21.15.23:0/1596181135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c0095f0 con 0x7f9550100c60 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f954ea00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f954000e970 tx=0x7f954000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540009790 con 0x7f9550100c60 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f954000c7e0 con 0x7f9550100c60 2024-03-30T03:59:47.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95501a0f10 con 0x7f9550100c60 2024-03-30T03:59:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540003c70 con 0x7f9550100c60 2024-03-30T03:59:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.167+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95501a1340 con 0x7f9550100c60 2024-03-30T03:59:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.168+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95500679a0 con 0x7f9550100c60 2024-03-30T03:59:47.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.170+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f954000cc50 con 0x7f9550100c60 2024-03-30T03:59:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.171+0000 7f9554a00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 0x7f95340a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.171+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9540014070 con 0x7f9550100c60 2024-03-30T03:59:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.171+0000 7f954e000700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 0x7f95340a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.172+0000 7f954e000700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 0x7f95340a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f953c003730 tx=0x7f953c00b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.175+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9540096190 con 0x7f9550100c60 2024-03-30T03:59:47.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:46 smithi023 ceph-mon[38254]: pgmap v239: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2024-03-30T03:59:47.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/333947502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/153994692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.429+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f955010c4c0 con 0x7f9550100c60 2024-03-30T03:59:47.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.431+0000 7f9554a00700 1 -- 172.21.15.23:0/1596181135 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7f9540003dd0 con 0x7f9550100c60 2024-03-30T03:59:47.430 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:47.430 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:46 smithi154 ceph-mon[79161]: pgmap v239: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2024-03-30T03:59:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/333947502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/153994692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.433+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 msgr2=0x7f95340a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.433+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 0x7f95340a3660 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f953c003730 tx=0x7f953c00b540 comp rx=0 tx=0).stop 2024-03-30T03:59:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.433+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 msgr2=0x7f9550198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.433+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f954000e970 tx=0x7f954000ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:47.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 shutdown_connections 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95501027f0 0x7f95501a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9550100c60 0x7f9550198e30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f95340a11a0 0x7f95340a3660 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 --2- 172.21.15.23:0/1596181135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9550101e50 0x7f9550199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 >> 172.21.15.23:0/1596181135 conn(0x7f95500fc1f0 msgr2=0x7f9550072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 shutdown_connections 2024-03-30T03:59:47.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.434+0000 7f954fe00700 1 -- 172.21.15.23:0/1596181135 wait complete. 2024-03-30T03:59:47.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-30T03:59:40.432862+0000', 'modified': '2024-03-30T03:59:44.968704+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24569}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/2911311868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2911311868}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2911311868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-30T03:59:47.453 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-30T03:59:47.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T03:59:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:46 smithi193 ceph-mon[79581]: pgmap v239: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 8 op/s 2024-03-30T03:59:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/333947502' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/153994692' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T03:59:47.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.654+0000 7fbd51400700 1 -- 172.21.15.23:0/1479571484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 msgr2=0x7fbd4c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.654+0000 7fbd51400700 1 --2- 172.21.15.23:0/1479571484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c1023d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd380099c0 tx=0x7fbd38009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 -- 172.21.15.23:0/1479571484 shutdown_connections 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 --2- 172.21.15.23:0/1479571484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c102910 0x7fbd4c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 --2- 172.21.15.23:0/1479571484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c100d80 0x7fbd4c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 --2- 172.21.15.23:0/1479571484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c1023d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 -- 172.21.15.23:0/1479571484 >> 172.21.15.23:0/1479571484 conn(0x7fbd4c0fc2f0 msgr2=0x7fbd4c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 -- 172.21.15.23:0/1479571484 shutdown_connections 2024-03-30T03:59:47.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.655+0000 7fbd51400700 1 -- 172.21.15.23:0/1479571484 wait complete. 2024-03-30T03:59:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 Processor -- start 2024-03-30T03:59:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 -- start start 2024-03-30T03:59:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 0x7fbd4c196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 0x7fbd4c19e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c19e830 con 0x7fbd4c101f70 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c19e9a0 con 0x7fbd4c102910 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd51400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4c19eb60 con 0x7fbd4c100d80 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd49e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd4a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 0x7fbd4c196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd49e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33116/0 (socket says 172.21.15.23:33116) 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.656+0000 7fbd49e00700 1 -- 172.21.15.23:0/167408281 learned_addr learned my addr 172.21.15.23:0/167408281 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd4b200700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 0x7fbd4c19e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 -- 172.21.15.23:0/167408281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 msgr2=0x7fbd4c196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 0x7fbd4c196d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 -- 172.21.15.23:0/167408281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 msgr2=0x7fbd4c19e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 0x7fbd4c19e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 -- 172.21.15.23:0/167408281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd380095f0 con 0x7fbd4c101f70 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd4a800700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 0x7fbd4c196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:47.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd4b200700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 0x7fbd4c19e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd49e00700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd380058e0 tx=0x7fbd3800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd43400700 1 -- 172.21.15.23:0/167408281 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3801d070 con 0x7fbd4c101f70 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd43400700 1 -- 172.21.15.23:0/167408281 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd38022470 con 0x7fbd4c101f70 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4c19ede0 con 0x7fbd4c101f70 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd43400700 1 -- 172.21.15.23:0/167408281 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd38017650 con 0x7fbd4c101f70 2024-03-30T03:59:47.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.657+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4c19f1b0 con 0x7fbd4c101f70 2024-03-30T03:59:47.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.662+0000 7fbd43400700 1 -- 172.21.15.23:0/167408281 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fbd38022850 con 0x7fbd4c101f70 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 Processor -- start 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 -- start start 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd43400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 0x7fbd300a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c634d0 0x558191c63d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191c647f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd43400700 1 -- 172.21.15.23:0/167408281 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd380cdd30 con 0x7fbd4c101f70 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191bb6940 con 0x558191c634d0 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191bb4300 con 0x558191c64370 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191bafbf0 con 0x558191c62b70 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.663+0000 7fbd4b200700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 0x7fbd300a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd49e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c634d0 0x558191c63d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191c647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55212/0 (socket says 172.21.15.23:55212) 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 -- 172.21.15.23:0/535252823 learned_addr learned my addr 172.21.15.23:0/535252823 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 -- 172.21.15.23:0/535252823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 msgr2=0x558191c647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191c647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 -- 172.21.15.23:0/535252823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c634d0 msgr2=0x558191c63d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c634d0 0x558191c63d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 -- 172.21.15.23:0/535252823 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558191a31360 con 0x558191c62b70 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4b200700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 0x7fbd300a35e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd440094f0 tx=0x7fbd44009240 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.664+0000 7fbd4a800700 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c011540 tx=0x7fbd3c011850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbea7b926f071181 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd37400700 1 -- 172.21.15.23:0/535252823 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c004310 con 0x558191c62b70 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd37400700 1 -- 172.21.15.23:0/535252823 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd3c004470 con 0x558191c62b70 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd37400700 1 -- 172.21.15.23:0/535252823 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c003680 con 0x558191c62b70 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/535252823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 msgr2=0x558191c62f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c011540 tx=0x7fbd3c011850 comp rx=0 tx=0).stop 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/535252823 shutdown_connections 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191c647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c62b70 0x558191c62f90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/535252823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c634d0 0x558191c63d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.665+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/535252823 >> 172.21.15.23:0/535252823 conn(0x558191bbd650 msgr2=0x558191c59860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.666+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/535252823 shutdown_connections 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.666+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/535252823 wait complete. 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.666+0000 7fbd64ed0b80 1 -- start start 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.666+0000 7fbd64ed0b80 1 Processor -- start 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c6c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c6cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191b4bd30 con 0x558191c64370 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191b4b9d0 con 0x558191c62b70 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191a33650 con 0x558191c634d0 2024-03-30T03:59:47.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd64ed0b80 10 client.0 _ll_register_callbacks cb 0x558191b55420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T03:59:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd4b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd49e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c6cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd4b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33136/0 (socket says 172.21.15.23:33136) 2024-03-30T03:59:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd4b200700 1 -- 172.21.15.23:0/2148964301 learned_addr learned my addr 172.21.15.23:0/2148964301 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.667+0000 7fbd4a800700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c6c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 -- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 msgr2=0x558191c6c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c6c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 -- 172.21.15.23:0/2148964301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 msgr2=0x558191c73f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 -- 172.21.15.23:0/2148964301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3c009d10 con 0x558191c634d0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd4b200700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd49e00700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c6cf10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c010 tx=0x7fbd3800fda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1488bfdded2c67 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd2be00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a10b0 con 0x558191c634d0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd3809b6c0 con 0x558191c634d0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.668+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558191b70330 con 0x558191c634d0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd3809afc0 con 0x558191c634d0 2024-03-30T03:59:47.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 10 client.24670 fetch_fsmap learned FSMap version 16 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x5581919ddb10 con 0x558191c634d0 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558191b70330 con 0x558191c634d0 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fbd38095bd0 con 0x558191c634d0 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 10 client.24670 fetch_fsmap finished waiting for FSMap version 16 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 10 client.24670 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 20 client.24670 populate_metadata read hostname 'smithi023' 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191c77700 0x558191c7ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 4 client.24670 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558191ba86b0 con 0x558191c77700 2024-03-30T03:59:47.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.669+0000 7fbd4a800700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191c77700 0x558191c7ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.670+0000 7fbd4a800700 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191c77700 0x558191c7ca00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef2e0439af9bc2f server_cookie=501819a8fc1a3e9b in_seq=0 out_seq=0 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.670+0000 7fbd2be00700 10 client.24670 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd2be00700 1 -- 172.21.15.23:0/2148964301 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558191ba86b0 con 0x558191c77700 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd2be00700 10 client.24670 handle_command_reply: tid=0 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 client.24670 shutdown 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 msgr2=0x558191c6cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c6cf10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800c010 tx=0x7fbd3800fda0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 shutdown_connections 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c73f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c6c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191c77700 0x558191c7ca00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/2148964301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c6cf10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd2be00700 0 client.24670 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 >> 172.21.15.23:0/2148964301 conn(0x558191bbd650 msgr2=0x558191c61530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 shutdown_connections 2024-03-30T03:59:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.671+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/2148964301 wait complete. 2024-03-30T03:59:47.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.672+0000 7fbd64ed0b80 20 client.24670 trim_cache size 0 max 16384 2024-03-30T03:59:47.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.690+0000 7fbd64ed0b80 1 Processor -- start 2024-03-30T03:59:47.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.690+0000 7fbd64ed0b80 1 -- start start 2024-03-30T03:59:47.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.690+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c62f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c58ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191ba86b0 con 0x558191c64370 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191b70330 con 0x558191c62b70 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191b6dad0 con 0x558191c634d0 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33142/0 (socket says 172.21.15.23:33142) 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 -- 172.21.15.23:0/3240762066 learned_addr learned my addr 172.21.15.23:0/3240762066 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4a800700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c62f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd49e00700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c58ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 -- 172.21.15.23:0/3240762066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 msgr2=0x558191c58ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c58ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 -- 172.21.15.23:0/3240762066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 msgr2=0x558191c62f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c62f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd4b200700 1 -- 172.21.15.23:0/3240762066 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558191b75e50 con 0x558191c64370 2024-03-30T03:59:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.691+0000 7fbd49e00700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c58ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd4b200700 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd44019ec0 tx=0x7fbd4401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b829f5d7b4f16289 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd37e00700 1 -- 172.21.15.23:0/3240762066 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd4400e050 con 0x558191c64370 2024-03-30T03:59:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd37e00700 1 -- 172.21.15.23:0/3240762066 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd4400f040 con 0x558191c64370 2024-03-30T03:59:47.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd37e00700 1 -- 172.21.15.23:0/3240762066 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd44026c00 con 0x558191c64370 2024-03-30T03:59:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/3240762066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 msgr2=0x558191c6bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd44019ec0 tx=0x7fbd4401f000 comp rx=0 tx=0).stop 2024-03-30T03:59:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/3240762066 shutdown_connections 2024-03-30T03:59:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c64370 0x558191c6bee0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c62b70 0x558191c62f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/3240762066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c58ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.692+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/3240762066 >> 172.21.15.23:0/3240762066 conn(0x558191bbd650 msgr2=0x558191c67580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.693+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/3240762066 shutdown_connections 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.693+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/3240762066 wait complete. 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.693+0000 7fbd64ed0b80 1 -- start start 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.693+0000 7fbd64ed0b80 1 Processor -- start 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c710a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191ccbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191c740c0 con 0x558191c62b70 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191c74230 con 0x558191c64370 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558191c743f0 con 0x558191c634d0 2024-03-30T03:59:47.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd64ed0b80 10 client.0 _ll_register_callbacks cb 0x558191b55420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33148/0 (socket says 172.21.15.23:33148) 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 -- 172.21.15.23:0/1471674086 learned_addr learned my addr 172.21.15.23:0/1471674086 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd49e00700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c710a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4b200700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191ccbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 -- 172.21.15.23:0/1471674086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 msgr2=0x558191c710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c710a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 -- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 msgr2=0x558191ccbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191ccbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.694+0000 7fbd4a800700 1 -- 172.21.15.23:0/1471674086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd44019c00 con 0x558191c62b70 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd49e00700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd4a800700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009340 tx=0x7fbd3c0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972bb7b891a8eb70 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:47.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd2be00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c02b040 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd3c0055b0 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbd3c004470 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd3c003680 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd64ed0b80 10 client.14676 fetch_fsmap learned FSMap version 16 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.695+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x558191a35c80 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd3c004470 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 16) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fbd3c02f780 con 0x558191c62b70 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 10 client.14676 fetch_fsmap finished waiting for FSMap version 16 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 10 client.14676 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 20 client.14676 populate_metadata read hostname 'smithi023' 2024-03-30T03:59:47.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191cd11b0 0x558191ccdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 4 client.14676 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5581919feca0 con 0x558191cd11b0 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.696+0000 7fbd49e00700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191cd11b0 0x558191ccdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd49e00700 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191cd11b0 0x558191ccdae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5597d88d71b65fcc server_cookie=b6ea7cfeddc3f81 in_seq=0 out_seq=0 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd2be00700 10 client.14676 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd2be00700 1 -- 172.21.15.23:0/1471674086 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5581919feca0 con 0x558191cd11b0 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd2be00700 10 client.14676 handle_command_reply: tid=0 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 client.14676 shutdown 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 msgr2=0x558191c70b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009340 tx=0x7fbd3c0092d0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 shutdown_connections 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558191c64370 0x558191ccbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558191c62b70 0x558191c70b60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558191cd11b0 0x558191ccdae0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 --2- 172.21.15.23:0/1471674086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558191c634d0 0x558191c710a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd2be00700 0 client.14676 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 >> 172.21.15.23:0/1471674086 conn(0x558191bbd650 msgr2=0x558191c59490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 shutdown_connections 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.697+0000 7fbd64ed0b80 1 -- 172.21.15.23:0/1471674086 wait complete. 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd64ed0b80 20 client.14676 trim_cache size 0 max 16384 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24569, 2024-03-30T03:59:47.701 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 7.2613941410000002, 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 16, 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 84, 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 83, 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 14.381229332 2024-03-30T03:59:47.702 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 msgr2=0x7fbd300a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 0x7fbd300a35e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd440094f0 tx=0x7fbd44009240 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 msgr2=0x7fbd4c197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd380058e0 tx=0x7fbd3800f690 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 shutdown_connections 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd4c102910 0x7fbd4c19e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd300a1120 0x7fbd300a35e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd4c100d80 0x7fbd4c196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 --2- 172.21.15.23:0/167408281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd4c101f70 0x7fbd4c197280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 >> 172.21.15.23:0/167408281 conn(0x7fbd4c0fc2f0 msgr2=0x7fbd4c1052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 shutdown_connections 2024-03-30T03:59:47.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:47.698+0000 7fbd51400700 1 -- 172.21.15.23:0/167408281 wait complete. 2024-03-30T03:59:47.715 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T03:59:47.715 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:47.715 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T03:59:48.079 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T03:59:48.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:47 smithi023 ceph-mon[38254]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T03:59:48.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:47 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T03:59:48.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:47 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1596181135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:47 smithi154 ceph-mon[79161]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T03:59:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:47 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T03:59:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:47 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1596181135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:48.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:47 smithi193 ceph-mon[79581]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-30T03:59:48.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:47 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T03:59:48.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:47 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1596181135' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:49.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 -- 172.21.15.23:0/628189318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c101820 msgr2=0x7f681c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 --2- 172.21.15.23:0/628189318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c101820 0x7f681c10dc10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f681400b0d0 tx=0x7f681400b3e0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 -- 172.21.15.23:0/628189318 shutdown_connections 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 --2- 172.21.15.23:0/628189318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c101820 0x7f681c10dc10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 --2- 172.21.15.23:0/628189318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c106ef0 0x7f681c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 --2- 172.21.15.23:0/628189318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 0x7f681c1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 -- 172.21.15.23:0/628189318 >> 172.21.15.23:0/628189318 conn(0x7f681c077fa0 msgr2=0x7f681c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.239+0000 7f6822c00700 1 -- 172.21.15.23:0/628189318 shutdown_connections 2024-03-30T03:59:49.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.240+0000 7f6822c00700 1 -- 172.21.15.23:0/628189318 wait complete. 2024-03-30T03:59:49.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 Processor -- start 2024-03-30T03:59:49.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 -- start start 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 0x7f681c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 0x7f681c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a5660 con 0x7f681c106ef0 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a57d0 con 0x7f681c100e60 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f6822c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1a5990 con 0x7f681c101820 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.241+0000 7f681b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 0x7f681c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f681b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55140/0 (socket says 172.21.15.23:55140) 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f681b400700 1 -- 172.21.15.23:0/582532566 learned_addr learned my addr 172.21.15.23:0/582532566 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f681be00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 0x7f681c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 -- 172.21.15.23:0/582532566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 msgr2=0x7f681c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 -- 172.21.15.23:0/582532566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 msgr2=0x7f681c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 0x7f681c0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 -- 172.21.15.23:0/582532566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814009d00 con 0x7f681c106ef0 2024-03-30T03:59:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f681b400700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6820c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 0x7f681c073b30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f681400b770 tx=0x7f681400bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:49.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681400e040 con 0x7f681c106ef0 2024-03-30T03:59:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.242+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c1a5c10 con 0x7f681c106ef0 2024-03-30T03:59:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.243+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f681401b460 con 0x7f681c106ef0 2024-03-30T03:59:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.243+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6814005290 con 0x7f681c106ef0 2024-03-30T03:59:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.243+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1a6080 con 0x7f681c106ef0 2024-03-30T03:59:49.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.244+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c1092f0 con 0x7f681c106ef0 2024-03-30T03:59:49.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.246+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f681401b5d0 con 0x7f681c106ef0 2024-03-30T03:59:49.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.247+0000 7f6818c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 0x7f68000a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:49.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.247+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f68140cd9a0 con 0x7f681c106ef0 2024-03-30T03:59:49.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.247+0000 7f681be00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 0x7f68000a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:49.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.248+0000 7f681be00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 0x7f68000a35e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f680c009c60 tx=0x7f680c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:49.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.251+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6814096f70 con 0x7f681c106ef0 2024-03-30T03:59:49.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:49 smithi023 ceph-mon[38254]: pgmap v240: 129 pgs: 7 creating+peering, 122 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-30T03:59:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:49 smithi154 ceph-mon[79161]: pgmap v240: 129 pgs: 7 creating+peering, 122 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-30T03:59:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:49 smithi193 ceph-mon[79581]: pgmap v240: 129 pgs: 7 creating+peering, 122 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s wr, 6 op/s 2024-03-30T03:59:49.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.510+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f681c074590 con 0x7f681c106ef0 2024-03-30T03:59:49.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.510+0000 7f6818c00700 1 -- 172.21.15.23:0/582532566 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f681409a590 con 0x7f681c106ef0 2024-03-30T03:59:49.509 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-30T03:59:49.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 msgr2=0x7f68000a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:49.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 0x7f68000a35e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f680c009c60 tx=0x7f680c009260 comp rx=0 tx=0).stop 2024-03-30T03:59:49.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 msgr2=0x7f681c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 0x7f681c073b30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f681400b770 tx=0x7f681400bcd0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 shutdown_connections 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c106ef0 0x7f681c073b30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c100e60 0x7f681c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f68000a1120 0x7f68000a35e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 --2- 172.21.15.23:0/582532566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c101820 0x7f681c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 >> 172.21.15.23:0/582532566 conn(0x7f681c077fa0 msgr2=0x7f681c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 shutdown_connections 2024-03-30T03:59:49.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:49.513+0000 7f6822c00700 1 -- 172.21.15.23:0/582532566 wait complete. 2024-03-30T03:59:49.919 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-30T03:59:49.919 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T03:59:49.920 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-30T03:59:49.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:49.920 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-30T03:59:49.963 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41807sec preferred_lft 41807sec 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T03:59:49.964 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T03:59:49.965 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-30T03:59:49.965 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-30T03:59:49.966 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T03:59:49.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:49.966 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T03:59:49.966 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T03:59:49.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-30T03:59:49.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-30T03:59:49.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-30T03:59:49.967 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T03:59:49.967 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T03:59:50.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:50 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/582532566' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T03:59:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:50 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/582532566' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T03:59:50.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:50.445 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T03:59:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:50 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/582532566' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T03:59:50.511 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-30T03:59:50.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:50.517 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-30T03:59:50.570 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-30T03:59:50.571 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T03:59:50.571 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-30T03:59:50.571 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-30T03:59:50.571 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T03:59:50.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:50.572 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T03:59:50.572 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T03:59:50.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-30T03:59:50.572 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-30T03:59:50.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-30T03:59:50.573 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T03:59:51.226 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:51 smithi023 ceph-mon[38254]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 900 B/s wr, 5 op/s 2024-03-30T03:59:51.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:51.229 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-30T03:59:51.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:51.273 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-30T03:59:51.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:51.356 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T03:59:51.356 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T03:59:51.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-30T03:59:51.356 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-30T03:59:51.356 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T03:59:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:51 smithi154 ceph-mon[79161]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 900 B/s wr, 5 op/s 2024-03-30T03:59:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:51 smithi193 ceph-mon[79581]: pgmap v241: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 900 B/s wr, 5 op/s 2024-03-30T03:59:51.535 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-30T03:59:51.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:51.535 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T03:59:51.535 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T03:59:51.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-30T03:59:51.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T03:59:51.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-30T03:59:51.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-30T03:59:51.536 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-30T03:59:51.536 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T03:59:51.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:51.808 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T03:59:51.808 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T03:59:51.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-30T03:59:51.808 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T03:59:51.808 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T03:59:51.977 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-30T03:59:51.977 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T03:59:51.977 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:52.019 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-30T03:59:52.019 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T03:59:52.019 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:52.101 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-30T03:59:52.208 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T03:59:52.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T03:59:52.274 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T03:59:52.276 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T03:59:52.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:52.277 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-30T03:59:52.318 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-30T03:59:52.318 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-30T03:59:52.363 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-30T03:59:52.430 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T03:59:52.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T03:59:52.501 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T03:59:52.505 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T03:59:52.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:52.506 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-30T03:59:52.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2024-03-30T03:59:52.548+0000 7fae4aad55c0 -1 init, newargv = 0x55ba1fe181c0 newargc=15 2024-03-30T03:59:52.547 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[91113]: starting ceph client 2024-03-30T03:59:52.557 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[91113]: starting fuse 2024-03-30T03:59:53.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-30T03:59:53.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:53 smithi154 ceph-mon[79161]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-03-30T03:59:53.715 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:53 smithi023 ceph-mon[38254]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-03-30T03:59:53.725 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-30T03:59:53.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-30T03:59:53.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:53 smithi193 ceph-mon[79581]: pgmap v242: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 3 op/s 2024-03-30T03:59:53.791 INFO:teuthology.orchestra.run.smithi023.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-30T03:59:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-30T03:59:53.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:53.794 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-30T03:59:53.836 INFO:teuthology.orchestra.run.smithi023.stdout:122 2024-03-30T03:59:53.836 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [122] 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> import re 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> import os 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-30T03:59:53.837 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> return f 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-30T03:59:53.838 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2024-03-30T03:59:53.839 DEBUG:teuthology.orchestra.run.smithi023:> ' 2024-03-30T03:59:54.019 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.91113.asok 2024-03-30T03:59:54.196 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.91113.asok 2024-03-30T03:59:54.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:54.196 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.91113.asok status 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "470100a4ccbcf7769052bf09a5413881ccf1b97f", 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 18.2.2-1018-g470100a4 (470100a4ccbcf7769052bf09a5413881ccf1b97f) reef (stable)", 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2024-03-30T03:59:54.424 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "91113", 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24698, 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 24698 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-30T03:59:54.425 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1727927136 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1727927136 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.24698 172.21.15.23:0/1727927136", 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "172.21.15.23:0/1727927136", 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 16, 2024-03-30T03:59:54.426 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 84, 2024-03-30T03:59:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2024-03-30T03:59:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2024-03-30T03:59:54.427 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2024-03-30T03:59:54.427 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T03:59:54.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:54.439 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2024-03-30T03:59:54.439 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:54.482 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2024-03-30T03:59:54.482 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:54.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:54.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-30T03:59:54.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T03:59:54.591 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-30T03:59:54.720 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2024-03-30T03:59:54.721 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-30T03:59:54.721 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:54.722 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:59:54.722 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T03:59:54.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 -- 172.21.15.23:0/816307719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20101e70 msgr2=0x7f6c201022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 --2- 172.21.15.23:0/816307719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20101e70 0x7f6c201022d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c0099c0 tx=0x7f6c0c009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 -- 172.21.15.23:0/816307719 shutdown_connections 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 --2- 172.21.15.23:0/816307719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 0x7f6c201090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 --2- 172.21.15.23:0/816307719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20100c80 0x7f6c201010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 --2- 172.21.15.23:0/816307719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20101e70 0x7f6c201022d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 -- 172.21.15.23:0/816307719 >> 172.21.15.23:0/816307719 conn(0x7f6c200fc1f0 msgr2=0x7f6c200fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 -- 172.21.15.23:0/816307719 shutdown_connections 2024-03-30T03:59:55.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.003+0000 7f6c26800700 1 -- 172.21.15.23:0/816307719 wait complete. 2024-03-30T03:59:55.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.004+0000 7f6c26800700 1 Processor -- start 2024-03-30T03:59:55.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 -- start start 2024-03-30T03:59:55.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20101e70 0x7f6c20199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 0x7f6c201a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201a0950 con 0x7f6c20100c80 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201a0a90 con 0x7f6c20102810 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c26800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201a0c20 con 0x7f6c20101e70 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c1f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c1f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53604/0 (socket says 172.21.15.23:53604) 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c1f400700 1 -- 172.21.15.23:0/1484357480 learned_addr learned my addr 172.21.15.23:0/1484357480 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c1ea00700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20101e70 0x7f6c20199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.005+0000 7f6c1fe00700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 0x7f6c201a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 -- 172.21.15.23:0/1484357480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20101e70 msgr2=0x7f6c20199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20101e70 0x7f6c20199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 -- 172.21.15.23:0/1484357480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 msgr2=0x7f6c201a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 0x7f6c201a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 -- 172.21.15.23:0/1484357480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c0095f0 con 0x7f6c20100c80 2024-03-30T03:59:55.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c1f400700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000e970 tx=0x7f6c1000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10009790 con 0x7f6c20100c80 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c1000c7e0 con 0x7f6c20100c80 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c201a0f00 con 0x7f6c20100c80 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.006+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10003c70 con 0x7f6c20100c80 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.007+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c201a1320 con 0x7f6c20100c80 2024-03-30T03:59:55.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.008+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c20067a80 con 0x7f6c20100c80 2024-03-30T03:59:55.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.009+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6c1000cc50 con 0x7f6c20100c80 2024-03-30T03:59:55.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.011+0000 7f6c17e00700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 0x7f6c040a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.011+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6c10014070 con 0x7f6c20100c80 2024-03-30T03:59:55.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.011+0000 7f6c1ea00700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 0x7f6c040a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.012+0000 7f6c1ea00700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 0x7f6c040a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c003730 tx=0x7f6c0c00b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.015+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6c20067a80 con 0x7f6c20100c80 2024-03-30T03:59:55.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.273+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c2010c4e0 con 0x7f6c20100c80 2024-03-30T03:59:55.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.274+0000 7f6c17e00700 1 -- 172.21.15.23:0/1484357480 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7f6c10096230 con 0x7f6c20100c80 2024-03-30T03:59:55.273 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:55.274 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 msgr2=0x7f6c040a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 0x7f6c040a35e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c003730 tx=0x7f6c0c00b540 comp rx=0 tx=0).stop 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 msgr2=0x7f6c20198e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000e970 tx=0x7f6c1000ec80 comp rx=0 tx=0).stop 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 shutdown_connections 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c20102810 0x7f6c201a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6c20100c80 0x7f6c20198e20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6c040a1120 0x7f6c040a35e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 --2- 172.21.15.23:0/1484357480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c20101e70 0x7f6c20199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 >> 172.21.15.23:0/1484357480 conn(0x7f6c200fc1f0 msgr2=0x7f6c20072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:55.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 shutdown_connections 2024-03-30T03:59:55.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.277+0000 7f6c26800700 1 -- 172.21.15.23:0/1484357480 wait complete. 2024-03-30T03:59:55.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T03:59:55.295 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.497+0000 7f442c800700 1 -- 172.21.15.23:0/1851122104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424100d80 msgr2=0x7f44241011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.497+0000 7f442c800700 1 --2- 172.21.15.23:0/1851122104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424100d80 0x7f44241011a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4418009920 tx=0x7f4418009c30 comp rx=0 tx=0).stop 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 -- 172.21.15.23:0/1851122104 shutdown_connections 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 --2- 172.21.15.23:0/1851122104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424102910 0x7f44241091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 --2- 172.21.15.23:0/1851122104 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424100d80 0x7f44241011a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 --2- 172.21.15.23:0/1851122104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424101f70 0x7f44241023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 -- 172.21.15.23:0/1851122104 >> 172.21.15.23:0/1851122104 conn(0x7f44240fc2f0 msgr2=0x7f44240fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 -- 172.21.15.23:0/1851122104 shutdown_connections 2024-03-30T03:59:55.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.498+0000 7f442c800700 1 -- 172.21.15.23:0/1851122104 wait complete. 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 Processor -- start 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 -- start start 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 0x7f4424198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424102910 0x7f44241a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44241a0af0 con 0x7f4424101f70 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44241a0c30 con 0x7f4424100d80 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.499+0000 7f442c800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44241a0dc0 con 0x7f4424102910 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 0x7f4424198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53622/0 (socket says 172.21.15.23:53622) 2024-03-30T03:59:55.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 -- 172.21.15.23:0/1129642015 learned_addr learned my addr 172.21.15.23:0/1129642015 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f442a800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424102910 0x7f44241a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 -- 172.21.15.23:0/1129642015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424102910 msgr2=0x7f44241a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424102910 0x7f44241a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 -- 172.21.15.23:0/1129642015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 msgr2=0x7f4424198fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 0x7f4424198fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 -- 172.21.15.23:0/1129642015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009520 con 0x7f4424101f70 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429e00700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 0x7f4424198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.500+0000 7f4429400700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f441400e9c0 tx=0x7f441400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.501+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414004a30 con 0x7f4424101f70 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.501+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f441400cc80 con 0x7f4424101f70 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.501+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44140054e0 con 0x7f4424101f70 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.501+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44241a10a0 con 0x7f4424101f70 2024-03-30T03:59:55.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.501+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44241a15b0 con 0x7f4424101f70 2024-03-30T03:59:55.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.502+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4424067af0 con 0x7f4424101f70 2024-03-30T03:59:55.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.504+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4414004b90 con 0x7f4424101f70 2024-03-30T03:59:55.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.505+0000 7f441ea00700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 0x7f440c0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.506+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4414014070 con 0x7f4424101f70 2024-03-30T03:59:55.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.506+0000 7f4429e00700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 0x7f440c0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.506+0000 7f4429e00700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 0x7f440c0a3660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f44180098f0 tx=0x7f4418019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.509+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f44140960c0 con 0x7f4424101f70 2024-03-30T03:59:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:55 smithi154 ceph-mon[79161]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s wr, 2 op/s 2024-03-30T03:59:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:55 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1484357480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:55.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.726+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f442410c7b0 con 0x7f4424101f70 2024-03-30T03:59:55.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.726+0000 7f441ea00700 1 -- 172.21.15.23:0/1129642015 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44140996e0 con 0x7f4424101f70 2024-03-30T03:59:55.725 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T03:59:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.728+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 msgr2=0x7f440c0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.728+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 0x7f440c0a3660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f44180098f0 tx=0x7f4418019040 comp rx=0 tx=0).stop 2024-03-30T03:59:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 msgr2=0x7f4424199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f441400e9c0 tx=0x7f441400ecd0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 shutdown_connections 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f440c0a11a0 0x7f440c0a3660 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4424102910 0x7f44241a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4424100d80 0x7f4424198fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 --2- 172.21.15.23:0/1129642015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4424101f70 0x7f4424199500 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 >> 172.21.15.23:0/1129642015 conn(0x7f44240fc2f0 msgr2=0x7f4424107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 shutdown_connections 2024-03-30T03:59:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.729+0000 7f442c800700 1 -- 172.21.15.23:0/1129642015 wait complete. 2024-03-30T03:59:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:55 smithi193 ceph-mon[79581]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s wr, 2 op/s 2024-03-30T03:59:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1484357480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:55.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:55 smithi023 ceph-mon[38254]: pgmap v243: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s wr, 2 op/s 2024-03-30T03:59:55.744 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1484357480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:55.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T03:59:55.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 -- 172.21.15.23:0/814833008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028074df0 msgr2=0x7f2028073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 --2- 172.21.15.23:0/814833008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028074df0 0x7f2028073240 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f201c009990 tx=0x7f201c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 -- 172.21.15.23:0/814833008 shutdown_connections 2024-03-30T03:59:55.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 --2- 172.21.15.23:0/814833008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f202810b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 --2- 172.21.15.23:0/814833008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028074df0 0x7f2028073240 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 --2- 172.21.15.23:0/814833008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028073810 0x7f2028073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 -- 172.21.15.23:0/814833008 >> 172.21.15.23:0/814833008 conn(0x7f20280fc2f0 msgr2=0x7f20280fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:55.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 -- 172.21.15.23:0/814833008 shutdown_connections 2024-03-30T03:59:55.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.951+0000 7f202fa00700 1 -- 172.21.15.23:0/814833008 wait complete. 2024-03-30T03:59:55.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.952+0000 7f202fa00700 1 Processor -- start 2024-03-30T03:59:55.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.952+0000 7f202fa00700 1 -- start start 2024-03-30T03:59:55.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.952+0000 7f202fa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202fa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028074df0 0x7f2028199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202fa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f20281a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202fa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20281a0a20 con 0x7f2028073810 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202fa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20281a0b60 con 0x7f2028108c40 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202fa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20281a0cf0 con 0x7f2028074df0 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202da00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f20281a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53634/0 (socket says 172.21.15.23:53634) 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 -- 172.21.15.23:0/3647359020 learned_addr learned my addr 172.21.15.23:0/3647359020 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f2027e00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028074df0 0x7f2028199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 -- 172.21.15.23:0/3647359020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028074df0 msgr2=0x7f2028199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028074df0 0x7f2028199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 -- 172.21.15.23:0/3647359020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 msgr2=0x7f20281a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f20281a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202d000700 1 -- 172.21.15.23:0/3647359020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c0095f0 con 0x7f2028073810 2024-03-30T03:59:55.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.953+0000 7f202da00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f20281a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f202d000700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f201c000c00 tx=0x7f201c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c01d070 con 0x7f2028073810 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f201c021410 con 0x7f2028073810 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20281a0f70 con 0x7f2028073810 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201c017650 con 0x7f2028073810 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.954+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20281a1330 con 0x7f2028073810 2024-03-30T03:59:55.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.955+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2028107a00 con 0x7f2028073810 2024-03-30T03:59:55.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.957+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f201c0217f0 con 0x7f2028073810 2024-03-30T03:59:55.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.958+0000 7f2025600700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 0x7f20100a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:55.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.958+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f201c0cdf20 con 0x7f2028073810 2024-03-30T03:59:55.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.958+0000 7f2027e00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 0x7f20100a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:55.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.959+0000 7f2027e00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 0x7f20100a3660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2028074b10 tx=0x7f201400b410 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:55.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:55.962+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f201c0974f0 con 0x7f2028073810 2024-03-30T03:59:56.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.178+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f202802ccb0 con 0x7f2028073810 2024-03-30T03:59:56.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.178+0000 7f2025600700 1 -- 172.21.15.23:0/3647359020 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f201c09ab10 con 0x7f2028073810 2024-03-30T03:59:56.177 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 msgr2=0x7f20100a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 0x7f20100a3660 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2028074b10 tx=0x7f201400b410 comp rx=0 tx=0).stop 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 msgr2=0x7f2028198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f201c000c00 tx=0x7f201c005740 comp rx=0 tx=0).stop 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 shutdown_connections 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2028108c40 0x7f20281a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2028073810 0x7f2028198ef0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f20100a11a0 0x7f20100a3660 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 --2- 172.21.15.23:0/3647359020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2028074df0 0x7f2028199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 >> 172.21.15.23:0/3647359020 conn(0x7f20280fc2f0 msgr2=0x7f2028104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:56.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 shutdown_connections 2024-03-30T03:59:56.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.181+0000 7f202fa00700 1 -- 172.21.15.23:0/3647359020 wait complete. 2024-03-30T03:59:56.198 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T03:59:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 -- 172.21.15.23:0/1995358875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 msgr2=0x7f7c741023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 --2- 172.21.15.23:0/1995358875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c741023d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7c600099c0 tx=0x7f7c60009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 -- 172.21.15.23:0/1995358875 shutdown_connections 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 --2- 172.21.15.23:0/1995358875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 0x7f7c741091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 --2- 172.21.15.23:0/1995358875 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c741011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 --2- 172.21.15.23:0/1995358875 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c741023d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.402+0000 7f7c7a600700 1 -- 172.21.15.23:0/1995358875 >> 172.21.15.23:0/1995358875 conn(0x7f7c740fc2f0 msgr2=0x7f7c740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.403+0000 7f7c7a600700 1 -- 172.21.15.23:0/1995358875 shutdown_connections 2024-03-30T03:59:56.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.403+0000 7f7c7a600700 1 -- 172.21.15.23:0/1995358875 wait complete. 2024-03-30T03:59:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 Processor -- start 2024-03-30T03:59:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 -- start start 2024-03-30T03:59:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 0x7f7c741a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741a0a50 con 0x7f7c74101f70 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741a0b90 con 0x7f7c74100d80 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c7a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c741a0d20 con 0x7f7c74102910 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c73400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c72a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c73400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56504/0 (socket says 172.21.15.23:56504) 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c73400700 1 -- 172.21.15.23:0/2884950901 learned_addr learned my addr 172.21.15.23:0/2884950901 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.404+0000 7f7c72a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53654/0 (socket says 172.21.15.23:53654) 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73e00700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 0x7f7c741a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 -- 172.21.15.23:0/2884950901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 msgr2=0x7f7c741a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 0x7f7c741a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 -- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 msgr2=0x7f7c74199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 -- 172.21.15.23:0/2884950901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c64009520 con 0x7f7c74100d80 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c72a00700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c73400700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400ea50 tx=0x7f7c6400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.405+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400cbd0 con 0x7f7c74100d80 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.406+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c600095f0 con 0x7f7c74100d80 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.406+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6400cd30 con 0x7f7c74100d80 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.406+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64018680 con 0x7f7c74100d80 2024-03-30T03:59:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.406+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c741a1290 con 0x7f7c74100d80 2024-03-30T03:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.407+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74067af0 con 0x7f7c74100d80 2024-03-30T03:59:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.409+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f7c64010430 con 0x7f7c74100d80 2024-03-30T03:59:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.410+0000 7f7c6be00700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 0x7f7c580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.410+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7c64014070 con 0x7f7c74100d80 2024-03-30T03:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.410+0000 7f7c72a00700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 0x7f7c580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.411+0000 7f7c72a00700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 0x7f7c580a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7c60003730 tx=0x7f7c6001a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:56.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.414+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7c64096540 con 0x7f7c74100d80 2024-03-30T03:59:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:56 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1129642015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:56 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3647359020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.636+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c741a15a0 con 0x7f7c74100d80 2024-03-30T03:59:56.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.636+0000 7f7c6be00700 1 -- 172.21.15.23:0/2884950901 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c64099b60 con 0x7f7c74100d80 2024-03-30T03:59:56.635 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 msgr2=0x7f7c580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 0x7f7c580a35e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7c60003730 tx=0x7f7c6001a040 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 msgr2=0x7f7c74198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400ea50 tx=0x7f7c6400ed60 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 shutdown_connections 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f7c580a1120 0x7f7c580a35e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c74102910 0x7f7c741a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c74100d80 0x7f7c74198f20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 --2- 172.21.15.23:0/2884950901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c74101f70 0x7f7c74199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 >> 172.21.15.23:0/2884950901 conn(0x7f7c740fc2f0 msgr2=0x7f7c74107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 shutdown_connections 2024-03-30T03:59:56.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.639+0000 7f7c7a600700 1 -- 172.21.15.23:0/2884950901 wait complete. 2024-03-30T03:59:56.656 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T03:59:56.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1129642015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3647359020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:56 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1129642015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:56 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3647359020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:56.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 -- 172.21.15.23:0/2794461925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0104110 msgr2=0x7f89a0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 --2- 172.21.15.23:0/2794461925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0104110 0x7f89a0106500 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f898c009990 tx=0x7f898c009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 -- 172.21.15.23:0/2794461925 shutdown_connections 2024-03-30T03:59:56.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 --2- 172.21.15.23:0/2794461925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a0106a40 0x7f89a0108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 --2- 172.21.15.23:0/2794461925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a01017e0 0x7f89a0103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 --2- 172.21.15.23:0/2794461925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0104110 0x7f89a0106500 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 -- 172.21.15.23:0/2794461925 >> 172.21.15.23:0/2794461925 conn(0x7f89a00fb1d0 msgr2=0x7f89a00fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:56.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 -- 172.21.15.23:0/2794461925 shutdown_connections 2024-03-30T03:59:56.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.858+0000 7f89a4a00700 1 -- 172.21.15.23:0/2794461925 wait complete. 2024-03-30T03:59:56.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.859+0000 7f89a4a00700 1 Processor -- start 2024-03-30T03:59:56.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.859+0000 7f89a4a00700 1 -- start start 2024-03-30T03:59:56.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 0x7f89a0196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 0x7f89a019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a019e7f0 con 0x7f89a0106a40 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a019e960 con 0x7f89a0104110 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f89a4a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a019eb20 con 0x7f89a01017e0 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f899d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f899de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 0x7f89a0196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f899d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56512/0 (socket says 172.21.15.23:56512) 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f899d400700 1 -- 172.21.15.23:0/3485158238 learned_addr learned my addr 172.21.15.23:0/3485158238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.860+0000 7f899e800700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 0x7f89a019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 -- 172.21.15.23:0/3485158238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 msgr2=0x7f89a0196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 0x7f89a0196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 -- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 msgr2=0x7f89a019e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:56.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 0x7f89a019e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 -- 172.21.15.23:0/3485158238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0095f0 con 0x7f89a0104110 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899de00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 0x7f89a0196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899e800700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 0x7f89a019e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f899d400700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f898c005910 tx=0x7f898c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c01d070 con 0x7f89a0104110 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f898c021410 con 0x7f89a0104110 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c005720 con 0x7f89a0104110 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.861+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a019eda0 con 0x7f89a0104110 2024-03-30T03:59:56.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.862+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a019f170 con 0x7f89a0104110 2024-03-30T03:59:56.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.863+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a00fcd50 con 0x7f89a0104110 2024-03-30T03:59:56.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.865+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f898c021580 con 0x7f89a0104110 2024-03-30T03:59:56.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.866+0000 7f8996a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 0x7f89840a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:56.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.866+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f898c0ce480 con 0x7f89a0104110 2024-03-30T03:59:56.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.866+0000 7f899de00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 0x7f89840a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:56.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.867+0000 7f899de00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 0x7f89840a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f89a0102f70 tx=0x7f8990009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:56.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:56.868+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f898c097a50 con 0x7f89a0104110 2024-03-30T03:59:57.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.085+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f89a010c5a0 con 0x7f89a0104110 2024-03-30T03:59:57.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.086+0000 7f8996a00700 1 -- 172.21.15.23:0/3485158238 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v84) v1 ==== 103+0+11 (secure 0 0 0) 0x7f898c026020 con 0x7f89a0104110 2024-03-30T03:59:57.085 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 msgr2=0x7f89840a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 0x7f89840a3660 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f89a0102f70 tx=0x7f8990009260 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 msgr2=0x7f89a0197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f898c005910 tx=0x7f898c009580 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 shutdown_connections 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89a0106a40 0x7f89a019e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f89840a11a0 0x7f89840a3660 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89a01017e0 0x7f89a0196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 --2- 172.21.15.23:0/3485158238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f89a0104110 0x7f89a0197210 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 >> 172.21.15.23:0/3485158238 conn(0x7f89a00fb1d0 msgr2=0x7f89a0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:57.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 shutdown_connections 2024-03-30T03:59:57.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.089+0000 7f89a4a00700 1 -- 172.21.15.23:0/3485158238 wait complete. 2024-03-30T03:59:57.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 -- 172.21.15.23:0/3815771607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c101f70 msgr2=0x7f821c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 --2- 172.21.15.23:0/3815771607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c101f70 0x7f821c1023d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f82080099c0 tx=0x7f8208009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 -- 172.21.15.23:0/3815771607 shutdown_connections 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 --2- 172.21.15.23:0/3815771607 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 0x7f821c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 --2- 172.21.15.23:0/3815771607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c100d80 0x7f821c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 --2- 172.21.15.23:0/3815771607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c101f70 0x7f821c1023d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 -- 172.21.15.23:0/3815771607 >> 172.21.15.23:0/3815771607 conn(0x7f821c0fc2f0 msgr2=0x7f821c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 -- 172.21.15.23:0/3815771607 shutdown_connections 2024-03-30T03:59:57.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.306+0000 7f8223e00700 1 -- 172.21.15.23:0/3815771607 wait complete. 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.307+0000 7f8223e00700 1 Processor -- start 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.307+0000 7f8223e00700 1 -- start start 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c101f70 0x7f821c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 0x7f821c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c073e80 con 0x7f821c100d80 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c073ff0 con 0x7f821c102910 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8223e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c1a0db0 con 0x7f821c101f70 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53694/0 (socket says 172.21.15.23:53694) 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 -- 172.21.15.23:0/2664291592 learned_addr learned my addr 172.21.15.23:0/2664291592 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8220a00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c101f70 0x7f821c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 0x7f821c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 -- 172.21.15.23:0/2664291592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c101f70 msgr2=0x7f821c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c101f70 0x7f821c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 -- 172.21.15.23:0/2664291592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 msgr2=0x7f821c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 0x7f821c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 -- 172.21.15.23:0/2664291592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080095f0 con 0x7f821c100d80 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.308+0000 7f8221400700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f821000e970 tx=0x7f821000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.309+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210009790 con 0x7f821c100d80 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.309+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82100044e0 con 0x7f821c100d80 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.309+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210003d80 con 0x7f821c100d80 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.309+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c1a0fe0 con 0x7f821c100d80 2024-03-30T03:59:57.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.309+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1a1410 con 0x7f821c100d80 2024-03-30T03:59:57.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.310+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c108250 con 0x7f821c100d80 2024-03-30T03:59:57.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.312+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8210003ee0 con 0x7f821c100d80 2024-03-30T03:59:57.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.313+0000 7f821a000700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 0x7f82040a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.313+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8210014070 con 0x7f821c100d80 2024-03-30T03:59:57.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.313+0000 7f8220a00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 0x7f82040a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.314+0000 7f8220a00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 0x7f82040a3660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f820800b580 tx=0x7f820801a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:57.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.317+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8210096290 con 0x7f821c100d80 2024-03-30T03:59:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:57 smithi023 ceph-mon[38254]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 181 B/s wr, 1 op/s 2024-03-30T03:59:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:57 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2884950901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:57 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3485158238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.592+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f821c067af0 con 0x7f821c100d80 2024-03-30T03:59:57.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.593+0000 7f821a000700 1 -- 172.21.15.23:0/2664291592 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v84) v1 ==== 81+0+588 (secure 0 0 0) 0x7f8210004190 con 0x7f821c100d80 2024-03-30T03:59:57.591 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1161676542 2024-03-31T03:54:16.197829+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/443135190 2024-03-31T03:54:16.197829+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2173844842 2024-03-31T03:53:51.749812+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2471664699 2024-03-31T03:53:51.749812+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/783324330 2024-03-31T03:53:51.749812+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1219945839 2024-03-31T03:54:16.197829+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T03:59:57.592 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T03:59:57.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.595+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 msgr2=0x7f82040a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.595+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 0x7f82040a3660 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f820800b580 tx=0x7f820801a040 comp rx=0 tx=0).stop 2024-03-30T03:59:57.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.595+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 msgr2=0x7f821c074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.595+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f821000e970 tx=0x7f821000ed30 comp rx=0 tx=0).stop 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.595+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 shutdown_connections 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f821c102910 0x7f821c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f821c100d80 0x7f821c074b10 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f82040a11a0 0x7f82040a3660 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 --2- 172.21.15.23:0/2664291592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f821c101f70 0x7f821c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 >> 172.21.15.23:0/2664291592 conn(0x7f821c0fc2f0 msgr2=0x7f821c105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 shutdown_connections 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.596+0000 7f8223e00700 1 -- 172.21.15.23:0/2664291592 wait complete. 2024-03-30T03:59:57.595 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-30T03:59:57.613 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T03:59:57.613 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T03:59:57.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T03:59:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:57 smithi154 ceph-mon[79161]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 181 B/s wr, 1 op/s 2024-03-30T03:59:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:57 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2884950901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:57 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3485158238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:57 smithi193 ceph-mon[79581]: pgmap v244: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 181 B/s wr, 1 op/s 2024-03-30T03:59:57.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:57 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2884950901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:57 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3485158238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T03:59:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 -- 172.21.15.23:0/2111407769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 msgr2=0x7f4d881010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2111407769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d881010a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4d780099c0 tx=0x7f4d78009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 -- 172.21.15.23:0/2111407769 shutdown_connections 2024-03-30T03:59:57.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2111407769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88102810 0x7f4d881090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2111407769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d881010a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2111407769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88101e70 0x7f4d881022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 -- 172.21.15.23:0/2111407769 >> 172.21.15.23:0/2111407769 conn(0x7f4d880fc1f0 msgr2=0x7f4d880fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 -- 172.21.15.23:0/2111407769 shutdown_connections 2024-03-30T03:59:57.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.894+0000 7f4d8e000700 1 -- 172.21.15.23:0/2111407769 wait complete. 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.895+0000 7f4d8e000700 1 Processor -- start 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.895+0000 7f4d8e000700 1 -- start start 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101e70 0x7f4d880731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88102810 0x7f4d880736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88073e80 con 0x7f4d88100c80 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88073ff0 con 0x7f4d88101e70 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d8e000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d881a0c60 con 0x7f4d88102810 2024-03-30T03:59:57.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d87400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d87400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53720/0 (socket says 172.21.15.23:53720) 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d87400700 1 -- 172.21.15.23:0/2820774166 learned_addr learned my addr 172.21.15.23:0/2820774166 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d86a00700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101e70 0x7f4d880731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.896+0000 7f4d87e00700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88102810 0x7f4d880736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 -- 172.21.15.23:0/2820774166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88102810 msgr2=0x7f4d880736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88102810 0x7f4d880736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 -- 172.21.15.23:0/2820774166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101e70 msgr2=0x7f4d880731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101e70 0x7f4d880731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 -- 172.21.15.23:0/2820774166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780095f0 con 0x7f4d88100c80 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d87400700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4d78003730 tx=0x7f4d7800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7801d070 con 0x7f4d88100c80 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d78005050 con 0x7f4d88100c80 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d881a0e90 con 0x7f4d88100c80 2024-03-30T03:59:57.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004a50 con 0x7f4d88100c80 2024-03-30T03:59:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.897+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d881a1200 con 0x7f4d88100c80 2024-03-30T03:59:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.898+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88108100 con 0x7f4d88100c80 2024-03-30T03:59:57.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.900+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f4d780051c0 con 0x7f4d88100c80 2024-03-30T03:59:57.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.901+0000 7f4d7fe00700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 0x7f4d6c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:57.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.901+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4d780cd890 con 0x7f4d88100c80 2024-03-30T03:59:57.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.902+0000 7f4d86a00700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 0x7f4d6c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:57.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.902+0000 7f4d86a00700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 0x7f4d6c0a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009c60 tx=0x7f4d74009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:57.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:57.905+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4d78096e60 con 0x7f4d88100c80 2024-03-30T03:59:58.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.161+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d88067a10 con 0x7f4d88100c80 2024-03-30T03:59:58.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.162+0000 7f4d7fe00700 1 -- 172.21.15.23:0/2820774166 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7f4d7809a480 con 0x7f4d88100c80 2024-03-30T03:59:58.161 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T03:59:58.161 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 msgr2=0x7f4d6c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 0x7f4d6c0a35e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4d74009c60 tx=0x7f4d74009260 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 msgr2=0x7f4d88074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4d78003730 tx=0x7f4d7800f690 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 shutdown_connections 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f4d6c0a1120 0x7f4d6c0a35e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4d88102810 0x7f4d880736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4d88100c80 0x7f4d88074b10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 --2- 172.21.15.23:0/2820774166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d88101e70 0x7f4d880731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 >> 172.21.15.23:0/2820774166 conn(0x7f4d880fc1f0 msgr2=0x7f4d88105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 shutdown_connections 2024-03-30T03:59:58.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.165+0000 7f4d8e000700 1 -- 172.21.15.23:0/2820774166 wait complete. 2024-03-30T03:59:58.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T03:59:58.182 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T03:59:58.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 -- 172.21.15.23:0/3121338355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 msgr2=0x7fc0c01091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/3121338355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c01091c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800b0d0 tx=0x7fc0b800b3e0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 -- 172.21.15.23:0/3121338355 shutdown_connections 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/3121338355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c01091c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/3121338355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 0x7fc0c01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/3121338355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 -- 172.21.15.23:0/3121338355 >> 172.21.15.23:0/3121338355 conn(0x7fc0c00fc2f0 msgr2=0x7fc0c00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 -- 172.21.15.23:0/3121338355 shutdown_connections 2024-03-30T03:59:58.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.388+0000 7fc0c6c00700 1 -- 172.21.15.23:0/3121338355 wait complete. 2024-03-30T03:59:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 Processor -- start 2024-03-30T03:59:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 -- start start 2024-03-30T03:59:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 0x7fc0c0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0073e80 con 0x7fc0c0102910 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0073ff0 con 0x7fc0c0101f70 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c6c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c01a0eb0 con 0x7fc0c0100d80 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59230/0 (socket says 172.21.15.23:59230) 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0c4c00700 1 -- 172.21.15.23:0/1929040238 learned_addr learned my addr 172.21.15.23:0/1929040238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.390+0000 7fc0bfe00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 0x7fc0c0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0bf400700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 -- 172.21.15.23:0/1929040238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 msgr2=0x7fc0c0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 0x7fc0c0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 -- 172.21.15.23:0/1929040238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 msgr2=0x7fc0c00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 -- 172.21.15.23:0/1929040238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b8009d00 con 0x7fc0c0102910 2024-03-30T03:59:58.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0bf400700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c4c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8007b60 tx=0x7fc0b8008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800e040 con 0x7fc0c0102910 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0b801d930 con 0x7fc0c0102910 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c01a10e0 con 0x7fc0c0102910 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b801ba40 con 0x7fc0c0102910 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.391+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c01a1500 con 0x7fc0c0102910 2024-03-30T03:59:58.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.393+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0108180 con 0x7fc0c0102910 2024-03-30T03:59:58.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.394+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc0b801daa0 con 0x7fc0c0102910 2024-03-30T03:59:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.396+0000 7fc0bcc00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 0x7fc0a40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.396+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc0b80cdff0 con 0x7fc0c0102910 2024-03-30T03:59:58.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.396+0000 7fc0bfe00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 0x7fc0a40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.397+0000 7fc0bfe00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 0x7fc0a40a3660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0101dd0 tx=0x7fc0b0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:58.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.400+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc0b80975c0 con 0x7fc0c0102910 2024-03-30T03:59:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:58 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2664291592' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T03:59:58.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:58 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2820774166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.618+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0c010c7f0 con 0x7fc0c0102910 2024-03-30T03:59:58.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.618+0000 7fc0bcc00700 1 -- 172.21.15.23:0/1929040238 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0b809abe0 con 0x7fc0c0102910 2024-03-30T03:59:58.618 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 msgr2=0x7fc0a40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 0x7fc0a40a3660 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0101dd0 tx=0x7fc0b0009260 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 msgr2=0x7fc0c00736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8007b60 tx=0x7fc0b8008e40 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 shutdown_connections 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0c0102910 0x7fc0c00736e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0a40a11a0 0x7fc0a40a3660 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0c0100d80 0x7fc0c0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 --2- 172.21.15.23:0/1929040238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0c0101f70 0x7fc0c00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 >> 172.21.15.23:0/1929040238 conn(0x7fc0c00fc2f0 msgr2=0x7fc0c01052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:58.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 shutdown_connections 2024-03-30T03:59:58.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.621+0000 7fc0c6c00700 1 -- 172.21.15.23:0/1929040238 wait complete. 2024-03-30T03:59:58.638 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T03:59:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:58 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2664291592' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T03:59:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:58 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2820774166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:58.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:58 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2664291592' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T03:59:58.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:58 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2820774166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 -- 172.21.15.23:0/3858531937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68100d80 msgr2=0x7f8d681011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/3858531937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68100d80 0x7f8d681011a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009920 tx=0x7f8d58009c30 comp rx=0 tx=0).stop 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 -- 172.21.15.23:0/3858531937 shutdown_connections 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/3858531937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 0x7f8d681091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/3858531937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68100d80 0x7f8d681011a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/3858531937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68101f70 0x7f8d681023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 -- 172.21.15.23:0/3858531937 >> 172.21.15.23:0/3858531937 conn(0x7f8d680fc2f0 msgr2=0x7f8d680fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:58.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 -- 172.21.15.23:0/3858531937 shutdown_connections 2024-03-30T03:59:58.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.842+0000 7f8d6ce00700 1 -- 172.21.15.23:0/3858531937 wait complete. 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.843+0000 7f8d6ce00700 1 Processor -- start 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.843+0000 7f8d6ce00700 1 -- start start 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68101f70 0x7f8d681994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 0x7f8d681a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d681a0ad0 con 0x7f8d68100d80 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d681a0c10 con 0x7f8d68101f70 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d6ce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d681a0da0 con 0x7f8d68102910 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d65e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d65e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59246/0 (socket says 172.21.15.23:59246) 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d65e00700 1 -- 172.21.15.23:0/2026826129 learned_addr learned my addr 172.21.15.23:0/2026826129 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d65400700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68101f70 0x7f8d681994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.844+0000 7f8d66800700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 0x7f8d681a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 -- 172.21.15.23:0/2026826129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 msgr2=0x7f8d681a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 0x7f8d681a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 -- 172.21.15.23:0/2026826129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68101f70 msgr2=0x7f8d681994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68101f70 0x7f8d681994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 -- 172.21.15.23:0/2026826129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d540095f0 con 0x7f8d68100d80 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d65e00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009230 tx=0x7f8d5800bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5801c070 con 0x7f8d68100d80 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d5800bb20 con 0x7f8d68100d80 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d58009520 con 0x7f8d68100d80 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d58004950 con 0x7f8d68100d80 2024-03-30T03:59:58.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.845+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d681a12b0 con 0x7f8d68100d80 2024-03-30T03:59:58.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.846+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6810ced0 con 0x7f8d68100d80 2024-03-30T03:59:58.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.848+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f8d58020070 con 0x7f8d68100d80 2024-03-30T03:59:58.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.850+0000 7f8d5ea00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 0x7f8d4c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:58.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.850+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8d580cd920 con 0x7f8d68100d80 2024-03-30T03:59:58.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.850+0000 7f8d65400700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 0x7f8d4c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:58.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.851+0000 7f8d65400700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 0x7f8d4c0a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005fd0 tx=0x7f8d54009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:58.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:58.852+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d58097010 con 0x7f8d68100d80 2024-03-30T03:59:59.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.072+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d6810c7c0 con 0x7f8d68100d80 2024-03-30T03:59:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.072+0000 7f8d5ea00700 1 -- 172.21.15.23:0/2026826129 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d58027020 con 0x7f8d68100d80 2024-03-30T03:59:59.071 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T03:59:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 msgr2=0x7f8d4c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 0x7f8d4c0a35e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8d54005fd0 tx=0x7f8d54009310 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 msgr2=0x7f8d68198fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d58009230 tx=0x7f8d5800bef0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 shutdown_connections 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d4c0a1120 0x7f8d4c0a35e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d68102910 0x7f8d681a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d68100d80 0x7f8d68198fa0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 --2- 172.21.15.23:0/2026826129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d68101f70 0x7f8d681994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 >> 172.21.15.23:0/2026826129 conn(0x7f8d680fc2f0 msgr2=0x7f8d68107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 shutdown_connections 2024-03-30T03:59:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.075+0000 7f8d6ce00700 1 -- 172.21.15.23:0/2026826129 wait complete. 2024-03-30T03:59:59.092 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 -- 172.21.15.23:0/85902543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44100d80 msgr2=0x7f3a441011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 --2- 172.21.15.23:0/85902543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44100d80 0x7f3a441011a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3a340099c0 tx=0x7f3a34009cd0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 -- 172.21.15.23:0/85902543 shutdown_connections 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 --2- 172.21.15.23:0/85902543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 0x7f3a441091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 --2- 172.21.15.23:0/85902543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44100d80 0x7f3a441011a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 --2- 172.21.15.23:0/85902543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44101f70 0x7f3a441023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 -- 172.21.15.23:0/85902543 >> 172.21.15.23:0/85902543 conn(0x7f3a440fc2f0 msgr2=0x7f3a440fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:59.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 -- 172.21.15.23:0/85902543 shutdown_connections 2024-03-30T03:59:59.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.294+0000 7f3a49600700 1 -- 172.21.15.23:0/85902543 wait complete. 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.295+0000 7f3a49600700 1 Processor -- start 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.295+0000 7f3a49600700 1 -- start start 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44100d80 0x7f3a44198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 0x7f3a441a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a441a0ab0 con 0x7f3a44101f70 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a441a0bf0 con 0x7f3a44100d80 2024-03-30T03:59:59.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a49600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a441a0d80 con 0x7f3a44102910 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59262/0 (socket says 172.21.15.23:59262) 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 -- 172.21.15.23:0/1938879422 learned_addr learned my addr 172.21.15.23:0/1938879422 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42a00700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44100d80 0x7f3a44198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a43400700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 0x7f3a441a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 -- 172.21.15.23:0/1938879422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 msgr2=0x7f3a441a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 0x7f3a441a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 -- 172.21.15.23:0/1938879422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44100d80 msgr2=0x7f3a44198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44100d80 0x7f3a44198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.296+0000 7f3a42000700 1 -- 172.21.15.23:0/1938879422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a30009520 con 0x7f3a44101f70 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a42000700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000eb30 tx=0x7f3a3000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30009880 con 0x7f3a44101f70 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a300044e0 con 0x7f3a44101f70 2024-03-30T03:59:59.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a340095f0 con 0x7f3a44101f70 2024-03-30T03:59:59.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30003d80 con 0x7f3a44101f70 2024-03-30T03:59:59.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.297+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a441a13e0 con 0x7f3a44101f70 2024-03-30T03:59:59.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.298+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a44067260 con 0x7f3a44101f70 2024-03-30T03:59:59.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.300+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f3a30003f20 con 0x7f3a44101f70 2024-03-30T03:59:59.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.301+0000 7f3a3b400700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 0x7f3a280a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.301+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3a30014070 con 0x7f3a44101f70 2024-03-30T03:59:59.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.302+0000 7f3a42a00700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 0x7f3a280a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.302+0000 7f3a42a00700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 0x7f3a280a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009990 tx=0x7f3a34000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:59.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.305+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3a300965e0 con 0x7f3a44101f70 2024-03-30T03:59:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.524+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3a441a1760 con 0x7f3a44101f70 2024-03-30T03:59:59.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.525+0000 7f3a3b400700 1 -- 172.21.15.23:0/1938879422 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3a30099c00 con 0x7f3a44101f70 2024-03-30T03:59:59.524 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 msgr2=0x7f3a280a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 0x7f3a280a35e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009990 tx=0x7f3a34000bc0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 msgr2=0x7f3a441994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000eb30 tx=0x7f3a3000ee40 comp rx=0 tx=0).stop 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 shutdown_connections 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f3a280a1120 0x7f3a280a35e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a44102910 0x7f3a441a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a44100d80 0x7f3a44198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 --2- 172.21.15.23:0/1938879422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a44101f70 0x7f3a441994c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 >> 172.21.15.23:0/1938879422 conn(0x7f3a440fc2f0 msgr2=0x7f3a44072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.527+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 shutdown_connections 2024-03-30T03:59:59.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.528+0000 7f3a49600700 1 -- 172.21.15.23:0/1938879422 wait complete. 2024-03-30T03:59:59.544 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T03:59:59.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:59 smithi023 ceph-mon[38254]: pgmap v245: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T03:59:59.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1929040238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.601 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 03:59:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2026826129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:59 smithi154 ceph-mon[79161]: pgmap v245: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T03:59:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1929040238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 03:59:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2026826129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:59 smithi193 ceph-mon[79581]: pgmap v245: 129 pgs: 129 active+clean; 579 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-30T03:59:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1929040238' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 03:59:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2026826129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 -- 172.21.15.23:0/1004136856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 msgr2=0x7fbedc073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 --2- 172.21.15.23:0/1004136856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc073240 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009990 tx=0x7fbecc009ca0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 -- 172.21.15.23:0/1004136856 shutdown_connections 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 --2- 172.21.15.23:0/1004136856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 0x7fbedc10b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 --2- 172.21.15.23:0/1004136856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc073240 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 --2- 172.21.15.23:0/1004136856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 -- 172.21.15.23:0/1004136856 >> 172.21.15.23:0/1004136856 conn(0x7fbedc0fc2f0 msgr2=0x7fbedc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:59.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 -- 172.21.15.23:0/1004136856 shutdown_connections 2024-03-30T03:59:59.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.746+0000 7fbee2c00700 1 -- 172.21.15.23:0/1004136856 wait complete. 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.747+0000 7fbee2c00700 1 Processor -- start 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.747+0000 7fbee2c00700 1 -- start start 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 0x7fbedc1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1a0a40 con 0x7fbedc073810 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1a0b80 con 0x7fbedc108c40 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee2c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc1a0d10 con 0x7fbedc074df0 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59280/0 (socket says 172.21.15.23:59280) 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 -- 172.21.15.23:0/154953303 learned_addr learned my addr 172.21.15.23:0/154953303 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T03:59:59.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbee0c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 0x7fbedc1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 -- 172.21.15.23:0/154953303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 msgr2=0x7fbedc199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 -- 172.21.15.23:0/154953303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 msgr2=0x7fbedc1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 0x7fbedc1a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.748+0000 7fbedbe00700 1 -- 172.21.15.23:0/154953303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc0095f0 con 0x7fbedc073810 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbedbe00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0098e0 tx=0x7fbecc005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01d070 con 0x7fbedc073810 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbecc022470 con 0x7fbedc073810 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc1a0f90 con 0x7fbedc073810 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc017650 con 0x7fbedc073810 2024-03-30T03:59:59.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.749+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc1a1350 con 0x7fbedc073810 2024-03-30T03:59:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.750+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc107a00 con 0x7fbedc073810 2024-03-30T03:59:59.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.752+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fbecc022890 con 0x7fbedc073810 2024-03-30T03:59:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.753+0000 7fbed8c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T03:59:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.753+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbecc0ce100 con 0x7fbedc073810 2024-03-30T03:59:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.753+0000 7fbedb400700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T03:59:59.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.754+0000 7fbedb400700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 0x7fbec00a3660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbedc074b10 tx=0x7fbec8009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T03:59:59.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.757+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbecc0977f0 con 0x7fbedc073810 2024-03-30T03:59:59.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.975+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbedc02ccb0 con 0x7fbedc073810 2024-03-30T03:59:59.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.975+0000 7fbed8c00700 1 -- 172.21.15.23:0/154953303 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v84) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbecc027020 con 0x7fbedc073810 2024-03-30T03:59:59.974 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 msgr2=0x7fbec00a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 0x7fbec00a3660 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbedc074b10 tx=0x7fbec8009260 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 msgr2=0x7fbedc198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbecc0098e0 tx=0x7fbecc005740 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 shutdown_connections 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbedc108c40 0x7fbedc1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbedc073810 0x7fbedc198f10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbec00a11a0 0x7fbec00a3660 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 --2- 172.21.15.23:0/154953303 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbedc074df0 0x7fbedc199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 >> 172.21.15.23:0/154953303 conn(0x7fbedc0fc2f0 msgr2=0x7fbedc104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 shutdown_connections 2024-03-30T03:59:59.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T03:59:59.978+0000 7fbee2c00700 1 -- 172.21.15.23:0/154953303 wait complete. 2024-03-30T03:59:59.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T04:00:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 -- 172.21.15.23:0/2872685593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4101f70 msgr2=0x7fc7f41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2872685593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4101f70 0x7fc7f41023d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00099c0 tx=0x7fc7e0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 -- 172.21.15.23:0/2872685593 shutdown_connections 2024-03-30T04:00:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2872685593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4102910 0x7fc7f41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2872685593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 0x7fc7f41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2872685593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4101f70 0x7fc7f41023d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 -- 172.21.15.23:0/2872685593 >> 172.21.15.23:0/2872685593 conn(0x7fc7f40fc2f0 msgr2=0x7fc7f40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 -- 172.21.15.23:0/2872685593 shutdown_connections 2024-03-30T04:00:00.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.198+0000 7fc7f9600700 1 -- 172.21.15.23:0/2872685593 wait complete. 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.199+0000 7fc7f9600700 1 Processor -- start 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.199+0000 7fc7f9600700 1 -- start start 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 0x7fc7f4196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4101f70 0x7fc7f41972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f419e860 con 0x7fc7f4102910 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f419e9d0 con 0x7fc7f4100d80 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f9600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f419eb90 con 0x7fc7f4101f70 2024-03-30T04:00:00.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 0x7fc7f4196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59294/0 (socket says 172.21.15.23:59294) 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 -- 172.21.15.23:0/2140034517 learned_addr learned my addr 172.21.15.23:0/2140034517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f2000700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4101f70 0x7fc7f41972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 -- 172.21.15.23:0/2140034517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4101f70 msgr2=0x7fc7f41972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4101f70 0x7fc7f41972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 -- 172.21.15.23:0/2140034517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 msgr2=0x7fc7f4196d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 0x7fc7f4196d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.200+0000 7fc7f3400700 1 -- 172.21.15.23:0/2140034517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00095f0 con 0x7fc7f4102910 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7f3400700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00ef30 tx=0x7fc7ec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec018070 con 0x7fc7f4102910 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7ec00f040 con 0x7fc7f4102910 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ec00cd00 con 0x7fc7f4102910 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f419ee10 con 0x7fc7f4102910 2024-03-30T04:00:00.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.201+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f419f230 con 0x7fc7f4102910 2024-03-30T04:00:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.202+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f4108250 con 0x7fc7f4102910 2024-03-30T04:00:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.204+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc7ec003a00 con 0x7fc7f4102910 2024-03-30T04:00:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.205+0000 7fc7eb400700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 0x7fc7d80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.205+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc7ec0cc620 con 0x7fc7f4102910 2024-03-30T04:00:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.206+0000 7fc7f2a00700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 0x7fc7d80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.206+0000 7fc7f2a00700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 0x7fc7d80a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc7f41981f0 tx=0x7fc7e4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.210+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc7ec095bf0 con 0x7fc7f4102910 2024-03-30T04:00:00.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.498+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7f4067af0 con 0x7fc7f4102910 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.498+0000 7fc7eb400700 1 -- 172.21.15.23:0/2140034517 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v84) v1 ==== 81+0+588 (secure 0 0 0) 0x7fc7ec099210 con 0x7fc7f4102910 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1161676542 2024-03-31T03:54:16.197829+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/443135190 2024-03-31T03:54:16.197829+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2173844842 2024-03-31T03:53:51.749812+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2471664699 2024-03-31T03:53:51.749812+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/783324330 2024-03-31T03:53:51.749812+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1219945839 2024-03-31T03:54:16.197829+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T04:00:00.497 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 msgr2=0x7fc7d80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 0x7fc7d80a35e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc7f41981f0 tx=0x7fc7e4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 msgr2=0x7fc7f419e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec00ef30 tx=0x7fc7ec00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 shutdown_connections 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7f4102910 0x7fc7f419e290 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7f4100d80 0x7fc7f4196d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc7d80a1120 0x7fc7d80a35e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 --2- 172.21.15.23:0/2140034517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc7f4101f70 0x7fc7f41972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 >> 172.21.15.23:0/2140034517 conn(0x7fc7f40fc2f0 msgr2=0x7fc7f4105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 shutdown_connections 2024-03-30T04:00:00.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:00.501+0000 7fc7f9600700 1 -- 172.21.15.23:0/2140034517 wait complete. 2024-03-30T04:00:00.501 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-30T04:00:00.518 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2024-03-30T04:00:00.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:00:00.518 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-30T04:00:00.583 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[91113]: fuse finished with error 0 and tester_r 0 2024-03-30T04:00:00.586 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T04:00:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1938879422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/154953303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:00 smithi154 ceph-mon[79161]: overall HEALTH_OK 2024-03-30T04:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1938879422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/154953303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:00 smithi023 ceph-mon[38254]: overall HEALTH_OK 2024-03-30T04:00:00.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1938879422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/154953303' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:00:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:00 smithi193 ceph-mon[79581]: overall HEALTH_OK 2024-03-30T04:00:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:01 smithi154 ceph-mon[79161]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:01 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2140034517' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:00:01.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:01 smithi193 ceph-mon[79581]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:01.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:01 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2140034517' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:00:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:01 smithi023 ceph-mon[38254]: pgmap v246: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:01 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2140034517' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:00:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:03 smithi154 ceph-mon[79161]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:03.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:03 smithi193 ceph-mon[79581]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:03.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:03 smithi023 ceph-mon[38254]: pgmap v247: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:05 smithi154 ceph-mon[79161]: pgmap v248: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:05.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:05 smithi193 ceph-mon[79581]: pgmap v248: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:05.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:05 smithi023 ceph-mon[38254]: pgmap v248: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-30T04:00:06.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:00:06.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:00:06.689 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-30T04:00:06.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 -- 172.21.15.23:0/4095464108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc100c80 msgr2=0x7fb8cc1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 --2- 172.21.15.23:0/4095464108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc100c80 0x7fb8cc1010a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc0099c0 tx=0x7fb8bc009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 -- 172.21.15.23:0/4095464108 shutdown_connections 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 --2- 172.21.15.23:0/4095464108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc102810 0x7fb8cc1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 --2- 172.21.15.23:0/4095464108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc100c80 0x7fb8cc1010a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 --2- 172.21.15.23:0/4095464108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc101e70 0x7fb8cc1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 -- 172.21.15.23:0/4095464108 >> 172.21.15.23:0/4095464108 conn(0x7fb8cc0fc1f0 msgr2=0x7fb8cc0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 -- 172.21.15.23:0/4095464108 shutdown_connections 2024-03-30T04:00:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.958+0000 7fb8d2600700 1 -- 172.21.15.23:0/4095464108 wait complete. 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.959+0000 7fb8d2600700 1 Processor -- start 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.959+0000 7fb8d2600700 1 -- start start 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc101e70 0x7fb8cc199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 0x7fb8cc1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1a0960 con 0x7fb8cc100c80 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1a0aa0 con 0x7fb8cc102810 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8d2600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc1a0c30 con 0x7fb8cc101e70 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8cb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8cb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59310/0 (socket says 172.21.15.23:59310) 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8cb400700 1 -- 172.21.15.23:0/1281443001 learned_addr learned my addr 172.21.15.23:0/1281443001 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8caa00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc101e70 0x7fb8cc199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.960+0000 7fb8cbe00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 0x7fb8cc1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 -- 172.21.15.23:0/1281443001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc101e70 msgr2=0x7fb8cc199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc101e70 0x7fb8cc199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 -- 172.21.15.23:0/1281443001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 msgr2=0x7fb8cc1a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 0x7fb8cc1a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 -- 172.21.15.23:0/1281443001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc0095f0 con 0x7fb8cc100c80 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cbe00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 0x7fb8cc1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8cb400700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc003730 tx=0x7fb8bc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc01d070 con 0x7fb8cc100c80 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8bc005050 con 0x7fb8cc100c80 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc004a50 con 0x7fb8cc100c80 2024-03-30T04:00:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc1a0eb0 con 0x7fb8cc100c80 2024-03-30T04:00:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.961+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc1a1220 con 0x7fb8cc100c80 2024-03-30T04:00:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.962+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8cc067a10 con 0x7fb8cc100c80 2024-03-30T04:00:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.964+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb8bc0051c0 con 0x7fb8cc100c80 2024-03-30T04:00:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.965+0000 7fb8c3e00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 0x7fb8b00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.966+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb8bc0cdc60 con 0x7fb8cc100c80 2024-03-30T04:00:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.966+0000 7fb8caa00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 0x7fb8b00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.966+0000 7fb8caa00700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 0x7fb8b00a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009c60 tx=0x7fb8b8009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:06.968+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb8bc097230 con 0x7fb8cc100c80 2024-03-30T04:00:07.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.225+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8cc10c730 con 0x7fb8cc100c80 2024-03-30T04:00:07.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.227+0000 7fb8c3e00700 1 -- 172.21.15.23:0/1281443001 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7fb8bc09a850 con 0x7fb8cc100c80 2024-03-30T04:00:07.226 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:07.226 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 msgr2=0x7fb8b00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 0x7fb8b00a35e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009c60 tx=0x7fb8b8009260 comp rx=0 tx=0).stop 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 msgr2=0x7fb8cc198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc003730 tx=0x7fb8bc00f690 comp rx=0 tx=0).stop 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 shutdown_connections 2024-03-30T04:00:07.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8cc102810 0x7fb8cc1a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb8cc100c80 0x7fb8cc198e30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb8b00a1120 0x7fb8b00a35e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 --2- 172.21.15.23:0/1281443001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb8cc101e70 0x7fb8cc199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 >> 172.21.15.23:0/1281443001 conn(0x7fb8cc0fc1f0 msgr2=0x7fb8cc107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 shutdown_connections 2024-03-30T04:00:07.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.229+0000 7fb8d2600700 1 -- 172.21.15.23:0/1281443001 wait complete. 2024-03-30T04:00:07.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:06 smithi193 ceph-mon[79581]: pgmap v249: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:07.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:06 smithi023 ceph-mon[38254]: pgmap v249: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:07.247 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:00:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:06 smithi154 ceph-mon[79161]: pgmap v249: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 -- 172.21.15.23:0/3820812890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28100c80 msgr2=0x7f9b281010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 --2- 172.21.15.23:0/3820812890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28100c80 0x7f9b281010a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9b180099c0 tx=0x7f9b18009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 -- 172.21.15.23:0/3820812890 shutdown_connections 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 --2- 172.21.15.23:0/3820812890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28102810 0x7f9b281090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 --2- 172.21.15.23:0/3820812890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28100c80 0x7f9b281010a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 --2- 172.21.15.23:0/3820812890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28101e70 0x7f9b281022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 -- 172.21.15.23:0/3820812890 >> 172.21.15.23:0/3820812890 conn(0x7f9b280fc1f0 msgr2=0x7f9b280fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 -- 172.21.15.23:0/3820812890 shutdown_connections 2024-03-30T04:00:07.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.448+0000 7f9b2d400700 1 -- 172.21.15.23:0/3820812890 wait complete. 2024-03-30T04:00:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.449+0000 7f9b2d400700 1 Processor -- start 2024-03-30T04:00:07.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 -- start start 2024-03-30T04:00:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28100c80 0x7f9b28074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28102810 0x7f9b280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28073e80 con 0x7f9b28101e70 2024-03-30T04:00:07.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b28073ff0 con 0x7f9b28100c80 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b2d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b281a0c40 con 0x7f9b28102810 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b25e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b25e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59326/0 (socket says 172.21.15.23:59326) 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.450+0000 7f9b25e00700 1 -- 172.21.15.23:0/1563630493 learned_addr learned my addr 172.21.15.23:0/1563630493 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b26800700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28100c80 0x7f9b28074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b27200700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28102810 0x7f9b280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 -- 172.21.15.23:0/1563630493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28102810 msgr2=0x7f9b280736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28102810 0x7f9b280736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 -- 172.21.15.23:0/1563630493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28100c80 msgr2=0x7f9b28074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28100c80 0x7f9b28074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 -- 172.21.15.23:0/1563630493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b20009c40 con 0x7f9b28101e70 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b25e00700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400e9a0 tx=0x7f9b1400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:07.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14009790 con 0x7f9b28101e70 2024-03-30T04:00:07.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b180095f0 con 0x7f9b28101e70 2024-03-30T04:00:07.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.451+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b1400ce20 con 0x7f9b28101e70 2024-03-30T04:00:07.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.452+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b14018420 con 0x7f9b28101e70 2024-03-30T04:00:07.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.452+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b281a11f0 con 0x7f9b28101e70 2024-03-30T04:00:07.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.453+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b28108150 con 0x7f9b28101e70 2024-03-30T04:00:07.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.455+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f9b140106a0 con 0x7f9b28101e70 2024-03-30T04:00:07.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.456+0000 7f9b1f400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 0x7f9b0c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.456+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9b14014070 con 0x7f9b28101e70 2024-03-30T04:00:07.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.456+0000 7f9b26800700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 0x7f9b0c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.457+0000 7f9b26800700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 0x7f9b0c0a35e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009990 tx=0x7f9b18000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:07.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.460+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9b140529f0 con 0x7f9b28101e70 2024-03-30T04:00:07.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.720+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b28067a10 con 0x7f9b28101e70 2024-03-30T04:00:07.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.721+0000 7f9b1f400700 1 -- 172.21.15.23:0/1563630493 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+23958 (secure 0 0 0) 0x7f9b14099c70 con 0x7f9b28101e70 2024-03-30T04:00:07.720 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:07.721 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":84,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T03:59:44.968467+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 msgr2=0x7f9b0c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 0x7f9b0c0a35e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9b18009990 tx=0x7f9b18000bc0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 msgr2=0x7f9b280731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400e9a0 tx=0x7f9b1400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 shutdown_connections 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9b0c0a1120 0x7f9b0c0a35e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b28102810 0x7f9b280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b28100c80 0x7f9b28074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 --2- 172.21.15.23:0/1563630493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b28101e70 0x7f9b280731a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 >> 172.21.15.23:0/1563630493 conn(0x7f9b280fc1f0 msgr2=0x7f9b28105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:07.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 shutdown_connections 2024-03-30T04:00:07.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.724+0000 7f9b2d400700 1 -- 172.21.15.23:0/1563630493 wait complete. 2024-03-30T04:00:07.742 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 -- 172.21.15.23:0/1152232039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78102f90 msgr2=0x7fca781033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 --2- 172.21.15.23:0/1152232039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78102f90 0x7fca781033b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0098f0 tx=0x7fca6c009c00 comp rx=0 tx=0).stop 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 -- 172.21.15.23:0/1152232039 shutdown_connections 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 --2- 172.21.15.23:0/1152232039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78104b20 0x7fca7810b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 --2- 172.21.15.23:0/1152232039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78102f90 0x7fca781033b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 --2- 172.21.15.23:0/1152232039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104180 0x7fca781045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 -- 172.21.15.23:0/1152232039 >> 172.21.15.23:0/1152232039 conn(0x7fca780fe500 msgr2=0x7fca78100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:07.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 -- 172.21.15.23:0/1152232039 shutdown_connections 2024-03-30T04:00:07.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.945+0000 7fca7fe00700 1 -- 172.21.15.23:0/1152232039 wait complete. 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.946+0000 7fca7fe00700 1 Processor -- start 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.946+0000 7fca7fe00700 1 -- start start 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.946+0000 7fca7fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.946+0000 7fca7fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78104180 0x7fca781994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104b20 0x7fca781a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7fe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca781a0ae0 con 0x7fca78102f90 2024-03-30T04:00:07.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7fe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca781a0c20 con 0x7fca78104b20 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca781a0db0 con 0x7fca78104180 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78104180 0x7fca781994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59330/0 (socket says 172.21.15.23:59330) 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 -- 172.21.15.23:0/3255546276 learned_addr learned my addr 172.21.15.23:0/3255546276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7de00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104b20 0x7fca781a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 -- 172.21.15.23:0/3255546276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78104180 msgr2=0x7fca781994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78104180 0x7fca781994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 -- 172.21.15.23:0/3255546276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104b20 msgr2=0x7fca781a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:07.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104b20 0x7fca781a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.947+0000 7fca7d400700 1 -- 172.21.15.23:0/3255546276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6c009520 con 0x7fca78102f90 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca7d400700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009810 tx=0x7fca6c00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c01c070 con 0x7fca78102f90 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca6c005040 con 0x7fca78102f90 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca781a1030 con 0x7fca78102f90 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0038c0 con 0x7fca78102f90 2024-03-30T04:00:07.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.948+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca781a1450 con 0x7fca78102f90 2024-03-30T04:00:07.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.949+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7810a4d0 con 0x7fca78102f90 2024-03-30T04:00:07.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.951+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fca6c020030 con 0x7fca78102f90 2024-03-30T04:00:07.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.952+0000 7fca76000700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 0x7fca600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:07.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.952+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fca6c0cd960 con 0x7fca78102f90 2024-03-30T04:00:07.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.952+0000 7fca7ca00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 0x7fca600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:07.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.953+0000 7fca7ca00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 0x7fca600a35e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca64005fd0 tx=0x7fca64009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:07.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:07.956+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fca6c096f30 con 0x7fca78102f90 2024-03-30T04:00:08.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1281443001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1563630493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.224+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fca7810ea20 con 0x7fca78102f90 2024-03-30T04:00:08.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1281443001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1563630493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1281443001' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1563630493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:08.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.948+0000 7fca76000700 1 -- 172.21.15.23:0/3255546276 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fca6c0051b0 con 0x7fca78102f90 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 msgr2=0x7fca600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 0x7fca600a35e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fca64005fd0 tx=0x7fca64009310 comp rx=0 tx=0).stop 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 msgr2=0x7fca78198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009810 tx=0x7fca6c00bf60 comp rx=0 tx=0).stop 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 shutdown_connections 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca78104b20 0x7fca781a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:08.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca78102f90 0x7fca78198fb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:08.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca600a1120 0x7fca600a35e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:08.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 --2- 172.21.15.23:0/3255546276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca78104180 0x7fca781994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:08.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 >> 172.21.15.23:0/3255546276 conn(0x7fca780fe500 msgr2=0x7fca78107610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:08.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 shutdown_connections 2024-03-30T04:00:08.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:08.951+0000 7fca7fe00700 1 -- 172.21.15.23:0/3255546276 wait complete. 2024-03-30T04:00:08.970 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-30T04:00:08.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 -- 172.21.15.23:0/2566579975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610101f70 msgr2=0x7fb6101023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 --2- 172.21.15.23:0/2566579975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610101f70 0x7fb6101023d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0099c0 tx=0x7fb5fc009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 -- 172.21.15.23:0/2566579975 shutdown_connections 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 --2- 172.21.15.23:0/2566579975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610102910 0x7fb6101091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 --2- 172.21.15.23:0/2566579975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610100d80 0x7fb6101011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 --2- 172.21.15.23:0/2566579975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610101f70 0x7fb6101023d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 -- 172.21.15.23:0/2566579975 >> 172.21.15.23:0/2566579975 conn(0x7fb6100fc2f0 msgr2=0x7fb6100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:09.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 -- 172.21.15.23:0/2566579975 shutdown_connections 2024-03-30T04:00:09.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.176+0000 7fb617c00700 1 -- 172.21.15.23:0/2566579975 wait complete. 2024-03-30T04:00:09.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 Processor -- start 2024-03-30T04:00:09.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 -- start start 2024-03-30T04:00:09.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610101f70 0x7fb610197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 0x7fb61019e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61019e820 con 0x7fb610102910 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61019e990 con 0x7fb610101f70 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.178+0000 7fb617c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61019eb50 con 0x7fb610100d80 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 0x7fb61019e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:53766/0 (socket says 172.21.15.23:53766) 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 -- 172.21.15.23:0/796116889 learned_addr learned my addr 172.21.15.23:0/796116889 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb60fe00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610101f70 0x7fb610197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:09.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 -- 172.21.15.23:0/796116889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610101f70 msgr2=0x7fb610197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610101f70 0x7fb610197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 -- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 msgr2=0x7fb61019e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 0x7fb61019e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 -- 172.21.15.23:0/796116889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc0095f0 con 0x7fb610100d80 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 0x7fb61019e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:09.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.179+0000 7fb615200700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e970 tx=0x7fb60400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.180+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604009790 con 0x7fb610100d80 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.180+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61019ee30 con 0x7fb610100d80 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.180+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb60400c7e0 con 0x7fb610100d80 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.180+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb604003c70 con 0x7fb610100d80 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.180+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61019f260 con 0x7fb610100d80 2024-03-30T04:00:09.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.181+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6101081d0 con 0x7fb610100d80 2024-03-30T04:00:09.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.183+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fb60400cc50 con 0x7fb610100d80 2024-03-30T04:00:09.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.184+0000 7fb60d600700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 0x7fb5f80a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.184+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb604014070 con 0x7fb610100d80 2024-03-30T04:00:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.184+0000 7fb60fe00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 0x7fb5f80a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:09.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.185+0000 7fb60fe00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 0x7fb5f80a3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006010 tx=0x7fb5fc00b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:09.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.186+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb604096350 con 0x7fb610100d80 2024-03-30T04:00:09.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:08 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T04:00:09.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:08 smithi193 ceph-mon[79581]: pgmap v250: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T04:00:09.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:08 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T04:00:09.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:08 smithi023 ceph-mon[38254]: pgmap v250: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T04:00:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:08 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-30T04:00:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:08 smithi154 ceph-mon[79161]: pgmap v250: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-30T04:00:09.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.454+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fb61010c5e0 con 0x7fb610100d80 2024-03-30T04:00:09.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.967+0000 7fb60d600700 1 -- 172.21.15.23:0/796116889 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb604099970 con 0x7fb610100d80 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 msgr2=0x7fb5f80a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 0x7fb5f80a3660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc006010 tx=0x7fb5fc00b540 comp rx=0 tx=0).stop 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 msgr2=0x7fb610196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb60400e970 tx=0x7fb60400ec80 comp rx=0 tx=0).stop 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 shutdown_connections 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb610102910 0x7fb61019e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5f80a11a0 0x7fb5f80a3660 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb610100d80 0x7fb610196d00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 --2- 172.21.15.23:0/796116889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb610101f70 0x7fb610197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:09.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 >> 172.21.15.23:0/796116889 conn(0x7fb6100fc2f0 msgr2=0x7fb610105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:09.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 shutdown_connections 2024-03-30T04:00:09.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:09.970+0000 7fb617c00700 1 -- 172.21.15.23:0/796116889 wait complete. 2024-03-30T04:00:09.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-30T04:00:10.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 -- 172.21.15.23:0/2538844818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 msgr2=0x7f614c103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:10.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 --2- 172.21.15.23:0/2538844818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c103c00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f613c009990 tx=0x7f613c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 -- 172.21.15.23:0/2538844818 shutdown_connections 2024-03-30T04:00:10.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 --2- 172.21.15.23:0/2538844818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 --2- 172.21.15.23:0/2538844818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c103c00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 --2- 172.21.15.23:0/2538844818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 -- 172.21.15.23:0/2538844818 >> 172.21.15.23:0/2538844818 conn(0x7f614c0fb200 msgr2=0x7f614c0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:10.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 -- 172.21.15.23:0/2538844818 shutdown_connections 2024-03-30T04:00:10.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.195+0000 7f6152400700 1 -- 172.21.15.23:0/2538844818 wait complete. 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.196+0000 7f6152400700 1 Processor -- start 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.196+0000 7f6152400700 1 -- start start 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c0ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:10.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1004a0 con 0x7f614c104140 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c100610 con 0x7f614c101810 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f6152400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1a0d70 con 0x7f614c106a70 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35452/0 (socket says 172.21.15.23:35452) 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 -- 172.21.15.23:0/422774797 learned_addr learned my addr 172.21.15.23:0/422774797 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614b400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614be00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c0ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 -- 172.21.15.23:0/422774797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 msgr2=0x7f614c0ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c0ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 -- 172.21.15.23:0/422774797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 msgr2=0x7f614c101130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c101130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614aa00700 1 -- 172.21.15.23:0/422774797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6138009520 con 0x7f614c104140 2024-03-30T04:00:10.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614be00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.197+0000 7f614b400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f614aa00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f613800eb30 tx=0x7f613800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138009880 con 0x7f614c104140 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61380044e0 con 0x7f614c104140 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138003df0 con 0x7f614c104140 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613c0095f0 con 0x7f614c104140 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.198+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c1a12a0 con 0x7f614c104140 2024-03-30T04:00:10.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.199+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c0fcd80 con 0x7f614c104140 2024-03-30T04:00:10.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.201+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6138003fb0 con 0x7f614c104140 2024-03-30T04:00:10.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.203+0000 7f6143e00700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 0x7f61300a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:10.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.203+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6138014070 con 0x7f614c104140 2024-03-30T04:00:10.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.203+0000 7f614b400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 0x7f61300a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:10.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.203+0000 7f614b400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 0x7f61300a35e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b2e0 tx=0x7f613c005fb0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:10.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.206+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f61380964d0 con 0x7f614c104140 2024-03-30T04:00:10.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:09 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T04:00:10.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:09 smithi193 ceph-mon[79581]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:10.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:09 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/796116889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:09 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:09 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T04:00:10.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:09 smithi023 ceph-mon[38254]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:10.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:09 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/796116889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:09 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:09 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3255546276' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-30T04:00:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:09 smithi154 ceph-mon[79161]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:09 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/796116889' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:09 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-30T04:00:10.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.476+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f614c067260 con 0x7f614c104140 2024-03-30T04:00:10.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.928+0000 7f6143e00700 1 -- 172.21.15.23:0/422774797 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f6138099af0 con 0x7f614c104140 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 msgr2=0x7f61300a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 0x7f61300a35e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f613c00b2e0 tx=0x7f613c005fb0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 msgr2=0x7f614c0ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f613800eb30 tx=0x7f613800ee40 comp rx=0 tx=0).stop 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 shutdown_connections 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f61300a1120 0x7f61300a35e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f614c106a70 0x7f614c0ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f614c101810 0x7f614c101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 --2- 172.21.15.23:0/422774797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f614c104140 0x7f614c0ff7c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:10.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 >> 172.21.15.23:0/422774797 conn(0x7f614c0fb200 msgr2=0x7f614c1036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:10.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 shutdown_connections 2024-03-30T04:00:10.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:10.931+0000 7f6152400700 1 -- 172.21.15.23:0/422774797 wait complete. 2024-03-30T04:00:10.955 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-30T04:00:11.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 -- 172.21.15.23:0/309955314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec101f70 msgr2=0x7fa9ec1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:11.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 --2- 172.21.15.23:0/309955314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec101f70 0x7fa9ec1023d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009990 tx=0x7fa9d8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 -- 172.21.15.23:0/309955314 shutdown_connections 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 --2- 172.21.15.23:0/309955314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec102910 0x7fa9ec1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 --2- 172.21.15.23:0/309955314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec100d80 0x7fa9ec1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 --2- 172.21.15.23:0/309955314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec101f70 0x7fa9ec1023d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 -- 172.21.15.23:0/309955314 >> 172.21.15.23:0/309955314 conn(0x7fa9ec0fc2f0 msgr2=0x7fa9ec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 -- 172.21.15.23:0/309955314 shutdown_connections 2024-03-30T04:00:11.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.160+0000 7fa9f4600700 1 -- 172.21.15.23:0/309955314 wait complete. 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.161+0000 7fa9f4600700 1 Processor -- start 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 -- start start 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 0x7fa9ec198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec101f70 0x7fa9ec199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1a0a50 con 0x7fa9ec102910 2024-03-30T04:00:11.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1a0b90 con 0x7fa9ec101f70 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f4600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec1a0d20 con 0x7fa9ec100d80 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35472/0 (socket says 172.21.15.23:35472) 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f2600700 1 -- 172.21.15.23:0/692688427 learned_addr learned my addr 172.21.15.23:0/692688427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f1c00700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 0x7fa9ec198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.162+0000 7fa9f1200700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec101f70 0x7fa9ec199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 -- 172.21.15.23:0/692688427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 msgr2=0x7fa9ec198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 0x7fa9ec198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 -- 172.21.15.23:0/692688427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec101f70 msgr2=0x7fa9ec199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec101f70 0x7fa9ec199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 -- 172.21.15.23:0/692688427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d80095f0 con 0x7fa9ec102910 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f1c00700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 0x7fa9ec198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:11.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f2600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4007ee0 tx=0x7fa9e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e4004db0 con 0x7fa9ec102910 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa9e400f040 con 0x7fa9ec102910 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9ec1a0fa0 con 0x7fa9ec102910 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e40044e0 con 0x7fa9ec102910 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.163+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9ec1a14d0 con 0x7fa9ec102910 2024-03-30T04:00:11.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.164+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9ec067af0 con 0x7fa9ec102910 2024-03-30T04:00:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.168+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa9e4004640 con 0x7fa9ec102910 2024-03-30T04:00:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.169+0000 7fa9ea000700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 0x7fa9d40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.169+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa9e40cce80 con 0x7fa9ec102910 2024-03-30T04:00:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.169+0000 7fa9f1c00700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 0x7fa9d40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.170+0000 7fa9f1c00700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 0x7fa9d40a3660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec101dd0 tx=0x7fa9e0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:11.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.172+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa9e40963c0 con 0x7fa9ec102910 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: pgmap v251: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:active 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T04:00:11.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T04:00:11.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-30T04:00:11.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:11.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:10 smithi193 ceph-mon[79581]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: pgmap v251: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:active 2024-03-30T04:00:11.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T04:00:11.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T04:00:11.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-30T04:00:11.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:11.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:10 smithi023 ceph-mon[38254]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-30T04:00:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-30T04:00:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-30T04:00:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-30T04:00:11.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: pgmap v251: 129 pgs: 129 active+clean; 581 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] up:active 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/422774797' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 2 up:standby-replay 8 up:standby 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:11.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:10 smithi154 ceph-mon[79161]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-03-30T04:00:11.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.438+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fa9ec10c5e0 con 0x7fa9ec102910 2024-03-30T04:00:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.940+0000 7fa9ea000700 1 -- 172.21.15.23:0/692688427 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa9e4019070 con 0x7fa9ec102910 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 msgr2=0x7fa9d40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 0x7fa9d40a3660 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec101dd0 tx=0x7fa9e0009260 comp rx=0 tx=0).stop 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 msgr2=0x7fa9ec1a0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4007ee0 tx=0x7fa9e400d3b0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 shutdown_connections 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa9ec102910 0x7fa9ec1a0480 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa9d40a11a0 0x7fa9d40a3660 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa9ec100d80 0x7fa9ec198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 --2- 172.21.15.23:0/692688427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa9ec101f70 0x7fa9ec199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 >> 172.21.15.23:0/692688427 conn(0x7fa9ec0fc2f0 msgr2=0x7fa9ec072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 shutdown_connections 2024-03-30T04:00:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:11.943+0000 7fa9f4600700 1 -- 172.21.15.23:0/692688427 wait complete. 2024-03-30T04:00:11.967 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-30T04:00:11.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-30T04:00:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 -- 172.21.15.23:0/228645458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 msgr2=0x7fbeb81011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 --2- 172.21.15.23:0/228645458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb81011a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbea80099c0 tx=0x7fbea8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 -- 172.21.15.23:0/228645458 shutdown_connections 2024-03-30T04:00:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 --2- 172.21.15.23:0/228645458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 --2- 172.21.15.23:0/228645458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb81011a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 --2- 172.21.15.23:0/228645458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 -- 172.21.15.23:0/228645458 >> 172.21.15.23:0/228645458 conn(0x7fbeb80fc2f0 msgr2=0x7fbeb80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 -- 172.21.15.23:0/228645458 shutdown_connections 2024-03-30T04:00:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.175+0000 7fbebee00700 1 -- 172.21.15.23:0/228645458 wait complete. 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.176+0000 7fbebee00700 1 Processor -- start 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.176+0000 7fbebee00700 1 -- start start 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8073de0 con 0x7fbeb8100d80 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8073f50 con 0x7fbeb8102910 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebee00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81a0dc0 con 0x7fbeb8101f70 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:12.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35496/0 (socket says 172.21.15.23:35496) 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 -- 172.21.15.23:0/3321639327 learned_addr learned my addr 172.21.15.23:0/3321639327 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebce00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7400700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 -- 172.21.15.23:0/3321639327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 msgr2=0x7fbeb80731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb80731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 -- 172.21.15.23:0/3321639327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 msgr2=0x7fbeb80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7e00700 1 -- 172.21.15.23:0/3321639327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea80095f0 con 0x7fbeb8100d80 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbeb7400700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb80731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.177+0000 7fbebce00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbeb7e00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbea8006010 tx=0x7fbea800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea801d070 con 0x7fbeb8100d80 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbea80054e0 con 0x7fbeb8100d80 2024-03-30T04:00:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb81a0fc0 con 0x7fbeb8100d80 2024-03-30T04:00:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea8004c10 con 0x7fbeb8100d80 2024-03-30T04:00:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.178+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb81a1330 con 0x7fbeb8100d80 2024-03-30T04:00:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.179+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb8108250 con 0x7fbeb8100d80 2024-03-30T04:00:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.181+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fbea800fbb0 con 0x7fbeb8100d80 2024-03-30T04:00:12.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.182+0000 7fbeb4c00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 0x7fbe9c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:12.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.182+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbea80cd690 con 0x7fbeb8100d80 2024-03-30T04:00:12.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.182+0000 7fbeb7400700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 0x7fbe9c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:12.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.183+0000 7fbeb7400700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 0x7fbe9c0a35e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009cb0 tx=0x7fbea4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:12.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.186+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbea8096c60 con 0x7fbeb8100d80 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:active 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:12.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:11 smithi193 ceph-mon[79581]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-30T04:00:12.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:active 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:12.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:11 smithi023 ceph-mon[38254]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] up:active 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/692688427' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 2 up:standby-replay 7 up:standby 2024-03-30T04:00:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:12.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:11 smithi154 ceph-mon[79161]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-30T04:00:12.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:12.443+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeb802cd70 con 0x7fbe9c0a1120 2024-03-30T04:00:12.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:00:12.450+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:00:12.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:00:12.450+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:00:12.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:00:12.450+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:00:12.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:00:12.450+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:00:12.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:00:12.450+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: pgmap v252: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:active 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-30T04:00:13.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:12 smithi193 ceph-mon[79581]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: pgmap v252: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:active 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-30T04:00:13.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:12 smithi023 ceph-mon[38254]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-30T04:00:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: pgmap v252: 129 pgs: 129 active+clean; 582 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 1.1 KiB/s wr, 3 op/s 2024-03-30T04:00:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] up:active 2024-03-30T04:00:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-30T04:00:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-30T04:00:13.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T04:00:13.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-30T04:00:13.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-30T04:00:13.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:12 smithi154 ceph-mon[79161]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-30T04:00:13.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.973+0000 7fbeb4c00700 1 -- 172.21.15.23:0/3321639327 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbeb802cd70 con 0x7fbe9c0a1120 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.976+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 msgr2=0x7fbe9c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.976+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 0x7fbe9c0a35e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbea4009cb0 tx=0x7fbea4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.976+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 msgr2=0x7fbeb8074ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.976+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbea8006010 tx=0x7fbea800f740 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 shutdown_connections 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbeb8102910 0x7fbeb80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbeb8100d80 0x7fbeb8074ac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbe9c0a1120 0x7fbe9c0a35e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 --2- 172.21.15.23:0/3321639327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeb8101f70 0x7fbeb80731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 >> 172.21.15.23:0/3321639327 conn(0x7fbeb80fc2f0 msgr2=0x7fbeb8105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:13.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 shutdown_connections 2024-03-30T04:00:13.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:13.977+0000 7fbebee00700 1 -- 172.21.15.23:0/3321639327 wait complete. 2024-03-30T04:00:14.000 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-30T04:00:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 -- 172.21.15.23:0/234725702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c104110 msgr2=0x7ff99c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 --2- 172.21.15.23:0/234725702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c104110 0x7ff99c106500 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff988009990 tx=0x7ff988009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 -- 172.21.15.23:0/234725702 shutdown_connections 2024-03-30T04:00:14.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 --2- 172.21.15.23:0/234725702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 0x7ff99c108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 --2- 172.21.15.23:0/234725702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c1017e0 0x7ff99c103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 --2- 172.21.15.23:0/234725702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c104110 0x7ff99c106500 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 -- 172.21.15.23:0/234725702 >> 172.21.15.23:0/234725702 conn(0x7ff99c0fb1d0 msgr2=0x7ff99c0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 -- 172.21.15.23:0/234725702 shutdown_connections 2024-03-30T04:00:14.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.210+0000 7ff9a2800700 1 -- 172.21.15.23:0/234725702 wait complete. 2024-03-30T04:00:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.211+0000 7ff9a2800700 1 Processor -- start 2024-03-30T04:00:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.211+0000 7ff9a2800700 1 -- start start 2024-03-30T04:00:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 0x7ff99c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 0x7ff99c1a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1a0a70 con 0x7ff99c1017e0 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1a0bb0 con 0x7ff99c106a40 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff9a2800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99c1a0d40 con 0x7ff99c104110 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff99b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff99b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35532/0 (socket says 172.21.15.23:35532) 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff99b400700 1 -- 172.21.15.23:0/233411821 learned_addr learned my addr 172.21.15.23:0/233411821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff99aa00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 0x7ff99c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.212+0000 7ff99be00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 0x7ff99c1a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 -- 172.21.15.23:0/233411821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 msgr2=0x7ff99c199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 0x7ff99c199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 -- 172.21.15.23:0/233411821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 msgr2=0x7ff99c1a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 0x7ff99c1a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 -- 172.21.15.23:0/233411821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c009520 con 0x7ff99c1017e0 2024-03-30T04:00:14.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99aa00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 0x7ff99c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff99b400700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00eb30 tx=0x7ff98c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c009880 con 0x7ff99c1017e0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff98c0044e0 con 0x7ff99c1017e0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c003d80 con 0x7ff99c1017e0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff994009c40 con 0x7ff99c1017e0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.213+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9880095f0 con 0x7ff99c1017e0 2024-03-30T04:00:14.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.214+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99c108610 con 0x7ff99c1017e0 2024-03-30T04:00:14.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.216+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ff98c003ee0 con 0x7ff99c1017e0 2024-03-30T04:00:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.218+0000 7ff993e00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 0x7ff9800a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.219+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff98c014070 con 0x7ff99c1017e0 2024-03-30T04:00:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.219+0000 7ff99aa00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 0x7ff9800a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.219+0000 7ff99aa00700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 0x7ff9800a35e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff98800b2e0 tx=0x7ff988005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:14.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.223+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff98c096460 con 0x7ff99c1017e0 2024-03-30T04:00:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:13 smithi193 ceph-mon[79581]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:13 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:active 2024-03-30T04:00:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:13 smithi193 ceph-mon[79581]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-30T04:00:14.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:13 smithi193 ceph-mon[79581]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:00:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:13 smithi023 ceph-mon[38254]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:13 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:active 2024-03-30T04:00:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:13 smithi023 ceph-mon[38254]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-30T04:00:14.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:13 smithi023 ceph-mon[38254]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:00:14.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:13 smithi154 ceph-mon[79161]: from='client.14802 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:14.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:13 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] up:active 2024-03-30T04:00:14.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:13 smithi154 ceph-mon[79161]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-30T04:00:14.282 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:13 smithi154 ceph-mon[79161]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:00:14.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.475+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7ff99c067ac0 con 0x7ff9800a1120 2024-03-30T04:00:14.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.502+0000 7ff993e00700 1 -- 172.21.15.23:0/233411821 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff99c067ac0 con 0x7ff9800a1120 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 msgr2=0x7ff9800a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 0x7ff9800a35e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff98800b2e0 tx=0x7ff988005fd0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 msgr2=0x7ff99c198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00eb30 tx=0x7ff98c00ee40 comp rx=0 tx=0).stop 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 shutdown_connections 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff99c106a40 0x7ff99c1a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff99c1017e0 0x7ff99c198f40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff9800a1120 0x7ff9800a35e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 --2- 172.21.15.23:0/233411821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff99c104110 0x7ff99c199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 >> 172.21.15.23:0/233411821 conn(0x7ff99c0fb1d0 msgr2=0x7ff99c100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:14.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 shutdown_connections 2024-03-30T04:00:14.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.505+0000 7ff9a2800700 1 -- 172.21.15.23:0/233411821 wait complete. 2024-03-30T04:00:14.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-30T04:00:14.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:00:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:00:14.282+0000 7f7e1571a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-30T04:00:14.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:00:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:00:14.282+0000 7f7e1571a700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-30T04:00:14.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:00:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:00:14.282+0000 7f7e1571a700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-30T04:00:14.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:00:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:00:14.282+0000 7f7e1571a700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-30T04:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 -- 172.21.15.23:0/1803723951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100100d80 msgr2=0x7f81001011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 --2- 172.21.15.23:0/1803723951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100100d80 0x7f81001011a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f80f00099c0 tx=0x7f80f0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 -- 172.21.15.23:0/1803723951 shutdown_connections 2024-03-30T04:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 --2- 172.21.15.23:0/1803723951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100102910 0x7f81001091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 --2- 172.21.15.23:0/1803723951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100100d80 0x7f81001011a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 --2- 172.21.15.23:0/1803723951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100101f70 0x7f81001023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 -- 172.21.15.23:0/1803723951 >> 172.21.15.23:0/1803723951 conn(0x7f81000fc2f0 msgr2=0x7f81000fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 -- 172.21.15.23:0/1803723951 shutdown_connections 2024-03-30T04:00:14.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.728+0000 7f8105800700 1 -- 172.21.15.23:0/1803723951 wait complete. 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.729+0000 7f8105800700 1 Processor -- start 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 -- start start 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100100d80 0x7f8100074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100102910 0x7f81000736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100073e80 con 0x7f8100101f70 2024-03-30T04:00:14.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100073ff0 con 0x7f8100100d80 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f8105800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81001a0dc0 con 0x7f8100102910 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f80fe000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f80fe000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35538/0 (socket says 172.21.15.23:35538) 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f80fe000700 1 -- 172.21.15.23:0/694462152 learned_addr learned my addr 172.21.15.23:0/694462152 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f80fea00700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100100d80 0x7f8100074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.730+0000 7f80ff400700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100102910 0x7f81000736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 -- 172.21.15.23:0/694462152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100102910 msgr2=0x7f81000736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100102910 0x7f81000736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 -- 172.21.15.23:0/694462152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100100d80 msgr2=0x7f8100074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100100d80 0x7f8100074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 -- 172.21.15.23:0/694462152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00095f0 con 0x7f8100101f70 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80fe000700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9a0 tx=0x7f80ec00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec009790 con 0x7f8100101f70 2024-03-30T04:00:14.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80ec00cb70 con 0x7f8100101f70 2024-03-30T04:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ec018680 con 0x7f8100101f70 2024-03-30T04:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81001a0ff0 con 0x7f8100101f70 2024-03-30T04:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.731+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81001a14d0 con 0x7f8100101f70 2024-03-30T04:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.732+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8100108220 con 0x7f8100101f70 2024-03-30T04:00:14.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.734+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f80ec00cce0 con 0x7f8100101f70 2024-03-30T04:00:14.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.735+0000 7f80f7400700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 0x7f80e40a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:14.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.736+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f80ec014070 con 0x7f8100101f70 2024-03-30T04:00:14.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.736+0000 7f80fea00700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 0x7f80e40a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:14.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.736+0000 7f80fea00700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 0x7f80e40a3660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009990 tx=0x7f80f001a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:14.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:14.739+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f80ec0529f0 con 0x7f8100101f70 2024-03-30T04:00:14.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.000+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f80ec096690 con 0x7f8100101f70 2024-03-30T04:00:15.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.007+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f810002cd20 con 0x7f80e40a11a0 2024-03-30T04:00:15.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.011+0000 7f80f7400700 1 -- 172.21.15.23:0/694462152 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f810002cd20 con 0x7f80e40a11a0 2024-03-30T04:00:15.010 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_0/e6634e5f-e0cf-41e8-a018-96d29ab877f8 2024-03-30T04:00:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 msgr2=0x7f80e40a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 0x7f80e40a3660 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009990 tx=0x7f80f001a040 comp rx=0 tx=0).stop 2024-03-30T04:00:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 msgr2=0x7f81000731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00e9a0 tx=0x7f80ec00ed60 comp rx=0 tx=0).stop 2024-03-30T04:00:15.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 shutdown_connections 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f80e40a11a0 0x7f80e40a3660 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8100102910 0x7f81000736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8100100d80 0x7f8100074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 --2- 172.21.15.23:0/694462152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8100101f70 0x7f81000731a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 >> 172.21.15.23:0/694462152 conn(0x7f81000fc2f0 msgr2=0x7f8100105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 shutdown_connections 2024-03-30T04:00:15.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.014+0000 7f8105800700 1 -- 172.21.15.23:0/694462152 wait complete. 2024-03-30T04:00:15.031 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-30T04:00:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 -- 172.21.15.23:0/1276643512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24104110 msgr2=0x7fbd24106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 --2- 172.21.15.23:0/1276643512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24104110 0x7fbd24106500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009990 tx=0x7fbd10009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 -- 172.21.15.23:0/1276643512 shutdown_connections 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 --2- 172.21.15.23:0/1276643512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd24106a40 0x7fbd24108eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 --2- 172.21.15.23:0/1276643512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd241017e0 0x7fbd24103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 --2- 172.21.15.23:0/1276643512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24104110 0x7fbd24106500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.236+0000 7fbd29600700 1 -- 172.21.15.23:0/1276643512 >> 172.21.15.23:0/1276643512 conn(0x7fbd240fb1d0 msgr2=0x7fbd240fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.237+0000 7fbd29600700 1 -- 172.21.15.23:0/1276643512 shutdown_connections 2024-03-30T04:00:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.237+0000 7fbd29600700 1 -- 172.21.15.23:0/1276643512 wait complete. 2024-03-30T04:00:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 Processor -- start 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 -- start start 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd241017e0 0x7fbd24198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd24104110 0x7fbd24199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a0a40 con 0x7fbd24106a40 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a0b80 con 0x7fbd24104110 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd29600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241a0d10 con 0x7fbd241017e0 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd23400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd23400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35560/0 (socket says 172.21.15.23:35560) 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd23400700 1 -- 172.21.15.23:0/4179309845 learned_addr learned my addr 172.21.15.23:0/4179309845 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd22000700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd24104110 0x7fbd24199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.238+0000 7fbd22a00700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd241017e0 0x7fbd24198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 -- 172.21.15.23:0/4179309845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd241017e0 msgr2=0x7fbd24198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd241017e0 0x7fbd24198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 -- 172.21.15.23:0/4179309845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd24104110 msgr2=0x7fbd24199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd24104110 0x7fbd24199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 -- 172.21.15.23:0/4179309845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd100095f0 con 0x7fbd24106a40 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd23400700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c007ee0 tx=0x7fbd1c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c017070 con 0x7fbd24106a40 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd1c00f040 con 0x7fbd24106a40 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd241a0ff0 con 0x7fbd24106a40 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c005580 con 0x7fbd24106a40 2024-03-30T04:00:15.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.239+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd241a14c0 con 0x7fbd24106a40 2024-03-30T04:00:15.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.241+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24108610 con 0x7fbd24106a40 2024-03-30T04:00:15.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.242+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbd1c00db20 con 0x7fbd24106a40 2024-03-30T04:00:15.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:14 smithi193 ceph-mon[79581]: pgmap v253: 129 pgs: 129 active+clean; 591 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 10 op/s 2024-03-30T04:00:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.244+0000 7fbd1b400700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 0x7fbd080a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.245+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd1c0cc340 con 0x7fbd24106a40 2024-03-30T04:00:15.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.245+0000 7fbd22a00700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 0x7fbd080a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.245+0000 7fbd22a00700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 0x7fbd080a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009c60 tx=0x7fbd14009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:15.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.249+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbd1c095720 con 0x7fbd24106a40 2024-03-30T04:00:15.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:14 smithi023 ceph-mon[38254]: pgmap v253: 129 pgs: 129 active+clean; 591 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 10 op/s 2024-03-30T04:00:15.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:14 smithi154 ceph-mon[79161]: pgmap v253: 129 pgs: 129 active+clean; 591 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 3.5 KiB/s wr, 10 op/s 2024-03-30T04:00:15.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.509+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd2402cd20 con 0x7fbd080a1230 2024-03-30T04:00:15.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.536+0000 7fbd1b400700 1 -- 172.21.15.23:0/4179309845 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbd2402cd20 con 0x7fbd080a1230 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 msgr2=0x7fbd080a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 0x7fbd080a36f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009c60 tx=0x7fbd14009260 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 msgr2=0x7fbd241a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c007ee0 tx=0x7fbd1c00d3b0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 shutdown_connections 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbd24106a40 0x7fbd241a0470 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbd080a1230 0x7fbd080a36f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd241017e0 0x7fbd24198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 --2- 172.21.15.23:0/4179309845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd24104110 0x7fbd24199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 >> 172.21.15.23:0/4179309845 conn(0x7fbd240fb1d0 msgr2=0x7fbd24101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:15.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 shutdown_connections 2024-03-30T04:00:15.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.539+0000 7fbd29600700 1 -- 172.21.15.23:0/4179309845 wait complete. 2024-03-30T04:00:15.556 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-30T04:00:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 -- 172.21.15.23:0/2899643063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4101f70 msgr2=0x7f06a41023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 --2- 172.21.15.23:0/2899643063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4101f70 0x7f06a41023d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f06900099c0 tx=0x7f0690009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 -- 172.21.15.23:0/2899643063 shutdown_connections 2024-03-30T04:00:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 --2- 172.21.15.23:0/2899643063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4102910 0x7f06a41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 --2- 172.21.15.23:0/2899643063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 --2- 172.21.15.23:0/2899643063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4101f70 0x7f06a41023d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.760+0000 7f06aa200700 1 -- 172.21.15.23:0/2899643063 >> 172.21.15.23:0/2899643063 conn(0x7f06a40fc2f0 msgr2=0x7f06a40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:15.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.761+0000 7f06aa200700 1 -- 172.21.15.23:0/2899643063 shutdown_connections 2024-03-30T04:00:15.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.761+0000 7f06aa200700 1 -- 172.21.15.23:0/2899643063 wait complete. 2024-03-30T04:00:15.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 Processor -- start 2024-03-30T04:00:15.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 -- start start 2024-03-30T04:00:15.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a4198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4101f70 0x7f06a41994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 0x7f06a41a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a41a0a60 con 0x7f06a4102910 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a41a0ba0 con 0x7f06a4101f70 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06aa200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a41a0d30 con 0x7f06a4100d80 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06a3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 0x7f06a41a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06a3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 0x7f06a41a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35582/0 (socket says 172.21.15.23:35582) 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06a3e00700 1 -- 172.21.15.23:0/3391887675 learned_addr learned my addr 172.21.15.23:0/3391887675 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.762+0000 7f06a3400700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a4198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a2a00700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4101f70 0x7f06a41994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 -- 172.21.15.23:0/3391887675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4101f70 msgr2=0x7f06a41994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4101f70 0x7f06a41994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 -- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 msgr2=0x7f06a41a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:15.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 0x7f06a41a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:15.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 -- 172.21.15.23:0/3391887675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06900095f0 con 0x7f06a4100d80 2024-03-30T04:00:15.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06a3400700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a4198f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f069400e9c0 tx=0x7f069400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:15.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069400cbd0 con 0x7f06a4100d80 2024-03-30T04:00:15.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f069400eeb0 con 0x7f06a4100d80 2024-03-30T04:00:15.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a41a0fb0 con 0x7f06a4100d80 2024-03-30T04:00:15.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.763+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694003c70 con 0x7f06a4100d80 2024-03-30T04:00:15.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.764+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a41a1430 con 0x7f06a4100d80 2024-03-30T04:00:15.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.765+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a4067af0 con 0x7f06a4100d80 2024-03-30T04:00:15.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.767+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f0694004870 con 0x7f06a4100d80 2024-03-30T04:00:15.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.768+0000 7f069be00700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 0x7f06880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:15.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.768+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0694014070 con 0x7f06a4100d80 2024-03-30T04:00:15.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.768+0000 7f06a2a00700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 0x7f06880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:15.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.769+0000 7f06a2a00700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 0x7f06880a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f069000b580 tx=0x7f069001a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:15.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:15.771+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0694096a70 con 0x7f06a4100d80 2024-03-30T04:00:16.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.034+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a4074700 con 0x7f06880a11b0 2024-03-30T04:00:16.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.038+0000 7f069be00700 1 -- 172.21.15.23:0/3391887675 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f06a4074700 con 0x7f06880a11b0 2024-03-30T04:00:16.036 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_1/51da4228-0048-4ff1-b829-59e3acc5f4e7 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 msgr2=0x7f06880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 0x7f06880a3670 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f069000b580 tx=0x7f069001a040 comp rx=0 tx=0).stop 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 msgr2=0x7f06a4198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a4198f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f069400e9c0 tx=0x7f069400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 shutdown_connections 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06a4102910 0x7f06a41a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06880a11b0 0x7f06880a3670 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06a4100d80 0x7f06a4198f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 --2- 172.21.15.23:0/3391887675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a4101f70 0x7f06a41994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 >> 172.21.15.23:0/3391887675 conn(0x7f06a40fc2f0 msgr2=0x7f06a4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 shutdown_connections 2024-03-30T04:00:16.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.040+0000 7f06aa200700 1 -- 172.21.15.23:0/3391887675 wait complete. 2024-03-30T04:00:16.057 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:00:16.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:15 smithi193 ceph-mon[79581]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:15 smithi193 ceph-mon[79581]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T04:00:16.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:15 smithi193 ceph-mon[79581]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.258+0000 7fe107a00700 1 -- 172.21.15.23:0/2211708556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 msgr2=0x7fe100073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.258+0000 7fe107a00700 1 --2- 172.21.15.23:0/2211708556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100073c90 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009990 tx=0x7fe0ec009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 -- 172.21.15.23:0/2211708556 shutdown_connections 2024-03-30T04:00:16.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 --2- 172.21.15.23:0/2211708556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100108ad0 0x7fe10010af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 --2- 172.21.15.23:0/2211708556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100074df0 0x7fe100073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 --2- 172.21.15.23:0/2211708556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100073c90 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 -- 172.21.15.23:0/2211708556 >> 172.21.15.23:0/2211708556 conn(0x7fe1000fc160 msgr2=0x7fe1000fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 -- 172.21.15.23:0/2211708556 shutdown_connections 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.259+0000 7fe107a00700 1 -- 172.21.15.23:0/2211708556 wait complete. 2024-03-30T04:00:16.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.260+0000 7fe107a00700 1 Processor -- start 2024-03-30T04:00:16.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.260+0000 7fe107a00700 1 -- start start 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100074df0 0x7fe1001992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100108ad0 0x7fe1001a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001a08d0 con 0x7fe100073810 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001a0a10 con 0x7fe100074df0 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe107a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001a0ba0 con 0x7fe100108ad0 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35594/0 (socket says 172.21.15.23:35594) 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 -- 172.21.15.23:0/3273322907 learned_addr learned my addr 172.21.15.23:0/3273322907 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe0ffe00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100074df0 0x7fe1001992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100108ad0 0x7fe1001a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 -- 172.21.15.23:0/3273322907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100108ad0 msgr2=0x7fe1001a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100108ad0 0x7fe1001a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 -- 172.21.15.23:0/3273322907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100074df0 msgr2=0x7fe1001992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.261+0000 7fe105000700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100074df0 0x7fe1001992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe105000700 1 -- 172.21.15.23:0/3273322907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0095f0 con 0x7fe100073810 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe105000700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400d950 tx=0x7fe0f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400f890 con 0x7fe100073810 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0f4010460 con 0x7fe100073810 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1001a0e80 con 0x7fe100073810 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f400e610 con 0x7fe100073810 2024-03-30T04:00:16.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.262+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1001a12b0 con 0x7fe100073810 2024-03-30T04:00:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.263+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100107890 con 0x7fe100073810 2024-03-30T04:00:16.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.265+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe0f40108d0 con 0x7fe100073810 2024-03-30T04:00:16.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.267+0000 7fe0fd600700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 0x7fe0e80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.267+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe0f40cc9d0 con 0x7fe100073810 2024-03-30T04:00:16.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.268+0000 7fe0ffe00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 0x7fe0e80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.268+0000 7fe0ffe00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 0x7fe0e80a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b5c0 tx=0x7fe0ec003790 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:16.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.272+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0f4095db0 con 0x7fe100073810 2024-03-30T04:00:16.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:15 smithi023 ceph-mon[38254]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:15 smithi023 ceph-mon[38254]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T04:00:16.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:15 smithi023 ceph-mon[38254]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:15 smithi154 ceph-mon[79161]: from='client.14814 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:15 smithi154 ceph-mon[79161]: mgrmap e17: x(active, since 5m), standbys: z, y 2024-03-30T04:00:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:15 smithi154 ceph-mon[79161]: from='client.14820 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:16.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.541+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1000679a0 con 0x7fe100073810 2024-03-30T04:00:16.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.541+0000 7fe0fd600700 1 -- 172.21.15.23:0/3273322907 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fe0f4016090 con 0x7fe100073810 2024-03-30T04:00:16.540 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:16.540 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 msgr2=0x7fe0e80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 0x7fe0e80a36f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00b5c0 tx=0x7fe0ec003790 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 msgr2=0x7fe100198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe0f400d950 tx=0x7fe0f400dc60 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 shutdown_connections 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe0e80a1230 0x7fe0e80a36f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe100108ad0 0x7fe1001a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe100073810 0x7fe100198da0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 --2- 172.21.15.23:0/3273322907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe100074df0 0x7fe1001992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 >> 172.21.15.23:0/3273322907 conn(0x7fe1000fc160 msgr2=0x7fe1001049d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 shutdown_connections 2024-03-30T04:00:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.544+0000 7fe107a00700 1 -- 172.21.15.23:0/3273322907 wait complete. 2024-03-30T04:00:16.544 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:00:16.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 -- 172.21.15.23:0/54281954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 msgr2=0x7fe7c4102260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 --2- 172.21.15.23:0/54281954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4102260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe7b00099c0 tx=0x7fe7b0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 -- 172.21.15.23:0/54281954 shutdown_connections 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 --2- 172.21.15.23:0/54281954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c41027a0 0x7fe7c4109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 --2- 172.21.15.23:0/54281954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c4100c10 0x7fe7c4101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 --2- 172.21.15.23:0/54281954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4102260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 -- 172.21.15.23:0/54281954 >> 172.21.15.23:0/54281954 conn(0x7fe7c40fc160 msgr2=0x7fe7c40fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 -- 172.21.15.23:0/54281954 shutdown_connections 2024-03-30T04:00:16.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.762+0000 7fe7cb800700 1 -- 172.21.15.23:0/54281954 wait complete. 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 Processor -- start 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 -- start start 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 0x7fe7c41a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41a0900 con 0x7fe7c4101e00 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41a0a70 con 0x7fe7c4100c10 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.763+0000 7fe7cb800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c41a0c30 con 0x7fe7c41027a0 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:45502/0 (socket says 172.21.15.23:45502) 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 -- 172.21.15.23:0/3405781622 learned_addr learned my addr 172.21.15.23:0/3405781622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:16.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c3e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c9800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 0x7fe7c41a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 -- 172.21.15.23:0/3405781622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 msgr2=0x7fe7c41a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 0x7fe7c41a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 -- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 msgr2=0x7fe7c4199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 -- 172.21.15.23:0/3405781622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00095f0 con 0x7fe7c4100c10 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c9800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 0x7fe7c41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.764+0000 7fe7c8e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e970 tx=0x7fe7b800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.765+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8009790 con 0x7fe7c4100c10 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.765+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b800eec0 con 0x7fe7c4100c10 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.765+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c41a0f10 con 0x7fe7c4100c10 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.765+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8003cf0 con 0x7fe7c4100c10 2024-03-30T04:00:16.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.765+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c41a1330 con 0x7fe7c4100c10 2024-03-30T04:00:16.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.766+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c40679a0 con 0x7fe7c4100c10 2024-03-30T04:00:16.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.768+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe7b800ccd0 con 0x7fe7c4100c10 2024-03-30T04:00:16.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.769+0000 7fe7c1600700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 0x7fe7ac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:16.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.769+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7b8014070 con 0x7fe7c4100c10 2024-03-30T04:00:16.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.769+0000 7fe7c3e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 0x7fe7ac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:16.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.770+0000 7fe7c3e00700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 0x7fe7ac0a3670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006010 tx=0x7fe7b000b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:16.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:16.772+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe7b8096450 con 0x7fe7c4100c10 2024-03-30T04:00:17.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.034+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7c410c470 con 0x7fe7c4100c10 2024-03-30T04:00:17.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.036+0000 7fe7c1600700 1 -- 172.21.15.23:0/3405781622 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+24428 (secure 0 0 0) 0x7fe7b809ee60 con 0x7fe7c4100c10 2024-03-30T04:00:17.035 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:17.035 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":85,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:16.989840+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:17.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 msgr2=0x7fe7ac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 0x7fe7ac0a3670 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006010 tx=0x7fe7b000b540 comp rx=0 tx=0).stop 2024-03-30T04:00:17.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 msgr2=0x7fe7c4198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800e970 tx=0x7fe7b800ec80 comp rx=0 tx=0).stop 2024-03-30T04:00:17.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 shutdown_connections 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe7ac0a11b0 0x7fe7ac0a3670 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7c41027a0 0x7fe7c41a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7c4100c10 0x7fe7c4198e10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 --2- 172.21.15.23:0/3405781622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe7c4101e00 0x7fe7c4199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 >> 172.21.15.23:0/3405781622 conn(0x7fe7c40fc160 msgr2=0x7fe7c4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 shutdown_connections 2024-03-30T04:00:17.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.038+0000 7fe7cb800700 1 -- 172.21.15.23:0/3405781622 wait complete. 2024-03-30T04:00:17.056 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-30T04:00:17.056 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T04:00:17.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 -- 172.21.15.23:0/3561755888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4100c80 msgr2=0x7f59e41010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 --2- 172.21.15.23:0/3561755888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4100c80 0x7f59e41010a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f59d40099c0 tx=0x7f59d4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 -- 172.21.15.23:0/3561755888 shutdown_connections 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 --2- 172.21.15.23:0/3561755888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 --2- 172.21.15.23:0/3561755888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4100c80 0x7f59e41010a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 --2- 172.21.15.23:0/3561755888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4101e70 0x7f59e41022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 -- 172.21.15.23:0/3561755888 >> 172.21.15.23:0/3561755888 conn(0x7f59e40fc1f0 msgr2=0x7f59e40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.257+0000 7f59eaa00700 1 -- 172.21.15.23:0/3561755888 shutdown_connections 2024-03-30T04:00:17.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.258+0000 7f59eaa00700 1 -- 172.21.15.23:0/3561755888 wait complete. 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 Processor -- start 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 -- start start 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4100c80 0x7f59e4198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4101e70 0x7f59e41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e41a09c0 con 0x7f59e4102810 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e41a0b00 con 0x7f59e4101e70 2024-03-30T04:00:17.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59eaa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e41a0c90 con 0x7f59e4100c80 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59e8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59e8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35624/0 (socket says 172.21.15.23:35624) 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59e8a00700 1 -- 172.21.15.23:0/2710149726 learned_addr learned my addr 172.21.15.23:0/2710149726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59e3400700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4101e70 0x7f59e41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.259+0000 7f59e3e00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4100c80 0x7f59e4198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 -- 172.21.15.23:0/2710149726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4100c80 msgr2=0x7f59e4198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4100c80 0x7f59e4198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 -- 172.21.15.23:0/2710149726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4101e70 msgr2=0x7f59e41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4101e70 0x7f59e41993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 -- 172.21.15.23:0/2710149726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59dc009c40 con 0x7f59e4102810 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e8a00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c370 tx=0x7f59dc00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc017070 con 0x7f59e4102810 2024-03-30T04:00:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59dc00f040 con 0x7f59e4102810 2024-03-30T04:00:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d40095f0 con 0x7f59e4102810 2024-03-30T04:00:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00e050 con 0x7f59e4102810 2024-03-30T04:00:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.260+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e41a1230 con 0x7f59e4102810 2024-03-30T04:00:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.261+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e4067a10 con 0x7f59e4102810 2024-03-30T04:00:17.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.263+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f59dc00cb60 con 0x7f59e4102810 2024-03-30T04:00:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.265+0000 7f59e0c00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 0x7f59c80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.265+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59dc0cc6d0 con 0x7f59e4102810 2024-03-30T04:00:17.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.265+0000 7f59e3e00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 0x7f59c80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.265+0000 7f59e3e00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 0x7f59c80a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f59d4003730 tx=0x7f59d4000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.269+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f59dc095990 con 0x7f59e4102810 2024-03-30T04:00:17.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: pgmap v254: 129 pgs: 129 active+clean; 593 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 15 op/s 2024-03-30T04:00:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:16 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3273322907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:00:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: pgmap v254: 129 pgs: 129 active+clean; 593 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 15 op/s 2024-03-30T04:00:17.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:16 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3273322907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:00:17.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.482+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59e4074700 con 0x7f59c80a11b0 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='client.14826 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='client.24832 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]: dispatch 2024-03-30T04:00:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: pgmap v254: 129 pgs: 129 active+clean; 593 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 4.8 KiB/s wr, 15 op/s 2024-03-30T04:00:17.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:16 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3273322907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:00:17.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.498+0000 7f59e0c00700 1 -- 172.21.15.23:0/2710149726 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422895 (secure 0 0 0) 0x7f59e4074700 con 0x7f59c80a11b0 2024-03-30T04:00:17.503 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.503+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 msgr2=0x7f59c80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.503+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 0x7f59c80a3670 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f59d4003730 tx=0x7f59d4000bc0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.503+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 msgr2=0x7f59e41a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.503+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f59dc00c370 tx=0x7f59dc00c680 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 shutdown_connections 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59e4102810 0x7f59e41a03f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f59c80a11b0 0x7f59c80a3670 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59e4100c80 0x7f59e4198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 --2- 172.21.15.23:0/2710149726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e4101e70 0x7f59e41993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 >> 172.21.15.23:0/2710149726 conn(0x7f59e40fc1f0 msgr2=0x7f59e4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 shutdown_connections 2024-03-30T04:00:17.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.504+0000 7f59eaa00700 1 -- 172.21.15.23:0/2710149726 wait complete. 2024-03-30T04:00:17.508 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:00:17.521 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-03-30T04:00:16.995845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606966,"num_objects":94,"num_object_clones":0,"num_object_copies":283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":182,"num_read_kb":159,"num_write":382,"num_write_kb":3014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369908,"kb_used_data":12948,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124571916,"statfs":{"total":1151940427776,"available":1151561641984,"internally_reserved":0,"allocated":13258752,"data_stored":7220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12004,"num_objects":70,"num_object_clones":0,"num_object_copies":211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":152,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.722198"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454371+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T03:59:44.454371+0000","last_peered":"2024-03-30T03:59:44.454371+0000","last_clean":"2024-03-30T03:59:44.454371+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T03:59:44.454371+0000","last_undegraded":"2024-03-30T03:59:44.454371+0000","last_fullsized":"2024-03-30T03:59:44.454371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465480+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T03:59:44.465480+0000","last_peered":"2024-03-30T03:59:44.465480+0000","last_clean":"2024-03-30T03:59:44.465480+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T03:59:44.465480+0000","last_undegraded":"2024-03-30T03:59:44.465480+0000","last_fullsized":"2024-03-30T03:59:44.465480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.464161+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T03:59:44.464161+0000","last_peered":"2024-03-30T03:59:44.464161+0000","last_clean":"2024-03-30T03:59:44.464161+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T03:59:44.464161+0000","last_undegraded":"2024-03-30T03:59:44.464161+0000","last_fullsized":"2024-03-30T03:59:44.464161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455406+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T03:59:44.455406+0000","last_peered":"2024-03-30T03:59:44.455406+0000","last_clean":"2024-03-30T03:59:44.455406+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T03:59:44.455406+0000","last_undegraded":"2024-03-30T03:59:44.455406+0000","last_fullsized":"2024-03-30T03:59:44.455406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.452773+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T03:59:44.452773+0000","last_peered":"2024-03-30T03:59:44.452773+0000","last_clean":"2024-03-30T03:59:44.452773+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T03:59:44.452773+0000","last_undegraded":"2024-03-30T03:59:44.452773+0000","last_fullsized":"2024-03-30T03:59:44.452773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597214+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T03:59:44.597214+0000","last_peered":"2024-03-30T03:59:44.597214+0000","last_clean":"2024-03-30T03:59:44.597214+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T03:59:44.597214+0000","last_undegraded":"2024-03-30T03:59:44.597214+0000","last_fullsized":"2024-03-30T03:59:44.597214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454004+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T03:59:44.454004+0000","last_peered":"2024-03-30T03:59:44.454004+0000","last_clean":"2024-03-30T03:59:44.454004+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T03:59:44.454004+0000","last_undegraded":"2024-03-30T03:59:44.454004+0000","last_fullsized":"2024-03-30T03:59:44.454004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.462218+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T03:59:44.462218+0000","last_peered":"2024-03-30T03:59:44.462218+0000","last_clean":"2024-03-30T03:59:44.462218+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T03:59:44.462218+0000","last_undegraded":"2024-03-30T03:59:44.462218+0000","last_fullsized":"2024-03-30T03:59:44.462218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900192+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T03:59:44.900192+0000","last_peered":"2024-03-30T03:59:44.900192+0000","last_clean":"2024-03-30T03:59:44.900192+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T03:59:44.900192+0000","last_undegraded":"2024-03-30T03:59:44.900192+0000","last_fullsized":"2024-03-30T03:59:44.900192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597527+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T03:59:44.597527+0000","last_peered":"2024-03-30T03:59:44.597527+0000","last_clean":"2024-03-30T03:59:44.597527+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T03:59:44.597527+0000","last_undegraded":"2024-03-30T03:59:44.597527+0000","last_fullsized":"2024-03-30T03:59:44.597527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454226+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T03:59:44.454226+0000","last_peered":"2024-03-30T03:59:44.454226+0000","last_clean":"2024-03-30T03:59:44.454226+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T03:59:44.454226+0000","last_undegraded":"2024-03-30T03:59:44.454226+0000","last_fullsized":"2024-03-30T03:59:44.454226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455210+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T03:59:44.455210+0000","last_peered":"2024-03-30T03:59:44.455210+0000","last_clean":"2024-03-30T03:59:44.455210+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T03:59:44.455210+0000","last_undegraded":"2024-03-30T03:59:44.455210+0000","last_fullsized":"2024-03-30T03:59:44.455210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455966+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T03:59:44.455966+0000","last_peered":"2024-03-30T03:59:44.455966+0000","last_clean":"2024-03-30T03:59:44.455966+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T03:59:44.455966+0000","last_undegraded":"2024-03-30T03:59:44.455966+0000","last_fullsized":"2024-03-30T03:59:44.455966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.459215+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T03:59:44.459215+0000","last_peered":"2024-03-30T03:59:44.459215+0000","last_clean":"2024-03-30T03:59:44.459215+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T03:59:44.459215+0000","last_undegraded":"2024-03-30T03:59:44.459215+0000","last_fullsized":"2024-03-30T03:59:44.459215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.459129+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T03:59:44.459129+0000","last_peered":"2024-03-30T03:59:44.459129+0000","last_clean":"2024-03-30T03:59:44.459129+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T03:59:44.459129+0000","last_undegraded":"2024-03-30T03:59:44.459129+0000","last_fullsized":"2024-03-30T03:59:44.459129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900031+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T03:59:44.900031+0000","last_peered":"2024-03-30T03:59:44.900031+0000","last_clean":"2024-03-30T03:59:44.900031+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T03:59:44.900031+0000","last_undegraded":"2024-03-30T03:59:44.900031+0000","last_fullsized":"2024-03-30T03:59:44.900031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453913+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T03:59:44.453913+0000","last_peered":"2024-03-30T03:59:44.453913+0000","last_clean":"2024-03-30T03:59:44.453913+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T03:59:44.453913+0000","last_undegraded":"2024-03-30T03:59:44.453913+0000","last_fullsized":"2024-03-30T03:59:44.453913+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.448635+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T03:59:44.448635+0000","last_peered":"2024-03-30T03:59:44.448635+0000","last_clean":"2024-03-30T03:59:44.448635+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T03:59:44.448635+0000","last_undegraded":"2024-03-30T03:59:44.448635+0000","last_fullsized":"2024-03-30T03:59:44.448635+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449739+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T03:59:44.449739+0000","last_peered":"2024-03-30T03:59:44.449739+0000","last_clean":"2024-03-30T03:59:44.449739+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T03:59:44.449739+0000","last_undegraded":"2024-03-30T03:59:44.449739+0000","last_fullsized":"2024-03-30T03:59:44.449739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109974+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T03:59:45.109974+0000","last_peered":"2024-03-30T03:59:45.109974+0000","last_clean":"2024-03-30T03:59:45.109974+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T03:59:45.109974+0000","last_undegraded":"2024-03-30T03:59:45.109974+0000","last_fullsized":"2024-03-30T03:59:45.109974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'13","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:14.974764+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:00:14.974764+0000","last_peered":"2024-03-30T04:00:14.974764+0000","last_clean":"2024-03-30T04:00:14.974764+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:00:14.974764+0000","last_undegraded":"2024-03-30T04:00:14.974764+0000","last_fullsized":"2024-03-30T04:00:14.974764+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.463238+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T03:59:44.463238+0000","last_peered":"2024-03-30T03:59:44.463238+0000","last_clean":"2024-03-30T03:59:44.463238+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T03:59:44.463238+0000","last_undegraded":"2024-03-30T03:59:44.463238+0000","last_fullsized":"2024-03-30T03:59:44.463238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109079+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T03:59:45.109079+0000","last_peered":"2024-03-30T03:59:45.109079+0000","last_clean":"2024-03-30T03:59:45.109079+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T03:59:45.109079+0000","last_undegraded":"2024-03-30T03:59:45.109079+0000","last_fullsized":"2024-03-30T03:59:45.109079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110620+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T03:59:45.110620+0000","last_peered":"2024-03-30T03:59:45.110620+0000","last_clean":"2024-03-30T03:59:45.110620+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T03:59:45.110620+0000","last_undegraded":"2024-03-30T03:59:45.110620+0000","last_fullsized":"2024-03-30T03:59:45.110620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465407+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T03:59:44.465407+0000","last_peered":"2024-03-30T03:59:44.465407+0000","last_clean":"2024-03-30T03:59:44.465407+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T03:59:44.465407+0000","last_undegraded":"2024-03-30T03:59:44.465407+0000","last_fullsized":"2024-03-30T03:59:44.465407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109167+0000","last_change":"2024-03-30T03:59:44.475488+0000","last_active":"2024-03-30T03:59:45.109167+0000","last_peered":"2024-03-30T03:59:45.109167+0000","last_clean":"2024-03-30T03:59:45.109167+0000","last_became_active":"2024-03-30T03:59:44.475178+0000","last_became_peered":"2024-03-30T03:59:44.475178+0000","last_unstale":"2024-03-30T03:59:45.109167+0000","last_undegraded":"2024-03-30T03:59:45.109167+0000","last_fullsized":"2024-03-30T03:59:45.109167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:46.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'11","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.963280+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:00:12.963280+0000","last_peered":"2024-03-30T04:00:12.963280+0000","last_clean":"2024-03-30T04:00:12.963280+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:00:12.963280+0000","last_undegraded":"2024-03-30T04:00:12.963280+0000","last_fullsized":"2024-03-30T04:00:12.963280+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900130+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T03:59:44.900130+0000","last_peered":"2024-03-30T03:59:44.900130+0000","last_clean":"2024-03-30T03:59:44.900130+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T03:59:44.900130+0000","last_undegraded":"2024-03-30T03:59:44.900130+0000","last_fullsized":"2024-03-30T03:59:44.900130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109675+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T03:59:45.109675+0000","last_peered":"2024-03-30T03:59:45.109675+0000","last_clean":"2024-03-30T03:59:45.109675+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T03:59:45.109675+0000","last_undegraded":"2024-03-30T03:59:45.109675+0000","last_fullsized":"2024-03-30T03:59:45.109675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.963588+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:00:12.963588+0000","last_peered":"2024-03-30T04:00:12.963588+0000","last_clean":"2024-03-30T04:00:12.963588+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:00:12.963588+0000","last_undegraded":"2024-03-30T04:00:12.963588+0000","last_fullsized":"2024-03-30T04:00:12.963588+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.457654+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T03:59:44.457654+0000","last_peered":"2024-03-30T03:59:44.457654+0000","last_clean":"2024-03-30T03:59:44.457654+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T03:59:44.457654+0000","last_undegraded":"2024-03-30T03:59:44.457654+0000","last_fullsized":"2024-03-30T03:59:44.457654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110805+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T03:59:45.110805+0000","last_peered":"2024-03-30T03:59:45.110805+0000","last_clean":"2024-03-30T03:59:45.110805+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T03:59:45.110805+0000","last_undegraded":"2024-03-30T03:59:45.110805+0000","last_fullsized":"2024-03-30T03:59:45.110805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'5","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.938244+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:00:11.938244+0000","last_peered":"2024-03-30T04:00:11.938244+0000","last_clean":"2024-03-30T04:00:11.938244+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:00:11.938244+0000","last_undegraded":"2024-03-30T04:00:11.938244+0000","last_fullsized":"2024-03-30T04:00:11.938244+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456966+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T03:59:44.456966+0000","last_peered":"2024-03-30T03:59:44.456966+0000","last_clean":"2024-03-30T03:59:44.456966+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T03:59:44.456966+0000","last_undegraded":"2024-03-30T03:59:44.456966+0000","last_fullsized":"2024-03-30T03:59:44.456966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110857+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T03:59:45.110857+0000","last_peered":"2024-03-30T03:59:45.110857+0000","last_clean":"2024-03-30T03:59:45.110857+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T03:59:45.110857+0000","last_undegraded":"2024-03-30T03:59:45.110857+0000","last_fullsized":"2024-03-30T03:59:45.110857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.953917+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:00:11.953917+0000","last_peered":"2024-03-30T04:00:11.953917+0000","last_clean":"2024-03-30T04:00:11.953917+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:00:11.953917+0000","last_undegraded":"2024-03-30T04:00:11.953917+0000","last_fullsized":"2024-03-30T04:00:11.953917+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453466+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T03:59:44.453466+0000","last_peered":"2024-03-30T03:59:44.453466+0000","last_clean":"2024-03-30T03:59:44.453466+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T03:59:44.453466+0000","last_undegraded":"2024-03-30T03:59:44.453466+0000","last_fullsized":"2024-03-30T03:59:44.453466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110164+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T03:59:45.110164+0000","last_peered":"2024-03-30T03:59:45.110164+0000","last_clean":"2024-03-30T03:59:45.110164+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T03:59:45.110164+0000","last_undegraded":"2024-03-30T03:59:45.110164+0000","last_fullsized":"2024-03-30T03:59:45.110164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.947519+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:00:10.947519+0000","last_peered":"2024-03-30T04:00:10.947519+0000","last_clean":"2024-03-30T04:00:10.947519+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:00:10.947519+0000","last_undegraded":"2024-03-30T04:00:10.947519+0000","last_fullsized":"2024-03-30T04:00:10.947519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.460236+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T03:59:44.460236+0000","last_peered":"2024-03-30T03:59:44.460236+0000","last_clean":"2024-03-30T03:59:44.460236+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T03:59:44.460236+0000","last_undegraded":"2024-03-30T03:59:44.460236+0000","last_fullsized":"2024-03-30T03:59:44.460236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109813+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T03:59:45.109813+0000","last_peered":"2024-03-30T03:59:45.109813+0000","last_clean":"2024-03-30T03:59:45.109813+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T03:59:45.109813+0000","last_undegraded":"2024-03-30T03:59:45.109813+0000","last_fullsized":"2024-03-30T03:59:45.109813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.952955+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:00:11.952955+0000","last_peered":"2024-03-30T04:00:11.952955+0000","last_clean":"2024-03-30T04:00:11.952955+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:00:11.952955+0000","last_undegraded":"2024-03-30T04:00:11.952955+0000","last_fullsized":"2024-03-30T04:00:11.952955+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.448383+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T03:59:44.448383+0000","last_peered":"2024-03-30T03:59:44.448383+0000","last_clean":"2024-03-30T03:59:44.448383+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T03:59:44.448383+0000","last_undegraded":"2024-03-30T03:59:44.448383+0000","last_fullsized":"2024-03-30T03:59:44.448383+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109294+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T03:59:45.109294+0000","last_peered":"2024-03-30T03:59:45.109294+0000","last_clean":"2024-03-30T03:59:45.109294+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T03:59:45.109294+0000","last_undegraded":"2024-03-30T03:59:45.109294+0000","last_fullsized":"2024-03-30T03:59:45.109294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.940412+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:00:13.940412+0000","last_peered":"2024-03-30T04:00:13.940412+0000","last_clean":"2024-03-30T04:00:13.940412+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:00:13.940412+0000","last_undegraded":"2024-03-30T04:00:13.940412+0000","last_fullsized":"2024-03-30T04:00:13.940412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900172+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T03:59:44.900172+0000","last_peered":"2024-03-30T03:59:44.900172+0000","last_clean":"2024-03-30T03:59:44.900172+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T03:59:44.900172+0000","last_undegraded":"2024-03-30T03:59:44.900172+0000","last_fullsized":"2024-03-30T03:59:44.900172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109929+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T03:59:45.109929+0000","last_peered":"2024-03-30T03:59:45.109929+0000","last_clean":"2024-03-30T03:59:45.109929+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T03:59:45.109929+0000","last_undegraded":"2024-03-30T03:59:45.109929+0000","last_fullsized":"2024-03-30T03:59:45.109929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.982193+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:00:09.982193+0000","last_peered":"2024-03-30T04:00:09.982193+0000","last_clean":"2024-03-30T04:00:09.982193+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:00:09.982193+0000","last_undegraded":"2024-03-30T04:00:09.982193+0000","last_fullsized":"2024-03-30T04:00:09.982193+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465522+0000","last_change":"2024-03-30T03:59:40.460524+0000","last_active":"2024-03-30T03:59:44.465522+0000","last_peered":"2024-03-30T03:59:44.465522+0000","last_clean":"2024-03-30T03:59:44.465522+0000","last_became_active":"2024-03-30T03:59:40.460206+0000","last_became_peered":"2024-03-30T03:59:40.460206+0000","last_unstale":"2024-03-30T03:59:44.465522+0000","last_undegraded":"2024-03-30T03:59:44.465522+0000","last_fullsized":"2024-03-30T03:59:44.465522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:50.133321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109383+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T03:59:45.109383+0000","last_peered":"2024-03-30T03:59:45.109383+0000","last_clean":"2024-03-30T03:59:45.109383+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T03:59:45.109383+0000","last_undegraded":"2024-03-30T03:59:45.109383+0000","last_fullsized":"2024-03-30T03:59:45.109383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.936238+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:00:10.936238+0000","last_peered":"2024-03-30T04:00:10.936238+0000","last_clean":"2024-03-30T04:00:10.936238+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:00:10.936238+0000","last_undegraded":"2024-03-30T04:00:10.936238+0000","last_fullsized":"2024-03-30T04:00:10.936238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900148+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T03:59:44.900148+0000","last_peered":"2024-03-30T03:59:44.900148+0000","last_clean":"2024-03-30T03:59:44.900148+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T03:59:44.900148+0000","last_undegraded":"2024-03-30T03:59:44.900148+0000","last_fullsized":"2024-03-30T03:59:44.900148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110347+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T03:59:45.110347+0000","last_peered":"2024-03-30T03:59:45.110347+0000","last_clean":"2024-03-30T03:59:45.110347+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T03:59:45.110347+0000","last_undegraded":"2024-03-30T03:59:45.110347+0000","last_fullsized":"2024-03-30T03:59:45.110347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"84'17","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.965221+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:00:13.965221+0000","last_peered":"2024-03-30T04:00:13.965221+0000","last_clean":"2024-03-30T04:00:13.965221+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:00:13.965221+0000","last_undegraded":"2024-03-30T04:00:13.965221+0000","last_fullsized":"2024-03-30T04:00:13.965221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10398,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":19,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453151+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T03:59:44.453151+0000","last_peered":"2024-03-30T03:59:44.453151+0000","last_clean":"2024-03-30T03:59:44.453151+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T03:59:44.453151+0000","last_undegraded":"2024-03-30T03:59:44.453151+0000","last_fullsized":"2024-03-30T03:59:44.453151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.478501+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T03:59:44.478501+0000","last_peered":"2024-03-30T03:59:44.478501+0000","last_clean":"2024-03-30T03:59:44.478501+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T03:59:44.478501+0000","last_undegraded":"2024-03-30T03:59:44.478501+0000","last_fullsized":"2024-03-30T03:59:44.478501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'7","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.945620+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:00:10.945620+0000","last_peered":"2024-03-30T04:00:10.945620+0000","last_clean":"2024-03-30T04:00:10.945620+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:00:10.945620+0000","last_undegraded":"2024-03-30T04:00:10.945620+0000","last_fullsized":"2024-03-30T04:00:10.945620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.457117+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T03:59:44.457117+0000","last_peered":"2024-03-30T03:59:44.457117+0000","last_clean":"2024-03-30T03:59:44.457117+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T03:59:44.457117+0000","last_undegraded":"2024-03-30T03:59:44.457117+0000","last_fullsized":"2024-03-30T03:59:44.457117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109816+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T03:59:45.109816+0000","last_peered":"2024-03-30T03:59:45.109816+0000","last_clean":"2024-03-30T03:59:45.109816+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T03:59:45.109816+0000","last_undegraded":"2024-03-30T03:59:45.109816+0000","last_fullsized":"2024-03-30T03:59:45.109816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.954901+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:00:11.954901+0000","last_peered":"2024-03-30T04:00:11.954901+0000","last_clean":"2024-03-30T04:00:11.954901+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:00:11.954901+0000","last_undegraded":"2024-03-30T04:00:11.954901+0000","last_fullsized":"2024-03-30T04:00:11.954901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453922+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T03:59:44.453922+0000","last_peered":"2024-03-30T03:59:44.453922+0000","last_clean":"2024-03-30T03:59:44.453922+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T03:59:44.453922+0000","last_undegraded":"2024-03-30T03:59:44.453922+0000","last_fullsized":"2024-03-30T03:59:44.453922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.111292+0000","last_change":"2024-03-30T03:59:44.478370+0000","last_active":"2024-03-30T03:59:45.111292+0000","last_peered":"2024-03-30T03:59:45.111292+0000","last_clean":"2024-03-30T03:59:45.111292+0000","last_became_active":"2024-03-30T03:59:44.477981+0000","last_became_peered":"2024-03-30T03:59:44.477981+0000","last_unstale":"2024-03-30T03:59:45.111292+0000","last_undegraded":"2024-03-30T03:59:45.111292+0000","last_fullsized":"2024-03-30T03:59:45.111292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:47:43.348222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.969680+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:00:13.969680+0000","last_peered":"2024-03-30T04:00:13.969680+0000","last_clean":"2024-03-30T04:00:13.969680+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:00:13.969680+0000","last_undegraded":"2024-03-30T04:00:13.969680+0000","last_fullsized":"2024-03-30T04:00:13.969680+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453863+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T03:59:44.453863+0000","last_peered":"2024-03-30T03:59:44.453863+0000","last_clean":"2024-03-30T03:59:44.453863+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T03:59:44.453863+0000","last_undegraded":"2024-03-30T03:59:44.453863+0000","last_fullsized":"2024-03-30T03:59:44.453863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.476461+0000","last_change":"2024-03-30T03:59:44.473996+0000","last_active":"2024-03-30T03:59:44.476461+0000","last_peered":"2024-03-30T03:59:44.476461+0000","last_clean":"2024-03-30T03:59:44.476461+0000","last_became_active":"2024-03-30T03:59:44.473604+0000","last_became_peered":"2024-03-30T03:59:44.473604+0000","last_unstale":"2024-03-30T03:59:44.476461+0000","last_undegraded":"2024-03-30T03:59:44.476461+0000","last_fullsized":"2024-03-30T03:59:44.476461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:10:01.350249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.961845+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:00:12.961845+0000","last_peered":"2024-03-30T04:00:12.961845+0000","last_clean":"2024-03-30T04:00:12.961845+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:00:12.961845+0000","last_undegraded":"2024-03-30T04:00:12.961845+0000","last_fullsized":"2024-03-30T04:00:12.961845+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455843+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T03:59:44.455843+0000","last_peered":"2024-03-30T03:59:44.455843+0000","last_clean":"2024-03-30T03:59:44.455843+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T03:59:44.455843+0000","last_undegraded":"2024-03-30T03:59:44.455843+0000","last_fullsized":"2024-03-30T03:59:44.455843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.472560+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T03:59:44.472560+0000","last_peered":"2024-03-30T03:59:44.472560+0000","last_clean":"2024-03-30T03:59:44.472560+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T03:59:44.472560+0000","last_undegraded":"2024-03-30T03:59:44.472560+0000","last_fullsized":"2024-03-30T03:59:44.472560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.934050+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:00:10.934050+0000","last_peered":"2024-03-30T04:00:10.934050+0000","last_clean":"2024-03-30T04:00:10.934050+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:00:10.934050+0000","last_undegraded":"2024-03-30T04:00:10.934050+0000","last_fullsized":"2024-03-30T04:00:10.934050+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453675+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T03:59:44.453675+0000","last_peered":"2024-03-30T03:59:44.453675+0000","last_clean":"2024-03-30T03:59:44.453675+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T03:59:44.453675+0000","last_undegraded":"2024-03-30T03:59:44.453675+0000","last_fullsized":"2024-03-30T03:59:44.453675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109700+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T03:59:45.109700+0000","last_peered":"2024-03-30T03:59:45.109700+0000","last_clean":"2024-03-30T03:59:45.109700+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T03:59:45.109700+0000","last_undegraded":"2024-03-30T03:59:45.109700+0000","last_fullsized":"2024-03-30T03:59:45.109700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110841+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T03:59:45.110841+0000","last_peered":"2024-03-30T03:59:45.110841+0000","last_clean":"2024-03-30T03:59:45.110841+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T03:59:45.110841+0000","last_undegraded":"2024-03-30T03:59:45.110841+0000","last_fullsized":"2024-03-30T03:59:45.110841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597510+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T03:59:44.597510+0000","last_peered":"2024-03-30T03:59:44.597510+0000","last_clean":"2024-03-30T03:59:44.597510+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T03:59:44.597510+0000","last_undegraded":"2024-03-30T03:59:44.597510+0000","last_fullsized":"2024-03-30T03:59:44.597510+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.476850+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T03:59:44.476850+0000","last_peered":"2024-03-30T03:59:44.476850+0000","last_clean":"2024-03-30T03:59:44.476850+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T03:59:44.476850+0000","last_undegraded":"2024-03-30T03:59:44.476850+0000","last_fullsized":"2024-03-30T03:59:44.476850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494910+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T03:59:45.494910+0000","last_peered":"2024-03-30T03:59:45.494910+0000","last_clean":"2024-03-30T03:59:45.494910+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T03:59:45.494910+0000","last_undegraded":"2024-03-30T03:59:45.494910+0000","last_fullsized":"2024-03-30T03:59:45.494910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597350+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T03:59:44.597350+0000","last_peered":"2024-03-30T03:59:44.597350+0000","last_clean":"2024-03-30T03:59:44.597350+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T03:59:44.597350+0000","last_undegraded":"2024-03-30T03:59:44.597350+0000","last_fullsized":"2024-03-30T03:59:44.597350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.599769+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T03:59:44.599769+0000","last_peered":"2024-03-30T03:59:44.599769+0000","last_clean":"2024-03-30T03:59:44.599769+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T03:59:44.599769+0000","last_undegraded":"2024-03-30T03:59:44.599769+0000","last_fullsized":"2024-03-30T03:59:44.599769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456481+0000","last_change":"2024-03-30T03:59:40.456721+0000","last_active":"2024-03-30T03:59:44.456481+0000","last_peered":"2024-03-30T03:59:44.456481+0000","last_clean":"2024-03-30T03:59:44.456481+0000","last_became_active":"2024-03-30T03:59:40.456324+0000","last_became_peered":"2024-03-30T03:59:40.456324+0000","last_unstale":"2024-03-30T03:59:44.456481+0000","last_undegraded":"2024-03-30T03:59:44.456481+0000","last_fullsized":"2024-03-30T03:59:44.456481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:16:51.772984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.947642+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:00:10.947642+0000","last_peered":"2024-03-30T04:00:10.947642+0000","last_clean":"2024-03-30T04:00:10.947642+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:00:10.947642+0000","last_undegraded":"2024-03-30T04:00:10.947642+0000","last_fullsized":"2024-03-30T04:00:10.947642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494933+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T03:59:45.494933+0000","last_peered":"2024-03-30T03:59:45.494933+0000","last_clean":"2024-03-30T03:59:45.494933+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T03:59:45.494933+0000","last_undegraded":"2024-03-30T03:59:45.494933+0000","last_fullsized":"2024-03-30T03:59:45.494933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456883+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T03:59:44.456883+0000","last_peered":"2024-03-30T03:59:44.456883+0000","last_clean":"2024-03-30T03:59:44.456883+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T03:59:44.456883+0000","last_undegraded":"2024-03-30T03:59:44.456883+0000","last_fullsized":"2024-03-30T03:59:44.456883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.965576+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:00:13.965576+0000","last_peered":"2024-03-30T04:00:13.965576+0000","last_clean":"2024-03-30T04:00:13.965576+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:00:13.965576+0000","last_undegraded":"2024-03-30T04:00:13.965576+0000","last_fullsized":"2024-03-30T04:00:13.965576+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":335,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494572+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T03:59:45.494572+0000","last_peered":"2024-03-30T03:59:45.494572+0000","last_clean":"2024-03-30T03:59:45.494572+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T03:59:45.494572+0000","last_undegraded":"2024-03-30T03:59:45.494572+0000","last_fullsized":"2024-03-30T03:59:45.494572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.599891+0000","last_change":"2024-03-30T03:59:44.598909+0000","last_active":"2024-03-30T03:59:44.599891+0000","last_peered":"2024-03-30T03:59:44.599891+0000","last_clean":"2024-03-30T03:59:44.599891+0000","last_became_active":"2024-03-30T03:59:44.598427+0000","last_became_peered":"2024-03-30T03:59:44.598427+0000","last_unstale":"2024-03-30T03:59:44.599891+0000","last_undegraded":"2024-03-30T03:59:44.599891+0000","last_fullsized":"2024-03-30T03:59:44.599891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:38:01.401346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455404+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T03:59:44.455404+0000","last_peered":"2024-03-30T03:59:44.455404+0000","last_clean":"2024-03-30T03:59:44.455404+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T03:59:44.455404+0000","last_undegraded":"2024-03-30T03:59:44.455404+0000","last_fullsized":"2024-03-30T03:59:44.455404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.945435+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:00:10.945435+0000","last_peered":"2024-03-30T04:00:10.945435+0000","last_clean":"2024-03-30T04:00:10.945435+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:00:10.945435+0000","last_undegraded":"2024-03-30T04:00:10.945435+0000","last_fullsized":"2024-03-30T04:00:10.945435+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109325+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T03:59:45.109325+0000","last_peered":"2024-03-30T03:59:45.109325+0000","last_clean":"2024-03-30T03:59:45.109325+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T03:59:45.109325+0000","last_undegraded":"2024-03-30T03:59:45.109325+0000","last_fullsized":"2024-03-30T03:59:45.109325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.955404+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:00:11.955404+0000","last_peered":"2024-03-30T04:00:11.955404+0000","last_clean":"2024-03-30T04:00:11.955404+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:00:11.955404+0000","last_undegraded":"2024-03-30T04:00:11.955404+0000","last_fullsized":"2024-03-30T04:00:11.955404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465339+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T03:59:44.465339+0000","last_peered":"2024-03-30T03:59:44.465339+0000","last_clean":"2024-03-30T03:59:44.465339+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T03:59:44.465339+0000","last_undegraded":"2024-03-30T03:59:44.465339+0000","last_fullsized":"2024-03-30T03:59:44.465339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109224+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T03:59:45.109224+0000","last_peered":"2024-03-30T03:59:45.109224+0000","last_clean":"2024-03-30T03:59:45.109224+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T03:59:45.109224+0000","last_undegraded":"2024-03-30T03:59:45.109224+0000","last_fullsized":"2024-03-30T03:59:45.109224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.956084+0000","last_change":"2024-03-30T03:59:39.932264+0000","last_active":"2024-03-30T04:00:11.956084+0000","last_peered":"2024-03-30T04:00:11.956084+0000","last_clean":"2024-03-30T04:00:11.956084+0000","last_became_active":"2024-03-30T03:59:39.931859+0000","last_became_peered":"2024-03-30T03:59:39.931859+0000","last_unstale":"2024-03-30T04:00:11.956084+0000","last_undegraded":"2024-03-30T04:00:11.956084+0000","last_fullsized":"2024-03-30T04:00:11.956084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:31:24.215248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465468+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T03:59:44.465468+0000","last_peered":"2024-03-30T03:59:44.465468+0000","last_clean":"2024-03-30T03:59:44.465468+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T03:59:44.465468+0000","last_undegraded":"2024-03-30T03:59:44.465468+0000","last_fullsized":"2024-03-30T03:59:44.465468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109009+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T03:59:45.109009+0000","last_peered":"2024-03-30T03:59:45.109009+0000","last_clean":"2024-03-30T03:59:45.109009+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T03:59:45.109009+0000","last_undegraded":"2024-03-30T03:59:45.109009+0000","last_fullsized":"2024-03-30T03:59:45.109009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.962495+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:00:12.962495+0000","last_peered":"2024-03-30T04:00:12.962495+0000","last_clean":"2024-03-30T04:00:12.962495+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:00:12.962495+0000","last_undegraded":"2024-03-30T04:00:12.962495+0000","last_fullsized":"2024-03-30T04:00:12.962495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465577+0000","last_change":"2024-03-30T03:59:40.464994+0000","last_active":"2024-03-30T03:59:44.465577+0000","last_peered":"2024-03-30T03:59:44.465577+0000","last_clean":"2024-03-30T03:59:44.465577+0000","last_became_active":"2024-03-30T03:59:40.462862+0000","last_became_peered":"2024-03-30T03:59:40.462862+0000","last_unstale":"2024-03-30T03:59:44.465577+0000","last_undegraded":"2024-03-30T03:59:44.465577+0000","last_fullsized":"2024-03-30T03:59:44.465577+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:29:48.413081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110065+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T03:59:45.110065+0000","last_peered":"2024-03-30T03:59:45.110065+0000","last_clean":"2024-03-30T03:59:45.110065+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T03:59:45.110065+0000","last_undegraded":"2024-03-30T03:59:45.110065+0000","last_fullsized":"2024-03-30T03:59:45.110065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.982804+0000","last_change":"2024-03-30T03:59:39.928250+0000","last_active":"2024-03-30T04:00:09.982804+0000","last_peered":"2024-03-30T04:00:09.982804+0000","last_clean":"2024-03-30T04:00:09.982804+0000","last_became_active":"2024-03-30T03:59:39.927915+0000","last_became_peered":"2024-03-30T03:59:39.927915+0000","last_unstale":"2024-03-30T04:00:09.982804+0000","last_undegraded":"2024-03-30T04:00:09.982804+0000","last_fullsized":"2024-03-30T04:00:09.982804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:02:47.307714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.452764+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T03:59:44.452764+0000","last_peered":"2024-03-30T03:59:44.452764+0000","last_clean":"2024-03-30T03:59:44.452764+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T03:59:44.452764+0000","last_undegraded":"2024-03-30T03:59:44.452764+0000","last_fullsized":"2024-03-30T03:59:44.452764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.111338+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T03:59:45.111338+0000","last_peered":"2024-03-30T03:59:45.111338+0000","last_clean":"2024-03-30T03:59:45.111338+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T03:59:45.111338+0000","last_undegraded":"2024-03-30T03:59:45.111338+0000","last_fullsized":"2024-03-30T03:59:45.111338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.964240+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:00:12.964240+0000","last_peered":"2024-03-30T04:00:12.964240+0000","last_clean":"2024-03-30T04:00:12.964240+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:00:12.964240+0000","last_undegraded":"2024-03-30T04:00:12.964240+0000","last_fullsized":"2024-03-30T04:00:12.964240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453800+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T03:59:44.453800+0000","last_peered":"2024-03-30T03:59:44.453800+0000","last_clean":"2024-03-30T03:59:44.453800+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T03:59:44.453800+0000","last_undegraded":"2024-03-30T03:59:44.453800+0000","last_fullsized":"2024-03-30T03:59:44.453800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.900306+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T03:59:45.900306+0000","last_peered":"2024-03-30T03:59:45.900306+0000","last_clean":"2024-03-30T03:59:45.900306+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T03:59:45.900306+0000","last_undegraded":"2024-03-30T03:59:45.900306+0000","last_fullsized":"2024-03-30T03:59:45.900306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110168+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T03:59:45.110168+0000","last_peered":"2024-03-30T03:59:45.110168+0000","last_clean":"2024-03-30T03:59:45.110168+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T03:59:45.110168+0000","last_undegraded":"2024-03-30T03:59:45.110168+0000","last_fullsized":"2024-03-30T03:59:45.110168+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.464621+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T03:59:44.464621+0000","last_peered":"2024-03-30T03:59:44.464621+0000","last_clean":"2024-03-30T03:59:44.464621+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T03:59:44.464621+0000","last_undegraded":"2024-03-30T03:59:44.464621+0000","last_fullsized":"2024-03-30T03:59:44.464621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110097+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T03:59:45.110097+0000","last_peered":"2024-03-30T03:59:45.110097+0000","last_clean":"2024-03-30T03:59:45.110097+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T03:59:45.110097+0000","last_undegraded":"2024-03-30T03:59:45.110097+0000","last_fullsized":"2024-03-30T03:59:45.110097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.981124+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:00:09.981124+0000","last_peered":"2024-03-30T04:00:09.981124+0000","last_clean":"2024-03-30T04:00:09.981124+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:00:09.981124+0000","last_undegraded":"2024-03-30T04:00:09.981124+0000","last_fullsized":"2024-03-30T04:00:09.981124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454242+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T03:59:44.454242+0000","last_peered":"2024-03-30T03:59:44.454242+0000","last_clean":"2024-03-30T03:59:44.454242+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T03:59:44.454242+0000","last_undegraded":"2024-03-30T03:59:44.454242+0000","last_fullsized":"2024-03-30T03:59:44.454242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900088+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T03:59:44.900088+0000","last_peered":"2024-03-30T03:59:44.900088+0000","last_clean":"2024-03-30T03:59:44.900088+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T03:59:44.900088+0000","last_undegraded":"2024-03-30T03:59:44.900088+0000","last_fullsized":"2024-03-30T03:59:44.900088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.953027+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:00:11.953027+0000","last_peered":"2024-03-30T04:00:11.953027+0000","last_clean":"2024-03-30T04:00:11.953027+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:00:11.953027+0000","last_undegraded":"2024-03-30T04:00:11.953027+0000","last_fullsized":"2024-03-30T04:00:11.953027+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454411+0000","last_change":"2024-03-30T03:59:40.462459+0000","last_active":"2024-03-30T03:59:44.454411+0000","last_peered":"2024-03-30T03:59:44.454411+0000","last_clean":"2024-03-30T03:59:44.454411+0000","last_became_active":"2024-03-30T03:59:40.461692+0000","last_became_peered":"2024-03-30T03:59:40.461692+0000","last_unstale":"2024-03-30T03:59:44.454411+0000","last_undegraded":"2024-03-30T03:59:44.454411+0000","last_fullsized":"2024-03-30T03:59:44.454411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:46.872671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109492+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T03:59:45.109492+0000","last_peered":"2024-03-30T03:59:45.109492+0000","last_clean":"2024-03-30T03:59:45.109492+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T03:59:45.109492+0000","last_undegraded":"2024-03-30T03:59:45.109492+0000","last_fullsized":"2024-03-30T03:59:45.109492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.955007+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:00:11.955007+0000","last_peered":"2024-03-30T04:00:11.955007+0000","last_clean":"2024-03-30T04:00:11.955007+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:00:11.955007+0000","last_undegraded":"2024-03-30T04:00:11.955007+0000","last_fullsized":"2024-03-30T04:00:11.955007+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453799+0000","last_change":"2024-03-30T03:59:40.467087+0000","last_active":"2024-03-30T03:59:44.453799+0000","last_peered":"2024-03-30T03:59:44.453799+0000","last_clean":"2024-03-30T03:59:44.453799+0000","last_became_active":"2024-03-30T03:59:40.466769+0000","last_became_peered":"2024-03-30T03:59:40.466769+0000","last_unstale":"2024-03-30T03:59:44.453799+0000","last_undegraded":"2024-03-30T03:59:44.453799+0000","last_fullsized":"2024-03-30T03:59:44.453799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:25:41.741807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.463545+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T03:59:44.463545+0000","last_peered":"2024-03-30T03:59:44.463545+0000","last_clean":"2024-03-30T03:59:44.463545+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T03:59:44.463545+0000","last_undegraded":"2024-03-30T03:59:44.463545+0000","last_fullsized":"2024-03-30T03:59:44.463545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.962772+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:00:12.962772+0000","last_peered":"2024-03-30T04:00:12.962772+0000","last_clean":"2024-03-30T04:00:12.962772+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:00:12.962772+0000","last_undegraded":"2024-03-30T04:00:12.962772+0000","last_fullsized":"2024-03-30T04:00:12.962772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454355+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T03:59:44.454355+0000","last_peered":"2024-03-30T03:59:44.454355+0000","last_clean":"2024-03-30T03:59:44.454355+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T03:59:44.454355+0000","last_undegraded":"2024-03-30T03:59:44.454355+0000","last_fullsized":"2024-03-30T03:59:44.454355+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453905+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T03:59:44.453905+0000","last_peered":"2024-03-30T03:59:44.453905+0000","last_clean":"2024-03-30T03:59:44.453905+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T03:59:44.453905+0000","last_undegraded":"2024-03-30T03:59:44.453905+0000","last_fullsized":"2024-03-30T03:59:44.453905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.450026+0000","last_change":"2024-03-30T03:59:40.458059+0000","last_active":"2024-03-30T03:59:44.450026+0000","last_peered":"2024-03-30T03:59:44.450026+0000","last_clean":"2024-03-30T03:59:44.450026+0000","last_became_active":"2024-03-30T03:59:40.457738+0000","last_became_peered":"2024-03-30T03:59:40.457738+0000","last_unstale":"2024-03-30T03:59:44.450026+0000","last_undegraded":"2024-03-30T03:59:44.450026+0000","last_fullsized":"2024-03-30T03:59:44.450026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:21.899370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449155+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T03:59:44.449155+0000","last_peered":"2024-03-30T03:59:44.449155+0000","last_clean":"2024-03-30T03:59:44.449155+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T03:59:44.449155+0000","last_undegraded":"2024-03-30T03:59:44.449155+0000","last_fullsized":"2024-03-30T03:59:44.449155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.458407+0000","last_change":"2024-03-30T03:59:40.454510+0000","last_active":"2024-03-30T03:59:44.458407+0000","last_peered":"2024-03-30T03:59:44.458407+0000","last_clean":"2024-03-30T03:59:44.458407+0000","last_became_active":"2024-03-30T03:59:40.454119+0000","last_became_peered":"2024-03-30T03:59:40.454119+0000","last_unstale":"2024-03-30T03:59:44.458407+0000","last_undegraded":"2024-03-30T03:59:44.458407+0000","last_fullsized":"2024-03-30T03:59:44.458407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:50:00.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465252+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T03:59:44.465252+0000","last_peered":"2024-03-30T03:59:44.465252+0000","last_clean":"2024-03-30T03:59:44.465252+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T03:59:44.465252+0000","last_undegraded":"2024-03-30T03:59:44.465252+0000","last_fullsized":"2024-03-30T03:59:44.465252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453459+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T03:59:44.453459+0000","last_peered":"2024-03-30T03:59:44.453459+0000","last_clean":"2024-03-30T03:59:44.453459+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T03:59:44.453459+0000","last_undegraded":"2024-03-30T03:59:44.453459+0000","last_fullsized":"2024-03-30T03:59:44.453459+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449683+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T03:59:44.449683+0000","last_peered":"2024-03-30T03:59:44.449683+0000","last_clean":"2024-03-30T03:59:44.449683+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T03:59:44.449683+0000","last_undegraded":"2024-03-30T03:59:44.449683+0000","last_fullsized":"2024-03-30T03:59:44.449683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.469792+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T03:59:44.469792+0000","last_peered":"2024-03-30T03:59:44.469792+0000","last_clean":"2024-03-30T03:59:44.469792+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T03:59:44.469792+0000","last_undegraded":"2024-03-30T03:59:44.469792+0000","last_fullsized":"2024-03-30T03:59:44.469792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456383+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T03:59:44.456383+0000","last_peered":"2024-03-30T03:59:44.456383+0000","last_clean":"2024-03-30T03:59:44.456383+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T03:59:44.456383+0000","last_undegraded":"2024-03-30T03:59:44.456383+0000","last_fullsized":"2024-03-30T03:59:44.456383+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.460984+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T03:59:44.460984+0000","last_peered":"2024-03-30T03:59:44.460984+0000","last_clean":"2024-03-30T03:59:44.460984+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T03:59:44.460984+0000","last_undegraded":"2024-03-30T03:59:44.460984+0000","last_fullsized":"2024-03-30T03:59:44.460984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900110+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T03:59:44.900110+0000","last_peered":"2024-03-30T03:59:44.900110+0000","last_clean":"2024-03-30T03:59:44.900110+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T03:59:44.900110+0000","last_undegraded":"2024-03-30T03:59:44.900110+0000","last_fullsized":"2024-03-30T03:59:44.900110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456058+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T03:59:44.456058+0000","last_peered":"2024-03-30T03:59:44.456058+0000","last_clean":"2024-03-30T03:59:44.456058+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T03:59:44.456058+0000","last_undegraded":"2024-03-30T03:59:44.456058+0000","last_fullsized":"2024-03-30T03:59:44.456058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454093+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T03:59:44.454093+0000","last_peered":"2024-03-30T03:59:44.454093+0000","last_clean":"2024-03-30T03:59:44.454093+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T03:59:44.454093+0000","last_undegraded":"2024-03-30T03:59:44.454093+0000","last_fullsized":"2024-03-30T03:59:44.454093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":16481,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":200,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":56418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":937984,"data_stored":451119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.9340000000000002,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.68799999999999994}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.71999999999999997}]},{"osd":10,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.73699999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776148,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1585152,"data_stored":1052822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47899999999999998}]},{"osd":3,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":8,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.66700000000000004}]},{"osd":9,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.70799999999999996}]},{"osd":11,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.74199999999999999}]}]},{"osd":9,"up_from":63,"seq":270582939673,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":970752,"data_stored":459289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.621}]},{"osd":2,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.69499999999999995}]},{"osd":4,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.74399999999999999}]},{"osd":5,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.4860000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.83299999999999996}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.72699999999999998}]},{"osd":8,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.86699999999999999}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.84299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168605,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":958464,"data_stored":451475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70599999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.432}]},{"osd":3,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.40899999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.82099999999999995}]},{"osd":9,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80800000000000005}]},{"osd":10,"last update":"Sat Mar 30 03:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.747}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":7,"up_from":49,"seq":210453397534,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1042039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.372,"15min":0.372},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61199999999999999,"15min":0.60799999999999998},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.625}]},{"osd":6,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86699999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.64200000000000002}]},{"osd":8,"last update":"Sat Mar 30 04:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.48299999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.82799999999999996}]},{"osd":9,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78700000000000003}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.73099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":6,"up_from":44,"seq":188978561059,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":450087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.625},"min":{"1min":0.47099999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89300000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95399999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.73699999999999999}]},{"osd":2,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68100000000000005,"15min":0.68300000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0289999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.67700000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0800000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.92900000000000005,"5min":0.995,"15min":0.995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.69599999999999995}]},{"osd":4,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.51300000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.746},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.59499999999999997,"5min":0.438,"15min":0.438},"max":{"1min":1.046,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.121,"5min":1.462,"15min":1.462},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.83499999999999996,"15min":0.83999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95399999999999996,"5min":1.472,"15min":1.472},"last":0.80600000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.96399999999999997}]},{"osd":8,"last update":"Sat Mar 30 04:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.85999999999999999,"15min":0.85099999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.88800000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.79500000000000004}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.97499999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36144,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709008,"statfs":{"total":95995035648,"available":95958024192,"internally_reserved":0,"allocated":958464,"data_stored":462470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54000000000000004,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.84999999999999998}]},{"osd":2,"last update":"Sat Mar 30 03:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.48499999999999999,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.51100000000000001,"15min":0.48699999999999999},"min":{"1min":0.35199999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.52000000000000002,"15min":0.49199999999999999},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79300000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.45200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66400000000000003}]},{"osd":4,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.32500000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.374},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.34300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66400000000000003,"15min":0.65600000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.82099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.74199999999999999}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.44900000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84699999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.55200000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.71999999999999997}]},{"osd":9,"last update":"Sat Mar 30 03:59:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78600000000000003}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.626}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":451453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.81000000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71799999999999997}]},{"osd":2,"last update":"Sat Mar 30 03:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.39900000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.504},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Mar 30 03:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48199999999999998},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91100000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.38300000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Mar 30 03:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85399999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68899999999999995,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90500000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Sat Mar 30 03:59:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.68100000000000005,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.67700000000000005,"15min":0.66100000000000003},"min":{"1min":0.57599999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.50800000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.73199999999999998,"15min":0.71499999999999997},"min":{"1min":0.505,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.84999999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.82699999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.68999999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.1180000000000001,"5min":1.333,"15min":1.333},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.67500000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.038,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.76500000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.97299999999999998}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.75}]},{"osd":11,"last update":"Sat Mar 30 04:00:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.78100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":455073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84099999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":1.133,"15min":1.133},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.41499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.66200000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.75,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38700000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.54500000000000004}]},{"osd":4,"last update":"Sat Mar 30 03:58:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.751},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.751,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82399999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67800000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66600000000000004}]},{"osd":9,"last update":"Sat Mar 30 03:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73399999999999999}]},{"osd":10,"last update":"Sat Mar 30 03:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":450405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.42899999999999999,"15min":0.40400000000000003},"min":{"1min":0.376,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45800000000000002,"15min":0.42999999999999999},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.762,"15min":1.762},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.64900000000000002,"15min":0.60599999999999998},"min":{"1min":0.58699999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0409999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.58499999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.181,"15min":1.181},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.89200000000000002,"5min":1.296,"15min":1.296},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88200000000000001,"5min":1.113,"15min":1.113},"last":0.501},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.70599999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.591,"15min":1.591},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70299999999999996,"15min":0.66500000000000004},"min":{"1min":0.61099999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93799999999999994,"5min":1.139,"15min":1.139},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.77000000000000002,"15min":0.72999999999999998},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0569999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.77200000000000002,"15min":0.754},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.87,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.59699999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.214,"15min":1.214},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1499999999999999,"5min":1.242,"15min":1.242},"last":0.84899999999999998}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:30 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.51700000000000002}]},{"osd":10,"last update":"Sat Mar 30 03:59:47 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":946176,"data_stored":451020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59999999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.307,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.52700000000000002},"min":{"1min":0.46000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83699999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Sat Mar 30 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.51400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.88700000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.748}]},{"osd":6,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.47199999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.51100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56899999999999995}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.45300000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93999999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.51900000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90000000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.70899999999999996}]},{"osd":9,"last update":"Sat Mar 30 03:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.59699999999999998}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.58399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1560576,"data_stored":1042784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.751,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.45300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.42699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.246,"15min":0.246},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":0.873},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.627,"15min":0.627},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.622,"15min":0.625},"min":{"1min":0.42299999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78200000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76300000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.80800000000000005}]},{"osd":8,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.77900000000000003}]},{"osd":9,"last update":"Sat Mar 30 03:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.83299999999999996}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":25194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":12479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:00:17.529 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T04:00:17.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 -- 172.21.15.23:0/2128952111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 msgr2=0x7f9c781022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/2128952111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c781022d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9c640099c0 tx=0x7f9c64009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 -- 172.21.15.23:0/2128952111 shutdown_connections 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/2128952111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78102810 0x7f9c781090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/2128952111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78100c80 0x7f9c781010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/2128952111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c781022d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 -- 172.21.15.23:0/2128952111 >> 172.21.15.23:0/2128952111 conn(0x7f9c780fc1f0 msgr2=0x7f9c780fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 -- 172.21.15.23:0/2128952111 shutdown_connections 2024-03-30T04:00:17.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.760+0000 7f9c7dc00700 1 -- 172.21.15.23:0/2128952111 wait complete. 2024-03-30T04:00:17.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.761+0000 7f9c7dc00700 1 Processor -- start 2024-03-30T04:00:17.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.761+0000 7f9c7dc00700 1 -- start start 2024-03-30T04:00:17.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78100c80 0x7f9c78074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78102810 0x7f9c780736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78073e80 con 0x7f9c78101e70 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c78073ff0 con 0x7f9c78100c80 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c7dc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c781a0c20 con 0x7f9c78102810 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35634/0 (socket says 172.21.15.23:35634) 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 -- 172.21.15.23:0/3227958213 learned_addr learned my addr 172.21.15.23:0/3227958213 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c77400700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78102810 0x7f9c780736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76a00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78100c80 0x7f9c78074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 -- 172.21.15.23:0/3227958213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78102810 msgr2=0x7f9c780736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78102810 0x7f9c780736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 -- 172.21.15.23:0/3227958213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78100c80 msgr2=0x7f9c78074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:17.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.762+0000 7f9c76000700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78100c80 0x7f9c78074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c76000700 1 -- 172.21.15.23:0/3227958213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c640095f0 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c76000700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c64003730 tx=0x7f9c6400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6401d070 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c64005050 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c781a0e50 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64004b10 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.763+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c781a11c0 con 0x7f9c78101e70 2024-03-30T04:00:17.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.764+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c78067a80 con 0x7f9c78101e70 2024-03-30T04:00:17.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.766+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f9c640051c0 con 0x7f9c78101e70 2024-03-30T04:00:17.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.767+0000 7f9c6f400700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 0x7f9c5c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:17.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.768+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9c640cdb40 con 0x7f9c78101e70 2024-03-30T04:00:17.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.768+0000 7f9c76a00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 0x7f9c5c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:17.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.768+0000 7f9c76a00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 0x7f9c5c0a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9c78101cd0 tx=0x7f9c68009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:17.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.771+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9c78067a80 con 0x7f9c78101e70 2024-03-30T04:00:17.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.984+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c78067a10 con 0x7f9c5c0a11b0 2024-03-30T04:00:17.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:17.998+0000 7f9c6f400700 1 -- 172.21.15.23:0/3227958213 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422895 (secure 0 0 0) 0x7f9c78067a10 con 0x7f9c5c0a11b0 2024-03-30T04:00:18.002 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 msgr2=0x7f9c5c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 0x7f9c5c0a3670 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9c78101cd0 tx=0x7f9c68009260 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 msgr2=0x7f9c780731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c64003730 tx=0x7f9c6400f690 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 shutdown_connections 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f9c5c0a11b0 0x7f9c5c0a3670 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9c78102810 0x7f9c780736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c78100c80 0x7f9c78074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 --2- 172.21.15.23:0/3227958213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c78101e70 0x7f9c780731a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 >> 172.21.15.23:0/3227958213 conn(0x7f9c780fc1f0 msgr2=0x7f9c78105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:18.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 shutdown_connections 2024-03-30T04:00:18.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:18.003+0000 7f9c7dc00700 1 -- 172.21.15.23:0/3227958213 wait complete. 2024-03-30T04:00:18.007 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:00:18.027 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-03-30T04:00:16.995845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606966,"num_objects":94,"num_object_clones":0,"num_object_copies":283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":182,"num_read_kb":159,"num_write":382,"num_write_kb":3014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":313,"ondisk_log_size":313,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369908,"kb_used_data":12948,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124571916,"statfs":{"total":1151940427776,"available":1151561641984,"internally_reserved":0,"allocated":13258752,"data_stored":7220036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12004,"num_objects":70,"num_object_clones":0,"num_object_copies":211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":152,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.722198"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454371+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T03:59:44.454371+0000","last_peered":"2024-03-30T03:59:44.454371+0000","last_clean":"2024-03-30T03:59:44.454371+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T03:59:44.454371+0000","last_undegraded":"2024-03-30T03:59:44.454371+0000","last_fullsized":"2024-03-30T03:59:44.454371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465480+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T03:59:44.465480+0000","last_peered":"2024-03-30T03:59:44.465480+0000","last_clean":"2024-03-30T03:59:44.465480+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T03:59:44.465480+0000","last_undegraded":"2024-03-30T03:59:44.465480+0000","last_fullsized":"2024-03-30T03:59:44.465480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.464161+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T03:59:44.464161+0000","last_peered":"2024-03-30T03:59:44.464161+0000","last_clean":"2024-03-30T03:59:44.464161+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T03:59:44.464161+0000","last_undegraded":"2024-03-30T03:59:44.464161+0000","last_fullsized":"2024-03-30T03:59:44.464161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455406+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T03:59:44.455406+0000","last_peered":"2024-03-30T03:59:44.455406+0000","last_clean":"2024-03-30T03:59:44.455406+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T03:59:44.455406+0000","last_undegraded":"2024-03-30T03:59:44.455406+0000","last_fullsized":"2024-03-30T03:59:44.455406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.452773+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T03:59:44.452773+0000","last_peered":"2024-03-30T03:59:44.452773+0000","last_clean":"2024-03-30T03:59:44.452773+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T03:59:44.452773+0000","last_undegraded":"2024-03-30T03:59:44.452773+0000","last_fullsized":"2024-03-30T03:59:44.452773+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597214+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T03:59:44.597214+0000","last_peered":"2024-03-30T03:59:44.597214+0000","last_clean":"2024-03-30T03:59:44.597214+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T03:59:44.597214+0000","last_undegraded":"2024-03-30T03:59:44.597214+0000","last_fullsized":"2024-03-30T03:59:44.597214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454004+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T03:59:44.454004+0000","last_peered":"2024-03-30T03:59:44.454004+0000","last_clean":"2024-03-30T03:59:44.454004+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T03:59:44.454004+0000","last_undegraded":"2024-03-30T03:59:44.454004+0000","last_fullsized":"2024-03-30T03:59:44.454004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.462218+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T03:59:44.462218+0000","last_peered":"2024-03-30T03:59:44.462218+0000","last_clean":"2024-03-30T03:59:44.462218+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T03:59:44.462218+0000","last_undegraded":"2024-03-30T03:59:44.462218+0000","last_fullsized":"2024-03-30T03:59:44.462218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900192+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T03:59:44.900192+0000","last_peered":"2024-03-30T03:59:44.900192+0000","last_clean":"2024-03-30T03:59:44.900192+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T03:59:44.900192+0000","last_undegraded":"2024-03-30T03:59:44.900192+0000","last_fullsized":"2024-03-30T03:59:44.900192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597527+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T03:59:44.597527+0000","last_peered":"2024-03-30T03:59:44.597527+0000","last_clean":"2024-03-30T03:59:44.597527+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T03:59:44.597527+0000","last_undegraded":"2024-03-30T03:59:44.597527+0000","last_fullsized":"2024-03-30T03:59:44.597527+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454226+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T03:59:44.454226+0000","last_peered":"2024-03-30T03:59:44.454226+0000","last_clean":"2024-03-30T03:59:44.454226+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T03:59:44.454226+0000","last_undegraded":"2024-03-30T03:59:44.454226+0000","last_fullsized":"2024-03-30T03:59:44.454226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455210+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T03:59:44.455210+0000","last_peered":"2024-03-30T03:59:44.455210+0000","last_clean":"2024-03-30T03:59:44.455210+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T03:59:44.455210+0000","last_undegraded":"2024-03-30T03:59:44.455210+0000","last_fullsized":"2024-03-30T03:59:44.455210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455966+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T03:59:44.455966+0000","last_peered":"2024-03-30T03:59:44.455966+0000","last_clean":"2024-03-30T03:59:44.455966+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T03:59:44.455966+0000","last_undegraded":"2024-03-30T03:59:44.455966+0000","last_fullsized":"2024-03-30T03:59:44.455966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.459215+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T03:59:44.459215+0000","last_peered":"2024-03-30T03:59:44.459215+0000","last_clean":"2024-03-30T03:59:44.459215+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T03:59:44.459215+0000","last_undegraded":"2024-03-30T03:59:44.459215+0000","last_fullsized":"2024-03-30T03:59:44.459215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.459129+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T03:59:44.459129+0000","last_peered":"2024-03-30T03:59:44.459129+0000","last_clean":"2024-03-30T03:59:44.459129+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T03:59:44.459129+0000","last_undegraded":"2024-03-30T03:59:44.459129+0000","last_fullsized":"2024-03-30T03:59:44.459129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900031+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T03:59:44.900031+0000","last_peered":"2024-03-30T03:59:44.900031+0000","last_clean":"2024-03-30T03:59:44.900031+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T03:59:44.900031+0000","last_undegraded":"2024-03-30T03:59:44.900031+0000","last_fullsized":"2024-03-30T03:59:44.900031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453913+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T03:59:44.453913+0000","last_peered":"2024-03-30T03:59:44.453913+0000","last_clean":"2024-03-30T03:59:44.453913+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T03:59:44.453913+0000","last_undegraded":"2024-03-30T03:59:44.453913+0000","last_fullsized":"2024-03-30T03:59:44.453913+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.448635+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T03:59:44.448635+0000","last_peered":"2024-03-30T03:59:44.448635+0000","last_clean":"2024-03-30T03:59:44.448635+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T03:59:44.448635+0000","last_undegraded":"2024-03-30T03:59:44.448635+0000","last_fullsized":"2024-03-30T03:59:44.448635+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449739+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T03:59:44.449739+0000","last_peered":"2024-03-30T03:59:44.449739+0000","last_clean":"2024-03-30T03:59:44.449739+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T03:59:44.449739+0000","last_undegraded":"2024-03-30T03:59:44.449739+0000","last_fullsized":"2024-03-30T03:59:44.449739+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109974+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T03:59:45.109974+0000","last_peered":"2024-03-30T03:59:45.109974+0000","last_clean":"2024-03-30T03:59:45.109974+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T03:59:45.109974+0000","last_undegraded":"2024-03-30T03:59:45.109974+0000","last_fullsized":"2024-03-30T03:59:45.109974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'13","reported_seq":33,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:14.974764+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:00:14.974764+0000","last_peered":"2024-03-30T04:00:14.974764+0000","last_clean":"2024-03-30T04:00:14.974764+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:00:14.974764+0000","last_undegraded":"2024-03-30T04:00:14.974764+0000","last_fullsized":"2024-03-30T04:00:14.974764+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.463238+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T03:59:44.463238+0000","last_peered":"2024-03-30T03:59:44.463238+0000","last_clean":"2024-03-30T03:59:44.463238+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T03:59:44.463238+0000","last_undegraded":"2024-03-30T03:59:44.463238+0000","last_fullsized":"2024-03-30T03:59:44.463238+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109079+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T03:59:45.109079+0000","last_peered":"2024-03-30T03:59:45.109079+0000","last_clean":"2024-03-30T03:59:45.109079+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T03:59:45.109079+0000","last_undegraded":"2024-03-30T03:59:45.109079+0000","last_fullsized":"2024-03-30T03:59:45.109079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110620+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T03:59:45.110620+0000","last_peered":"2024-03-30T03:59:45.110620+0000","last_clean":"2024-03-30T03:59:45.110620+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T03:59:45.110620+0000","last_undegraded":"2024-03-30T03:59:45.110620+0000","last_fullsized":"2024-03-30T03:59:45.110620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465407+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T03:59:44.465407+0000","last_peered":"2024-03-30T03:59:44.465407+0000","last_clean":"2024-03-30T03:59:44.465407+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T03:59:44.465407+0000","last_undegraded":"2024-03-30T03:59:44.465407+0000","last_fullsized":"2024-03-30T03:59:44.465407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109167+0000","last_change":"2024-03-30T03:59:44.475488+0000","last_active":"2024-03-30T03:59:45.109167+0000","last_peered":"2024-03-30T03:59:45.109167+0000","last_clean":"2024-03-30T03:59:45.109167+0000","last_became_active":"2024-03-30T03:59:44.475178+0000","last_became_peered":"2024-03-30T03:59:44.475178+0000","last_unstale":"2024-03-30T03:59:45.109167+0000","last_undegraded":"2024-03-30T03:59:45.109167+0000","last_fullsized":"2024-03-30T03:59:45.109167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:46.632391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'11","reported_seq":28,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.963280+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:00:12.963280+0000","last_peered":"2024-03-30T04:00:12.963280+0000","last_clean":"2024-03-30T04:00:12.963280+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:00:12.963280+0000","last_undegraded":"2024-03-30T04:00:12.963280+0000","last_fullsized":"2024-03-30T04:00:12.963280+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900130+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T03:59:44.900130+0000","last_peered":"2024-03-30T03:59:44.900130+0000","last_clean":"2024-03-30T03:59:44.900130+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T03:59:44.900130+0000","last_undegraded":"2024-03-30T03:59:44.900130+0000","last_fullsized":"2024-03-30T03:59:44.900130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109675+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T03:59:45.109675+0000","last_peered":"2024-03-30T03:59:45.109675+0000","last_clean":"2024-03-30T03:59:45.109675+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T03:59:45.109675+0000","last_undegraded":"2024-03-30T03:59:45.109675+0000","last_fullsized":"2024-03-30T03:59:45.109675+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.963588+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:00:12.963588+0000","last_peered":"2024-03-30T04:00:12.963588+0000","last_clean":"2024-03-30T04:00:12.963588+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:00:12.963588+0000","last_undegraded":"2024-03-30T04:00:12.963588+0000","last_fullsized":"2024-03-30T04:00:12.963588+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.457654+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T03:59:44.457654+0000","last_peered":"2024-03-30T03:59:44.457654+0000","last_clean":"2024-03-30T03:59:44.457654+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T03:59:44.457654+0000","last_undegraded":"2024-03-30T03:59:44.457654+0000","last_fullsized":"2024-03-30T03:59:44.457654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110805+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T03:59:45.110805+0000","last_peered":"2024-03-30T03:59:45.110805+0000","last_clean":"2024-03-30T03:59:45.110805+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T03:59:45.110805+0000","last_undegraded":"2024-03-30T03:59:45.110805+0000","last_fullsized":"2024-03-30T03:59:45.110805+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'5","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.938244+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:00:11.938244+0000","last_peered":"2024-03-30T04:00:11.938244+0000","last_clean":"2024-03-30T04:00:11.938244+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:00:11.938244+0000","last_undegraded":"2024-03-30T04:00:11.938244+0000","last_fullsized":"2024-03-30T04:00:11.938244+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456966+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T03:59:44.456966+0000","last_peered":"2024-03-30T03:59:44.456966+0000","last_clean":"2024-03-30T03:59:44.456966+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T03:59:44.456966+0000","last_undegraded":"2024-03-30T03:59:44.456966+0000","last_fullsized":"2024-03-30T03:59:44.456966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110857+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T03:59:45.110857+0000","last_peered":"2024-03-30T03:59:45.110857+0000","last_clean":"2024-03-30T03:59:45.110857+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T03:59:45.110857+0000","last_undegraded":"2024-03-30T03:59:45.110857+0000","last_fullsized":"2024-03-30T03:59:45.110857+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.953917+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:00:11.953917+0000","last_peered":"2024-03-30T04:00:11.953917+0000","last_clean":"2024-03-30T04:00:11.953917+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:00:11.953917+0000","last_undegraded":"2024-03-30T04:00:11.953917+0000","last_fullsized":"2024-03-30T04:00:11.953917+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453466+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T03:59:44.453466+0000","last_peered":"2024-03-30T03:59:44.453466+0000","last_clean":"2024-03-30T03:59:44.453466+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T03:59:44.453466+0000","last_undegraded":"2024-03-30T03:59:44.453466+0000","last_fullsized":"2024-03-30T03:59:44.453466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110164+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T03:59:45.110164+0000","last_peered":"2024-03-30T03:59:45.110164+0000","last_clean":"2024-03-30T03:59:45.110164+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T03:59:45.110164+0000","last_undegraded":"2024-03-30T03:59:45.110164+0000","last_fullsized":"2024-03-30T03:59:45.110164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.947519+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:00:10.947519+0000","last_peered":"2024-03-30T04:00:10.947519+0000","last_clean":"2024-03-30T04:00:10.947519+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:00:10.947519+0000","last_undegraded":"2024-03-30T04:00:10.947519+0000","last_fullsized":"2024-03-30T04:00:10.947519+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.460236+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T03:59:44.460236+0000","last_peered":"2024-03-30T03:59:44.460236+0000","last_clean":"2024-03-30T03:59:44.460236+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T03:59:44.460236+0000","last_undegraded":"2024-03-30T03:59:44.460236+0000","last_fullsized":"2024-03-30T03:59:44.460236+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109813+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T03:59:45.109813+0000","last_peered":"2024-03-30T03:59:45.109813+0000","last_clean":"2024-03-30T03:59:45.109813+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T03:59:45.109813+0000","last_undegraded":"2024-03-30T03:59:45.109813+0000","last_fullsized":"2024-03-30T03:59:45.109813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.952955+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:00:11.952955+0000","last_peered":"2024-03-30T04:00:11.952955+0000","last_clean":"2024-03-30T04:00:11.952955+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:00:11.952955+0000","last_undegraded":"2024-03-30T04:00:11.952955+0000","last_fullsized":"2024-03-30T04:00:11.952955+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.448383+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T03:59:44.448383+0000","last_peered":"2024-03-30T03:59:44.448383+0000","last_clean":"2024-03-30T03:59:44.448383+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T03:59:44.448383+0000","last_undegraded":"2024-03-30T03:59:44.448383+0000","last_fullsized":"2024-03-30T03:59:44.448383+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109294+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T03:59:45.109294+0000","last_peered":"2024-03-30T03:59:45.109294+0000","last_clean":"2024-03-30T03:59:45.109294+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T03:59:45.109294+0000","last_undegraded":"2024-03-30T03:59:45.109294+0000","last_fullsized":"2024-03-30T03:59:45.109294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.940412+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:00:13.940412+0000","last_peered":"2024-03-30T04:00:13.940412+0000","last_clean":"2024-03-30T04:00:13.940412+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:00:13.940412+0000","last_undegraded":"2024-03-30T04:00:13.940412+0000","last_fullsized":"2024-03-30T04:00:13.940412+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900172+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T03:59:44.900172+0000","last_peered":"2024-03-30T03:59:44.900172+0000","last_clean":"2024-03-30T03:59:44.900172+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T03:59:44.900172+0000","last_undegraded":"2024-03-30T03:59:44.900172+0000","last_fullsized":"2024-03-30T03:59:44.900172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109929+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T03:59:45.109929+0000","last_peered":"2024-03-30T03:59:45.109929+0000","last_clean":"2024-03-30T03:59:45.109929+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T03:59:45.109929+0000","last_undegraded":"2024-03-30T03:59:45.109929+0000","last_fullsized":"2024-03-30T03:59:45.109929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.982193+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:00:09.982193+0000","last_peered":"2024-03-30T04:00:09.982193+0000","last_clean":"2024-03-30T04:00:09.982193+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:00:09.982193+0000","last_undegraded":"2024-03-30T04:00:09.982193+0000","last_fullsized":"2024-03-30T04:00:09.982193+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465522+0000","last_change":"2024-03-30T03:59:40.460524+0000","last_active":"2024-03-30T03:59:44.465522+0000","last_peered":"2024-03-30T03:59:44.465522+0000","last_clean":"2024-03-30T03:59:44.465522+0000","last_became_active":"2024-03-30T03:59:40.460206+0000","last_became_peered":"2024-03-30T03:59:40.460206+0000","last_unstale":"2024-03-30T03:59:44.465522+0000","last_undegraded":"2024-03-30T03:59:44.465522+0000","last_fullsized":"2024-03-30T03:59:44.465522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:42:50.133321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109383+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T03:59:45.109383+0000","last_peered":"2024-03-30T03:59:45.109383+0000","last_clean":"2024-03-30T03:59:45.109383+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T03:59:45.109383+0000","last_undegraded":"2024-03-30T03:59:45.109383+0000","last_fullsized":"2024-03-30T03:59:45.109383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":22,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.936238+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:00:10.936238+0000","last_peered":"2024-03-30T04:00:10.936238+0000","last_clean":"2024-03-30T04:00:10.936238+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:00:10.936238+0000","last_undegraded":"2024-03-30T04:00:10.936238+0000","last_fullsized":"2024-03-30T04:00:10.936238+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900148+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T03:59:44.900148+0000","last_peered":"2024-03-30T03:59:44.900148+0000","last_clean":"2024-03-30T03:59:44.900148+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T03:59:44.900148+0000","last_undegraded":"2024-03-30T03:59:44.900148+0000","last_fullsized":"2024-03-30T03:59:44.900148+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110347+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T03:59:45.110347+0000","last_peered":"2024-03-30T03:59:45.110347+0000","last_clean":"2024-03-30T03:59:45.110347+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T03:59:45.110347+0000","last_undegraded":"2024-03-30T03:59:45.110347+0000","last_fullsized":"2024-03-30T03:59:45.110347+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"84'17","reported_seq":44,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.965221+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:00:13.965221+0000","last_peered":"2024-03-30T04:00:13.965221+0000","last_clean":"2024-03-30T04:00:13.965221+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:00:13.965221+0000","last_undegraded":"2024-03-30T04:00:13.965221+0000","last_fullsized":"2024-03-30T04:00:13.965221+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10398,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":19,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453151+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T03:59:44.453151+0000","last_peered":"2024-03-30T03:59:44.453151+0000","last_clean":"2024-03-30T03:59:44.453151+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T03:59:44.453151+0000","last_undegraded":"2024-03-30T03:59:44.453151+0000","last_fullsized":"2024-03-30T03:59:44.453151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.478501+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T03:59:44.478501+0000","last_peered":"2024-03-30T03:59:44.478501+0000","last_clean":"2024-03-30T03:59:44.478501+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T03:59:44.478501+0000","last_undegraded":"2024-03-30T03:59:44.478501+0000","last_fullsized":"2024-03-30T03:59:44.478501+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'7","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.945620+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:00:10.945620+0000","last_peered":"2024-03-30T04:00:10.945620+0000","last_clean":"2024-03-30T04:00:10.945620+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:00:10.945620+0000","last_undegraded":"2024-03-30T04:00:10.945620+0000","last_fullsized":"2024-03-30T04:00:10.945620+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.457117+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T03:59:44.457117+0000","last_peered":"2024-03-30T03:59:44.457117+0000","last_clean":"2024-03-30T03:59:44.457117+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T03:59:44.457117+0000","last_undegraded":"2024-03-30T03:59:44.457117+0000","last_fullsized":"2024-03-30T03:59:44.457117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109816+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T03:59:45.109816+0000","last_peered":"2024-03-30T03:59:45.109816+0000","last_clean":"2024-03-30T03:59:45.109816+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T03:59:45.109816+0000","last_undegraded":"2024-03-30T03:59:45.109816+0000","last_fullsized":"2024-03-30T03:59:45.109816+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.954901+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:00:11.954901+0000","last_peered":"2024-03-30T04:00:11.954901+0000","last_clean":"2024-03-30T04:00:11.954901+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:00:11.954901+0000","last_undegraded":"2024-03-30T04:00:11.954901+0000","last_fullsized":"2024-03-30T04:00:11.954901+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453922+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T03:59:44.453922+0000","last_peered":"2024-03-30T03:59:44.453922+0000","last_clean":"2024-03-30T03:59:44.453922+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T03:59:44.453922+0000","last_undegraded":"2024-03-30T03:59:44.453922+0000","last_fullsized":"2024-03-30T03:59:44.453922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.111292+0000","last_change":"2024-03-30T03:59:44.478370+0000","last_active":"2024-03-30T03:59:45.111292+0000","last_peered":"2024-03-30T03:59:45.111292+0000","last_clean":"2024-03-30T03:59:45.111292+0000","last_became_active":"2024-03-30T03:59:44.477981+0000","last_became_peered":"2024-03-30T03:59:44.477981+0000","last_unstale":"2024-03-30T03:59:45.111292+0000","last_undegraded":"2024-03-30T03:59:45.111292+0000","last_fullsized":"2024-03-30T03:59:45.111292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:47:43.348222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":26,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.969680+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:00:13.969680+0000","last_peered":"2024-03-30T04:00:13.969680+0000","last_clean":"2024-03-30T04:00:13.969680+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:00:13.969680+0000","last_undegraded":"2024-03-30T04:00:13.969680+0000","last_fullsized":"2024-03-30T04:00:13.969680+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453863+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T03:59:44.453863+0000","last_peered":"2024-03-30T03:59:44.453863+0000","last_clean":"2024-03-30T03:59:44.453863+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T03:59:44.453863+0000","last_undegraded":"2024-03-30T03:59:44.453863+0000","last_fullsized":"2024-03-30T03:59:44.453863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.476461+0000","last_change":"2024-03-30T03:59:44.473996+0000","last_active":"2024-03-30T03:59:44.476461+0000","last_peered":"2024-03-30T03:59:44.476461+0000","last_clean":"2024-03-30T03:59:44.476461+0000","last_became_active":"2024-03-30T03:59:44.473604+0000","last_became_peered":"2024-03-30T03:59:44.473604+0000","last_unstale":"2024-03-30T03:59:44.476461+0000","last_undegraded":"2024-03-30T03:59:44.476461+0000","last_fullsized":"2024-03-30T03:59:44.476461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:10:01.350249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":34,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.961845+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:00:12.961845+0000","last_peered":"2024-03-30T04:00:12.961845+0000","last_clean":"2024-03-30T04:00:12.961845+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:00:12.961845+0000","last_undegraded":"2024-03-30T04:00:12.961845+0000","last_fullsized":"2024-03-30T04:00:12.961845+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455843+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T03:59:44.455843+0000","last_peered":"2024-03-30T03:59:44.455843+0000","last_clean":"2024-03-30T03:59:44.455843+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T03:59:44.455843+0000","last_undegraded":"2024-03-30T03:59:44.455843+0000","last_fullsized":"2024-03-30T03:59:44.455843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.472560+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T03:59:44.472560+0000","last_peered":"2024-03-30T03:59:44.472560+0000","last_clean":"2024-03-30T03:59:44.472560+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T03:59:44.472560+0000","last_undegraded":"2024-03-30T03:59:44.472560+0000","last_fullsized":"2024-03-30T03:59:44.472560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.934050+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:00:10.934050+0000","last_peered":"2024-03-30T04:00:10.934050+0000","last_clean":"2024-03-30T04:00:10.934050+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:00:10.934050+0000","last_undegraded":"2024-03-30T04:00:10.934050+0000","last_fullsized":"2024-03-30T04:00:10.934050+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453675+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T03:59:44.453675+0000","last_peered":"2024-03-30T03:59:44.453675+0000","last_clean":"2024-03-30T03:59:44.453675+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T03:59:44.453675+0000","last_undegraded":"2024-03-30T03:59:44.453675+0000","last_fullsized":"2024-03-30T03:59:44.453675+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109700+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T03:59:45.109700+0000","last_peered":"2024-03-30T03:59:45.109700+0000","last_clean":"2024-03-30T03:59:45.109700+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T03:59:45.109700+0000","last_undegraded":"2024-03-30T03:59:45.109700+0000","last_fullsized":"2024-03-30T03:59:45.109700+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110841+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T03:59:45.110841+0000","last_peered":"2024-03-30T03:59:45.110841+0000","last_clean":"2024-03-30T03:59:45.110841+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T03:59:45.110841+0000","last_undegraded":"2024-03-30T03:59:45.110841+0000","last_fullsized":"2024-03-30T03:59:45.110841+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597510+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T03:59:44.597510+0000","last_peered":"2024-03-30T03:59:44.597510+0000","last_clean":"2024-03-30T03:59:44.597510+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T03:59:44.597510+0000","last_undegraded":"2024-03-30T03:59:44.597510+0000","last_fullsized":"2024-03-30T03:59:44.597510+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.476850+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T03:59:44.476850+0000","last_peered":"2024-03-30T03:59:44.476850+0000","last_clean":"2024-03-30T03:59:44.476850+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T03:59:44.476850+0000","last_undegraded":"2024-03-30T03:59:44.476850+0000","last_fullsized":"2024-03-30T03:59:44.476850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494910+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T03:59:45.494910+0000","last_peered":"2024-03-30T03:59:45.494910+0000","last_clean":"2024-03-30T03:59:45.494910+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T03:59:45.494910+0000","last_undegraded":"2024-03-30T03:59:45.494910+0000","last_fullsized":"2024-03-30T03:59:45.494910+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.597350+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T03:59:44.597350+0000","last_peered":"2024-03-30T03:59:44.597350+0000","last_clean":"2024-03-30T03:59:44.597350+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T03:59:44.597350+0000","last_undegraded":"2024-03-30T03:59:44.597350+0000","last_fullsized":"2024-03-30T03:59:44.597350+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.599769+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T03:59:44.599769+0000","last_peered":"2024-03-30T03:59:44.599769+0000","last_clean":"2024-03-30T03:59:44.599769+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T03:59:44.599769+0000","last_undegraded":"2024-03-30T03:59:44.599769+0000","last_fullsized":"2024-03-30T03:59:44.599769+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456481+0000","last_change":"2024-03-30T03:59:40.456721+0000","last_active":"2024-03-30T03:59:44.456481+0000","last_peered":"2024-03-30T03:59:44.456481+0000","last_clean":"2024-03-30T03:59:44.456481+0000","last_became_active":"2024-03-30T03:59:40.456324+0000","last_became_peered":"2024-03-30T03:59:40.456324+0000","last_unstale":"2024-03-30T03:59:44.456481+0000","last_undegraded":"2024-03-30T03:59:44.456481+0000","last_fullsized":"2024-03-30T03:59:44.456481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:16:51.772984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.947642+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:00:10.947642+0000","last_peered":"2024-03-30T04:00:10.947642+0000","last_clean":"2024-03-30T04:00:10.947642+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:00:10.947642+0000","last_undegraded":"2024-03-30T04:00:10.947642+0000","last_fullsized":"2024-03-30T04:00:10.947642+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494933+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T03:59:45.494933+0000","last_peered":"2024-03-30T03:59:45.494933+0000","last_clean":"2024-03-30T03:59:45.494933+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T03:59:45.494933+0000","last_undegraded":"2024-03-30T03:59:45.494933+0000","last_fullsized":"2024-03-30T03:59:45.494933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456883+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T03:59:44.456883+0000","last_peered":"2024-03-30T03:59:44.456883+0000","last_clean":"2024-03-30T03:59:44.456883+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T03:59:44.456883+0000","last_undegraded":"2024-03-30T03:59:44.456883+0000","last_fullsized":"2024-03-30T03:59:44.456883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":25,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:13.965576+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:00:13.965576+0000","last_peered":"2024-03-30T04:00:13.965576+0000","last_clean":"2024-03-30T04:00:13.965576+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:00:13.965576+0000","last_undegraded":"2024-03-30T04:00:13.965576+0000","last_fullsized":"2024-03-30T04:00:13.965576+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":335,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.494572+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T03:59:45.494572+0000","last_peered":"2024-03-30T03:59:45.494572+0000","last_clean":"2024-03-30T03:59:45.494572+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T03:59:45.494572+0000","last_undegraded":"2024-03-30T03:59:45.494572+0000","last_fullsized":"2024-03-30T03:59:45.494572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.599891+0000","last_change":"2024-03-30T03:59:44.598909+0000","last_active":"2024-03-30T03:59:44.599891+0000","last_peered":"2024-03-30T03:59:44.599891+0000","last_clean":"2024-03-30T03:59:44.599891+0000","last_became_active":"2024-03-30T03:59:44.598427+0000","last_became_peered":"2024-03-30T03:59:44.598427+0000","last_unstale":"2024-03-30T03:59:44.599891+0000","last_undegraded":"2024-03-30T03:59:44.599891+0000","last_fullsized":"2024-03-30T03:59:44.599891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:38:01.401346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.455404+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T03:59:44.455404+0000","last_peered":"2024-03-30T03:59:44.455404+0000","last_clean":"2024-03-30T03:59:44.455404+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T03:59:44.455404+0000","last_undegraded":"2024-03-30T03:59:44.455404+0000","last_fullsized":"2024-03-30T03:59:44.455404+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'3","reported_seq":19,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:10.945435+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:00:10.945435+0000","last_peered":"2024-03-30T04:00:10.945435+0000","last_clean":"2024-03-30T04:00:10.945435+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:00:10.945435+0000","last_undegraded":"2024-03-30T04:00:10.945435+0000","last_fullsized":"2024-03-30T04:00:10.945435+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109325+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T03:59:45.109325+0000","last_peered":"2024-03-30T03:59:45.109325+0000","last_clean":"2024-03-30T03:59:45.109325+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T03:59:45.109325+0000","last_undegraded":"2024-03-30T03:59:45.109325+0000","last_fullsized":"2024-03-30T03:59:45.109325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.955404+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:00:11.955404+0000","last_peered":"2024-03-30T04:00:11.955404+0000","last_clean":"2024-03-30T04:00:11.955404+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:00:11.955404+0000","last_undegraded":"2024-03-30T04:00:11.955404+0000","last_fullsized":"2024-03-30T04:00:11.955404+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465339+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T03:59:44.465339+0000","last_peered":"2024-03-30T03:59:44.465339+0000","last_clean":"2024-03-30T03:59:44.465339+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T03:59:44.465339+0000","last_undegraded":"2024-03-30T03:59:44.465339+0000","last_fullsized":"2024-03-30T03:59:44.465339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109224+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T03:59:45.109224+0000","last_peered":"2024-03-30T03:59:45.109224+0000","last_clean":"2024-03-30T03:59:45.109224+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T03:59:45.109224+0000","last_undegraded":"2024-03-30T03:59:45.109224+0000","last_fullsized":"2024-03-30T03:59:45.109224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.956084+0000","last_change":"2024-03-30T03:59:39.932264+0000","last_active":"2024-03-30T04:00:11.956084+0000","last_peered":"2024-03-30T04:00:11.956084+0000","last_clean":"2024-03-30T04:00:11.956084+0000","last_became_active":"2024-03-30T03:59:39.931859+0000","last_became_peered":"2024-03-30T03:59:39.931859+0000","last_unstale":"2024-03-30T04:00:11.956084+0000","last_undegraded":"2024-03-30T04:00:11.956084+0000","last_fullsized":"2024-03-30T04:00:11.956084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:31:24.215248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465468+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T03:59:44.465468+0000","last_peered":"2024-03-30T03:59:44.465468+0000","last_clean":"2024-03-30T03:59:44.465468+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T03:59:44.465468+0000","last_undegraded":"2024-03-30T03:59:44.465468+0000","last_fullsized":"2024-03-30T03:59:44.465468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109009+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T03:59:45.109009+0000","last_peered":"2024-03-30T03:59:45.109009+0000","last_clean":"2024-03-30T03:59:45.109009+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T03:59:45.109009+0000","last_undegraded":"2024-03-30T03:59:45.109009+0000","last_fullsized":"2024-03-30T03:59:45.109009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":24,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.962495+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:00:12.962495+0000","last_peered":"2024-03-30T04:00:12.962495+0000","last_clean":"2024-03-30T04:00:12.962495+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:00:12.962495+0000","last_undegraded":"2024-03-30T04:00:12.962495+0000","last_fullsized":"2024-03-30T04:00:12.962495+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465577+0000","last_change":"2024-03-30T03:59:40.464994+0000","last_active":"2024-03-30T03:59:44.465577+0000","last_peered":"2024-03-30T03:59:44.465577+0000","last_clean":"2024-03-30T03:59:44.465577+0000","last_became_active":"2024-03-30T03:59:40.462862+0000","last_became_peered":"2024-03-30T03:59:40.462862+0000","last_unstale":"2024-03-30T03:59:44.465577+0000","last_undegraded":"2024-03-30T03:59:44.465577+0000","last_fullsized":"2024-03-30T03:59:44.465577+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:29:48.413081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110065+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T03:59:45.110065+0000","last_peered":"2024-03-30T03:59:45.110065+0000","last_clean":"2024-03-30T03:59:45.110065+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T03:59:45.110065+0000","last_undegraded":"2024-03-30T03:59:45.110065+0000","last_fullsized":"2024-03-30T03:59:45.110065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":18,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.982804+0000","last_change":"2024-03-30T03:59:39.928250+0000","last_active":"2024-03-30T04:00:09.982804+0000","last_peered":"2024-03-30T04:00:09.982804+0000","last_clean":"2024-03-30T04:00:09.982804+0000","last_became_active":"2024-03-30T03:59:39.927915+0000","last_became_peered":"2024-03-30T03:59:39.927915+0000","last_unstale":"2024-03-30T04:00:09.982804+0000","last_undegraded":"2024-03-30T04:00:09.982804+0000","last_fullsized":"2024-03-30T04:00:09.982804+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:02:47.307714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.452764+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T03:59:44.452764+0000","last_peered":"2024-03-30T03:59:44.452764+0000","last_clean":"2024-03-30T03:59:44.452764+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T03:59:44.452764+0000","last_undegraded":"2024-03-30T03:59:44.452764+0000","last_fullsized":"2024-03-30T03:59:44.452764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.111338+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T03:59:45.111338+0000","last_peered":"2024-03-30T03:59:45.111338+0000","last_clean":"2024-03-30T03:59:45.111338+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T03:59:45.111338+0000","last_undegraded":"2024-03-30T03:59:45.111338+0000","last_fullsized":"2024-03-30T03:59:45.111338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":23,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.964240+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:00:12.964240+0000","last_peered":"2024-03-30T04:00:12.964240+0000","last_clean":"2024-03-30T04:00:12.964240+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:00:12.964240+0000","last_undegraded":"2024-03-30T04:00:12.964240+0000","last_fullsized":"2024-03-30T04:00:12.964240+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453800+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T03:59:44.453800+0000","last_peered":"2024-03-30T03:59:44.453800+0000","last_clean":"2024-03-30T03:59:44.453800+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T03:59:44.453800+0000","last_undegraded":"2024-03-30T03:59:44.453800+0000","last_fullsized":"2024-03-30T03:59:44.453800+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":13,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.900306+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T03:59:45.900306+0000","last_peered":"2024-03-30T03:59:45.900306+0000","last_clean":"2024-03-30T03:59:45.900306+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T03:59:45.900306+0000","last_undegraded":"2024-03-30T03:59:45.900306+0000","last_fullsized":"2024-03-30T03:59:45.900306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'1","reported_seq":17,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110168+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T03:59:45.110168+0000","last_peered":"2024-03-30T03:59:45.110168+0000","last_clean":"2024-03-30T03:59:45.110168+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T03:59:45.110168+0000","last_undegraded":"2024-03-30T03:59:45.110168+0000","last_fullsized":"2024-03-30T03:59:45.110168+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.464621+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T03:59:44.464621+0000","last_peered":"2024-03-30T03:59:44.464621+0000","last_clean":"2024-03-30T03:59:44.464621+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T03:59:44.464621+0000","last_undegraded":"2024-03-30T03:59:44.464621+0000","last_fullsized":"2024-03-30T03:59:44.464621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.110097+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T03:59:45.110097+0000","last_peered":"2024-03-30T03:59:45.110097+0000","last_clean":"2024-03-30T03:59:45.110097+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T03:59:45.110097+0000","last_undegraded":"2024-03-30T03:59:45.110097+0000","last_fullsized":"2024-03-30T03:59:45.110097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:09.981124+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:00:09.981124+0000","last_peered":"2024-03-30T04:00:09.981124+0000","last_clean":"2024-03-30T04:00:09.981124+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:00:09.981124+0000","last_undegraded":"2024-03-30T04:00:09.981124+0000","last_fullsized":"2024-03-30T04:00:09.981124+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454242+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T03:59:44.454242+0000","last_peered":"2024-03-30T03:59:44.454242+0000","last_clean":"2024-03-30T03:59:44.454242+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T03:59:44.454242+0000","last_undegraded":"2024-03-30T03:59:44.454242+0000","last_fullsized":"2024-03-30T03:59:44.454242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900088+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T03:59:44.900088+0000","last_peered":"2024-03-30T03:59:44.900088+0000","last_clean":"2024-03-30T03:59:44.900088+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T03:59:44.900088+0000","last_undegraded":"2024-03-30T03:59:44.900088+0000","last_fullsized":"2024-03-30T03:59:44.900088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.953027+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:00:11.953027+0000","last_peered":"2024-03-30T04:00:11.953027+0000","last_clean":"2024-03-30T04:00:11.953027+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:00:11.953027+0000","last_undegraded":"2024-03-30T04:00:11.953027+0000","last_fullsized":"2024-03-30T04:00:11.953027+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454411+0000","last_change":"2024-03-30T03:59:40.462459+0000","last_active":"2024-03-30T03:59:44.454411+0000","last_peered":"2024-03-30T03:59:44.454411+0000","last_clean":"2024-03-30T03:59:44.454411+0000","last_became_active":"2024-03-30T03:59:40.461692+0000","last_became_peered":"2024-03-30T03:59:40.461692+0000","last_unstale":"2024-03-30T03:59:44.454411+0000","last_undegraded":"2024-03-30T03:59:44.454411+0000","last_fullsized":"2024-03-30T03:59:44.454411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:46.872671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T03:59:45.109492+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T03:59:45.109492+0000","last_peered":"2024-03-30T03:59:45.109492+0000","last_clean":"2024-03-30T03:59:45.109492+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T03:59:45.109492+0000","last_undegraded":"2024-03-30T03:59:45.109492+0000","last_fullsized":"2024-03-30T03:59:45.109492+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":20,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:11.955007+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:00:11.955007+0000","last_peered":"2024-03-30T04:00:11.955007+0000","last_clean":"2024-03-30T04:00:11.955007+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:00:11.955007+0000","last_undegraded":"2024-03-30T04:00:11.955007+0000","last_fullsized":"2024-03-30T04:00:11.955007+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453799+0000","last_change":"2024-03-30T03:59:40.467087+0000","last_active":"2024-03-30T03:59:44.453799+0000","last_peered":"2024-03-30T03:59:44.453799+0000","last_clean":"2024-03-30T03:59:44.453799+0000","last_became_active":"2024-03-30T03:59:40.466769+0000","last_became_peered":"2024-03-30T03:59:40.466769+0000","last_unstale":"2024-03-30T03:59:44.453799+0000","last_undegraded":"2024-03-30T03:59:44.453799+0000","last_fullsized":"2024-03-30T03:59:44.453799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:25:41.741807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.463545+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T03:59:44.463545+0000","last_peered":"2024-03-30T03:59:44.463545+0000","last_clean":"2024-03-30T03:59:44.463545+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T03:59:44.463545+0000","last_undegraded":"2024-03-30T03:59:44.463545+0000","last_fullsized":"2024-03-30T03:59:44.463545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":21,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-30T04:00:12.962772+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:00:12.962772+0000","last_peered":"2024-03-30T04:00:12.962772+0000","last_clean":"2024-03-30T04:00:12.962772+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:00:12.962772+0000","last_undegraded":"2024-03-30T04:00:12.962772+0000","last_fullsized":"2024-03-30T04:00:12.962772+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454355+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T03:59:44.454355+0000","last_peered":"2024-03-30T03:59:44.454355+0000","last_clean":"2024-03-30T03:59:44.454355+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T03:59:44.454355+0000","last_undegraded":"2024-03-30T03:59:44.454355+0000","last_fullsized":"2024-03-30T03:59:44.454355+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453905+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T03:59:44.453905+0000","last_peered":"2024-03-30T03:59:44.453905+0000","last_clean":"2024-03-30T03:59:44.453905+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T03:59:44.453905+0000","last_undegraded":"2024-03-30T03:59:44.453905+0000","last_fullsized":"2024-03-30T03:59:44.453905+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.450026+0000","last_change":"2024-03-30T03:59:40.458059+0000","last_active":"2024-03-30T03:59:44.450026+0000","last_peered":"2024-03-30T03:59:44.450026+0000","last_clean":"2024-03-30T03:59:44.450026+0000","last_became_active":"2024-03-30T03:59:40.457738+0000","last_became_peered":"2024-03-30T03:59:40.457738+0000","last_unstale":"2024-03-30T03:59:44.450026+0000","last_undegraded":"2024-03-30T03:59:44.450026+0000","last_fullsized":"2024-03-30T03:59:44.450026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:21.899370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449155+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T03:59:44.449155+0000","last_peered":"2024-03-30T03:59:44.449155+0000","last_clean":"2024-03-30T03:59:44.449155+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T03:59:44.449155+0000","last_undegraded":"2024-03-30T03:59:44.449155+0000","last_fullsized":"2024-03-30T03:59:44.449155+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.458407+0000","last_change":"2024-03-30T03:59:40.454510+0000","last_active":"2024-03-30T03:59:44.458407+0000","last_peered":"2024-03-30T03:59:44.458407+0000","last_clean":"2024-03-30T03:59:44.458407+0000","last_became_active":"2024-03-30T03:59:40.454119+0000","last_became_peered":"2024-03-30T03:59:40.454119+0000","last_unstale":"2024-03-30T03:59:44.458407+0000","last_undegraded":"2024-03-30T03:59:44.458407+0000","last_fullsized":"2024-03-30T03:59:44.458407+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:50:00.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.465252+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T03:59:44.465252+0000","last_peered":"2024-03-30T03:59:44.465252+0000","last_clean":"2024-03-30T03:59:44.465252+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T03:59:44.465252+0000","last_undegraded":"2024-03-30T03:59:44.465252+0000","last_fullsized":"2024-03-30T03:59:44.465252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.453459+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T03:59:44.453459+0000","last_peered":"2024-03-30T03:59:44.453459+0000","last_clean":"2024-03-30T03:59:44.453459+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T03:59:44.453459+0000","last_undegraded":"2024-03-30T03:59:44.453459+0000","last_fullsized":"2024-03-30T03:59:44.453459+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.449683+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T03:59:44.449683+0000","last_peered":"2024-03-30T03:59:44.449683+0000","last_clean":"2024-03-30T03:59:44.449683+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T03:59:44.449683+0000","last_undegraded":"2024-03-30T03:59:44.449683+0000","last_fullsized":"2024-03-30T03:59:44.449683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.469792+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T03:59:44.469792+0000","last_peered":"2024-03-30T03:59:44.469792+0000","last_clean":"2024-03-30T03:59:44.469792+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T03:59:44.469792+0000","last_undegraded":"2024-03-30T03:59:44.469792+0000","last_fullsized":"2024-03-30T03:59:44.469792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456383+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T03:59:44.456383+0000","last_peered":"2024-03-30T03:59:44.456383+0000","last_clean":"2024-03-30T03:59:44.456383+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T03:59:44.456383+0000","last_undegraded":"2024-03-30T03:59:44.456383+0000","last_fullsized":"2024-03-30T03:59:44.456383+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.460984+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T03:59:44.460984+0000","last_peered":"2024-03-30T03:59:44.460984+0000","last_clean":"2024-03-30T03:59:44.460984+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T03:59:44.460984+0000","last_undegraded":"2024-03-30T03:59:44.460984+0000","last_fullsized":"2024-03-30T03:59:44.460984+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.900110+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T03:59:44.900110+0000","last_peered":"2024-03-30T03:59:44.900110+0000","last_clean":"2024-03-30T03:59:44.900110+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T03:59:44.900110+0000","last_undegraded":"2024-03-30T03:59:44.900110+0000","last_fullsized":"2024-03-30T03:59:44.900110+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.456058+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T03:59:44.456058+0000","last_peered":"2024-03-30T03:59:44.456058+0000","last_clean":"2024-03-30T03:59:44.456058+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T03:59:44.456058+0000","last_undegraded":"2024-03-30T03:59:44.456058+0000","last_fullsized":"2024-03-30T03:59:44.456058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-03-30T03:59:44.454093+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T03:59:44.454093+0000","last_peered":"2024-03-30T03:59:44.454093+0000","last_clean":"2024-03-30T03:59:44.454093+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T03:59:44.454093+0000","last_undegraded":"2024-03-30T03:59:44.454093+0000","last_fullsized":"2024-03-30T03:59:44.454093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":16481,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":46,"num_read_kb":40,"num_write":200,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":56418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175,"ondisk_log_size":175,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547217,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32028,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713124,"statfs":{"total":95995035648,"available":95962238976,"internally_reserved":0,"allocated":937984,"data_stored":451119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.59399999999999997}]},{"osd":1,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48999999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.9340000000000002,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.66900000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.68799999999999994}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.71999999999999997}]},{"osd":10,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.73699999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776148,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1585152,"data_stored":1052822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44400000000000001}]},{"osd":2,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47899999999999998}]},{"osd":3,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68100000000000005}]},{"osd":4,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.52200000000000002}]},{"osd":5,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.71699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":8,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.66700000000000004}]},{"osd":9,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.70799999999999996}]},{"osd":11,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.74199999999999999}]}]},{"osd":9,"up_from":63,"seq":270582939673,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":970752,"data_stored":459289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.54900000000000004}]},{"osd":1,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.621}]},{"osd":2,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.66800000000000004}]},{"osd":3,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.294,"5min":1.294,"15min":1.294},"last":0.69499999999999995}]},{"osd":4,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.872,"15min":0.872},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4179999999999999,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.74399999999999999}]},{"osd":5,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":2.4860000000000002,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.47,"5min":1.47,"15min":1.47},"last":0.83299999999999996}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.03,"15min":1.03},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2.2240000000000002,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.72699999999999998}]},{"osd":8,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.86699999999999999}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.331,"5min":1.331,"15min":1.331},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.84299999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168605,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":958464,"data_stored":451475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70599999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.50900000000000001}]},{"osd":2,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.432}]},{"osd":3,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.71999999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.40899999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.82099999999999995}]},{"osd":9,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.80800000000000005}]},{"osd":10,"last update":"Sat Mar 30 03:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.747}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":7,"up_from":49,"seq":210453397534,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1042039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.372,"15min":0.372},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54800000000000004}]},{"osd":3,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65700000000000003}]},{"osd":4,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61199999999999999,"15min":0.60799999999999998},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.625}]},{"osd":6,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86699999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.64200000000000002}]},{"osd":8,"last update":"Sat Mar 30 04:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.48299999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.82799999999999996}]},{"osd":9,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78700000000000003}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.73099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":6,"up_from":44,"seq":188978561059,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":450087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.36499999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94099999999999995,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.629,"15min":0.625},"min":{"1min":0.47099999999999997,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89300000000000002,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.66800000000000004,"15min":0.66200000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95399999999999996,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.73699999999999999}]},{"osd":2,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68100000000000005,"15min":0.68300000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0289999999999999,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.67700000000000005,"15min":0.67100000000000004},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0800000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.76800000000000002}]},{"osd":3,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.61399999999999999,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.92900000000000005,"5min":0.995,"15min":0.995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.69599999999999995}]},{"osd":4,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.51300000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90400000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.746},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.748,"15min":0.74199999999999999},"min":{"1min":0.59499999999999997,"5min":0.438,"15min":0.438},"max":{"1min":1.046,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":5,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.83999999999999997,"15min":0.84499999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.121,"5min":1.462,"15min":1.462},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.83499999999999996,"15min":0.83999999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95399999999999996,"5min":1.472,"15min":1.472},"last":0.80600000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.96399999999999997}]},{"osd":8,"last update":"Sat Mar 30 04:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.85999999999999999,"15min":0.85099999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.88800000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:30 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.79500000000000004}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.97499999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36144,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93709008,"statfs":{"total":95995035648,"available":95958024192,"internally_reserved":0,"allocated":958464,"data_stored":462470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54000000000000004,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.84999999999999998}]},{"osd":2,"last update":"Sat Mar 30 03:59:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.48499999999999999,"15min":0.45900000000000002},"min":{"1min":0.32900000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.51100000000000001,"15min":0.48699999999999999},"min":{"1min":0.35199999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.52000000000000002,"15min":0.49199999999999999},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.79300000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.56000000000000005,"15min":0.52700000000000002},"min":{"1min":0.45200000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.66400000000000003}]},{"osd":4,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.32500000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.374},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.34300000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:35 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66400000000000003,"15min":0.65600000000000003},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64100000000000001,"15min":0.63700000000000001},"min":{"1min":0.47999999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.82099999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.74199999999999999}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.44900000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84699999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.55200000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.71999999999999997}]},{"osd":9,"last update":"Sat Mar 30 03:59:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.78600000000000003}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.626}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":451453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.81000000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71799999999999997}]},{"osd":2,"last update":"Sat Mar 30 03:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.39900000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.504},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.59299999999999997,"15min":0.57299999999999995},"min":{"1min":0.44,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65700000000000003}]},{"osd":3,"last update":"Sat Mar 30 03:58:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48199999999999998},"min":{"1min":0.35499999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.91100000000000003,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.38300000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Sat Mar 30 03:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85399999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68899999999999995,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90500000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.92500000000000004}]},{"osd":5,"last update":"Sat Mar 30 03:59:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.68100000000000005,"15min":0.66200000000000003},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.67700000000000005,"15min":0.66100000000000003},"min":{"1min":0.57599999999999996,"5min":0.497,"15min":0.497},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Mar 30 03:59:37 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.50800000000000001,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.73199999999999998,"15min":0.71499999999999997},"min":{"1min":0.505,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.84999999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.82699999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.68999999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.1180000000000001,"5min":1.333,"15min":1.333},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.67500000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.038,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.76500000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.97299999999999998}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.75}]},{"osd":11,"last update":"Sat Mar 30 04:00:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.78100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":455073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56399999999999995,"15min":0.56000000000000005},"min":{"1min":0.36699999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84099999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.57999999999999996},"min":{"1min":0.35699999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.79600000000000004,"5min":1.133,"15min":1.133},"last":0.46800000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.41499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.66200000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.51400000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40600000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.75,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38700000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.54500000000000004}]},{"osd":4,"last update":"Sat Mar 30 03:58:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.751},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70199999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.751,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.82399999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67800000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.79000000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66600000000000004}]},{"osd":9,"last update":"Sat Mar 30 03:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73399999999999999}]},{"osd":10,"last update":"Sat Mar 30 03:59:44 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.83399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182446,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":450405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.42899999999999999,"15min":0.40400000000000003},"min":{"1min":0.376,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.45800000000000002,"15min":0.42999999999999999},"min":{"1min":0.38,"5min":0.312,"15min":0.312},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.879,"5min":1.762,"15min":1.762},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.64900000000000002,"15min":0.60599999999999998},"min":{"1min":0.58699999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.0409999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.58499999999999996,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90300000000000002,"5min":1.181,"15min":1.181},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63800000000000001,"15min":0.623},"min":{"1min":0.443,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.89200000000000002,"5min":1.296,"15min":1.296},"last":0.59499999999999997}]},{"osd":4,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88200000000000001,"5min":1.113,"15min":1.113},"last":0.501},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.70599999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.591,"15min":1.591},"last":0.70799999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70299999999999996,"15min":0.66500000000000004},"min":{"1min":0.61099999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93799999999999994,"5min":1.139,"15min":1.139},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.77000000000000002,"15min":0.72999999999999998},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0569999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.75700000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.77200000000000002,"15min":0.754},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.87,"5min":0.745,"15min":0.72399999999999998},"min":{"1min":0.59699999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.47599999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.214,"15min":1.214},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1499999999999999,"5min":1.242,"15min":1.242},"last":0.84899999999999998}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 03:59:30 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.51700000000000002}]},{"osd":10,"last update":"Sat Mar 30 03:59:47 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.88800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32036,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713116,"statfs":{"total":95995035648,"available":95962230784,"internally_reserved":0,"allocated":946176,"data_stored":451020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59999999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.307,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.34200000000000003}]},{"osd":2,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.52700000000000002},"min":{"1min":0.46000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53100000000000003}]},{"osd":3,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83699999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Sat Mar 30 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.51400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.88700000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.748}]},{"osd":6,"last update":"Sat Mar 30 03:59:31 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.47199999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.65200000000000002,"15min":0.63900000000000001},"min":{"1min":0.51100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.56899999999999995}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.44},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.45300000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93999999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.51900000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90000000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.70899999999999996}]},{"osd":9,"last update":"Sat Mar 30 03:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.59699999999999998}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.58399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]},{"osd":5,"up_from":37,"seq":158913789991,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32636,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712516,"statfs":{"total":95995035648,"available":95961616384,"internally_reserved":0,"allocated":1560576,"data_stored":1042784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.751,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.45300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.42699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.495},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.246,"15min":0.246},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":0.873},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Mar 30 03:59:32 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.627,"15min":0.627},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82399999999999995,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.622,"15min":0.625},"min":{"1min":0.42299999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78200000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.76300000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.80800000000000005}]},{"osd":8,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.77900000000000003}]},{"osd":9,"last update":"Sat Mar 30 03:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.83299999999999996}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":25194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":12479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:00:18.034 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-30T04:00:18.034 INFO:tasks.cephadm:Setting up client nodes... 2024-03-30T04:00:18.035 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-30T04:00:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T04:00:18.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:17 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3405781622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-30T04:00:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-30T04:00:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-30T04:00:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-30T04:00:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T04:00:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:17 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3405781622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:18.437 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [2, 11]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-30T04:00:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2024-03-30T04:00:18.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 10]}]': finished 2024-03-30T04:00:18.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 8]}]': finished 2024-03-30T04:00:18.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: osdmap e85: 12 total, 12 up, 12 in 2024-03-30T04:00:18.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:17 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3405781622' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:19 smithi023 ceph-mon[38254]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:19 smithi023 ceph-mon[38254]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:19 smithi023 ceph-mon[38254]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T04:00:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:19 smithi023 ceph-mon[38254]: pgmap v257: 129 pgs: 3 peering, 126 active+clean; 594 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.4 KiB/s wr, 27 op/s 2024-03-30T04:00:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:19 smithi154 ceph-mon[79161]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:19 smithi154 ceph-mon[79161]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:19 smithi154 ceph-mon[79161]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T04:00:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:19 smithi154 ceph-mon[79161]: pgmap v257: 129 pgs: 3 peering, 126 active+clean; 594 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.4 KiB/s wr, 27 op/s 2024-03-30T04:00:19.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:19 smithi193 ceph-mon[79581]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:19 smithi193 ceph-mon[79581]: from='client.14856 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:19.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:19 smithi193 ceph-mon[79581]: osdmap e86: 12 total, 12 up, 12 in 2024-03-30T04:00:19.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:19 smithi193 ceph-mon[79581]: pgmap v257: 129 pgs: 3 peering, 126 active+clean; 594 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 7.4 KiB/s wr, 27 op/s 2024-03-30T04:00:19.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 -- 172.21.15.23:0/3652100034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 msgr2=0x7efee0109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 --2- 172.21.15.23:0/3652100034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee0109210 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efed800b0d0 tx=0x7efed800b3e0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 -- 172.21.15.23:0/3652100034 shutdown_connections 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 --2- 172.21.15.23:0/3652100034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee0109210 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 --2- 172.21.15.23:0/3652100034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0100dc0 0x7efee01011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 --2- 172.21.15.23:0/3652100034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0101fc0 0x7efee0102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 -- 172.21.15.23:0/3652100034 >> 172.21.15.23:0/3652100034 conn(0x7efee00fc360 msgr2=0x7efee00fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.636+0000 7efee7000700 1 -- 172.21.15.23:0/3652100034 shutdown_connections 2024-03-30T04:00:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.637+0000 7efee7000700 1 -- 172.21.15.23:0/3652100034 wait complete. 2024-03-30T04:00:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.638+0000 7efee7000700 1 Processor -- start 2024-03-30T04:00:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.638+0000 7efee7000700 1 -- start start 2024-03-30T04:00:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0101fc0 0x7efee01972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee019e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:19.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee019e8f0 con 0x7efee0102960 2024-03-30T04:00:19.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee019ea30 con 0x7efee0101fc0 2024-03-30T04:00:19.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee7000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee019ebc0 con 0x7efee0100dc0 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efedfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efedfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33686/0 (socket says 172.21.15.23:33686) 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efedfe00700 1 -- 172.21.15.23:0/2724657616 learned_addr learned my addr 172.21.15.23:0/2724657616 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efedf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0101fc0 0x7efee01972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.639+0000 7efee5000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee019e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 -- 172.21.15.23:0/2724657616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0101fc0 msgr2=0x7efee01972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0101fc0 0x7efee01972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 -- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 msgr2=0x7efee019e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee019e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 -- 172.21.15.23:0/2724657616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed8009d00 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efee5000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee019e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedfe00700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efed000e9a0 tx=0x7efed000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0009790 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.640+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee019eea0 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.641+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efed000c810 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.641+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee019f2c0 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.641+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0003be0 con 0x7efee0100dc0 2024-03-30T04:00:19.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.642+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee01081a0 con 0x7efee0100dc0 2024-03-30T04:00:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.644+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7efed0003d40 con 0x7efee0100dc0 2024-03-30T04:00:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.645+0000 7efedcc00700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 0x7efec40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.645+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efed0014070 con 0x7efee0100dc0 2024-03-30T04:00:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.645+0000 7efedf400700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 0x7efec40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.646+0000 7efedf400700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 0x7efec40a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efecc009c60 tx=0x7efecc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.647+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efed0096980 con 0x7efee0100dc0 2024-03-30T04:00:19.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.944+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7efee004ea80 con 0x7efee0100dc0 2024-03-30T04:00:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.951+0000 7efedcc00700 1 -- 172.21.15.23:0/2724657616 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7efed0099fa0 con 0x7efee0100dc0 2024-03-30T04:00:19.950 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-30T04:00:19.950 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQBTjgdmNYtuOBAAmgINlwmZ0m7hMtaET/ciiA== 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 msgr2=0x7efec40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 0x7efec40a3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efecc009c60 tx=0x7efecc009260 comp rx=0 tx=0).stop 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 msgr2=0x7efee0196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efed000e9a0 tx=0x7efed000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 shutdown_connections 2024-03-30T04:00:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.953+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee0102960 0x7efee019e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efec40a11b0 0x7efec40a3670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee0100dc0 0x7efee0196db0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 --2- 172.21.15.23:0/2724657616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee0101fc0 0x7efee01972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 >> 172.21.15.23:0/2724657616 conn(0x7efee00fc360 msgr2=0x7efee01052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 shutdown_connections 2024-03-30T04:00:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:19.954+0000 7efee7000700 1 -- 172.21.15.23:0/2724657616 wait complete. 2024-03-30T04:00:20.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:20 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2724657616' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:20 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:20 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:00:20.345 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T04:00:20.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-30T04:00:20.345 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T04:00:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:20 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2724657616' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:20 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:20 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:00:20.441 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T04:00:20.441 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-30T04:00:20.441 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph mgr dump --format=json 2024-03-30T04:00:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:20 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2724657616' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:20 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-30T04:00:20.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:20 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-30T04:00:20.847 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:21.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:21 smithi023 ceph-mon[38254]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 28 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:21 smithi154 ceph-mon[79161]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 28 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:21 smithi193 ceph-mon[79581]: pgmap v258: 129 pgs: 3 peering, 126 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 28 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:22.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 -- 172.21.15.23:0/4235985960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 msgr2=0x7fb1dc1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/4235985960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc1072d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc0099c0 tx=0x7fb1cc009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 -- 172.21.15.23:0/4235985960 shutdown_connections 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/4235985960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 0x7fb1dc10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/4235985960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc1072d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/4235985960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 0x7fb1dc1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 -- 172.21.15.23:0/4235985960 >> 172.21.15.23:0/4235985960 conn(0x7fb1dc077fa0 msgr2=0x7fb1dc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 -- 172.21.15.23:0/4235985960 shutdown_connections 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.019+0000 7fb1e1a00700 1 -- 172.21.15.23:0/4235985960 wait complete. 2024-03-30T04:00:22.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.020+0000 7fb1e1a00700 1 Processor -- start 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.020+0000 7fb1e1a00700 1 -- start start 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 0x7fb1dc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 0x7fb1dc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a5580 con 0x7fb1dc106ef0 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a56f0 con 0x7fb1dc100e60 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1e1a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1a58b0 con 0x7fb1dc101820 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1db400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1db400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40230/0 (socket says 172.21.15.23:40230) 2024-03-30T04:00:22.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1db400700 1 -- 172.21.15.23:0/1830990025 learned_addr learned my addr 172.21.15.23:0/1830990025 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1daa00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 0x7fb1dc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.021+0000 7fb1da000700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 0x7fb1dc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 -- 172.21.15.23:0/1830990025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 msgr2=0x7fb1dc0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 0x7fb1dc0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 -- 172.21.15.23:0/1830990025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 msgr2=0x7fb1dc0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 0x7fb1dc0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 -- 172.21.15.23:0/1830990025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc0095f0 con 0x7fb1dc106ef0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1db400700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400efc0 tx=0x7fb1d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4018070 con 0x7fb1dc106ef0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc1a5b30 con 0x7fb1dc106ef0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1d400f040 con 0x7fb1dc106ef0 2024-03-30T04:00:22.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400cbc0 con 0x7fb1dc106ef0 2024-03-30T04:00:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.022+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc1a6080 con 0x7fb1dc106ef0 2024-03-30T04:00:22.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.023+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc1092d0 con 0x7fb1dc106ef0 2024-03-30T04:00:22.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.025+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb1d400cd40 con 0x7fb1dc106ef0 2024-03-30T04:00:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.026+0000 7fb1d3400700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 0x7fb1c00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.027+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb1d40cc6e0 con 0x7fb1dc106ef0 2024-03-30T04:00:22.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.027+0000 7fb1daa00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 0x7fb1c00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:22.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.027+0000 7fb1daa00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 0x7fb1c00a36f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009990 tx=0x7fb1cc019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:22.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.031+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb1d4095a30 con 0x7fb1dc106ef0 2024-03-30T04:00:22.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.304+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb1dc1024f0 con 0x7fb1dc106ef0 2024-03-30T04:00:22.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.310+0000 7fb1d3400700 1 -- 172.21.15.23:0/1830990025 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260275 (secure 0 0 0) 0x7fb1d4099050 con 0x7fb1dc106ef0 2024-03-30T04:00:22.313 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 msgr2=0x7fb1c00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 0x7fb1c00a36f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009990 tx=0x7fb1cc019040 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 msgr2=0x7fb1dc073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400efc0 tx=0x7fb1d400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 shutdown_connections 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1dc106ef0 0x7fb1dc073b30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1dc100e60 0x7fb1dc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb1c00a1230 0x7fb1c00a36f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 --2- 172.21.15.23:0/1830990025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1dc101820 0x7fb1dc0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 >> 172.21.15.23:0/1830990025 conn(0x7fb1dc077fa0 msgr2=0x7fb1dc0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 shutdown_connections 2024-03-30T04:00:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:22.314+0000 7fb1e1a00700 1 -- 172.21.15.23:0/1830990025 wait complete. 2024-03-30T04:00:22.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:22 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1830990025' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T04:00:22.707 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:22 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1830990025' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T04:00:22.731 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"active_gid":14154,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":2266065890},{"type":"v1","addr":"172.21.15.23:6801","nonce":2266065890}]},"active_addr":"172.21.15.23:6801/2266065890","active_change":"2024-03-30T03:54:16.197981+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1767310006}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3829741445}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":4064094531}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":354023371}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1274030050}]}]} 2024-03-30T04:00:22.736 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-30T04:00:22.736 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-30T04:00:22.736 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T04:00:22.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:22 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1830990025' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-30T04:00:23.107 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:23.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:23 smithi023 ceph-mon[38254]: pgmap v259: 129 pgs: 2 peering, 127 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:23 smithi154 ceph-mon[79161]: pgmap v259: 129 pgs: 2 peering, 127 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:23.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:23 smithi193 ceph-mon[79581]: pgmap v259: 129 pgs: 2 peering, 127 active+clean; 629 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.6 KiB/s wr, 21 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:24.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.382+0000 7f3608e00700 1 -- 172.21.15.23:0/4034838037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604101820 msgr2=0x7f360410dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:24.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.382+0000 7f3608e00700 1 --2- 172.21.15.23:0/4034838037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604101820 0x7f360410dc10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00b0d0 tx=0x7f35fc00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.382+0000 7f3608e00700 1 -- 172.21.15.23:0/4034838037 shutdown_connections 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 --2- 172.21.15.23:0/4034838037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604101820 0x7f360410dc10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 --2- 172.21.15.23:0/4034838037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604106ef0 0x7f36041072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 --2- 172.21.15.23:0/4034838037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 -- 172.21.15.23:0/4034838037 >> 172.21.15.23:0/4034838037 conn(0x7f3604077fa0 msgr2=0x7f36040783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 -- 172.21.15.23:0/4034838037 shutdown_connections 2024-03-30T04:00:24.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.383+0000 7f3608e00700 1 -- 172.21.15.23:0/4034838037 wait complete. 2024-03-30T04:00:24.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 Processor -- start 2024-03-30T04:00:24.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 -- start start 2024-03-30T04:00:24.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:24.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 0x7f3604199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604106ef0 0x7f360419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360419e3e0 con 0x7f3604106ef0 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360419e550 con 0x7f3604101820 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3608e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360419e710 con 0x7f3604100e60 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3601e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3601e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33722/0 (socket says 172.21.15.23:33722) 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3601e00700 1 -- 172.21.15.23:0/3999635128 learned_addr learned my addr 172.21.15.23:0/3999635128 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3601400700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 0x7f3604199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.384+0000 7f3602800700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604106ef0 0x7f360419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 -- 172.21.15.23:0/3999635128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 msgr2=0x7f3604199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 0x7f3604199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 -- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604106ef0 msgr2=0x7f360419ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604106ef0 0x7f360419ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 -- 172.21.15.23:0/3999635128 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35fc009d00 con 0x7f3604100e60 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601400700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 0x7f3604199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f3601e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35f400e9a0 tx=0x7f35f400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.385+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4009790 con 0x7f3604100e60 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.386+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35f4010430 con 0x7f3604100e60 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.386+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f360419e9f0 con 0x7f3604100e60 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.386+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f4018680 con 0x7f3604100e60 2024-03-30T04:00:24.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.386+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36041a6250 con 0x7f3604100e60 2024-03-30T04:00:24.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.387+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36041024f0 con 0x7f3604100e60 2024-03-30T04:00:24.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.388+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f35f40105a0 con 0x7f3604100e60 2024-03-30T04:00:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.390+0000 7f35faa00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 0x7f35e80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.390+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f35f4014070 con 0x7f3604100e60 2024-03-30T04:00:24.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.390+0000 7f3601400700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 0x7f35e80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:24.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.391+0000 7f3601400700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 0x7f35e80a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f360419aa40 tx=0x7f35f0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:24.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.393+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f35f4096230 con 0x7f3604100e60 2024-03-30T04:00:24.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.609+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f360419a7f0 con 0x7f3604100e60 2024-03-30T04:00:24.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.611+0000 7f35faa00700 1 -- 172.21.15.23:0/3999635128 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+24428 (secure 0 0 0) 0x7f35f4099850 con 0x7f3604100e60 2024-03-30T04:00:24.610 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:24.610 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:17.995781+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:24.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 msgr2=0x7f35e80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 0x7f35e80a3670 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f360419aa40 tx=0x7f35f0009260 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 msgr2=0x7f36041994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f35f400e9a0 tx=0x7f35f400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 shutdown_connections 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3604106ef0 0x7f360419ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f35e80a11b0 0x7f35e80a3670 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3604100e60 0x7f36041994e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 --2- 172.21.15.23:0/3999635128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3604101820 0x7f3604199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 >> 172.21.15.23:0/3999635128 conn(0x7f3604077fa0 msgr2=0x7f360410c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:24.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 shutdown_connections 2024-03-30T04:00:24.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:24.614+0000 7f3608e00700 1 -- 172.21.15.23:0/3999635128 wait complete. 2024-03-30T04:00:25.038 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-30T04:00:25.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd dump --format=json 2024-03-30T04:00:25.409 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:25.628 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:25 smithi023 ceph-mon[38254]: pgmap v260: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:25.628 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:25 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3999635128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:25 smithi154 ceph-mon[79161]: pgmap v260: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:25 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3999635128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:25.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:25 smithi193 ceph-mon[79581]: pgmap v260: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.5 KiB/s wr, 18 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:25.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:25 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3999635128' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:26.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 -- 172.21.15.23:0/729154863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298106ef0 msgr2=0x7f12981072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:26.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 --2- 172.21.15.23:0/729154863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298106ef0 0x7f12981072d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f128c0099c0 tx=0x7f128c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 -- 172.21.15.23:0/729154863 shutdown_connections 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 --2- 172.21.15.23:0/729154863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298101820 0x7f129810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 --2- 172.21.15.23:0/729154863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298106ef0 0x7f12981072d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 --2- 172.21.15.23:0/729154863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298100e60 0x7f12981012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 -- 172.21.15.23:0/729154863 >> 172.21.15.23:0/729154863 conn(0x7f1298077fa0 msgr2=0x7f12980783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 -- 172.21.15.23:0/729154863 shutdown_connections 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.616+0000 7f12a0800700 1 -- 172.21.15.23:0/729154863 wait complete. 2024-03-30T04:00:26.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 Processor -- start 2024-03-30T04:00:26.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 -- start start 2024-03-30T04:00:26.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:26.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 0x7f1298197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:26.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 0x7f129819ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129819c0a0 con 0x7f1298101820 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.618+0000 7f12a0800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129819c210 con 0x7f1298106ef0 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f12a0800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129819c3d0 con 0x7f1298100e60 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 0x7f1298197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 0x7f129819ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33734/0 (socket says 172.21.15.23:33734) 2024-03-30T04:00:26.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 -- 172.21.15.23:0/1451655297 learned_addr learned my addr 172.21.15.23:0/1451655297 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 -- 172.21.15.23:0/1451655297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 msgr2=0x7f129819ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 0x7f129819ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 -- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 msgr2=0x7f1298197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 0x7f1298197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129de00700 1 -- 172.21.15.23:0/1451655297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c0095f0 con 0x7f1298100e60 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129e800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 0x7f129819ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:26.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.619+0000 7f129d400700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 0x7f1298197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f129de00700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f128c006010 tx=0x7f128c00b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c01d070 con 0x7f1298100e60 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129819c650 con 0x7f1298100e60 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f128c00be00 con 0x7f1298100e60 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c00fa90 con 0x7f1298100e60 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.620+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12981a3f10 con 0x7f1298100e60 2024-03-30T04:00:26.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.621+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12981092d0 con 0x7f1298100e60 2024-03-30T04:00:26.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.623+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f128c003680 con 0x7f1298100e60 2024-03-30T04:00:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.624+0000 7f1292a00700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 0x7f12800a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.624+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f128c0cdfd0 con 0x7f1298100e60 2024-03-30T04:00:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.625+0000 7f129d400700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 0x7f12800a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:26.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.625+0000 7f129d400700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 0x7f12800a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1298198700 tx=0x7f1288009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:26.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.627+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f128c097320 con 0x7f1298100e60 2024-03-30T04:00:26.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.840+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12981024f0 con 0x7f1298100e60 2024-03-30T04:00:26.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.841+0000 7f1292a00700 1 -- 172.21.15.23:0/1451655297 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+24428 (secure 0 0 0) 0x7f128c09a940 con 0x7f1298100e60 2024-03-30T04:00:26.841 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:26.841 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:17.995781+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 msgr2=0x7f12800a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 0x7f12800a3670 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1298198700 tx=0x7f1288009260 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 msgr2=0x7f12981971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f128c006010 tx=0x7f128c00b9c0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 shutdown_connections 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1298106ef0 0x7f129819ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f12800a11b0 0x7f12800a3670 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1298100e60 0x7f12981971f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 --2- 172.21.15.23:0/1451655297 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1298101820 0x7f1298197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 >> 172.21.15.23:0/1451655297 conn(0x7f1298077fa0 msgr2=0x7f12980ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:26.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 shutdown_connections 2024-03-30T04:00:26.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:26.844+0000 7f12a0800700 1 -- 172.21.15.23:0/1451655297 wait complete. 2024-03-30T04:00:27.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:26 smithi023 ceph-mon[38254]: pgmap v261: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 14 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:27.229 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:26 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1451655297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:27.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:26 smithi193 ceph-mon[79581]: pgmap v261: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 14 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:27.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:26 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1451655297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:27.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-03-30T04:00:27.255 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-03-30T04:00:27.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-03-30T04:00:27.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-03-30T04:00:27.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-03-30T04:00:27.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-03-30T04:00:27.257 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-03-30T04:00:27.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-03-30T04:00:27.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.8 flush_pg_stats 2024-03-30T04:00:27.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.9 flush_pg_stats 2024-03-30T04:00:27.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.10 flush_pg_stats 2024-03-30T04:00:27.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph tell osd.11 flush_pg_stats 2024-03-30T04:00:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:26 smithi154 ceph-mon[79161]: pgmap v261: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 14 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:26 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1451655297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:29.373 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:29.427 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:29.577 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:29.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:29 smithi023 ceph-mon[38254]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:29 smithi154 ceph-mon[79161]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:29.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:29 smithi193 ceph-mon[79581]: pgmap v262: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.5 KiB/s wr, 12 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:30.259 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:30.435 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:30.997 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:31.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 -- 172.21.15.23:0/3743745473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 msgr2=0x7fba381071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 --2- 172.21.15.23:0/3743745473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba381071d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0098f0 tx=0x7fba2c009c00 comp rx=0 tx=0).stop 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 -- 172.21.15.23:0/3743745473 shutdown_connections 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 --2- 172.21.15.23:0/3743745473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba380690f0 0x7fba3810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 --2- 172.21.15.23:0/3743745473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba381071d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 --2- 172.21.15.23:0/3743745473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba38068750 0x7fba38068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.481+0000 7fba3f800700 1 -- 172.21.15.23:0/3743745473 >> 172.21.15.23:0/3743745473 conn(0x7fba38075d50 msgr2=0x7fba38076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.482+0000 7fba3f800700 1 -- 172.21.15.23:0/3743745473 shutdown_connections 2024-03-30T04:00:31.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.482+0000 7fba3f800700 1 -- 172.21.15.23:0/3743745473 wait complete. 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 Processor -- start 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 -- start start 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 0x7fba380ffde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba380690f0 0x7fba38100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba38100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381a5650 con 0x7fba38106df0 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381a57c0 con 0x7fba380690f0 2024-03-30T04:00:31.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fba3f800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba381a5980 con 0x7fba38068750 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba38100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 0x7fba380ffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba38100860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34026/0 (socket says 172.21.15.23:34026) 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 0x7fba380ffde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44056/0 (socket says 172.21.15.23:44056) 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3d800700 1 -- 172.21.15.23:0/3549130165 learned_addr learned my addr 172.21.15.23:0/3549130165 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba37e00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba380690f0 0x7fba38100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 -- 172.21.15.23:0/3549130165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba380690f0 msgr2=0x7fba38100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba380690f0 0x7fba38100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 -- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 msgr2=0x7fba38100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba38106df0 0x7fba38100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba3ce00700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c009520 con 0x7fba38068750 2024-03-30T04:00:31.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fba37e00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba380690f0 0x7fba38100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba3ce00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 0x7fba380ffde0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009fd0 tx=0x7fba2c00fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01c070 con 0x7fba38068750 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba381a5c00 con 0x7fba38068750 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba381a6070 con 0x7fba38068750 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba2c00bb20 con 0x7fba38068750 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0038e0 con 0x7fba38068750 2024-03-30T04:00:31.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.486+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fba18000fc0 con 0x7fba38068750 2024-03-30T04:00:31.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.489+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fba2c020030 con 0x7fba38068750 2024-03-30T04:00:31.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba35600700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fba200a1230 0x7fba200a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fba2c0cdf00 con 0x7fba38068750 2024-03-30T04:00:31.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba37e00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fba200a1230 0x7fba200a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba35600700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] conn(0x7fba200ac120 0x7fba200ae5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba200aec90 con 0x7fba200ac120 2024-03-30T04:00:31.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.490+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba2c0ce270 con 0x7fba38068750 2024-03-30T04:00:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.491+0000 7fba3d800700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] conn(0x7fba200ac120 0x7fba200ae5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.491+0000 7fba37e00700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fba200a1230 0x7fba200a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba381014c0 tx=0x7fba24009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.491+0000 7fba3d800700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] conn(0x7fba200ac120 0x7fba200ae5c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.493+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== osd.0 v2:172.21.15.23:6802/2819496083 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fba200aec90 con 0x7fba200ac120 2024-03-30T04:00:31.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.512+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 --> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fba18002d40 con 0x7fba200ac120 2024-03-30T04:00:31.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.513+0000 7fba35600700 1 -- 172.21.15.23:0/3549130165 <== osd.0 v2:172.21.15.23:6802/2819496083 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fba18002d40 con 0x7fba200ac120 2024-03-30T04:00:31.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.513+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] conn(0x7fba200ac120 msgr2=0x7fba200ae5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.513+0000 7fba3f800700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] conn(0x7fba200ac120 0x7fba200ae5c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.513+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fba200a1230 msgr2=0x7fba200a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fba200a1230 0x7fba200a36f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba381014c0 tx=0x7fba24009260 comp rx=0 tx=0).stop 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 msgr2=0x7fba380ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 --2- 172.21.15.23:0/3549130165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba38068750 0x7fba380ffde0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009fd0 tx=0x7fba2c00fa00 comp rx=0 tx=0).stop 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3d800700 1 -- 172.21.15.23:0/3549130165 reap_dead start 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 shutdown_connections 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 >> 172.21.15.23:0/3549130165 conn(0x7fba38075d50 msgr2=0x7fba380fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 shutdown_connections 2024-03-30T04:00:31.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.514+0000 7fba3f800700 1 -- 172.21.15.23:0/3549130165 wait complete. 2024-03-30T04:00:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:31 smithi154 ceph-mon[79161]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 -- 172.21.15.23:0/4144990449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 msgr2=0x7fb61c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 --2- 172.21.15.23:0/4144990449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c10dc10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb61400b0d0 tx=0x7fb61400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 -- 172.21.15.23:0/4144990449 shutdown_connections 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 --2- 172.21.15.23:0/4144990449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c10dc10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 --2- 172.21.15.23:0/4144990449 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 0x7fb61c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 --2- 172.21.15.23:0/4144990449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb61c100e60 0x7fb61c1012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 -- 172.21.15.23:0/4144990449 >> 172.21.15.23:0/4144990449 conn(0x7fb61c077fa0 msgr2=0x7fb61c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.455+0000 7fb622800700 1 -- 172.21.15.23:0/4144990449 shutdown_connections 2024-03-30T04:00:31.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.456+0000 7fb622800700 1 -- 172.21.15.23:0/4144990449 wait complete. 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 Processor -- start 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 -- start start 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb61c100e60 0x7fb61c199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 0x7fb61c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c19e430 con 0x7fb61c101820 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c19e5a0 con 0x7fb61c106ef0 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.457+0000 7fb622800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb61c19e760 con 0x7fb61c100e60 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 0x7fb61c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34014/0 (socket says 172.21.15.23:34014) 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 -- 172.21.15.23:0/859684742 learned_addr learned my addr 172.21.15.23:0/859684742 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61b400700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb61c100e60 0x7fb61c199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 -- 172.21.15.23:0/859684742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb61c100e60 msgr2=0x7fb61c199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb61c100e60 0x7fb61c199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 -- 172.21.15.23:0/859684742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 msgr2=0x7fb61c19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 0x7fb61c19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614009d00 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61be00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb61c106ef0 0x7fb61c19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.458+0000 7fb61aa00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c199a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb60800d950 tx=0x7fb60800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.459+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60800f890 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.459+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c19ea40 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.459+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb608010460 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.459+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb60800e610 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.459+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c1a62a0 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.460+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb61c111050 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.462+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb6080108d0 con 0x7fb61c101820 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb613e00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6000a11b0 0x7fb6000a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb6080cd0a0 con 0x7fb61c101820 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb61b400700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6000a11b0 0x7fb6000a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb613e00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] conn(0x7fb6000ac150 0x7fb6000ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6000aecc0 con 0x7fb6000ac150 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6080cd410 con 0x7fb61c101820 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.464+0000 7fb61be00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] conn(0x7fb6000ac150 0x7fb6000ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:31.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.465+0000 7fb61b400700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6000a11b0 0x7fb6000a3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009be0 tx=0x7fb60c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.465+0000 7fb61be00700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] conn(0x7fb6000ac150 0x7fb6000ae5f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.466+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== osd.6 v2:172.21.15.23:6818/4279410626 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb6000aecc0 con 0x7fb6000ac150 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.483+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 --> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb61c100190 con 0x7fb6000ac150 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.484+0000 7fb613e00700 1 -- 172.21.15.23:0/859684742 <== osd.6 v2:172.21.15.23:6818/4279410626 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb61c100190 con 0x7fb6000ac150 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] conn(0x7fb6000ac150 msgr2=0x7fb6000ae5f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] conn(0x7fb6000ac150 0x7fb6000ae5f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6000a11b0 msgr2=0x7fb6000a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6000a11b0 0x7fb6000a3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb60c009be0 tx=0x7fb60c009260 comp rx=0 tx=0).stop 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 msgr2=0x7fb61c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 --2- 172.21.15.23:0/859684742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb61c101820 0x7fb61c199a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb60800d950 tx=0x7fb60800dc60 comp rx=0 tx=0).stop 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb61be00700 1 -- 172.21.15.23:0/859684742 reap_dead start 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 shutdown_connections 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.485+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 >> 172.21.15.23:0/859684742 conn(0x7fb61c077fa0 msgr2=0x7fb61c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:31.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.486+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 shutdown_connections 2024-03-30T04:00:31.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:31.486+0000 7fb622800700 1 -- 172.21.15.23:0/859684742 wait complete. 2024-03-30T04:00:31.732 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:31 smithi023 ceph-mon[38254]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:31.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:31 smithi193 ceph-mon[79581]: pgmap v263: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:32.102 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:32.462 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:32.537 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:32.735 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 -- 172.21.15.23:0/1317382702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 msgr2=0x7f09e8106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 --2- 172.21.15.23:0/1317382702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e8106bd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009990 tx=0x7f09d4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 -- 172.21.15.23:0/1317382702 shutdown_connections 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 --2- 172.21.15.23:0/1317382702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 0x7f09e810dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 --2- 172.21.15.23:0/1317382702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 --2- 172.21.15.23:0/1317382702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e8106bd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 -- 172.21.15.23:0/1317382702 >> 172.21.15.23:0/1317382702 conn(0x7f09e80fb5f0 msgr2=0x7f09e80fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:33.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 -- 172.21.15.23:0/1317382702 shutdown_connections 2024-03-30T04:00:33.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.200+0000 7f09ed200700 1 -- 172.21.15.23:0/1317382702 wait complete. 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.201+0000 7f09ed200700 1 Processor -- start 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.201+0000 7f09ed200700 1 -- start start 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e80ffa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e80fff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 0x7f09e81004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8101a00 con 0x7f09e8101de0 2024-03-30T04:00:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81a9b00 con 0x7f09e81072e0 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09ed200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81a9cc0 con 0x7f09e8102790 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e80ffa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e80ffa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34040/0 (socket says 172.21.15.23:34040) 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 -- 172.21.15.23:0/902799835 learned_addr learned my addr 172.21.15.23:0/902799835 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e6a00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e80fff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7e00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 0x7f09e81004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 -- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 msgr2=0x7f09e80fff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e80fff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 -- 172.21.15.23:0/902799835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 msgr2=0x7f09e81004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 0x7f09e81004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7400700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d40095f0 con 0x7f09e8101de0 2024-03-30T04:00:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e6a00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e8102790 0x7f09e80fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.202+0000 7f09e7e00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e81072e0 0x7f09e81004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09e7400700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e80ffa40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09d800e9a0 tx=0x7f09d800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8009790 con 0x7f09e8101de0 2024-03-30T04:00:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09d800cbd0 con 0x7f09e8101de0 2024-03-30T04:00:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8018680 con 0x7f09e8101de0 2024-03-30T04:00:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81a9fa0 con 0x7f09e8101de0 2024-03-30T04:00:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.203+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e81aa3c0 con 0x7f09e8101de0 2024-03-30T04:00:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.204+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f09e8110f40 con 0x7f09e8101de0 2024-03-30T04:00:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.206+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f09d800cd40 con 0x7f09e8101de0 2024-03-30T04:00:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.207+0000 7f09dfe00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09cc0a11b0 0x7f09cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f09d8014070 con 0x7f09e8101de0 2024-03-30T04:00:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09e6a00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09cc0a11b0 0x7f09cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09dfe00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] conn(0x7f09cc0ac1d0 0x7f09cc0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f09cc0aed40 con 0x7f09cc0ac1d0 2024-03-30T04:00:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09d80cd6f0 con 0x7f09e8101de0 2024-03-30T04:00:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09e6a00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09cc0a11b0 0x7f09cc0a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09e8101170 tx=0x7f09d400b260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.208+0000 7f09e7e00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] conn(0x7f09cc0ac1d0 0x7f09cc0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:33.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.209+0000 7f09e7e00700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] conn(0x7f09cc0ac1d0 0x7f09cc0ae670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:33.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.210+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== osd.8 v2:172.21.15.193:6816/54947619 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f09cc0aed40 con 0x7f09cc0ac1d0 2024-03-30T04:00:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.227+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 --> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f09e8068fb0 con 0x7f09cc0ac1d0 2024-03-30T04:00:33.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.228+0000 7f09dfe00700 1 -- 172.21.15.23:0/902799835 <== osd.8 v2:172.21.15.193:6816/54947619 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f09e8068fb0 con 0x7f09cc0ac1d0 2024-03-30T04:00:33.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] conn(0x7f09cc0ac1d0 msgr2=0x7f09cc0ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] conn(0x7f09cc0ac1d0 0x7f09cc0ae670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09cc0a11b0 msgr2=0x7f09cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f09cc0a11b0 0x7f09cc0a3670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f09e8101170 tx=0x7f09d400b260 comp rx=0 tx=0).stop 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 msgr2=0x7f09e80ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 --2- 172.21.15.23:0/902799835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09e8101de0 0x7f09e80ffa40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f09d800e9a0 tx=0x7f09d800ed60 comp rx=0 tx=0).stop 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09e7e00700 1 -- 172.21.15.23:0/902799835 reap_dead start 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 shutdown_connections 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 >> 172.21.15.23:0/902799835 conn(0x7f09e80fb5f0 msgr2=0x7f09e8105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 shutdown_connections 2024-03-30T04:00:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:33.229+0000 7f09ed200700 1 -- 172.21.15.23:0/902799835 wait complete. 2024-03-30T04:00:33.500 INFO:teuthology.orchestra.run.smithi023.stdout:34359738430 2024-03-30T04:00:33.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-03-30T04:00:33.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:33 smithi023 ceph-mon[38254]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:33.675 INFO:teuthology.orchestra.run.smithi023.stdout:188978561063 2024-03-30T04:00:33.675 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-03-30T04:00:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:33 smithi154 ceph-mon[79161]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:33.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:33 smithi193 ceph-mon[79581]: pgmap v264: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-30T04:00:34.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 -- 172.21.15.23:0/2367698392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360101820 msgr2=0x7f636010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 --2- 172.21.15.23:0/2367698392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360101820 0x7f636010dc10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f635800b0d0 tx=0x7f635800b3e0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 -- 172.21.15.23:0/2367698392 shutdown_connections 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 --2- 172.21.15.23:0/2367698392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360101820 0x7f636010dc10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 --2- 172.21.15.23:0/2367698392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360106ef0 0x7f63601072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 --2- 172.21.15.23:0/2367698392 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360100e60 0x7f63601012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 -- 172.21.15.23:0/2367698392 >> 172.21.15.23:0/2367698392 conn(0x7f6360077fa0 msgr2=0x7f63600783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 -- 172.21.15.23:0/2367698392 shutdown_connections 2024-03-30T04:00:34.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.173+0000 7f6368200700 1 -- 172.21.15.23:0/2367698392 wait complete. 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.174+0000 7f6368200700 1 Processor -- start 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 -- start start 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360100e60 0x7f63601972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360101820 0x7f6360197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 0x7f636019bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636019c1c0 con 0x7f6360106ef0 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636019c330 con 0x7f6360101820 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6368200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636019c4f0 con 0x7f6360100e60 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6366200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 0x7f636019bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6366200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 0x7f636019bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34050/0 (socket says 172.21.15.23:34050) 2024-03-30T04:00:34.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6366200700 1 -- 172.21.15.23:0/3158166109 learned_addr learned my addr 172.21.15.23:0/3158166109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6365800700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360100e60 0x7f63601972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.175+0000 7f6364e00700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360101820 0x7f6360197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 -- 172.21.15.23:0/3158166109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360100e60 msgr2=0x7f63601972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360100e60 0x7f63601972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 -- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360101820 msgr2=0x7f6360197800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6360101820 0x7f6360197800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6358009d00 con 0x7f6360106ef0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6365800700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6360100e60 0x7f63601972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6366200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 0x7f636019bbb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f635800bcd0 tx=0x7f635800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635800e040 con 0x7f6360106ef0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6358005070 con 0x7f6360106ef0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635801b6c0 con 0x7f6360106ef0 2024-03-30T04:00:34.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636019c770 con 0x7f6360106ef0 2024-03-30T04:00:34.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.176+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63601a4030 con 0x7f6360106ef0 2024-03-30T04:00:34.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.177+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f636019ca40 con 0x7f6360106ef0 2024-03-30T04:00:34.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.180+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f63580051e0 con 0x7f6360106ef0 2024-03-30T04:00:34.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f635e000700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63480a1230 0x7f63480a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63580cdd10 con 0x7f6360106ef0 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f6365800700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63480a1230 0x7f63480a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f635e000700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] conn(0x7f63480ac250 0x7f63480ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f63480aedc0 con 0x7f63480ac250 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.181+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63580ce080 con 0x7f6360106ef0 2024-03-30T04:00:34.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.182+0000 7f6364e00700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] conn(0x7f63480ac250 0x7f63480ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:34.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.182+0000 7f6365800700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63480a1230 0x7f63480a36f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6354011490 tx=0x7f6354009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:34.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.182+0000 7f6364e00700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] conn(0x7f63480ac250 0x7f63480ae6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:34.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.183+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== osd.1 v2:172.21.15.154:6800/1863895073 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f63480aedc0 con 0x7f63480ac250 2024-03-30T04:00:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.200+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 --> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6360110fd0 con 0x7f63480ac250 2024-03-30T04:00:34.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f635e000700 1 -- 172.21.15.23:0/3158166109 <== osd.1 v2:172.21.15.154:6800/1863895073 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6360110fd0 con 0x7f63480ac250 2024-03-30T04:00:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] conn(0x7f63480ac250 msgr2=0x7f63480ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] conn(0x7f63480ac250 0x7f63480ae6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63480a1230 msgr2=0x7f63480a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63480a1230 0x7f63480a36f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6354011490 tx=0x7f6354009260 comp rx=0 tx=0).stop 2024-03-30T04:00:34.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 msgr2=0x7f636019bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6368200700 1 --2- 172.21.15.23:0/3158166109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6360106ef0 0x7f636019bbb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f635800bcd0 tx=0x7f635800bd00 comp rx=0 tx=0).stop 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.201+0000 7f6366200700 1 -- 172.21.15.23:0/3158166109 reap_dead start 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.202+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 shutdown_connections 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.202+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 >> 172.21.15.23:0/3158166109 conn(0x7f6360077fa0 msgr2=0x7f63600ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.202+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 shutdown_connections 2024-03-30T04:00:34.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.202+0000 7f6368200700 1 -- 172.21.15.23:0/3158166109 wait complete. 2024-03-30T04:00:35.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 -- 172.21.15.23:0/2187002699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 msgr2=0x7fc1901012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 --2- 172.21.15.23:0/2187002699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc1901012e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc17c0099c0 tx=0x7fc17c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 -- 172.21.15.23:0/2187002699 shutdown_connections 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 --2- 172.21.15.23:0/2187002699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190101820 0x7fc19010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 --2- 172.21.15.23:0/2187002699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc190106ef0 0x7fc1901072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 --2- 172.21.15.23:0/2187002699 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc1901012e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 -- 172.21.15.23:0/2187002699 >> 172.21.15.23:0/2187002699 conn(0x7fc190077fa0 msgr2=0x7fc1900783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 -- 172.21.15.23:0/2187002699 shutdown_connections 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.858+0000 7fc194e00700 1 -- 172.21.15.23:0/2187002699 wait complete. 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.859+0000 7fc194e00700 1 Processor -- start 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.859+0000 7fc194e00700 1 -- start start 2024-03-30T04:00:35.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc190199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190101820 0x7fc1901998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc190106ef0 0x7fc19019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19019e290 con 0x7fc190100e60 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19019e400 con 0x7fc190101820 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc194e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19019e5c0 con 0x7fc190106ef0 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc190199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc190199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34068/0 (socket says 172.21.15.23:34068) 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 -- 172.21.15.23:0/3238602120 learned_addr learned my addr 172.21.15.23:0/3238602120 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18d400700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190101820 0x7fc1901998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18e800700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc190106ef0 0x7fc19019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 -- 172.21.15.23:0/3238602120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc190106ef0 msgr2=0x7fc19019dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc190106ef0 0x7fc19019dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.860+0000 7fc18de00700 1 -- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190101820 msgr2=0x7fc1901998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc18de00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc190101820 0x7fc1901998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc18de00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0095f0 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc18de00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc190199390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9a0 tx=0x7fc18000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180009790 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc180010430 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc180018680 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19019e8a0 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.861+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1901a6100 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.862+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc1901024f0 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.864+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc1800105a0 con 0x7fc190100e60 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.865+0000 7fc186a00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1740a11b0 0x7fc1740a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:35.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc180014070 con 0x7fc190100e60 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc18d400700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1740a11b0 0x7fc1740a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc186a00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] conn(0x7fc1740ac1d0 0x7fc1740ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc1740aed40 con 0x7fc1740ac1d0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1800cd610 con 0x7fc190100e60 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc18e800700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] conn(0x7fc1740ac1d0 0x7fc1740ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.866+0000 7fc18d400700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1740a11b0 0x7fc1740a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009fd0 tx=0x7fc17c019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.867+0000 7fc18e800700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] conn(0x7fc1740ac1d0 0x7fc1740ae670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.868+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== osd.4 v2:172.21.15.154:6808/2815560551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc1740aed40 con 0x7fc1740ac1d0 2024-03-30T04:00:35.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.885+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 --> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc190110f80 con 0x7fc1740ac1d0 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.885+0000 7fc186a00700 1 -- 172.21.15.23:0/3238602120 <== osd.4 v2:172.21.15.154:6808/2815560551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc190110f80 con 0x7fc1740ac1d0 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] conn(0x7fc1740ac1d0 msgr2=0x7fc1740ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] conn(0x7fc1740ac1d0 0x7fc1740ae670 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1740a11b0 msgr2=0x7fc1740a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1740a11b0 0x7fc1740a3670 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009fd0 tx=0x7fc17c019040 comp rx=0 tx=0).stop 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 msgr2=0x7fc190199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 --2- 172.21.15.23:0/3238602120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc190100e60 0x7fc190199390 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc18000e9a0 tx=0x7fc18000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc18e800700 1 -- 172.21.15.23:0/3238602120 reap_dead start 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 shutdown_connections 2024-03-30T04:00:35.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 >> 172.21.15.23:0/3238602120 conn(0x7fc190077fa0 msgr2=0x7fc19010c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:35.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 shutdown_connections 2024-03-30T04:00:35.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:34.886+0000 7fc194e00700 1 -- 172.21.15.23:0/3238602120 wait complete. 2024-03-30T04:00:35.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:35 smithi023 ceph-mon[38254]: pgmap v265: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-30T04:00:35.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:35 smithi154 ceph-mon[79161]: pgmap v265: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-30T04:00:35.705 INFO:teuthology.orchestra.run.smithi023.stdout:240518168609 2024-03-30T04:00:35.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.8 2024-03-30T04:00:35.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:35 smithi193 ceph-mon[79581]: pgmap v265: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-30T04:00:36.657 INFO:teuthology.orchestra.run.smithi023.stdout:137438953520 2024-03-30T04:00:36.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-03-30T04:00:36.701 INFO:teuthology.orchestra.run.smithi023.stdout:55834574907 2024-03-30T04:00:36.701 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-03-30T04:00:37.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 -- 172.21.15.23:0/2586290993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 msgr2=0x7fa4241116e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 --2- 172.21.15.23:0/2586290993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 0x7fa4241116e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa410009990 tx=0x7fa410009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 -- 172.21.15.23:0/2586290993 shutdown_connections 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 --2- 172.21.15.23:0/2586290993 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa424111c20 0x7fa424114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 --2- 172.21.15.23:0/2586290993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4240752e0 0x7fa4240756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 --2- 172.21.15.23:0/2586290993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 0x7fa4241116e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 -- 172.21.15.23:0/2586290993 >> 172.21.15.23:0/2586290993 conn(0x7fa4240fea70 msgr2=0x7fa424100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 -- 172.21.15.23:0/2586290993 shutdown_connections 2024-03-30T04:00:37.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.134+0000 7fa42a600700 1 -- 172.21.15.23:0/2586290993 wait complete. 2024-03-30T04:00:37.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.136+0000 7fa42a600700 1 Processor -- start 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.136+0000 7fa42a600700 1 -- start start 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.136+0000 7fa42a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 0x7fa4241a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa42a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 0x7fa4241a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa42a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa424111c20 0x7fa42419d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa42a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42419dd20 con 0x7fa4240752e0 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa42a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42419de90 con 0x7fa424075c00 2024-03-30T04:00:37.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa42a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42419e050 con 0x7fa424111c20 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa423400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 0x7fa4241a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa423400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 0x7fa4241a3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34088/0 (socket says 172.21.15.23:34088) 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.137+0000 7fa423400700 1 -- 172.21.15.23:0/1169663011 learned_addr learned my addr 172.21.15.23:0/1169663011 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423e00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa424111c20 0x7fa42419d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa422a00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 0x7fa4241a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423400700 1 -- 172.21.15.23:0/1169663011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa424111c20 msgr2=0x7fa42419d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423400700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa424111c20 0x7fa42419d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423400700 1 -- 172.21.15.23:0/1169663011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 msgr2=0x7fa4241a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423400700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa424075c00 0x7fa4241a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.138+0000 7fa423400700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4100095f0 con 0x7fa4240752e0 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.139+0000 7fa423400700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 0x7fa4241a3580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0060b0 tx=0x7fa41c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.139+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c00ddd0 con 0x7fa4240752e0 2024-03-30T04:00:37.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.140+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa41c00f040 con 0x7fa4240752e0 2024-03-30T04:00:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.140+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa42419e330 con 0x7fa4240752e0 2024-03-30T04:00:37.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.140+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa42406a9a0 con 0x7fa4240752e0 2024-03-30T04:00:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.141+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa424117640 con 0x7fa4240752e0 2024-03-30T04:00:37.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.141+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa41c014600 con 0x7fa4240752e0 2024-03-30T04:00:37.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.145+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa41c015400 con 0x7fa4240752e0 2024-03-30T04:00:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.148+0000 7fa41be00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4080a1230 0x7fa4080a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.148+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa41c0cce50 con 0x7fa4240752e0 2024-03-30T04:00:37.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.149+0000 7fa422a00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4080a1230 0x7fa4080a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.149+0000 7fa41be00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] conn(0x7fa4080ac250 0x7fa4080ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.149+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4080aedc0 con 0x7fa4080ac250 2024-03-30T04:00:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.149+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa41c0cd1c0 con 0x7fa4240752e0 2024-03-30T04:00:37.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.150+0000 7fa423e00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] conn(0x7fa4080ac250 0x7fa4080ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.150+0000 7fa422a00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4080a1230 0x7fa4080a36f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa42419edd0 tx=0x7fa410005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.151+0000 7fa423e00700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] conn(0x7fa4080ac250 0x7fa4080ae6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.154+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== osd.3 v2:172.21.15.23:6810/1020396955 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa4080aedc0 con 0x7fa4080ac250 2024-03-30T04:00:37.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.188+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 --> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa424076a20 con 0x7fa4080ac250 2024-03-30T04:00:37.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.189+0000 7fa41be00700 1 -- 172.21.15.23:0/1169663011 <== osd.3 v2:172.21.15.23:6810/1020396955 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa424076a20 con 0x7fa4080ac250 2024-03-30T04:00:37.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.189+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] conn(0x7fa4080ac250 msgr2=0x7fa4080ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.189+0000 7fa42a600700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] conn(0x7fa4080ac250 0x7fa4080ae6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4080a1230 msgr2=0x7fa4080a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4080a1230 0x7fa4080a36f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa42419edd0 tx=0x7fa410005fd0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 msgr2=0x7fa4241a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 --2- 172.21.15.23:0/1169663011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4240752e0 0x7fa4241a3580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0060b0 tx=0x7fa41c00d3b0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa423e00700 1 -- 172.21.15.23:0/1169663011 reap_dead start 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 shutdown_connections 2024-03-30T04:00:37.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 >> 172.21.15.23:0/1169663011 conn(0x7fa4240fea70 msgr2=0x7fa424100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 shutdown_connections 2024-03-30T04:00:37.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.190+0000 7fa42a600700 1 -- 172.21.15.23:0/1169663011 wait complete. 2024-03-30T04:00:37.217 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:36 smithi023 ceph-mon[38254]: pgmap v266: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:37.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:36 smithi193 ceph-mon[79581]: pgmap v266: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:36 smithi154 ceph-mon[79161]: pgmap v266: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:37.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 -- 172.21.15.23:0/841278048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd8068750 msgr2=0x7fbdd8068b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 --2- 172.21.15.23:0/841278048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd8068750 0x7fbdd8068b30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009990 tx=0x7fbdc8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 -- 172.21.15.23:0/841278048 shutdown_connections 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 --2- 172.21.15.23:0/841278048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbdd810d8c0 0x7fbdd810fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 --2- 172.21.15.23:0/841278048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd8068750 0x7fbdd8068b30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 --2- 172.21.15.23:0/841278048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdd8069070 0x7fbdd810d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 -- 172.21.15.23:0/841278048 >> 172.21.15.23:0/841278048 conn(0x7fbdd8075d50 msgr2=0x7fbdd8076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.272+0000 7fbddce00700 1 -- 172.21.15.23:0/841278048 shutdown_connections 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.273+0000 7fbddce00700 1 -- 172.21.15.23:0/841278048 wait complete. 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.273+0000 7fbddce00700 1 Processor -- start 2024-03-30T04:00:37.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 -- start start 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbdd8068750 0x7fbdd8199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdd8069070 0x7fbdd81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 0x7fbdd819dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd819e310 con 0x7fbdd810d8c0 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd819e480 con 0x7fbdd8069070 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbddce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd819e640 con 0x7fbdd8068750 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbdd6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 0x7fbdd819dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbdd6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 0x7fbdd819dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34096/0 (socket says 172.21.15.23:34096) 2024-03-30T04:00:37.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbdd6800700 1 -- 172.21.15.23:0/1716036216 learned_addr learned my addr 172.21.15.23:0/1716036216 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.274+0000 7fbdd5400700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdd8069070 0x7fbdd81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd5e00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbdd8068750 0x7fbdd8199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 -- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbdd8068750 msgr2=0x7fbdd8199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbdd8068750 0x7fbdd8199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 -- 172.21.15.23:0/1716036216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdd8069070 msgr2=0x7fbdd81999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbdd8069070 0x7fbdd81999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc80095f0 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdd6800700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 0x7fbdd819dd00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000ef90 tx=0x7fbdd000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.275+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0018070 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.276+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdd000f040 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.276+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd819e8c0 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.276+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd000cda0 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.276+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd81a6180 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.277+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdd81002f0 con 0x7fbdd810d8c0 2024-03-30T04:00:37.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.279+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbdd0003a00 con 0x7fbdd810d8c0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdcea00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbdbc0a1230 0x7fbdbc0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbdd00cd220 con 0x7fbdd810d8c0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdd5e00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbdbc0a1230 0x7fbdbc0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdcea00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] conn(0x7fbdbc0ac250 0x7fbdbc0ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdbc0aedc0 con 0x7fbdbc0ac250 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.282+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdd00cd590 con 0x7fbdd810d8c0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.283+0000 7fbdd5400700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] conn(0x7fbdbc0ac250 0x7fbdbc0ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.283+0000 7fbdd5e00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbdbc0a1230 0x7fbdbc0a36f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800b2e0 tx=0x7fbdc8005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.283+0000 7fbdd5400700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] conn(0x7fbdbc0ac250 0x7fbdbc0ae6f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.284+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== osd.11 v2:172.21.15.193:6824/2276508968 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fbdbc0aedc0 con 0x7fbdbc0ac250 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.311+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 --> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbdd81131f0 con 0x7fbdbc0ac250 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.312+0000 7fbdcea00700 1 -- 172.21.15.23:0/1716036216 <== osd.11 v2:172.21.15.193:6824/2276508968 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbdd81131f0 con 0x7fbdbc0ac250 2024-03-30T04:00:37.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] conn(0x7fbdbc0ac250 msgr2=0x7fbdbc0ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] conn(0x7fbdbc0ac250 0x7fbdbc0ae6f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbdbc0a1230 msgr2=0x7fbdbc0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbdbc0a1230 0x7fbdbc0a36f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800b2e0 tx=0x7fbdc8005fd0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 msgr2=0x7fbdd819dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 --2- 172.21.15.23:0/1716036216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdd810d8c0 0x7fbdd819dd00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000ef90 tx=0x7fbdd000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbdd6800700 1 -- 172.21.15.23:0/1716036216 reap_dead start 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 shutdown_connections 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.313+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 >> 172.21.15.23:0/1716036216 conn(0x7fbdd8075d50 msgr2=0x7fbdd810ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.314+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 shutdown_connections 2024-03-30T04:00:37.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.314+0000 7fbddce00700 1 -- 172.21.15.23:0/1716036216 wait complete. 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.827+0000 7fccb3400700 1 -- 172.21.15.23:0/75226076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac069070 msgr2=0x7fccac10d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.827+0000 7fccb3400700 1 --2- 172.21.15.23:0/75226076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac069070 0x7fccac10d380 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcc98009990 tx=0x7fcc98009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 -- 172.21.15.23:0/75226076 shutdown_connections 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 --2- 172.21.15.23:0/75226076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac10d8c0 0x7fccac10fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 --2- 172.21.15.23:0/75226076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac068750 0x7fccac068b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 --2- 172.21.15.23:0/75226076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac069070 0x7fccac10d380 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 -- 172.21.15.23:0/75226076 >> 172.21.15.23:0/75226076 conn(0x7fccac075d50 msgr2=0x7fccac076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 -- 172.21.15.23:0/75226076 shutdown_connections 2024-03-30T04:00:37.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.828+0000 7fccb3400700 1 -- 172.21.15.23:0/75226076 wait complete. 2024-03-30T04:00:37.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.829+0000 7fccb3400700 1 Processor -- start 2024-03-30T04:00:37.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 -- start start 2024-03-30T04:00:37.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac068750 0x7fccac199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac069070 0x7fccac199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 0x7fccac19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac19e400 con 0x7fccac068750 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac19e570 con 0x7fccac069070 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.830+0000 7fccb3400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccac19e730 con 0x7fccac10d8c0 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccabe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac069070 0x7fccac199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 0x7fccac19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb0a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac068750 0x7fccac199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 0x7fccac19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44184/0 (socket says 172.21.15.23:44184) 2024-03-30T04:00:37.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 -- 172.21.15.23:0/1750142365 learned_addr learned my addr 172.21.15.23:0/1750142365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 -- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac069070 msgr2=0x7fccac199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccac069070 0x7fccac199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 -- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac068750 msgr2=0x7fccac199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccac068750 0x7fccac199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.831+0000 7fccb1400700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc980095f0 con 0x7fccac10d8c0 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fccb1400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 0x7fccac19ddf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ef90 tx=0x7fcca400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4018070 con 0x7fccac10d8c0 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcca400f040 con 0x7fccac10d8c0 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac19e9b0 con 0x7fccac10d8c0 2024-03-30T04:00:37.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400cda0 con 0x7fccac10d8c0 2024-03-30T04:00:37.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.832+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac1a6270 con 0x7fccac10d8c0 2024-03-30T04:00:37.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.833+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fccac113210 con 0x7fccac10d8c0 2024-03-30T04:00:37.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.836+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fcca4003a00 con 0x7fccac10d8c0 2024-03-30T04:00:37.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.837+0000 7fcca9600700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fcc940a11b0 0x7fcc940a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcca40cc910 con 0x7fccac10d8c0 2024-03-30T04:00:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7fccb0a00700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fcc940a11b0 0x7fcc940a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7fcca9600700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] conn(0x7fcc940ac1d0 0x7fcc940ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc940aed40 con 0x7fcc940ac1d0 2024-03-30T04:00:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcca40d12a0 con 0x7fccac10d8c0 2024-03-30T04:00:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7fccabe00700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] conn(0x7fcc940ac1d0 0x7fcc940ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7fccb0a00700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fcc940a11b0 0x7fcc940a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009c60 tx=0x7fcca0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.840+0000 7fccabe00700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] conn(0x7fcc940ac1d0 0x7fcc940ae670 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.841+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== osd.10 v2:172.21.15.154:6824/1797593825 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fcc940aed40 con 0x7fcc940ac1d0 2024-03-30T04:00:37.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.859+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 --> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fccac1002f0 con 0x7fcc940ac1d0 2024-03-30T04:00:37.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.860+0000 7fcca9600700 1 -- 172.21.15.23:0/1750142365 <== osd.10 v2:172.21.15.154:6824/1797593825 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fccac1002f0 con 0x7fcc940ac1d0 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.861+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] conn(0x7fcc940ac1d0 msgr2=0x7fcc940ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.861+0000 7fccb3400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] conn(0x7fcc940ac1d0 0x7fcc940ae670 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fcc940a11b0 msgr2=0x7fcc940a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fcc940a11b0 0x7fcc940a3670 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcca0009c60 tx=0x7fcca0009260 comp rx=0 tx=0).stop 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 msgr2=0x7fccac19ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 --2- 172.21.15.23:0/1750142365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccac10d8c0 0x7fccac19ddf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcca400ef90 tx=0x7fcca400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb1400700 1 -- 172.21.15.23:0/1750142365 reap_dead start 2024-03-30T04:00:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 shutdown_connections 2024-03-30T04:00:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 >> 172.21.15.23:0/1750142365 conn(0x7fccac075d50 msgr2=0x7fccac10ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 shutdown_connections 2024-03-30T04:00:37.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.862+0000 7fccb3400700 1 -- 172.21.15.23:0/1750142365 wait complete. 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 -- 172.21.15.23:0/778246178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4073900 msgr2=0x7f6af410d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 --2- 172.21.15.23:0/778246178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4073900 0x7f6af410d2d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0009990 tx=0x7f6ae0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 -- 172.21.15.23:0/778246178 shutdown_connections 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 --2- 172.21.15.23:0/778246178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af410d810 0x7f6af410fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 --2- 172.21.15.23:0/778246178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af4072fe0 0x7f6af40733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 --2- 172.21.15.23:0/778246178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4073900 0x7f6af410d2d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 -- 172.21.15.23:0/778246178 >> 172.21.15.23:0/778246178 conn(0x7f6af4077fa0 msgr2=0x7f6af40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 -- 172.21.15.23:0/778246178 shutdown_connections 2024-03-30T04:00:37.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.948+0000 7f6afaa00700 1 -- 172.21.15.23:0/778246178 wait complete. 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.949+0000 7f6afaa00700 1 Processor -- start 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.949+0000 7f6afaa00700 1 -- start start 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4072fe0 0x7f6af4199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4073900 0x7f6af41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 0x7f6af419dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af419e380 con 0x7f6af410d810 2024-03-30T04:00:37.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af419e4f0 con 0x7f6af4073900 2024-03-30T04:00:37.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6afaa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af419e6b0 con 0x7f6af4072fe0 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 0x7f6af419dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4072fe0 0x7f6af4199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 0x7f6af419dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34180/0 (socket says 172.21.15.23:34180) 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 -- 172.21.15.23:0/3786195427 learned_addr learned my addr 172.21.15.23:0/3786195427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af3400700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4073900 0x7f6af41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 -- 172.21.15.23:0/3786195427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4072fe0 msgr2=0x7f6af4199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4072fe0 0x7f6af4199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 -- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4073900 msgr2=0x7f6af41999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6af4073900 0x7f6af41999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af8a00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00095f0 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.950+0000 7f6af3e00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6af4072fe0 0x7f6af4199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6af8a00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 0x7f6af419dd70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00efc0 tx=0x7f6aec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec018070 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6aec00f040 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af419e930 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00cd00 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.951+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af41a61f0 con 0x7f6af410d810 2024-03-30T04:00:37.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.952+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6af41132f0 con 0x7f6af410d810 2024-03-30T04:00:37.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.954+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f6aec003a00 con 0x7f6af410d810 2024-03-30T04:00:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af0c00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6ad80a11b0 0x7f6ad80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6aec0cd1a0 con 0x7f6af410d810 2024-03-30T04:00:37.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af3e00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6ad80a11b0 0x7f6ad80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af0c00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] conn(0x7f6ad80ac1d0 0x7f6ad80ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ad80aed40 con 0x7f6ad80ac1d0 2024-03-30T04:00:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.956+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aec0cd510 con 0x7f6af410d810 2024-03-30T04:00:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.957+0000 7f6af3400700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] conn(0x7f6ad80ac1d0 0x7f6ad80ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.957+0000 7f6af3e00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6ad80a11b0 0x7f6ad80a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4011490 tx=0x7f6ae4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.957+0000 7f6af3400700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] conn(0x7f6ad80ac1d0 0x7f6ad80ae670 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.958+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== osd.7 v2:172.21.15.154:6816/4282776176 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6ad80aed40 con 0x7f6ad80ac1d0 2024-03-30T04:00:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.975+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 --> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6af4074670 con 0x7f6ad80ac1d0 2024-03-30T04:00:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.975+0000 7f6af0c00700 1 -- 172.21.15.23:0/3786195427 <== osd.7 v2:172.21.15.154:6816/4282776176 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6af4074670 con 0x7f6ad80ac1d0 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] conn(0x7f6ad80ac1d0 msgr2=0x7f6ad80ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] conn(0x7f6ad80ac1d0 0x7f6ad80ae670 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6ad80a11b0 msgr2=0x7f6ad80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6ad80a11b0 0x7f6ad80a3670 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ae4011490 tx=0x7f6ae4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 msgr2=0x7f6af419dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 --2- 172.21.15.23:0/3786195427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6af410d810 0x7f6af419dd70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00efc0 tx=0x7f6aec00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6af8a00700 1 -- 172.21.15.23:0/3786195427 reap_dead start 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 shutdown_connections 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 >> 172.21.15.23:0/3786195427 conn(0x7f6af4077fa0 msgr2=0x7f6af40fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 shutdown_connections 2024-03-30T04:00:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.976+0000 7f6afaa00700 1 -- 172.21.15.23:0/3786195427 wait complete. 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 -- 172.21.15.23:0/2018390567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8105cf0 msgr2=0x7f11b8103780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 --2- 172.21.15.23:0/2018390567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8105cf0 0x7f11b8103780 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009990 tx=0x7f11ac009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 -- 172.21.15.23:0/2018390567 shutdown_connections 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 --2- 172.21.15.23:0/2018390567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b8068bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 --2- 172.21.15.23:0/2018390567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8105cf0 0x7f11b8103780 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 --2- 172.21.15.23:0/2018390567 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8103cc0 0x7f11b8104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.791+0000 7f11c0600700 1 -- 172.21.15.23:0/2018390567 >> 172.21.15.23:0/2018390567 conn(0x7f11b8076db0 msgr2=0x7f11b80771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.792+0000 7f11c0600700 1 -- 172.21.15.23:0/2018390567 shutdown_connections 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.792+0000 7f11c0600700 1 -- 172.21.15.23:0/2018390567 wait complete. 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.793+0000 7f11c0600700 1 Processor -- start 2024-03-30T04:00:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.793+0000 7f11c0600700 1 -- start start 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8103cc0 0x7f11b819dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8105cf0 0x7f11b81a0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81a15b0 con 0x7f11b8105cf0 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81a1720 con 0x7f11b8103cc0 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11c0600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b81a18e0 con 0x7f11b8068740 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11bdc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11bdc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b819d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44156/0 (socket says 172.21.15.23:44156) 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11bdc00700 1 -- 172.21.15.23:0/868041983 learned_addr learned my addr 172.21.15.23:0/868041983 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.794+0000 7f11be600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8105cf0 0x7f11b81a0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bd200700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8103cc0 0x7f11b819dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 -- 172.21.15.23:0/868041983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8103cc0 msgr2=0x7f11b819dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11b8103cc0 0x7f11b819dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 -- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8105cf0 msgr2=0x7f11b81a0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11b8105cf0 0x7f11b81a0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac0095f0 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11bdc00700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b819d770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.795+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac01d070 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.796+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b81a1b60 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.796+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11ac022470 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.796+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac017650 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.796+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b81aa510 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.797+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11b8117580 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.799+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f11ac0225e0 con 0x7f11b8068740 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.800+0000 7f11b6000700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f11a00a1230 0x7f11a00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f11ac0ce9b0 con 0x7f11b8068740 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11bd200700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f11a00a1230 0x7f11a00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11b6000700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] conn(0x7f11a00ac240 0x7f11a00ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11a00aedb0 con 0x7f11a00ac240 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11ac0ced20 con 0x7f11b8068740 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11bd200700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f11a00a1230 0x7f11a00a36f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f11b819ecd0 tx=0x7f11a4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.801+0000 7f11be600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] conn(0x7f11a00ac240 0x7f11a00ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.802+0000 7f11be600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] conn(0x7f11a00ac240 0x7f11a00ae6e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.803+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== osd.2 v2:172.21.15.193:6800/304660967 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f11a00aedb0 con 0x7f11a00ac240 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.837+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 --> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11b8100260 con 0x7f11a00ac240 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.838+0000 7f11b6000700 1 -- 172.21.15.23:0/868041983 <== osd.2 v2:172.21.15.193:6800/304660967 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f11b8100260 con 0x7f11a00ac240 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] conn(0x7f11a00ac240 msgr2=0x7f11a00ae6e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] conn(0x7f11a00ac240 0x7f11a00ae6e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f11a00a1230 msgr2=0x7f11a00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f11a00a1230 0x7f11a00a36f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f11b819ecd0 tx=0x7f11a4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 msgr2=0x7f11b819d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 --2- 172.21.15.23:0/868041983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11b8068740 0x7f11b819d770 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac005740 comp rx=0 tx=0).stop 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11be600700 1 -- 172.21.15.23:0/868041983 reap_dead start 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 shutdown_connections 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 >> 172.21.15.23:0/868041983 conn(0x7f11b8076db0 msgr2=0x7f11b80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.839+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 shutdown_connections 2024-03-30T04:00:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:37.840+0000 7f11c0600700 1 -- 172.21.15.23:0/868041983 wait complete. 2024-03-30T04:00:38.146 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:38 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:00:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:38 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:00:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:38 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:00:38.651 INFO:teuthology.orchestra.run.smithi023.stdout:77309411385 2024-03-30T04:00:38.652 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-03-30T04:00:38.826 INFO:teuthology.orchestra.run.smithi023.stdout:292057776153 2024-03-30T04:00:38.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.10 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 -- 172.21.15.23:0/1830250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8074d30 msgr2=0x7fc6b8072fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/1830250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8074d30 0x7fc6b8072fe0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098f0 tx=0x7fc6a8009c00 comp rx=0 tx=0).stop 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 -- 172.21.15.23:0/1830250 shutdown_connections 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/1830250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8111b30 0x7fc6b8113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/1830250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8074d30 0x7fc6b8072fe0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/1830250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6b8073520 0x7fc6b80739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 -- 172.21.15.23:0/1830250 >> 172.21.15.23:0/1830250 conn(0x7fc6b8077fa0 msgr2=0x7fc6b80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:39.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 -- 172.21.15.23:0/1830250 shutdown_connections 2024-03-30T04:00:39.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.036+0000 7fc6bdc00700 1 -- 172.21.15.23:0/1830250 wait complete. 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 Processor -- start 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 -- start start 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8073520 0x7fc6b81a3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 0x7fc6b81a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6b8111b30 0x7fc6b819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b819ddc0 con 0x7fc6b8073520 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b819df30 con 0x7fc6b8111b30 2024-03-30T04:00:39.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.037+0000 7fc6bdc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b819e0f0 con 0x7fc6b8074d30 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 0x7fc6b81a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6b8111b30 0x7fc6b819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 0x7fc6b81a3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36594/0 (socket says 172.21.15.23:36594) 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 -- 172.21.15.23:0/2329792640 learned_addr learned my addr 172.21.15.23:0/2329792640 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6a00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8073520 0x7fc6b81a3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 -- 172.21.15.23:0/2329792640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6b8111b30 msgr2=0x7fc6b819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6b8111b30 0x7fc6b819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 -- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8073520 msgr2=0x7fc6b81a3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8073520 0x7fc6b81a3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6000700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a8009520 con 0x7fc6b8074d30 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.038+0000 7fc6b6a00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc6b8073520 0x7fc6b81a3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:39.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6b6000700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 0x7fc6b81a3b60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400e9a0 tx=0x7fc6a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400ca10 con 0x7fc6b8074d30 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6a400cb70 con 0x7fc6b8074d30 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4018680 con 0x7fc6b8074d30 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b819e3d0 con 0x7fc6b8074d30 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.039+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b81aa610 con 0x7fc6b8074d30 2024-03-30T04:00:39.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.040+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6b8117700 con 0x7fc6b8074d30 2024-03-30T04:00:39.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.042+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc6a400cce0 con 0x7fc6b8074d30 2024-03-30T04:00:39.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.043+0000 7fc6af400700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc69c0a11b0 0x7fc69c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.044+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc6a4014070 con 0x7fc6b8074d30 2024-03-30T04:00:39.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.044+0000 7fc6b6a00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc69c0a11b0 0x7fc69c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.044+0000 7fc6af400700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] conn(0x7fc69c0ac1d0 0x7fc69c0ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.044+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc69c0aed40 con 0x7fc69c0ac1d0 2024-03-30T04:00:39.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.044+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a40cd700 con 0x7fc6b8074d30 2024-03-30T04:00:39.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.045+0000 7fc6b6a00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc69c0a11b0 0x7fc69c0a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098c0 tx=0x7fc6a8019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.045+0000 7fc6b7400700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] conn(0x7fc69c0ac1d0 0x7fc69c0ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.045+0000 7fc6b7400700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] conn(0x7fc69c0ac1d0 0x7fc69c0ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.046+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== osd.9 v2:172.21.15.23:6826/2836554643 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fc69c0aed40 con 0x7fc69c0ac1d0 2024-03-30T04:00:39.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.065+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 --> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6b8111150 con 0x7fc69c0ac1d0 2024-03-30T04:00:39.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6af400700 1 -- 172.21.15.23:0/2329792640 <== osd.9 v2:172.21.15.23:6826/2836554643 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc6b8111150 con 0x7fc69c0ac1d0 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] conn(0x7fc69c0ac1d0 msgr2=0x7fc69c0ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] conn(0x7fc69c0ac1d0 0x7fc69c0ae670 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc69c0a11b0 msgr2=0x7fc69c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc69c0a11b0 0x7fc69c0a3670 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc6a80098c0 tx=0x7fc6a8019040 comp rx=0 tx=0).stop 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 msgr2=0x7fc6b81a3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 --2- 172.21.15.23:0/2329792640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc6b8074d30 0x7fc6b81a3b60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400e9a0 tx=0x7fc6a400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6b7400700 1 -- 172.21.15.23:0/2329792640 reap_dead start 2024-03-30T04:00:39.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 shutdown_connections 2024-03-30T04:00:39.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 >> 172.21.15.23:0/2329792640 conn(0x7fc6b8077fa0 msgr2=0x7fc6b8110800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:39.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.066+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 shutdown_connections 2024-03-30T04:00:39.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.067+0000 7fc6bdc00700 1 -- 172.21.15.23:0/2329792640 wait complete. 2024-03-30T04:00:39.175 INFO:teuthology.orchestra.run.smithi023.stdout:107374182451 2024-03-30T04:00:39.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-03-30T04:00:39.239 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:39 smithi023 ceph-mon[38254]: pgmap v267: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:39.320 INFO:teuthology.orchestra.run.smithi023.stdout:210453397540 2024-03-30T04:00:39.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-03-30T04:00:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:39 smithi154 ceph-mon[79161]: pgmap v267: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:39.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:39 smithi193 ceph-mon[79581]: pgmap v267: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:39.645 INFO:teuthology.orchestra.run.smithi023.stdout:270582939678 2024-03-30T04:00:39.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.9 2024-03-30T04:00:39.662 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.020+0000 7f2f8aa00700 1 -- 172.21.15.23:0/958224519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 msgr2=0x7f2f8410dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.020+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/958224519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f8410dc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009f90 tx=0x7f2f7c00b330 comp rx=0 tx=0).stop 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 -- 172.21.15.23:0/958224519 shutdown_connections 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/958224519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f8410dc30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/958224519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f841071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/958224519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 0x7f2f84068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 -- 172.21.15.23:0/958224519 >> 172.21.15.23:0/958224519 conn(0x7f2f84075d50 msgr2=0x7f2f84076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 -- 172.21.15.23:0/958224519 shutdown_connections 2024-03-30T04:00:39.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.021+0000 7f2f8aa00700 1 -- 172.21.15.23:0/958224519 wait complete. 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.022+0000 7f2f8aa00700 1 Processor -- start 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.022+0000 7f2f8aa00700 1 -- start start 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 0x7f2f84199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f84199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f8419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8419e400 con 0x7f2f84068750 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8419e570 con 0x7f2f840690f0 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f8aa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8419e730 con 0x7f2f84106df0 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f84199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f88a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f8419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f84199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38540/0 (socket says 172.21.15.23:38540) 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f88a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f8419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36570/0 (socket says 172.21.15.23:36570) 2024-03-30T04:00:39.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f83400700 1 -- 172.21.15.23:0/4038437080 learned_addr learned my addr 172.21.15.23:0/4038437080 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.023+0000 7f2f83e00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 0x7f2f84199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 -- 172.21.15.23:0/4038437080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 msgr2=0x7f2f84199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f84199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 -- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 msgr2=0x7f2f84199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 0x7f2f84199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c009c40 con 0x7f2f84106df0 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f83400700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f840690f0 0x7f2f84199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f83e00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2f84068750 0x7f2f84199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f88a00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f8419ddf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c015000 tx=0x7f2f7c009470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c00e040 con 0x7f2f84106df0 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f7c00bbf0 con 0x7f2f84106df0 2024-03-30T04:00:39.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.024+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c005040 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.025+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8419e9b0 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.025+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f841a6270 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.026+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f84110f30 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.028+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f2f7c01b450 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.029+0000 7f2f80c00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2f680a1230 0x7f2f680a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2f7c0ce350 con 0x7f2f84106df0 2024-03-30T04:00:39.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f80c00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] conn(0x7f2f680ac250 0x7f2f680ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f680aedc0 con 0x7f2f680ac250 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f7c0ce6c0 con 0x7f2f84106df0 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f83e00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2f680a1230 0x7f2f680a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.030+0000 7f2f83400700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] conn(0x7f2f680ac250 0x7f2f680ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.031+0000 7f2f83e00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2f680a1230 0x7f2f680a36f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2f74005fd0 tx=0x7f2f74009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.031+0000 7f2f83400700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] conn(0x7f2f680ac250 0x7f2f680ae6f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.032+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== osd.5 v2:172.21.15.193:6808/1684695578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2f680aedc0 con 0x7f2f680ac250 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.060+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 --> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f841024a0 con 0x7f2f680ac250 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f80c00700 1 -- 172.21.15.23:0/4038437080 <== osd.5 v2:172.21.15.193:6808/1684695578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2f841024a0 con 0x7f2f680ac250 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] conn(0x7f2f680ac250 msgr2=0x7f2f680ae6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] conn(0x7f2f680ac250 0x7f2f680ae6f0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2f680a1230 msgr2=0x7f2f680a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f2f680a1230 0x7f2f680a36f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2f74005fd0 tx=0x7f2f74009310 comp rx=0 tx=0).stop 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 msgr2=0x7f2f8419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f8aa00700 1 --2- 172.21.15.23:0/4038437080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f84106df0 0x7f2f8419ddf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c015000 tx=0x7f2f7c009470 comp rx=0 tx=0).stop 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.061+0000 7f2f88a00700 1 -- 172.21.15.23:0/4038437080 reap_dead start 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.062+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 shutdown_connections 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.062+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 >> 172.21.15.23:0/4038437080 conn(0x7f2f84075d50 msgr2=0x7f2f8410bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.062+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 shutdown_connections 2024-03-30T04:00:39.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:39.062+0000 7f2f8aa00700 1 -- 172.21.15.23:0/4038437080 wait complete. 2024-03-30T04:00:40.213 INFO:teuthology.orchestra.run.smithi023.stdout:158913789997 2024-03-30T04:00:40.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-03-30T04:00:40.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:40 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:40 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.465 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:40.473 INFO:teuthology.orchestra.run.smithi023.stdout:322122547222 2024-03-30T04:00:40.474 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph osd last-stat-seq osd.11 2024-03-30T04:00:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:40 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:40.719 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:41.073 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:41.326 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:41.354 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:41 smithi023 ceph-mon[38254]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:41.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:41 smithi154 ceph-mon[79161]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:41.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:41 smithi193 ceph-mon[79581]: pgmap v268: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:42.386 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:43.481 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:43 smithi023 ceph-mon[38254]: pgmap v269: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:43 smithi154 ceph-mon[79161]: pgmap v269: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:43 smithi193 ceph-mon[79581]: pgmap v269: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-30T04:00:43.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.412+0000 7f1964c00700 1 -- 172.21.15.23:0/1921665807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960068750 msgr2=0x7f1960068b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:43.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.412+0000 7f1964c00700 1 --2- 172.21.15.23:0/1921665807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960068750 0x7f1960068b30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1950009990 tx=0x7f1950009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 -- 172.21.15.23:0/1921665807 shutdown_connections 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 --2- 172.21.15.23:0/1921665807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 0x7f196010fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 --2- 172.21.15.23:0/1921665807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960068750 0x7f1960068b30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 --2- 172.21.15.23:0/1921665807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960069070 0x7f196010d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 -- 172.21.15.23:0/1921665807 >> 172.21.15.23:0/1921665807 conn(0x7f1960075d50 msgr2=0x7f1960076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 -- 172.21.15.23:0/1921665807 shutdown_connections 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.413+0000 7f1964c00700 1 -- 172.21.15.23:0/1921665807 wait complete. 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.414+0000 7f1964c00700 1 Processor -- start 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.414+0000 7f1964c00700 1 -- start start 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960068750 0x7f1960199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:43.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 0x7f196019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196019e290 con 0x7f1960069070 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196019e400 con 0x7f196010d8c0 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f1964c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196019e5c0 con 0x7f1960068750 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54808/0 (socket says 172.21.15.23:54808) 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 -- 172.21.15.23:0/2208071324 learned_addr learned my addr 172.21.15.23:0/2208071324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195de00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960068750 0x7f1960199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195e800700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 0x7f196019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 -- 172.21.15.23:0/2208071324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960068750 msgr2=0x7f1960199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960068750 0x7f1960199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 -- 172.21.15.23:0/2208071324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 msgr2=0x7f196019dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:43.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 0x7f196019dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.415+0000 7f195d400700 1 -- 172.21.15.23:0/2208071324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19500095f0 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f195d400700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f194c00e9a0 tx=0x7f194c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c009790 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f194c010430 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196019e8a0 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194c018680 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.416+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19601a6100 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.417+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1960108ae0 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.419+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f194c0105a0 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.420+0000 7f1956a00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 0x7f19440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.420+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f194c014070 con 0x7f1960069070 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.421+0000 7f195de00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 0x7f19440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.421+0000 7f195de00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 0x7f19440a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000b260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:43.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.425+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f194c0529f0 con 0x7f1960069070 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stdout:77309411386 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.656+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f19601002f0 con 0x7f1960069070 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.656+0000 7f1956a00700 1 -- 172.21.15.23:0/2208071324 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f194c096250 con 0x7f1960069070 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 msgr2=0x7f19440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 0x7f19440a3670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1950006010 tx=0x7f195000b260 comp rx=0 tx=0).stop 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 msgr2=0x7f19601998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f194c00e9a0 tx=0x7f194c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 shutdown_connections 2024-03-30T04:00:43.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196010d8c0 0x7f196019dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f19440a11b0 0x7f19440a3670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1960068750 0x7f1960199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 --2- 172.21.15.23:0/2208071324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1960069070 0x7f19601998d0 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f194c00e9a0 tx=0x7f194c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.659+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 >> 172.21.15.23:0/2208071324 conn(0x7f1960075d50 msgr2=0x7f196010ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.660+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 shutdown_connections 2024-03-30T04:00:43.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.660+0000 7f1964c00700 1 -- 172.21.15.23:0/2208071324 wait complete. 2024-03-30T04:00:43.809 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 -- 172.21.15.23:0/2579715833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254106ed0 msgr2=0x7f72541072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 --2- 172.21.15.23:0/2579715833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254106ed0 0x7f72541072b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7244009990 tx=0x7f7244009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 -- 172.21.15.23:0/2579715833 shutdown_connections 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 --2- 172.21.15.23:0/2579715833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254101820 0x7f725410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 --2- 172.21.15.23:0/2579715833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254106ed0 0x7f72541072b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 --2- 172.21.15.23:0/2579715833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254100e60 0x7f72541012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.721+0000 7f725a600700 1 -- 172.21.15.23:0/2579715833 >> 172.21.15.23:0/2579715833 conn(0x7f7254077f60 msgr2=0x7f7254078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.722+0000 7f725a600700 1 -- 172.21.15.23:0/2579715833 shutdown_connections 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.722+0000 7f725a600700 1 -- 172.21.15.23:0/2579715833 wait complete. 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 Processor -- start 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 -- start start 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254101820 0x7f7254197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254106ed0 0x7f725419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725419c0a0 con 0x7f7254100e60 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725419c210 con 0x7f7254106ed0 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.723+0000 7f725a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725419c3d0 con 0x7f7254101820 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54812/0 (socket says 172.21.15.23:54812) 2024-03-30T04:00:44.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 -- 172.21.15.23:0/4077593774 learned_addr learned my addr 172.21.15.23:0/4077593774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253e00700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254106ed0 0x7f725419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7252a00700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254101820 0x7f7254197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 -- 172.21.15.23:0/4077593774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254101820 msgr2=0x7f7254197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254101820 0x7f7254197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 -- 172.21.15.23:0/4077593774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254106ed0 msgr2=0x7f725419ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254106ed0 0x7f725419ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 -- 172.21.15.23:0/4077593774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240009520 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.724+0000 7f7253400700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7244005910 tx=0x7f7244004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.725+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724401d070 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.725+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7244022470 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.725+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72440095f0 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.725+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7244017650 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.725+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725419c8d0 con 0x7f7254100e60 2024-03-30T04:00:44.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.726+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72541092f0 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.728+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f72440225e0 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.730+0000 7f724be00700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 0x7f72380a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.730+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f72440cde30 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.730+0000 7f7252a00700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 0x7f72380a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.730+0000 7f7252a00700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 0x7f72380a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7254198700 tx=0x7f7240009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.738+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f72440971e0 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.982+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f725404eab0 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.983+0000 7f724be00700 1 -- 172.21.15.23:0/4077593774 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f724409a800 con 0x7f7254100e60 2024-03-30T04:00:44.070 INFO:teuthology.orchestra.run.smithi023.stdout:137438953522 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 msgr2=0x7f72380a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 0x7f72380a3670 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7254198700 tx=0x7f7240009260 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 msgr2=0x7f72541971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7244005910 tx=0x7f7244004730 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 shutdown_connections 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7254106ed0 0x7f725419ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7254100e60 0x7f72541971f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f72380a11b0 0x7f72380a3670 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 --2- 172.21.15.23:0/4077593774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7254101820 0x7f7254197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:44.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 >> 172.21.15.23:0/4077593774 conn(0x7f7254077f60 msgr2=0x7f72540ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:44.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.986+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 shutdown_connections 2024-03-30T04:00:44.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:43.987+0000 7f725a600700 1 -- 172.21.15.23:0/4077593774 wait complete. 2024-03-30T04:00:44.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2208071324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T04:00:44.605 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4077593774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T04:00:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2208071324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T04:00:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4077593774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T04:00:44.736 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411385 got 77309411386 for osd.2 2024-03-30T04:00:44.736 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2208071324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-30T04:00:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4077593774' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-30T04:00:44.763 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:44.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953520 got 137438953522 for osd.4 2024-03-30T04:00:44.824 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 -- 172.21.15.23:0/1382603033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1460106df0 msgr2=0x7f14601071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 --2- 172.21.15.23:0/1382603033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1460106df0 0x7f14601071d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1450009990 tx=0x7f1450009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 -- 172.21.15.23:0/1382603033 shutdown_connections 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 --2- 172.21.15.23:0/1382603033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14600690f0 0x7f146010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 --2- 172.21.15.23:0/1382603033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1460106df0 0x7f14601071d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 --2- 172.21.15.23:0/1382603033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460068bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 -- 172.21.15.23:0/1382603033 >> 172.21.15.23:0/1382603033 conn(0x7f1460075d50 msgr2=0x7f1460076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 -- 172.21.15.23:0/1382603033 shutdown_connections 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.006+0000 7f1465400700 1 -- 172.21.15.23:0/1382603033 wait complete. 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.007+0000 7f1465400700 1 Processor -- start 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.007+0000 7f1465400700 1 -- start start 2024-03-30T04:00:45.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 0x7f1460197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 0x7f146019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146019c150 con 0x7f14600690f0 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146019c2c0 con 0x7f1460106df0 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f1465400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146019c480 con 0x7f1460068750 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 0x7f1460197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460197250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36670/0 (socket says 172.21.15.23:36670) 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145f200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 0x7f146019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145e800700 1 -- 172.21.15.23:0/4093397209 learned_addr learned my addr 172.21.15.23:0/4093397209 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.008+0000 7f145de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 0x7f1460197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54838/0 (socket says 172.21.15.23:54838) 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 -- 172.21.15.23:0/4093397209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 msgr2=0x7f1460197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 -- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 msgr2=0x7f1460197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 0x7f1460197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 -- 172.21.15.23:0/4093397209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458009c40 con 0x7f1460106df0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.009+0000 7f145f200700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 0x7f146019bb40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f145800efb0 tx=0x7f145800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.010+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458003a00 con 0x7f1460106df0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.010+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14500095f0 con 0x7f1460106df0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.010+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14600ffd10 con 0x7f1460106df0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.010+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f145800f040 con 0x7f1460106df0 2024-03-30T04:00:45.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.010+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004540 con 0x7f1460106df0 2024-03-30T04:00:45.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.011+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14400052f0 con 0x7f1460106df0 2024-03-30T04:00:45.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.013+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f1458004780 con 0x7f1460106df0 2024-03-30T04:00:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.014+0000 7f1457400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 0x7f14440a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.015+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f14580cc900 con 0x7f1460106df0 2024-03-30T04:00:45.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.015+0000 7f145e800700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 0x7f14440a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.016+0000 7f145e800700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 0x7f14440a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1450006010 tx=0x7f145000b280 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.019+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1458095c50 con 0x7f1460106df0 2024-03-30T04:00:45.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.290+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1440005160 con 0x7f1460106df0 2024-03-30T04:00:45.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f1457400700 1 -- 172.21.15.23:0/4093397209 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1458099270 con 0x7f1460106df0 2024-03-30T04:00:45.290 INFO:teuthology.orchestra.run.smithi023.stdout:188978561066 2024-03-30T04:00:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 msgr2=0x7f14440a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 0x7f14440a36f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1450006010 tx=0x7f145000b280 comp rx=0 tx=0).stop 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 msgr2=0x7f146019bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 0x7f146019bb40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f145800efb0 tx=0x7f145800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 shutdown_connections 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1460106df0 0x7f146019bb40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f14440a1230 0x7f14440a36f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1460068750 0x7f1460197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 --2- 172.21.15.23:0/4093397209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14600690f0 0x7f1460197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 >> 172.21.15.23:0/4093397209 conn(0x7f1460075d50 msgr2=0x7f14600febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 shutdown_connections 2024-03-30T04:00:45.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.294+0000 7f1465400700 1 -- 172.21.15.23:0/4093397209 wait complete. 2024-03-30T04:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 -- 172.21.15.23:0/1477580935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2540068750 msgr2=0x7f2540068b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 --2- 172.21.15.23:0/1477580935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2540068750 0x7f2540068b30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2530009990 tx=0x7f2530009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 -- 172.21.15.23:0/1477580935 shutdown_connections 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 --2- 172.21.15.23:0/1477580935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f254010d8c0 0x7f254010fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 --2- 172.21.15.23:0/1477580935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2540068750 0x7f2540068b30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 --2- 172.21.15.23:0/1477580935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540069070 0x7f254010d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 -- 172.21.15.23:0/1477580935 >> 172.21.15.23:0/1477580935 conn(0x7f2540075d50 msgr2=0x7f2540076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 -- 172.21.15.23:0/1477580935 shutdown_connections 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.986+0000 7f2545000700 1 -- 172.21.15.23:0/1477580935 wait complete. 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.987+0000 7f2545000700 1 Processor -- start 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 -- start start 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 0x7f25401999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f254010d8c0 0x7f254019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254019e340 con 0x7f2540068750 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254019e4b0 con 0x7f2540069070 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f2545000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254019e670 con 0x7f254010d8c0 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.988+0000 7f253de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54820/0 (socket says 172.21.15.23:54820) 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 -- 172.21.15.23:0/4088508714 learned_addr learned my addr 172.21.15.23:0/4088508714 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253d400700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 0x7f25401999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253e800700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f254010d8c0 0x7f254019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 -- 172.21.15.23:0/4088508714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f254010d8c0 msgr2=0x7f254019dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f254010d8c0 0x7f254019dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 -- 172.21.15.23:0/4088508714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 msgr2=0x7f25401999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 0x7f25401999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 -- 172.21.15.23:0/4088508714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25300095f0 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253d400700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 0x7f25401999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.989+0000 7f253de00700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2530000c00 tx=0x7f2530005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.990+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253001d070 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.990+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254019e8f0 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.990+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25401a61b0 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.990+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2530022470 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.990+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530017650 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.991+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251c0052f0 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.993+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f253000fb00 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.995+0000 7f2536a00700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 0x7f25240a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.995+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f25300ce650 con 0x7f2540068750 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.996+0000 7f253d400700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 0x7f25240a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.997+0000 7f253d400700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 0x7f25240a3670 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f254019a9a0 tx=0x7f2528009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:44.999+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f25300979a0 con 0x7f2540068750 2024-03-30T04:00:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.288+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f251c005160 con 0x7f2540068750 2024-03-30T04:00:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.288+0000 7f2536a00700 1 -- 172.21.15.23:0/4088508714 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f253009afc0 con 0x7f2540068750 2024-03-30T04:00:45.513 INFO:teuthology.orchestra.run.smithi023.stdout:55834574909 2024-03-30T04:00:45.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 msgr2=0x7f25240a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 0x7f25240a3670 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f254019a9a0 tx=0x7f2528009260 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 msgr2=0x7f2540199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2530000c00 tx=0x7f2530005740 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 shutdown_connections 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f25240a11b0 0x7f25240a3670 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f254010d8c0 0x7f254019dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540068750 0x7f2540199490 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 --2- 172.21.15.23:0/4088508714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2540069070 0x7f25401999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.291+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 >> 172.21.15.23:0/4088508714 conn(0x7f2540075d50 msgr2=0x7f254010ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.292+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 shutdown_connections 2024-03-30T04:00:45.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.292+0000 7f2545000700 1 -- 172.21.15.23:0/4088508714 wait complete. 2024-03-30T04:00:45.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 -- 172.21.15.23:0/2504394133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 msgr2=0x7fd2cc0739a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 --2- 172.21.15.23:0/2504394133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc0739a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009990 tx=0x7fd2b8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 -- 172.21.15.23:0/2504394133 shutdown_connections 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 --2- 172.21.15.23:0/2504394133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 0x7fd2cc113f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 --2- 172.21.15.23:0/2504394133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 0x7fd2cc072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 --2- 172.21.15.23:0/2504394133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc0739a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 -- 172.21.15.23:0/2504394133 >> 172.21.15.23:0/2504394133 conn(0x7fd2cc077fa0 msgr2=0x7fd2cc0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 -- 172.21.15.23:0/2504394133 shutdown_connections 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.569+0000 7fd2d2400700 1 -- 172.21.15.23:0/2504394133 wait complete. 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.570+0000 7fd2d2400700 1 Processor -- start 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.570+0000 7fd2d2400700 1 -- start start 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 0x7fd2cc1a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 0x7fd2cc19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc19dd00 con 0x7fd2cc073520 2024-03-30T04:00:45.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc19de70 con 0x7fd2cc111b30 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2d2400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc19e030 con 0x7fd2cc074d30 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2cb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2cb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54846/0 (socket says 172.21.15.23:54846) 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2cb400700 1 -- 172.21.15.23:0/1554720873 learned_addr learned my addr 172.21.15.23:0/1554720873 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2caa00700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 0x7fd2cc1a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.571+0000 7fd2cbe00700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 0x7fd2cc19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 -- 172.21.15.23:0/1554720873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 msgr2=0x7fd2cc1a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 0x7fd2cc1a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 -- 172.21.15.23:0/1554720873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 msgr2=0x7fd2cc19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 0x7fd2cc19d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 -- 172.21.15.23:0/1554720873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80095f0 con 0x7fd2cc073520 2024-03-30T04:00:45.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2cb400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00e9a0 tx=0x7fd2bc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc009790 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2bc00cbd0 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc19e310 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc018680 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.572+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc1aa500 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.573+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2cc074670 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.577+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd2bc00cd40 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.578+0000 7fd2c3e00700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 0x7fd2b00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.578+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd2bc014070 con 0x7fd2cc073520 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.578+0000 7fd2caa00700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 0x7fd2b00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.579+0000 7fd2caa00700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 0x7fd2b00a3670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006010 tx=0x7fd2b800b260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.582+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd2bc0529f0 con 0x7fd2cc073520 2024-03-30T04:00:45.679 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:45 smithi023 ceph-mon[38254]: pgmap v270: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:45.679 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4088508714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T04:00:45.679 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4093397209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T04:00:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:45 smithi154 ceph-mon[79161]: pgmap v270: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4088508714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T04:00:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4093397209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T04:00:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:45 smithi193 ceph-mon[79581]: pgmap v270: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-30T04:00:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4088508714' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-30T04:00:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4093397209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-30T04:00:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.823+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd2cc04ea80 con 0x7fd2cc073520 2024-03-30T04:00:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.824+0000 7fd2c3e00700 1 -- 172.21.15.23:0/1554720873 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd2bc099e50 con 0x7fd2cc073520 2024-03-30T04:00:45.823 INFO:teuthology.orchestra.run.smithi023.stdout:34359738433 2024-03-30T04:00:45.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.826+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 msgr2=0x7fd2b00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.826+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 0x7fd2b00a3670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8006010 tx=0x7fd2b800b260 comp rx=0 tx=0).stop 2024-03-30T04:00:45.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.826+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 msgr2=0x7fd2cc1a3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.826+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00e9a0 tx=0x7fd2bc00ed60 comp rx=0 tx=0).stop 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 shutdown_connections 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2cc111b30 0x7fd2cc19d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2cc073520 0x7fd2cc1a3510 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2b00a11b0 0x7fd2b00a3670 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 --2- 172.21.15.23:0/1554720873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2cc074d30 0x7fd2cc1a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 >> 172.21.15.23:0/1554720873 conn(0x7fd2cc077fa0 msgr2=0x7fd2cc1107a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 shutdown_connections 2024-03-30T04:00:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.827+0000 7fd2d2400700 1 -- 172.21.15.23:0/1554720873 wait complete. 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 -- 172.21.15.23:0/4070246194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c01017c0 msgr2=0x7f75c0105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 --2- 172.21.15.23:0/4070246194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c01017c0 0x7f75c0105810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009990 tx=0x7f75ac009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 -- 172.21.15.23:0/4070246194 shutdown_connections 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 --2- 172.21.15.23:0/4070246194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c0105ef0 0x7f75c010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 --2- 172.21.15.23:0/4070246194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c01011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 --2- 172.21.15.23:0/4070246194 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c01017c0 0x7f75c0105810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 -- 172.21.15.23:0/4070246194 >> 172.21.15.23:0/4070246194 conn(0x7f75c00fc6e0 msgr2=0x7f75c00feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 -- 172.21.15.23:0/4070246194 shutdown_connections 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.637+0000 7f75bf200700 1 -- 172.21.15.23:0/4070246194 wait complete. 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 Processor -- start 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 -- start start 2024-03-30T04:00:45.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 0x7f75c00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c0105ef0 0x7f75c0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01a5490 con 0x7f75c01017c0 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01a5600 con 0x7f75c0105ef0 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.638+0000 7f75bf200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c01a57c0 con 0x7f75c0100e10 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 0x7f75c00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36696/0 (socket says 172.21.15.23:36696) 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 -- 172.21.15.23:0/1735287979 learned_addr learned my addr 172.21.15.23:0/1735287979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75be800700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c0105ef0 0x7f75c0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 -- 172.21.15.23:0/1735287979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c0105ef0 msgr2=0x7f75c0073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c0105ef0 0x7f75c0073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 -- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 msgr2=0x7f75c00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 0x7f75c00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bde00700 1 -- 172.21.15.23:0/1735287979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac0095f0 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.639+0000 7f75bd400700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 0x7f75c00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75bde00700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9a0 tx=0x7f75b000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0009790 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c01a5aa0 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f75b0010430 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0018680 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.640+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c01a5fc0 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.641+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c01091b0 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.643+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f75b00105a0 con 0x7f75c0100e10 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.644+0000 7f75b6a00700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 0x7f75a40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:45.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.644+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f75b0014070 con 0x7f75c0100e10 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.645+0000 7f75bd400700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 0x7f75a40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.645+0000 7f75bd400700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 0x7f75a40a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00b2e0 tx=0x7f75ac005960 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.649+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f75b00529f0 con 0x7f75c0100e10 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.897+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f75c01022f0 con 0x7f75c0100e10 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.897+0000 7f75b6a00700 1 -- 172.21.15.23:0/1735287979 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f75b00961d0 con 0x7f75c0100e10 2024-03-30T04:00:45.904 INFO:teuthology.orchestra.run.smithi023.stdout:240518168611 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 msgr2=0x7f75a40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 0x7f75a40a36f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00b2e0 tx=0x7f75ac005960 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 msgr2=0x7f75c00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75b000e9a0 tx=0x7f75b000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 shutdown_connections 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f75c0105ef0 0x7f75c0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f75a40a1230 0x7f75a40a36f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c0100e10 0x7f75c00730b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 --2- 172.21.15.23:0/1735287979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75c01017c0 0x7f75c00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 >> 172.21.15.23:0/1735287979 conn(0x7f75c00fc6e0 msgr2=0x7f75c00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 shutdown_connections 2024-03-30T04:00:45.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:45.900+0000 7f75bf200700 1 -- 172.21.15.23:0/1735287979 wait complete. 2024-03-30T04:00:46.210 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:46.331 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:46.457 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:46.464 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1554720873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T04:00:46.464 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1735287979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T04:00:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1554720873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T04:00:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1735287979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T04:00:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1554720873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-30T04:00:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1735287979' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-30T04:00:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:47 smithi023 ceph-mon[38254]: pgmap v271: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:47.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168609 got 240518168611 for osd.8 2024-03-30T04:00:47.554 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:47.599 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561063 got 188978561066 for osd.6 2024-03-30T04:00:47.599 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:47 smithi154 ceph-mon[79161]: pgmap v271: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:47.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:47 smithi193 ceph-mon[79581]: pgmap v271: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 -- 172.21.15.23:0/3948240711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488068750 msgr2=0x7fc488068bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 --2- 172.21.15.23:0/3948240711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488068750 0x7fc488068bb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc474009990 tx=0x7fc474009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 -- 172.21.15.23:0/3948240711 shutdown_connections 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 --2- 172.21.15.23:0/3948240711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc48810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 --2- 172.21.15.23:0/3948240711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488106e10 0x7fc4881071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 --2- 172.21.15.23:0/3948240711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488068750 0x7fc488068bb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 -- 172.21.15.23:0/3948240711 >> 172.21.15.23:0/3948240711 conn(0x7fc488075d50 msgr2=0x7fc488076160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 -- 172.21.15.23:0/3948240711 shutdown_connections 2024-03-30T04:00:47.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.344+0000 7fc48ee00700 1 -- 172.21.15.23:0/3948240711 wait complete. 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.346+0000 7fc48ee00700 1 Processor -- start 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.346+0000 7fc48ee00700 1 -- start start 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc488100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488106e10 0x7fc488100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881a55c0 con 0x7fc4880690f0 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881a5730 con 0x7fc488068750 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ee00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881a58f0 con 0x7fc488106e10 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc487e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc487400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc488100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc487e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38704/0 (socket says 172.21.15.23:38704) 2024-03-30T04:00:47.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc487e00700 1 -- 172.21.15.23:0/439422916 learned_addr learned my addr 172.21.15.23:0/439422916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.347+0000 7fc48ce00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488106e10 0x7fc488100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 -- 172.21.15.23:0/439422916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488106e10 msgr2=0x7fc488100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488106e10 0x7fc488100860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 -- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 msgr2=0x7fc488100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc488100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 -- 172.21.15.23:0/439422916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4740095f0 con 0x7fc488068750 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487400700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc488100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc487e00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc47800d950 tx=0x7fc47800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.348+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800f890 con 0x7fc488068750 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.349+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4881a5bd0 con 0x7fc488068750 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.349+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc478010460 con 0x7fc488068750 2024-03-30T04:00:47.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.349+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800e610 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.349+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4881a60d0 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.350+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc488109330 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.352+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc4780108d0 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.354+0000 7fc484c00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 0x7fc46c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.354+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc4780ccac0 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.354+0000 7fc487400700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 0x7fc46c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.355+0000 7fc487400700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 0x7fc46c0a36f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc488101510 tx=0x7fc474005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.356+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc478095d80 con 0x7fc488068750 2024-03-30T04:00:47.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.596+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fc4881024c0 con 0x7fc488068750 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stdout:292057776155 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.597+0000 7fc484c00700 1 -- 172.21.15.23:0/439422916 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fc4780993a0 con 0x7fc488068750 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.599+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 msgr2=0x7fc46c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.599+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 0x7fc46c0a36f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc488101510 tx=0x7fc474005fd0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.599+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 msgr2=0x7fc4880ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.599+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc47800d950 tx=0x7fc47800dc60 comp rx=0 tx=0).stop 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 shutdown_connections 2024-03-30T04:00:47.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc46c0a1230 0x7fc46c0a36f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc488106e10 0x7fc488100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc488068750 0x7fc4880ffde0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 --2- 172.21.15.23:0/439422916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4880690f0 0x7fc488100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 >> 172.21.15.23:0/439422916 conn(0x7fc488075d50 msgr2=0x7fc4880febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 shutdown_connections 2024-03-30T04:00:47.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:47.600+0000 7fc48ee00700 1 -- 172.21.15.23:0/439422916 wait complete. 2024-03-30T04:00:48.259 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574907 got 55834574909 for osd.1 2024-03-30T04:00:48.259 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:48.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 -- 172.21.15.23:0/1664216121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 msgr2=0x7fd19c107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1664216121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c107bc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd188009990 tx=0x7fd188009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 -- 172.21.15.23:0/1664216121 shutdown_connections 2024-03-30T04:00:48.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1664216121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 0x7fd19c10fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1664216121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1664216121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c107bc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 -- 172.21.15.23:0/1664216121 >> 172.21.15.23:0/1664216121 conn(0x7fd19c0fea70 msgr2=0x7fd19c100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:48.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 -- 172.21.15.23:0/1664216121 shutdown_connections 2024-03-30T04:00:48.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.343+0000 7fd1a1200700 1 -- 172.21.15.23:0/1664216121 wait complete. 2024-03-30T04:00:48.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.344+0000 7fd1a1200700 1 Processor -- start 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.344+0000 7fd1a1200700 1 -- start start 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 0x7fd19c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c19e310 con 0x7fd19c103b70 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c19e480 con 0x7fd19c1031c0 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd1a1200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c19e640 con 0x7fd19c1082a0 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd199400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd199400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37050/0 (socket says 172.21.15.23:37050) 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd199e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd199400700 1 -- 172.21.15.23:0/1619890260 learned_addr learned my addr 172.21.15.23:0/1619890260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.345+0000 7fd19a800700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 0x7fd19c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 -- 172.21.15.23:0/1619890260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 msgr2=0x7fd19c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 0x7fd19c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 -- 172.21.15.23:0/1619890260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 msgr2=0x7fd19c199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 -- 172.21.15.23:0/1619890260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c009520 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199e00700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd199400700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd188000c00 tx=0x7fd188004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18801d070 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd188021410 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188017650 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1880095f0 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.346+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c19eaf0 con 0x7fd19c103b70 2024-03-30T04:00:48.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.347+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19c10b5c0 con 0x7fd19c103b70 2024-03-30T04:00:48.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.350+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd188021580 con 0x7fd19c103b70 2024-03-30T04:00:48.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.351+0000 7fd192a00700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 0x7fd1800a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.352+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd1880ce680 con 0x7fd19c103b70 2024-03-30T04:00:48.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.352+0000 7fd199e00700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 0x7fd1800a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.352+0000 7fd199e00700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 0x7fd1800a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009fd0 tx=0x7fd18c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:48.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.354+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd1880979d0 con 0x7fd19c103b70 2024-03-30T04:00:48.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738433 for osd.0 2024-03-30T04:00:48.448 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:48.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.590+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd19c1046a0 con 0x7fd19c103b70 2024-03-30T04:00:48.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.591+0000 7fd192a00700 1 -- 172.21.15.23:0/1619890260 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd18809aff0 con 0x7fd19c103b70 2024-03-30T04:00:48.590 INFO:teuthology.orchestra.run.smithi023.stdout:107374182453 2024-03-30T04:00:48.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 msgr2=0x7fd1800a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 0x7fd1800a3670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009fd0 tx=0x7fd18c009260 comp rx=0 tx=0).stop 2024-03-30T04:00:48.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 msgr2=0x7fd19c199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd188000c00 tx=0x7fd188004730 comp rx=0 tx=0).stop 2024-03-30T04:00:48.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 shutdown_connections 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd1800a11b0 0x7fd1800a3670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd19c1082a0 0x7fd19c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c1031c0 0x7fd19c199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 --2- 172.21.15.23:0/1619890260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd19c103b70 0x7fd19c199950 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.594+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 >> 172.21.15.23:0/1619890260 conn(0x7fd19c0fea70 msgr2=0x7fd19c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.595+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 shutdown_connections 2024-03-30T04:00:48.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.595+0000 7fd1a1200700 1 -- 172.21.15.23:0/1619890260 wait complete. 2024-03-30T04:00:48.608 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:48 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/439422916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T04:00:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:48 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/439422916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T04:00:48.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:48 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/439422916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-30T04:00:48.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 -- 172.21.15.23:0/2077769697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 msgr2=0x7fd6e8103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 --2- 172.21.15.23:0/2077769697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8103640 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8009990 tx=0x7fd6d8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 -- 172.21.15.23:0/2077769697 shutdown_connections 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 --2- 172.21.15.23:0/2077769697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8108340 0x7fd6e810ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 --2- 172.21.15.23:0/2077769697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8103640 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 --2- 172.21.15.23:0/2077769697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8103c10 0x7fd6e8107c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 -- 172.21.15.23:0/2077769697 >> 172.21.15.23:0/2077769697 conn(0x7fd6e80feb50 msgr2=0x7fd6e8100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 -- 172.21.15.23:0/2077769697 shutdown_connections 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.869+0000 7fd6eca00700 1 -- 172.21.15.23:0/2077769697 wait complete. 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.870+0000 7fd6eca00700 1 Processor -- start 2024-03-30T04:00:48.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 -- start start 2024-03-30T04:00:48.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 0x7fd6e81999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e819e380 con 0x7fd6e8103260 2024-03-30T04:00:48.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e819e4f0 con 0x7fd6e8108340 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6eca00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e819e6b0 con 0x7fd6e8103c10 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6e5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6e6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6e5400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 0x7fd6e81999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:53996/0 (socket says 172.21.15.23:53996) 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.871+0000 7fd6e5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37070/0 (socket says 172.21.15.23:37070) 2024-03-30T04:00:48.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e6800700 1 -- 172.21.15.23:0/3412287733 learned_addr learned my addr 172.21.15.23:0/3412287733 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 -- 172.21.15.23:0/3412287733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 msgr2=0x7fd6e819dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 -- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 msgr2=0x7fd6e8199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 -- 172.21.15.23:0/3412287733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d80095f0 con 0x7fd6e8103c10 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e6800700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:48.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.872+0000 7fd6e5400700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 0x7fd6e81999c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e9c0 tx=0x7fd6d400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.873+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400c9c0 con 0x7fd6e8103c10 2024-03-30T04:00:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.873+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6d4010430 con 0x7fd6e8103c10 2024-03-30T04:00:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.873+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e819e990 con 0x7fd6e8103c10 2024-03-30T04:00:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.873+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4018680 con 0x7fd6e8103c10 2024-03-30T04:00:48.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.873+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e81a61f0 con 0x7fd6e8103c10 2024-03-30T04:00:48.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.874+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e810b600 con 0x7fd6e8103c10 2024-03-30T04:00:48.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.876+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd6d400cb20 con 0x7fd6e8103c10 2024-03-30T04:00:48.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.877+0000 7fd6dea00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 0x7fd6cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:48.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.877+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd6d4014070 con 0x7fd6e8103c10 2024-03-30T04:00:48.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.878+0000 7fd6e5e00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 0x7fd6cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:48.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.878+0000 7fd6e5e00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 0x7fd6cc0a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b2e0 tx=0x7fd6d8005960 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:48.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:48.880+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd6d40961e0 con 0x7fd6e8103c10 2024-03-30T04:00:48.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776153 got 292057776155 for osd.10 2024-03-30T04:00:48.963 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:49.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.115+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd6e8104740 con 0x7fd6e8103c10 2024-03-30T04:00:49.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.116+0000 7fd6dea00700 1 -- 172.21.15.23:0/3412287733 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd6d4099800 con 0x7fd6e8103c10 2024-03-30T04:00:49.115 INFO:teuthology.orchestra.run.smithi023.stdout:158913789998 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 msgr2=0x7fd6cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 0x7fd6cc0a3670 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b2e0 tx=0x7fd6d8005960 comp rx=0 tx=0).stop 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 msgr2=0x7fd6e81999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 0x7fd6e81999c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400e9c0 tx=0x7fd6d400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 shutdown_connections 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e8108340 0x7fd6e819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6e8103260 0x7fd6e8199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd6cc0a11b0 0x7fd6cc0a3670 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 --2- 172.21.15.23:0/3412287733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6e8103c10 0x7fd6e81999c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.118+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 >> 172.21.15.23:0/3412287733 conn(0x7fd6e80feb50 msgr2=0x7fd6e80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:49.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.119+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 shutdown_connections 2024-03-30T04:00:49.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.119+0000 7fd6eca00700 1 -- 172.21.15.23:0/3412287733 wait complete. 2024-03-30T04:00:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:49 smithi023 ceph-mon[38254]: pgmap v272: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:49.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:49 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1619890260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T04:00:49.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:49 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3412287733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T04:00:49.560 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182451 got 107374182453 for osd.3 2024-03-30T04:00:49.561 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:49 smithi154 ceph-mon[79161]: pgmap v272: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:49 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1619890260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T04:00:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:49 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3412287733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T04:00:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:49 smithi193 ceph-mon[79581]: pgmap v272: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:49 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1619890260' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-30T04:00:49.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:49 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3412287733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-30T04:00:49.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 -- 172.21.15.23:0/461396355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf241017c0 msgr2=0x7fbf24105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:49.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 --2- 172.21.15.23:0/461396355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf241017c0 0x7fbf24105810 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009990 tx=0x7fbf10009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 -- 172.21.15.23:0/461396355 shutdown_connections 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 --2- 172.21.15.23:0/461396355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 0x7fbf2410dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 --2- 172.21.15.23:0/461396355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf24100e10 0x7fbf241011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 --2- 172.21.15.23:0/461396355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf241017c0 0x7fbf24105810 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 -- 172.21.15.23:0/461396355 >> 172.21.15.23:0/461396355 conn(0x7fbf240fc6e0 msgr2=0x7fbf240feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 -- 172.21.15.23:0/461396355 shutdown_connections 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.985+0000 7fbf2a200700 1 -- 172.21.15.23:0/461396355 wait complete. 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 Processor -- start 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 -- start start 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 0x7fbf24199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 0x7fbf2419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:49.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419e1f0 con 0x7fbf24100e10 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419e360 con 0x7fbf241017c0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.986+0000 7fbf2a200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2419e520 con 0x7fbf24105ef0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf28e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf28e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37094/0 (socket says 172.21.15.23:37094) 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf28e00700 1 -- 172.21.15.23:0/1934698994 learned_addr learned my addr 172.21.15.23:0/1934698994 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 0x7fbf24199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf29800700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 0x7fbf2419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 -- 172.21.15.23:0/1934698994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 msgr2=0x7fbf2419dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 0x7fbf2419dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 -- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 msgr2=0x7fbf241992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf23e00700 1 -- 172.21.15.23:0/1934698994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf100095f0 con 0x7fbf241017c0 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.987+0000 7fbf28e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:49.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf23e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 0x7fbf24199830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf100098e0 tx=0x7fbf10005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1001d070 con 0x7fbf241017c0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbf10022470 con 0x7fbf241017c0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2419e7a0 con 0x7fbf241017c0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf10017650 con 0x7fbf241017c0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.988+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf241a6060 con 0x7fbf241017c0 2024-03-30T04:00:49.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.989+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf241022f0 con 0x7fbf241017c0 2024-03-30T04:00:49.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.991+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fbf100225e0 con 0x7fbf241017c0 2024-03-30T04:00:49.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.992+0000 7fbf21600700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 0x7fbf0c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:49.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.993+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbf100ce140 con 0x7fbf241017c0 2024-03-30T04:00:49.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.993+0000 7fbf28e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 0x7fbf0c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:49.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.993+0000 7fbf28e00700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 0x7fbf0c0a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbf18011550 tx=0x7fbf18009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:49.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:49.997+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fbf10097490 con 0x7fbf241017c0 2024-03-30T04:00:50.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789997 got 158913789998 for osd.5 2024-03-30T04:00:50.071 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.266+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fbf2404ea80 con 0x7fbf241017c0 2024-03-30T04:00:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.267+0000 7fbf21600700 1 -- 172.21.15.23:0/1934698994 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fbf1009aab0 con 0x7fbf241017c0 2024-03-30T04:00:50.266 INFO:teuthology.orchestra.run.smithi023.stdout:322122547225 2024-03-30T04:00:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 msgr2=0x7fbf0c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 0x7fbf0c0a36f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbf18011550 tx=0x7fbf18009260 comp rx=0 tx=0).stop 2024-03-30T04:00:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 msgr2=0x7fbf24199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 0x7fbf24199830 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf100098e0 tx=0x7fbf10005740 comp rx=0 tx=0).stop 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 shutdown_connections 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fbf0c0a1230 0x7fbf0c0a36f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbf24105ef0 0x7fbf2419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbf24100e10 0x7fbf241992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 --2- 172.21.15.23:0/1934698994 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf241017c0 0x7fbf24199830 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.270+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 >> 172.21.15.23:0/1934698994 conn(0x7fbf240fc6e0 msgr2=0x7fbf240fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.271+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 shutdown_connections 2024-03-30T04:00:50.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.271+0000 7fbf2a200700 1 -- 172.21.15.23:0/1934698994 wait complete. 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 -- 172.21.15.23:0/691505057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 msgr2=0x7f990810dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 --2- 172.21.15.23:0/691505057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f990810dac0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f990000b0a0 tx=0x7f990000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 -- 172.21.15.23:0/691505057 shutdown_connections 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 --2- 172.21.15.23:0/691505057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f990810dac0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 --2- 172.21.15.23:0/691505057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9908100e10 0x7f99081011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 --2- 172.21.15.23:0/691505057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99081017c0 0x7f9908105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 -- 172.21.15.23:0/691505057 >> 172.21.15.23:0/691505057 conn(0x7f99080fc6e0 msgr2=0x7f99080feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 -- 172.21.15.23:0/691505057 shutdown_connections 2024-03-30T04:00:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.536+0000 7f990ec00700 1 -- 172.21.15.23:0/691505057 wait complete. 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.537+0000 7f990ec00700 1 Processor -- start 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.537+0000 7f990ec00700 1 -- start start 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.537+0000 7f990ec00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9908100e10 0x7f99080730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.537+0000 7f990ec00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99081017c0 0x7f99080735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990ec00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990ec00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99081a5540 con 0x7f9908105ef0 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990ec00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99081a56b0 con 0x7f9908100e10 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990ec00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99081a5870 con 0x7f99081017c0 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37104/0 (socket says 172.21.15.23:37104) 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 -- 172.21.15.23:0/3470233091 learned_addr learned my addr 172.21.15.23:0/3470233091 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:50.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990d800700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9908100e10 0x7f99080730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990ce00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99081017c0 0x7f99080735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 -- 172.21.15.23:0/3470233091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99081017c0 msgr2=0x7f99080735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99081017c0 0x7f99080735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 -- 172.21.15.23:0/3470233091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9908100e10 msgr2=0x7f99080730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9908100e10 0x7f99080730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 -- 172.21.15.23:0/3470233091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9900009d00 con 0x7f9908105ef0 2024-03-30T04:00:50.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.538+0000 7f990e200700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f990000b070 tx=0x7f99000078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.539+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990000e040 con 0x7f9908105ef0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.539+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f990001d470 con 0x7f9908105ef0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.539+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99000052a0 con 0x7f9908105ef0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.539+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99081a5af0 con 0x7f9908105ef0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.539+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99081a5f10 con 0x7f9908105ef0 2024-03-30T04:00:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.540+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9908109210 con 0x7f9908105ef0 2024-03-30T04:00:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.542+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f990001d850 con 0x7f9908105ef0 2024-03-30T04:00:50.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.543+0000 7f9906000700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 0x7f98f00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.543+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f99000cde60 con 0x7f9908105ef0 2024-03-30T04:00:50.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.544+0000 7f990d800700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 0x7f98f00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.544+0000 7f990d800700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 0x7f98f00a3670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009c60 tx=0x7f98fc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:50.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.547+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f99000971b0 con 0x7f9908105ef0 2024-03-30T04:00:50.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.771+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f99081022f0 con 0x7f9908105ef0 2024-03-30T04:00:50.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.772+0000 7f9906000700 1 -- 172.21.15.23:0/3470233091 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f990009a7d0 con 0x7f9908105ef0 2024-03-30T04:00:50.771 INFO:teuthology.orchestra.run.smithi023.stdout:210453397542 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 msgr2=0x7f98f00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 0x7f98f00a3670 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009c60 tx=0x7f98fc009260 comp rx=0 tx=0).stop 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 msgr2=0x7f9908073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f990000b070 tx=0x7f99000078c0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 shutdown_connections 2024-03-30T04:00:50.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9908105ef0 0x7f9908073b30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9908100e10 0x7f99080730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98f00a11b0 0x7f98f00a3670 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 --2- 172.21.15.23:0/3470233091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99081017c0 0x7f99080735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.774+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 >> 172.21.15.23:0/3470233091 conn(0x7f99080fc6e0 msgr2=0x7f99080fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.775+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 shutdown_connections 2024-03-30T04:00:50.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.775+0000 7f990ec00700 1 -- 172.21.15.23:0/3470233091 wait complete. 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 -- 172.21.15.23:0/2853863501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf410d3f0 msgr2=0x7fddf410f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 --2- 172.21.15.23:0/2853863501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf410d3f0 0x7fddf410f860 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fddec009f90 tx=0x7fddec00b330 comp rx=0 tx=0).stop 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 -- 172.21.15.23:0/2853863501 shutdown_connections 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 --2- 172.21.15.23:0/2853863501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf410d3f0 0x7fddf410f860 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 --2- 172.21.15.23:0/2853863501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf40731e0 0x7fddf40735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 --2- 172.21.15.23:0/2853863501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf4073b00 0x7fddf410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 -- 172.21.15.23:0/2853863501 >> 172.21.15.23:0/2853863501 conn(0x7fddf40fc6e0 msgr2=0x7fddf40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 -- 172.21.15.23:0/2853863501 shutdown_connections 2024-03-30T04:00:50.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.813+0000 7fddf3200700 1 -- 172.21.15.23:0/2853863501 wait complete. 2024-03-30T04:00:50.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.814+0000 7fddf3200700 1 Processor -- start 2024-03-30T04:00:50.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.814+0000 7fddf3200700 1 -- start start 2024-03-30T04:00:50.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf4073b00 0x7fddf41998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 0x7fddf419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419e260 con 0x7fddf410d3f0 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419e3d0 con 0x7fddf40731e0 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf3200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419e590 con 0x7fddf4073b00 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf2800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 0x7fddf419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54052/0 (socket says 172.21.15.23:54052) 2024-03-30T04:00:50.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf1e00700 1 -- 172.21.15.23:0/228678380 learned_addr learned my addr 172.21.15.23:0/228678380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:50.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf1400700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf4073b00 0x7fddf41998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf2800700 1 -- 172.21.15.23:0/228678380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf4073b00 msgr2=0x7fddf41998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.815+0000 7fddf2800700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf4073b00 0x7fddf41998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf2800700 1 -- 172.21.15.23:0/228678380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 msgr2=0x7fddf4199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf2800700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf2800700 1 -- 172.21.15.23:0/228678380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009520 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf1e00700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf2800700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 0x7fddf419dc50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fddec00b6c0 tx=0x7fddec0094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec00e040 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fddec007e10 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec005040 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.816+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec009c40 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.817+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41a5f00 con 0x7fddf410d3f0 2024-03-30T04:00:50.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.818+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4108600 con 0x7fddf410d3f0 2024-03-30T04:00:50.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.819+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fddec00bcf0 con 0x7fddf410d3f0 2024-03-30T04:00:50.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.821+0000 7fddeaa00700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 0x7fddd80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:50.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.821+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fddec0ce0e0 con 0x7fddf410d3f0 2024-03-30T04:00:50.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.821+0000 7fddf1e00700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 0x7fddd80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:50.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.822+0000 7fddf1e00700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 0x7fddd80a36f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009fd0 tx=0x7fdde4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:50.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:50.826+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fddec097430 con 0x7fddf410d3f0 2024-03-30T04:00:50.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547222 got 322122547225 for osd.11 2024-03-30T04:00:50.907 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:51.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.072+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fddf419a670 con 0x7fddf410d3f0 2024-03-30T04:00:51.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.072+0000 7fddeaa00700 1 -- 172.21.15.23:0/228678380 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fddec01bed0 con 0x7fddf410d3f0 2024-03-30T04:00:51.071 INFO:teuthology.orchestra.run.smithi023.stdout:270582939680 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 msgr2=0x7fddd80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 0x7fddd80a36f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009fd0 tx=0x7fdde4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 msgr2=0x7fddf419dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 0x7fddf419dc50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fddec00b6c0 tx=0x7fddec0094b0 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 shutdown_connections 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddf410d3f0 0x7fddf419dc50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf40731e0 0x7fddf4199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fddd80a1230 0x7fddd80a36f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 --2- 172.21.15.23:0/228678380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fddf4073b00 0x7fddf41998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 >> 172.21.15.23:0/228678380 conn(0x7fddf40fc6e0 msgr2=0x7fddf40fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 shutdown_connections 2024-03-30T04:00:51.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:51.075+0000 7fddf3200700 1 -- 172.21.15.23:0/228678380 wait complete. 2024-03-30T04:00:51.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397540 got 210453397542 for osd.7 2024-03-30T04:00:51.160 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1934698994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: pgmap v273: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3470233091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/228678380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:00:51.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:51 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1934698994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: pgmap v273: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3470233091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/228678380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:00:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:51 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 270582939678 got 270582939680 for osd.9 2024-03-30T04:00:51.455 DEBUG:teuthology.parallel:result is None 2024-03-30T04:00:51.455 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-30T04:00:51.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1934698994' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: pgmap v273: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3470233091' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/228678380' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:00:51.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:51 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' 2024-03-30T04:00:51.828 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 -- 172.21.15.23:0/655801140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8106ef0 msgr2=0x7f5de81072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 --2- 172.21.15.23:0/655801140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8106ef0 0x7f5de81072d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc0099c0 tx=0x7f5ddc009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 -- 172.21.15.23:0/655801140 shutdown_connections 2024-03-30T04:00:52.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 --2- 172.21.15.23:0/655801140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 --2- 172.21.15.23:0/655801140 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8106ef0 0x7f5de81072d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 --2- 172.21.15.23:0/655801140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8100e60 0x7f5de81012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 -- 172.21.15.23:0/655801140 >> 172.21.15.23:0/655801140 conn(0x7f5de8077fa0 msgr2=0x7f5de80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 -- 172.21.15.23:0/655801140 shutdown_connections 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.948+0000 7f5df0000700 1 -- 172.21.15.23:0/655801140 wait complete. 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.950+0000 7f5df0000700 1 Processor -- start 2024-03-30T04:00:52.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.950+0000 7f5df0000700 1 -- start start 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.950+0000 7f5df0000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.950+0000 7f5df0000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.950+0000 7f5df0000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8106ef0 0x7f5de8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5df0000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81a55a0 con 0x7f5de8100e60 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5df0000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81a5710 con 0x7f5de8106ef0 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5df0000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de81a58d0 con 0x7f5de8101820 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5decc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37142/0 (socket says 172.21.15.23:37142) 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 -- 172.21.15.23:0/2998601937 learned_addr learned my addr 172.21.15.23:0/2998601937 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5dee000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8106ef0 0x7f5de8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:52.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 -- 172.21.15.23:0/2998601937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 msgr2=0x7f5de80735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de80735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 -- 172.21.15.23:0/2998601937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8106ef0 msgr2=0x7f5de8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8106ef0 0x7f5de8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5ded600700 1 -- 172.21.15.23:0/2998601937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ddc0095f0 con 0x7f5de8100e60 2024-03-30T04:00:52.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.951+0000 7f5decc00700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5ded600700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005b40 tx=0x7f5ddc00b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc01d070 con 0x7f5de8100e60 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de81a5b50 con 0x7f5de8100e60 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ddc00be00 con 0x7f5de8100e60 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc00fb00 con 0x7f5de8100e60 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.952+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de81a5f60 con 0x7f5de8100e60 2024-03-30T04:00:52.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.953+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de81092d0 con 0x7f5de8100e60 2024-03-30T04:00:52.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.955+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5ddc003680 con 0x7f5de8100e60 2024-03-30T04:00:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.956+0000 7f5de6000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 0x7f5dd00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.957+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ddc0cdf60 con 0x7f5de8100e60 2024-03-30T04:00:52.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.957+0000 7f5decc00700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 0x7f5dd00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:52.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.957+0000 7f5decc00700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 0x7f5dd00a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5de80747e0 tx=0x7f5dd4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:52.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:52.959+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ddc0972b0 con 0x7f5de8100e60 2024-03-30T04:00:53.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.167+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5de8074590 con 0x7f5dd00a11b0 2024-03-30T04:00:53.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.181+0000 7f5de6000700 1 -- 172.21.15.23:0/2998601937 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426545 (secure 0 0 0) 0x7f5de8074590 con 0x7f5dd00a11b0 2024-03-30T04:00:53.188 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 msgr2=0x7f5dd00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 0x7f5dd00a3670 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5de80747e0 tx=0x7f5dd4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 msgr2=0x7f5de80730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc005b40 tx=0x7f5ddc00b9c0 comp rx=0 tx=0).stop 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 shutdown_connections 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5de8106ef0 0x7f5de8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:53.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5de8100e60 0x7f5de80730b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dd00a11b0 0x7f5dd00a3670 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 --2- 172.21.15.23:0/2998601937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5de8101820 0x7f5de80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.186+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 >> 172.21.15.23:0/2998601937 conn(0x7f5de8077fa0 msgr2=0x7f5de80ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.187+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 shutdown_connections 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:53.187+0000 7f5df0000700 1 -- 172.21.15.23:0/2998601937 wait complete. 2024-03-30T04:00:53.193 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:00:53.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:53 smithi023 ceph-mon[38254]: pgmap v274: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-30T04:00:53.583 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-03-30T04:00:52.291213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660128,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":548,"num_read_kb":402,"num_write":418,"num_write_kb":3078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370576,"kb_used_data":13564,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124571248,"statfs":{"total":1151940427776,"available":1151560957952,"internally_reserved":0,"allocated":13889536,"data_stored":7637539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":111,"num_read_kb":52,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003567+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T04:00:17.003567+0000","last_peered":"2024-03-30T04:00:17.003567+0000","last_clean":"2024-03-30T04:00:17.003567+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T04:00:17.003567+0000","last_undegraded":"2024-03-30T04:00:17.003567+0000","last_fullsized":"2024-03-30T04:00:17.003567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009193+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T04:00:17.009193+0000","last_peered":"2024-03-30T04:00:17.009193+0000","last_clean":"2024-03-30T04:00:17.009193+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T04:00:17.009193+0000","last_undegraded":"2024-03-30T04:00:17.009193+0000","last_fullsized":"2024-03-30T04:00:17.009193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009223+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T04:00:17.009223+0000","last_peered":"2024-03-30T04:00:17.009223+0000","last_clean":"2024-03-30T04:00:17.009223+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T04:00:17.009223+0000","last_undegraded":"2024-03-30T04:00:17.009223+0000","last_fullsized":"2024-03-30T04:00:17.009223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002560+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T04:00:17.002560+0000","last_peered":"2024-03-30T04:00:17.002560+0000","last_clean":"2024-03-30T04:00:17.002560+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T04:00:17.002560+0000","last_undegraded":"2024-03-30T04:00:17.002560+0000","last_fullsized":"2024-03-30T04:00:17.002560+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003129+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T04:00:17.003129+0000","last_peered":"2024-03-30T04:00:17.003129+0000","last_clean":"2024-03-30T04:00:17.003129+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T04:00:17.003129+0000","last_undegraded":"2024-03-30T04:00:17.003129+0000","last_fullsized":"2024-03-30T04:00:17.003129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007834+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T04:00:17.007834+0000","last_peered":"2024-03-30T04:00:17.007834+0000","last_clean":"2024-03-30T04:00:17.007834+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T04:00:17.007834+0000","last_undegraded":"2024-03-30T04:00:17.007834+0000","last_fullsized":"2024-03-30T04:00:17.007834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007464+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T04:00:17.007464+0000","last_peered":"2024-03-30T04:00:17.007464+0000","last_clean":"2024-03-30T04:00:17.007464+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T04:00:17.007464+0000","last_undegraded":"2024-03-30T04:00:17.007464+0000","last_fullsized":"2024-03-30T04:00:17.007464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002761+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T04:00:17.002761+0000","last_peered":"2024-03-30T04:00:17.002761+0000","last_clean":"2024-03-30T04:00:17.002761+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T04:00:17.002761+0000","last_undegraded":"2024-03-30T04:00:17.002761+0000","last_fullsized":"2024-03-30T04:00:17.002761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999414+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T04:00:16.999414+0000","last_peered":"2024-03-30T04:00:16.999414+0000","last_clean":"2024-03-30T04:00:16.999414+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T04:00:16.999414+0000","last_undegraded":"2024-03-30T04:00:16.999414+0000","last_fullsized":"2024-03-30T04:00:16.999414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008100+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T04:00:17.008100+0000","last_peered":"2024-03-30T04:00:17.008100+0000","last_clean":"2024-03-30T04:00:17.008100+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T04:00:17.008100+0000","last_undegraded":"2024-03-30T04:00:17.008100+0000","last_fullsized":"2024-03-30T04:00:17.008100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007963+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T04:00:17.007963+0000","last_peered":"2024-03-30T04:00:17.007963+0000","last_clean":"2024-03-30T04:00:17.007963+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T04:00:17.007963+0000","last_undegraded":"2024-03-30T04:00:17.007963+0000","last_fullsized":"2024-03-30T04:00:17.007963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002431+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T04:00:17.002431+0000","last_peered":"2024-03-30T04:00:17.002431+0000","last_clean":"2024-03-30T04:00:17.002431+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T04:00:17.002431+0000","last_undegraded":"2024-03-30T04:00:17.002431+0000","last_fullsized":"2024-03-30T04:00:17.002431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998864+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T04:00:16.998864+0000","last_peered":"2024-03-30T04:00:16.998864+0000","last_clean":"2024-03-30T04:00:16.998864+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T04:00:16.998864+0000","last_undegraded":"2024-03-30T04:00:16.998864+0000","last_fullsized":"2024-03-30T04:00:16.998864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007636+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T04:00:17.007636+0000","last_peered":"2024-03-30T04:00:17.007636+0000","last_clean":"2024-03-30T04:00:17.007636+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T04:00:17.007636+0000","last_undegraded":"2024-03-30T04:00:17.007636+0000","last_fullsized":"2024-03-30T04:00:17.007636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007543+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T04:00:17.007543+0000","last_peered":"2024-03-30T04:00:17.007543+0000","last_clean":"2024-03-30T04:00:17.007543+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T04:00:17.007543+0000","last_undegraded":"2024-03-30T04:00:17.007543+0000","last_fullsized":"2024-03-30T04:00:17.007543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998890+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T04:00:16.998890+0000","last_peered":"2024-03-30T04:00:16.998890+0000","last_clean":"2024-03-30T04:00:16.998890+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T04:00:16.998890+0000","last_undegraded":"2024-03-30T04:00:16.998890+0000","last_fullsized":"2024-03-30T04:00:16.998890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003437+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T04:00:17.003437+0000","last_peered":"2024-03-30T04:00:17.003437+0000","last_clean":"2024-03-30T04:00:17.003437+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T04:00:17.003437+0000","last_undegraded":"2024-03-30T04:00:17.003437+0000","last_fullsized":"2024-03-30T04:00:17.003437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005299+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T04:00:17.005299+0000","last_peered":"2024-03-30T04:00:17.005299+0000","last_clean":"2024-03-30T04:00:17.005299+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T04:00:17.005299+0000","last_undegraded":"2024-03-30T04:00:17.005299+0000","last_fullsized":"2024-03-30T04:00:17.005299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999974+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T04:00:16.999974+0000","last_peered":"2024-03-30T04:00:16.999974+0000","last_clean":"2024-03-30T04:00:16.999974+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T04:00:16.999974+0000","last_undegraded":"2024-03-30T04:00:16.999974+0000","last_fullsized":"2024-03-30T04:00:16.999974+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.000016+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T04:00:17.000016+0000","last_peered":"2024-03-30T04:00:17.000016+0000","last_clean":"2024-03-30T04:00:17.000016+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T04:00:17.000016+0000","last_undegraded":"2024-03-30T04:00:17.000016+0000","last_fullsized":"2024-03-30T04:00:17.000016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'13","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:50.024355+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:00:50.024355+0000","last_peered":"2024-03-30T04:00:50.024355+0000","last_clean":"2024-03-30T04:00:50.024355+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:00:50.024355+0000","last_undegraded":"2024-03-30T04:00:50.024355+0000","last_fullsized":"2024-03-30T04:00:50.024355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005839+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T04:00:17.005839+0000","last_peered":"2024-03-30T04:00:17.005839+0000","last_clean":"2024-03-30T04:00:17.005839+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T04:00:17.005839+0000","last_undegraded":"2024-03-30T04:00:17.005839+0000","last_fullsized":"2024-03-30T04:00:17.005839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002090+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T04:00:17.002090+0000","last_peered":"2024-03-30T04:00:17.002090+0000","last_clean":"2024-03-30T04:00:17.002090+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T04:00:17.002090+0000","last_undegraded":"2024-03-30T04:00:17.002090+0000","last_fullsized":"2024-03-30T04:00:17.002090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005933+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T04:00:17.005933+0000","last_peered":"2024-03-30T04:00:17.005933+0000","last_clean":"2024-03-30T04:00:17.005933+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T04:00:17.005933+0000","last_undegraded":"2024-03-30T04:00:17.005933+0000","last_fullsized":"2024-03-30T04:00:17.005933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009523+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T04:00:17.009523+0000","last_peered":"2024-03-30T04:00:17.009523+0000","last_clean":"2024-03-30T04:00:17.009523+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T04:00:17.009523+0000","last_undegraded":"2024-03-30T04:00:17.009523+0000","last_fullsized":"2024-03-30T04:00:17.009523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.019833+0000","last_change":"2024-03-30T04:00:18.019833+0000","last_active":"2024-03-30T04:00:18.019833+0000","last_peered":"2024-03-30T04:00:18.019833+0000","last_clean":"2024-03-30T04:00:18.019833+0000","last_became_active":"2024-03-30T04:00:18.019445+0000","last_became_peered":"2024-03-30T04:00:18.019445+0000","last_unstale":"2024-03-30T04:00:18.019833+0000","last_undegraded":"2024-03-30T04:00:18.019833+0000","last_fullsized":"2024-03-30T04:00:18.019833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:41:55.882417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.867086+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:00:17.867086+0000","last_peered":"2024-03-30T04:00:17.867086+0000","last_clean":"2024-03-30T04:00:17.867086+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:00:17.867086+0000","last_undegraded":"2024-03-30T04:00:17.867086+0000","last_fullsized":"2024-03-30T04:00:17.867086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999282+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T04:00:16.999282+0000","last_peered":"2024-03-30T04:00:16.999282+0000","last_clean":"2024-03-30T04:00:16.999282+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T04:00:16.999282+0000","last_undegraded":"2024-03-30T04:00:16.999282+0000","last_fullsized":"2024-03-30T04:00:16.999282+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002020+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T04:00:17.002020+0000","last_peered":"2024-03-30T04:00:17.002020+0000","last_clean":"2024-03-30T04:00:17.002020+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T04:00:17.002020+0000","last_undegraded":"2024-03-30T04:00:17.002020+0000","last_fullsized":"2024-03-30T04:00:17.002020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004349+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:00:17.004349+0000","last_peered":"2024-03-30T04:00:17.004349+0000","last_clean":"2024-03-30T04:00:17.004349+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:00:17.004349+0000","last_undegraded":"2024-03-30T04:00:17.004349+0000","last_fullsized":"2024-03-30T04:00:17.004349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004142+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T04:00:17.004142+0000","last_peered":"2024-03-30T04:00:17.004142+0000","last_clean":"2024-03-30T04:00:17.004142+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T04:00:17.004142+0000","last_undegraded":"2024-03-30T04:00:17.004142+0000","last_fullsized":"2024-03-30T04:00:17.004142+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999784+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T04:00:16.999784+0000","last_peered":"2024-03-30T04:00:16.999784+0000","last_clean":"2024-03-30T04:00:16.999784+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T04:00:16.999784+0000","last_undegraded":"2024-03-30T04:00:16.999784+0000","last_fullsized":"2024-03-30T04:00:16.999784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'5","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:46.988838+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:00:46.988838+0000","last_peered":"2024-03-30T04:00:46.988838+0000","last_clean":"2024-03-30T04:00:46.988838+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:00:46.988838+0000","last_undegraded":"2024-03-30T04:00:46.988838+0000","last_fullsized":"2024-03-30T04:00:46.988838+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003626+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T04:00:17.003626+0000","last_peered":"2024-03-30T04:00:17.003626+0000","last_clean":"2024-03-30T04:00:17.003626+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T04:00:17.003626+0000","last_undegraded":"2024-03-30T04:00:17.003626+0000","last_fullsized":"2024-03-30T04:00:17.003626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.000058+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T04:00:17.000058+0000","last_peered":"2024-03-30T04:00:17.000058+0000","last_clean":"2024-03-30T04:00:17.000058+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T04:00:17.000058+0000","last_undegraded":"2024-03-30T04:00:17.000058+0000","last_fullsized":"2024-03-30T04:00:17.000058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007735+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:00:17.007735+0000","last_peered":"2024-03-30T04:00:17.007735+0000","last_clean":"2024-03-30T04:00:17.007735+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:00:17.007735+0000","last_undegraded":"2024-03-30T04:00:17.007735+0000","last_fullsized":"2024-03-30T04:00:17.007735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003122+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T04:00:17.003122+0000","last_peered":"2024-03-30T04:00:17.003122+0000","last_clean":"2024-03-30T04:00:17.003122+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T04:00:17.003122+0000","last_undegraded":"2024-03-30T04:00:17.003122+0000","last_fullsized":"2024-03-30T04:00:17.003122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003360+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T04:00:17.003360+0000","last_peered":"2024-03-30T04:00:17.003360+0000","last_clean":"2024-03-30T04:00:17.003360+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T04:00:17.003360+0000","last_undegraded":"2024-03-30T04:00:17.003360+0000","last_fullsized":"2024-03-30T04:00:17.003360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998384+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:00:16.998384+0000","last_peered":"2024-03-30T04:00:16.998384+0000","last_clean":"2024-03-30T04:00:16.998384+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:00:16.998384+0000","last_undegraded":"2024-03-30T04:00:16.998384+0000","last_fullsized":"2024-03-30T04:00:16.998384+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003489+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T04:00:17.003489+0000","last_peered":"2024-03-30T04:00:17.003489+0000","last_clean":"2024-03-30T04:00:17.003489+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T04:00:17.003489+0000","last_undegraded":"2024-03-30T04:00:17.003489+0000","last_fullsized":"2024-03-30T04:00:17.003489+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009143+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T04:00:17.009143+0000","last_peered":"2024-03-30T04:00:17.009143+0000","last_clean":"2024-03-30T04:00:17.009143+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T04:00:17.009143+0000","last_undegraded":"2024-03-30T04:00:17.009143+0000","last_fullsized":"2024-03-30T04:00:17.009143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005605+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:00:17.005605+0000","last_peered":"2024-03-30T04:00:17.005605+0000","last_clean":"2024-03-30T04:00:17.005605+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:00:17.005605+0000","last_undegraded":"2024-03-30T04:00:17.005605+0000","last_fullsized":"2024-03-30T04:00:17.005605+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002874+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T04:00:17.002874+0000","last_peered":"2024-03-30T04:00:17.002874+0000","last_clean":"2024-03-30T04:00:17.002874+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T04:00:17.002874+0000","last_undegraded":"2024-03-30T04:00:17.002874+0000","last_fullsized":"2024-03-30T04:00:17.002874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003127+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T04:00:17.003127+0000","last_peered":"2024-03-30T04:00:17.003127+0000","last_clean":"2024-03-30T04:00:17.003127+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T04:00:17.003127+0000","last_undegraded":"2024-03-30T04:00:17.003127+0000","last_fullsized":"2024-03-30T04:00:17.003127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":97,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:48.991179+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:00:48.991179+0000","last_peered":"2024-03-30T04:00:48.991179+0000","last_clean":"2024-03-30T04:00:48.991179+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:00:48.991179+0000","last_undegraded":"2024-03-30T04:00:48.991179+0000","last_fullsized":"2024-03-30T04:00:48.991179+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999273+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T04:00:16.999273+0000","last_peered":"2024-03-30T04:00:16.999273+0000","last_clean":"2024-03-30T04:00:16.999273+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T04:00:16.999273+0000","last_undegraded":"2024-03-30T04:00:16.999273+0000","last_fullsized":"2024-03-30T04:00:16.999273+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007276+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T04:00:17.007276+0000","last_peered":"2024-03-30T04:00:17.007276+0000","last_clean":"2024-03-30T04:00:17.007276+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T04:00:17.007276+0000","last_undegraded":"2024-03-30T04:00:17.007276+0000","last_fullsized":"2024-03-30T04:00:17.007276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007850+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:00:17.007850+0000","last_peered":"2024-03-30T04:00:17.007850+0000","last_clean":"2024-03-30T04:00:17.007850+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:00:17.007850+0000","last_undegraded":"2024-03-30T04:00:17.007850+0000","last_fullsized":"2024-03-30T04:00:17.007850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.016147+0000","last_change":"2024-03-30T04:00:18.016147+0000","last_active":"2024-03-30T04:00:18.016147+0000","last_peered":"2024-03-30T04:00:18.016147+0000","last_clean":"2024-03-30T04:00:18.016147+0000","last_became_active":"2024-03-30T04:00:18.015780+0000","last_became_peered":"2024-03-30T04:00:18.015780+0000","last_unstale":"2024-03-30T04:00:18.016147+0000","last_undegraded":"2024-03-30T04:00:18.016147+0000","last_fullsized":"2024-03-30T04:00:18.016147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:00:33.540122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002787+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T04:00:17.002787+0000","last_peered":"2024-03-30T04:00:17.002787+0000","last_clean":"2024-03-30T04:00:17.002787+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T04:00:17.002787+0000","last_undegraded":"2024-03-30T04:00:17.002787+0000","last_fullsized":"2024-03-30T04:00:17.002787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004047+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:00:17.004047+0000","last_peered":"2024-03-30T04:00:17.004047+0000","last_clean":"2024-03-30T04:00:17.004047+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:00:17.004047+0000","last_undegraded":"2024-03-30T04:00:17.004047+0000","last_fullsized":"2024-03-30T04:00:17.004047+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999454+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T04:00:16.999454+0000","last_peered":"2024-03-30T04:00:16.999454+0000","last_clean":"2024-03-30T04:00:16.999454+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T04:00:16.999454+0000","last_undegraded":"2024-03-30T04:00:16.999454+0000","last_fullsized":"2024-03-30T04:00:16.999454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002904+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T04:00:17.002904+0000","last_peered":"2024-03-30T04:00:17.002904+0000","last_clean":"2024-03-30T04:00:17.002904+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T04:00:17.002904+0000","last_undegraded":"2024-03-30T04:00:17.002904+0000","last_fullsized":"2024-03-30T04:00:17.002904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":138,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:48.996548+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:00:48.996548+0000","last_peered":"2024-03-30T04:00:48.996548+0000","last_clean":"2024-03-30T04:00:48.996548+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:00:48.996548+0000","last_undegraded":"2024-03-30T04:00:48.996548+0000","last_fullsized":"2024-03-30T04:00:48.996548+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62510,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":86,"num_read_kb":105,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003952+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T04:00:17.003952+0000","last_peered":"2024-03-30T04:00:17.003952+0000","last_clean":"2024-03-30T04:00:17.003952+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T04:00:17.003952+0000","last_undegraded":"2024-03-30T04:00:17.003952+0000","last_fullsized":"2024-03-30T04:00:17.003952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004359+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T04:00:17.004359+0000","last_peered":"2024-03-30T04:00:17.004359+0000","last_clean":"2024-03-30T04:00:17.004359+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T04:00:17.004359+0000","last_undegraded":"2024-03-30T04:00:17.004359+0000","last_fullsized":"2024-03-30T04:00:17.004359+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999656+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:00:16.999656+0000","last_peered":"2024-03-30T04:00:16.999656+0000","last_clean":"2024-03-30T04:00:16.999656+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:00:16.999656+0000","last_undegraded":"2024-03-30T04:00:16.999656+0000","last_fullsized":"2024-03-30T04:00:16.999656+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998526+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T04:00:16.998526+0000","last_peered":"2024-03-30T04:00:16.998526+0000","last_clean":"2024-03-30T04:00:16.998526+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T04:00:16.998526+0000","last_undegraded":"2024-03-30T04:00:16.998526+0000","last_fullsized":"2024-03-30T04:00:16.998526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003982+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T04:00:17.003982+0000","last_peered":"2024-03-30T04:00:17.003982+0000","last_clean":"2024-03-30T04:00:17.003982+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T04:00:17.003982+0000","last_undegraded":"2024-03-30T04:00:17.003982+0000","last_fullsized":"2024-03-30T04:00:17.003982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003013+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:00:17.003013+0000","last_peered":"2024-03-30T04:00:17.003013+0000","last_clean":"2024-03-30T04:00:17.003013+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:00:17.003013+0000","last_undegraded":"2024-03-30T04:00:17.003013+0000","last_fullsized":"2024-03-30T04:00:17.003013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007223+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T04:00:17.007223+0000","last_peered":"2024-03-30T04:00:17.007223+0000","last_clean":"2024-03-30T04:00:17.007223+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T04:00:17.007223+0000","last_undegraded":"2024-03-30T04:00:17.007223+0000","last_fullsized":"2024-03-30T04:00:17.007223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.021325+0000","last_change":"2024-03-30T04:00:18.020869+0000","last_active":"2024-03-30T04:00:18.021325+0000","last_peered":"2024-03-30T04:00:18.021325+0000","last_clean":"2024-03-30T04:00:18.021325+0000","last_became_active":"2024-03-30T04:00:18.020410+0000","last_became_peered":"2024-03-30T04:00:18.020410+0000","last_unstale":"2024-03-30T04:00:18.021325+0000","last_undegraded":"2024-03-30T04:00:18.021325+0000","last_fullsized":"2024-03-30T04:00:18.021325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:53.675022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:49.023400+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:00:49.023400+0000","last_peered":"2024-03-30T04:00:49.023400+0000","last_clean":"2024-03-30T04:00:49.023400+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:00:49.023400+0000","last_undegraded":"2024-03-30T04:00:49.023400+0000","last_fullsized":"2024-03-30T04:00:49.023400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007454+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T04:00:17.007454+0000","last_peered":"2024-03-30T04:00:17.007454+0000","last_clean":"2024-03-30T04:00:17.007454+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T04:00:17.007454+0000","last_undegraded":"2024-03-30T04:00:17.007454+0000","last_fullsized":"2024-03-30T04:00:17.007454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.288237+0000","last_change":"2024-03-30T04:00:18.288237+0000","last_active":"2024-03-30T04:00:18.288237+0000","last_peered":"2024-03-30T04:00:18.288237+0000","last_clean":"2024-03-30T04:00:18.288237+0000","last_became_active":"2024-03-30T04:00:18.287748+0000","last_became_peered":"2024-03-30T04:00:18.287748+0000","last_unstale":"2024-03-30T04:00:18.288237+0000","last_undegraded":"2024-03-30T04:00:18.288237+0000","last_fullsized":"2024-03-30T04:00:18.288237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:50.893522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003013+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:00:17.003013+0000","last_peered":"2024-03-30T04:00:17.003013+0000","last_clean":"2024-03-30T04:00:17.003013+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:00:17.003013+0000","last_undegraded":"2024-03-30T04:00:17.003013+0000","last_fullsized":"2024-03-30T04:00:17.003013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005288+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T04:00:17.005288+0000","last_peered":"2024-03-30T04:00:17.005288+0000","last_clean":"2024-03-30T04:00:17.005288+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T04:00:17.005288+0000","last_undegraded":"2024-03-30T04:00:17.005288+0000","last_fullsized":"2024-03-30T04:00:17.005288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005097+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T04:00:17.005097+0000","last_peered":"2024-03-30T04:00:17.005097+0000","last_clean":"2024-03-30T04:00:17.005097+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T04:00:17.005097+0000","last_undegraded":"2024-03-30T04:00:17.005097+0000","last_fullsized":"2024-03-30T04:00:17.005097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002818+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:00:17.002818+0000","last_peered":"2024-03-30T04:00:17.002818+0000","last_clean":"2024-03-30T04:00:17.002818+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:00:17.002818+0000","last_undegraded":"2024-03-30T04:00:17.002818+0000","last_fullsized":"2024-03-30T04:00:17.002818+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002909+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T04:00:17.002909+0000","last_peered":"2024-03-30T04:00:17.002909+0000","last_clean":"2024-03-30T04:00:17.002909+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T04:00:17.002909+0000","last_undegraded":"2024-03-30T04:00:17.002909+0000","last_fullsized":"2024-03-30T04:00:17.002909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003547+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T04:00:17.003547+0000","last_peered":"2024-03-30T04:00:17.003547+0000","last_clean":"2024-03-30T04:00:17.003547+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T04:00:17.003547+0000","last_undegraded":"2024-03-30T04:00:17.003547+0000","last_fullsized":"2024-03-30T04:00:17.003547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007033+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T04:00:17.007033+0000","last_peered":"2024-03-30T04:00:17.007033+0000","last_clean":"2024-03-30T04:00:17.007033+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T04:00:17.007033+0000","last_undegraded":"2024-03-30T04:00:17.007033+0000","last_fullsized":"2024-03-30T04:00:17.007033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007538+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T04:00:17.007538+0000","last_peered":"2024-03-30T04:00:17.007538+0000","last_clean":"2024-03-30T04:00:17.007538+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T04:00:17.007538+0000","last_undegraded":"2024-03-30T04:00:17.007538+0000","last_fullsized":"2024-03-30T04:00:17.007538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998771+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T04:00:16.998771+0000","last_peered":"2024-03-30T04:00:16.998771+0000","last_clean":"2024-03-30T04:00:16.998771+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T04:00:16.998771+0000","last_undegraded":"2024-03-30T04:00:16.998771+0000","last_fullsized":"2024-03-30T04:00:16.998771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:46.019783+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T04:00:46.019783+0000","last_peered":"2024-03-30T04:00:46.019783+0000","last_clean":"2024-03-30T04:00:46.019783+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T04:00:46.019783+0000","last_undegraded":"2024-03-30T04:00:46.019783+0000","last_fullsized":"2024-03-30T04:00:46.019783+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007339+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T04:00:17.007339+0000","last_peered":"2024-03-30T04:00:17.007339+0000","last_clean":"2024-03-30T04:00:17.007339+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T04:00:17.007339+0000","last_undegraded":"2024-03-30T04:00:17.007339+0000","last_fullsized":"2024-03-30T04:00:17.007339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998956+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T04:00:16.998956+0000","last_peered":"2024-03-30T04:00:16.998956+0000","last_clean":"2024-03-30T04:00:16.998956+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T04:00:16.998956+0000","last_undegraded":"2024-03-30T04:00:16.998956+0000","last_fullsized":"2024-03-30T04:00:16.998956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.018515+0000","last_change":"2024-03-30T04:00:18.018515+0000","last_active":"2024-03-30T04:00:18.018515+0000","last_peered":"2024-03-30T04:00:18.018515+0000","last_clean":"2024-03-30T04:00:18.018515+0000","last_became_active":"2024-03-30T04:00:18.017823+0000","last_became_peered":"2024-03-30T04:00:18.017823+0000","last_unstale":"2024-03-30T04:00:18.018515+0000","last_undegraded":"2024-03-30T04:00:18.018515+0000","last_fullsized":"2024-03-30T04:00:18.018515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:15:54.957524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:47.019312+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:00:47.019312+0000","last_peered":"2024-03-30T04:00:47.019312+0000","last_clean":"2024-03-30T04:00:47.019312+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:00:47.019312+0000","last_undegraded":"2024-03-30T04:00:47.019312+0000","last_fullsized":"2024-03-30T04:00:47.019312+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007309+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T04:00:17.007309+0000","last_peered":"2024-03-30T04:00:17.007309+0000","last_clean":"2024-03-30T04:00:17.007309+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T04:00:17.007309+0000","last_undegraded":"2024-03-30T04:00:17.007309+0000","last_fullsized":"2024-03-30T04:00:17.007309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003405+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T04:00:17.003405+0000","last_peered":"2024-03-30T04:00:17.003405+0000","last_clean":"2024-03-30T04:00:17.003405+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T04:00:17.003405+0000","last_undegraded":"2024-03-30T04:00:17.003405+0000","last_fullsized":"2024-03-30T04:00:17.003405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002659+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:00:17.002659+0000","last_peered":"2024-03-30T04:00:17.002659+0000","last_clean":"2024-03-30T04:00:17.002659+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:00:17.002659+0000","last_undegraded":"2024-03-30T04:00:17.002659+0000","last_fullsized":"2024-03-30T04:00:17.002659+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.006941+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T04:00:17.006941+0000","last_peered":"2024-03-30T04:00:17.006941+0000","last_clean":"2024-03-30T04:00:17.006941+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T04:00:17.006941+0000","last_undegraded":"2024-03-30T04:00:17.006941+0000","last_fullsized":"2024-03-30T04:00:17.006941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.018249+0000","last_change":"2024-03-30T04:00:18.017573+0000","last_active":"2024-03-30T04:00:18.018249+0000","last_peered":"2024-03-30T04:00:18.018249+0000","last_clean":"2024-03-30T04:00:18.018249+0000","last_became_active":"2024-03-30T04:00:18.016937+0000","last_became_peered":"2024-03-30T04:00:18.016937+0000","last_unstale":"2024-03-30T04:00:18.018249+0000","last_undegraded":"2024-03-30T04:00:18.018249+0000","last_fullsized":"2024-03-30T04:00:18.018249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:26:20.863180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998323+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T04:00:16.998323+0000","last_peered":"2024-03-30T04:00:16.998323+0000","last_clean":"2024-03-30T04:00:16.998323+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T04:00:16.998323+0000","last_undegraded":"2024-03-30T04:00:16.998323+0000","last_fullsized":"2024-03-30T04:00:16.998323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007021+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:00:17.007021+0000","last_peered":"2024-03-30T04:00:17.007021+0000","last_clean":"2024-03-30T04:00:17.007021+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:00:17.007021+0000","last_undegraded":"2024-03-30T04:00:17.007021+0000","last_fullsized":"2024-03-30T04:00:17.007021+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002873+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T04:00:17.002873+0000","last_peered":"2024-03-30T04:00:17.002873+0000","last_clean":"2024-03-30T04:00:17.002873+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T04:00:17.002873+0000","last_undegraded":"2024-03-30T04:00:17.002873+0000","last_fullsized":"2024-03-30T04:00:17.002873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004064+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:00:17.004064+0000","last_peered":"2024-03-30T04:00:17.004064+0000","last_clean":"2024-03-30T04:00:17.004064+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:00:17.004064+0000","last_undegraded":"2024-03-30T04:00:17.004064+0000","last_fullsized":"2024-03-30T04:00:17.004064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999003+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T04:00:16.999003+0000","last_peered":"2024-03-30T04:00:16.999003+0000","last_clean":"2024-03-30T04:00:16.999003+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T04:00:16.999003+0000","last_undegraded":"2024-03-30T04:00:16.999003+0000","last_fullsized":"2024-03-30T04:00:16.999003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002570+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T04:00:17.002570+0000","last_peered":"2024-03-30T04:00:17.002570+0000","last_clean":"2024-03-30T04:00:17.002570+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T04:00:17.002570+0000","last_undegraded":"2024-03-30T04:00:17.002570+0000","last_fullsized":"2024-03-30T04:00:17.002570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.321082+0000","last_change":"2024-03-30T04:00:18.297312+0000","last_active":"2024-03-30T04:00:18.321082+0000","last_peered":"2024-03-30T04:00:18.321082+0000","last_clean":"2024-03-30T04:00:18.321082+0000","last_became_active":"2024-03-30T04:00:18.288640+0000","last_became_peered":"2024-03-30T04:00:18.288640+0000","last_unstale":"2024-03-30T04:00:18.321082+0000","last_undegraded":"2024-03-30T04:00:18.321082+0000","last_fullsized":"2024-03-30T04:00:18.321082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:21.664643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.006942+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T04:00:17.006942+0000","last_peered":"2024-03-30T04:00:17.006942+0000","last_clean":"2024-03-30T04:00:17.006942+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T04:00:17.006942+0000","last_undegraded":"2024-03-30T04:00:17.006942+0000","last_fullsized":"2024-03-30T04:00:17.006942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999181+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T04:00:16.999181+0000","last_peered":"2024-03-30T04:00:16.999181+0000","last_clean":"2024-03-30T04:00:16.999181+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T04:00:16.999181+0000","last_undegraded":"2024-03-30T04:00:16.999181+0000","last_fullsized":"2024-03-30T04:00:16.999181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002441+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:00:17.002441+0000","last_peered":"2024-03-30T04:00:17.002441+0000","last_clean":"2024-03-30T04:00:17.002441+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:00:17.002441+0000","last_undegraded":"2024-03-30T04:00:17.002441+0000","last_fullsized":"2024-03-30T04:00:17.002441+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009264+0000","last_change":"2024-03-30T03:59:40.464994+0000","last_active":"2024-03-30T04:00:17.009264+0000","last_peered":"2024-03-30T04:00:17.009264+0000","last_clean":"2024-03-30T04:00:17.009264+0000","last_became_active":"2024-03-30T03:59:40.462862+0000","last_became_peered":"2024-03-30T03:59:40.462862+0000","last_unstale":"2024-03-30T04:00:17.009264+0000","last_undegraded":"2024-03-30T04:00:17.009264+0000","last_fullsized":"2024-03-30T04:00:17.009264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:29:48.413081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002280+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T04:00:17.002280+0000","last_peered":"2024-03-30T04:00:17.002280+0000","last_clean":"2024-03-30T04:00:17.002280+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T04:00:17.002280+0000","last_undegraded":"2024-03-30T04:00:17.002280+0000","last_fullsized":"2024-03-30T04:00:17.002280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.291229+0000","last_change":"2024-03-30T04:00:18.291229+0000","last_active":"2024-03-30T04:00:18.291229+0000","last_peered":"2024-03-30T04:00:18.291229+0000","last_clean":"2024-03-30T04:00:18.291229+0000","last_became_active":"2024-03-30T04:00:18.287954+0000","last_became_peered":"2024-03-30T04:00:18.287954+0000","last_unstale":"2024-03-30T04:00:18.291229+0000","last_undegraded":"2024-03-30T04:00:18.291229+0000","last_fullsized":"2024-03-30T04:00:18.291229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:54.678907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002590+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T04:00:17.002590+0000","last_peered":"2024-03-30T04:00:17.002590+0000","last_clean":"2024-03-30T04:00:17.002590+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T04:00:17.002590+0000","last_undegraded":"2024-03-30T04:00:17.002590+0000","last_fullsized":"2024-03-30T04:00:17.002590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003702+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T04:00:17.003702+0000","last_peered":"2024-03-30T04:00:17.003702+0000","last_clean":"2024-03-30T04:00:17.003702+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T04:00:17.003702+0000","last_undegraded":"2024-03-30T04:00:17.003702+0000","last_fullsized":"2024-03-30T04:00:17.003702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002944+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:00:17.002944+0000","last_peered":"2024-03-30T04:00:17.002944+0000","last_clean":"2024-03-30T04:00:17.002944+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:00:17.002944+0000","last_undegraded":"2024-03-30T04:00:17.002944+0000","last_fullsized":"2024-03-30T04:00:17.002944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003830+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T04:00:17.003830+0000","last_peered":"2024-03-30T04:00:17.003830+0000","last_clean":"2024-03-30T04:00:17.003830+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T04:00:17.003830+0000","last_undegraded":"2024-03-30T04:00:17.003830+0000","last_fullsized":"2024-03-30T04:00:17.003830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002261+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T04:00:17.002261+0000","last_peered":"2024-03-30T04:00:17.002261+0000","last_clean":"2024-03-30T04:00:17.002261+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T04:00:17.002261+0000","last_undegraded":"2024-03-30T04:00:17.002261+0000","last_fullsized":"2024-03-30T04:00:17.002261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999378+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T04:00:16.999378+0000","last_peered":"2024-03-30T04:00:16.999378+0000","last_clean":"2024-03-30T04:00:16.999378+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T04:00:16.999378+0000","last_undegraded":"2024-03-30T04:00:16.999378+0000","last_fullsized":"2024-03-30T04:00:16.999378+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004857+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T04:00:17.004857+0000","last_peered":"2024-03-30T04:00:17.004857+0000","last_clean":"2024-03-30T04:00:17.004857+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T04:00:17.004857+0000","last_undegraded":"2024-03-30T04:00:17.004857+0000","last_fullsized":"2024-03-30T04:00:17.004857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002249+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T04:00:17.002249+0000","last_peered":"2024-03-30T04:00:17.002249+0000","last_clean":"2024-03-30T04:00:17.002249+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T04:00:17.002249+0000","last_undegraded":"2024-03-30T04:00:17.002249+0000","last_fullsized":"2024-03-30T04:00:17.002249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003241+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:00:17.003241+0000","last_peered":"2024-03-30T04:00:17.003241+0000","last_clean":"2024-03-30T04:00:17.003241+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:00:17.003241+0000","last_undegraded":"2024-03-30T04:00:17.003241+0000","last_fullsized":"2024-03-30T04:00:17.003241+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007355+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T04:00:17.007355+0000","last_peered":"2024-03-30T04:00:17.007355+0000","last_clean":"2024-03-30T04:00:17.007355+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T04:00:17.007355+0000","last_undegraded":"2024-03-30T04:00:17.007355+0000","last_fullsized":"2024-03-30T04:00:17.007355+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998944+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T04:00:16.998944+0000","last_peered":"2024-03-30T04:00:16.998944+0000","last_clean":"2024-03-30T04:00:16.998944+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T04:00:16.998944+0000","last_undegraded":"2024-03-30T04:00:16.998944+0000","last_fullsized":"2024-03-30T04:00:16.998944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:51.026108+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:00:51.026108+0000","last_peered":"2024-03-30T04:00:51.026108+0000","last_clean":"2024-03-30T04:00:51.026108+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:00:51.026108+0000","last_undegraded":"2024-03-30T04:00:51.026108+0000","last_fullsized":"2024-03-30T04:00:51.026108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008135+0000","last_change":"2024-03-30T03:59:40.462459+0000","last_active":"2024-03-30T04:00:17.008135+0000","last_peered":"2024-03-30T04:00:17.008135+0000","last_clean":"2024-03-30T04:00:17.008135+0000","last_became_active":"2024-03-30T03:59:40.461692+0000","last_became_peered":"2024-03-30T03:59:40.461692+0000","last_unstale":"2024-03-30T04:00:17.008135+0000","last_undegraded":"2024-03-30T04:00:17.008135+0000","last_fullsized":"2024-03-30T04:00:17.008135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:46.872671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007945+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T04:00:17.007945+0000","last_peered":"2024-03-30T04:00:17.007945+0000","last_clean":"2024-03-30T04:00:17.007945+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T04:00:17.007945+0000","last_undegraded":"2024-03-30T04:00:17.007945+0000","last_fullsized":"2024-03-30T04:00:17.007945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004251+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:00:17.004251+0000","last_peered":"2024-03-30T04:00:17.004251+0000","last_clean":"2024-03-30T04:00:17.004251+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:00:17.004251+0000","last_undegraded":"2024-03-30T04:00:17.004251+0000","last_fullsized":"2024-03-30T04:00:17.004251+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.016967+0000","last_change":"2024-03-30T04:00:18.016967+0000","last_active":"2024-03-30T04:00:18.016967+0000","last_peered":"2024-03-30T04:00:18.016967+0000","last_clean":"2024-03-30T04:00:18.016967+0000","last_became_active":"2024-03-30T04:00:18.016555+0000","last_became_peered":"2024-03-30T04:00:18.016555+0000","last_unstale":"2024-03-30T04:00:18.016967+0000","last_undegraded":"2024-03-30T04:00:18.016967+0000","last_fullsized":"2024-03-30T04:00:18.016967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:34:50.538682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005262+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T04:00:17.005262+0000","last_peered":"2024-03-30T04:00:17.005262+0000","last_clean":"2024-03-30T04:00:17.005262+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T04:00:17.005262+0000","last_undegraded":"2024-03-30T04:00:17.005262+0000","last_fullsized":"2024-03-30T04:00:17.005262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999106+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:00:16.999106+0000","last_peered":"2024-03-30T04:00:16.999106+0000","last_clean":"2024-03-30T04:00:16.999106+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:00:16.999106+0000","last_undegraded":"2024-03-30T04:00:16.999106+0000","last_fullsized":"2024-03-30T04:00:16.999106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009212+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T04:00:17.009212+0000","last_peered":"2024-03-30T04:00:17.009212+0000","last_clean":"2024-03-30T04:00:17.009212+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T04:00:17.009212+0000","last_undegraded":"2024-03-30T04:00:17.009212+0000","last_fullsized":"2024-03-30T04:00:17.009212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003921+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T04:00:17.003921+0000","last_peered":"2024-03-30T04:00:17.003921+0000","last_clean":"2024-03-30T04:00:17.003921+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T04:00:17.003921+0000","last_undegraded":"2024-03-30T04:00:17.003921+0000","last_fullsized":"2024-03-30T04:00:17.003921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.019138+0000","last_change":"2024-03-30T04:00:18.019138+0000","last_active":"2024-03-30T04:00:18.019138+0000","last_peered":"2024-03-30T04:00:18.019138+0000","last_clean":"2024-03-30T04:00:18.019138+0000","last_became_active":"2024-03-30T04:00:18.018528+0000","last_became_peered":"2024-03-30T04:00:18.018528+0000","last_unstale":"2024-03-30T04:00:18.019138+0000","last_undegraded":"2024-03-30T04:00:18.019138+0000","last_fullsized":"2024-03-30T04:00:18.019138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:27.603607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999251+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T04:00:16.999251+0000","last_peered":"2024-03-30T04:00:16.999251+0000","last_clean":"2024-03-30T04:00:16.999251+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T04:00:16.999251+0000","last_undegraded":"2024-03-30T04:00:16.999251+0000","last_fullsized":"2024-03-30T04:00:16.999251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008104+0000","last_change":"2024-03-30T03:59:40.454510+0000","last_active":"2024-03-30T04:00:17.008104+0000","last_peered":"2024-03-30T04:00:17.008104+0000","last_clean":"2024-03-30T04:00:17.008104+0000","last_became_active":"2024-03-30T03:59:40.454119+0000","last_became_peered":"2024-03-30T03:59:40.454119+0000","last_unstale":"2024-03-30T04:00:17.008104+0000","last_undegraded":"2024-03-30T04:00:17.008104+0000","last_fullsized":"2024-03-30T04:00:17.008104+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:50:00.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998916+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T04:00:16.998916+0000","last_peered":"2024-03-30T04:00:16.998916+0000","last_clean":"2024-03-30T04:00:16.998916+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T04:00:16.998916+0000","last_undegraded":"2024-03-30T04:00:16.998916+0000","last_fullsized":"2024-03-30T04:00:16.998916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003214+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T04:00:17.003214+0000","last_peered":"2024-03-30T04:00:17.003214+0000","last_clean":"2024-03-30T04:00:17.003214+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T04:00:17.003214+0000","last_undegraded":"2024-03-30T04:00:17.003214+0000","last_fullsized":"2024-03-30T04:00:17.003214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999916+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T04:00:16.999916+0000","last_peered":"2024-03-30T04:00:16.999916+0000","last_clean":"2024-03-30T04:00:16.999916+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T04:00:16.999916+0000","last_undegraded":"2024-03-30T04:00:16.999916+0000","last_fullsized":"2024-03-30T04:00:16.999916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008042+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T04:00:17.008042+0000","last_peered":"2024-03-30T04:00:17.008042+0000","last_clean":"2024-03-30T04:00:17.008042+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T04:00:17.008042+0000","last_undegraded":"2024-03-30T04:00:17.008042+0000","last_fullsized":"2024-03-30T04:00:17.008042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005659+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T04:00:17.005659+0000","last_peered":"2024-03-30T04:00:17.005659+0000","last_clean":"2024-03-30T04:00:17.005659+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T04:00:17.005659+0000","last_undegraded":"2024-03-30T04:00:17.005659+0000","last_fullsized":"2024-03-30T04:00:17.005659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003025+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T04:00:17.003025+0000","last_peered":"2024-03-30T04:00:17.003025+0000","last_clean":"2024-03-30T04:00:17.003025+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T04:00:17.003025+0000","last_undegraded":"2024-03-30T04:00:17.003025+0000","last_fullsized":"2024-03-30T04:00:17.003025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999278+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T04:00:16.999278+0000","last_peered":"2024-03-30T04:00:16.999278+0000","last_clean":"2024-03-30T04:00:16.999278+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T04:00:16.999278+0000","last_undegraded":"2024-03-30T04:00:16.999278+0000","last_fullsized":"2024-03-30T04:00:16.999278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998961+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T04:00:16.998961+0000","last_peered":"2024-03-30T04:00:16.998961+0000","last_clean":"2024-03-30T04:00:16.998961+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T04:00:16.998961+0000","last_undegraded":"2024-03-30T04:00:16.998961+0000","last_fullsized":"2024-03-30T04:00:16.998961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004029+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T04:00:17.004029+0000","last_peered":"2024-03-30T04:00:17.004029+0000","last_clean":"2024-03-30T04:00:17.004029+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T04:00:17.004029+0000","last_undegraded":"2024-03-30T04:00:17.004029+0000","last_fullsized":"2024-03-30T04:00:17.004029+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69526,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":412,"num_read_kb":283,"num_write":235,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547225,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":974848,"data_stored":474025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.501}]},{"osd":1,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.78500000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.625},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.65400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.71099999999999997}]},{"osd":5,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.9340000000000002,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.81100000000000005}]},{"osd":8,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.75},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.77100000000000002}]},{"osd":9,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.874}]},{"osd":10,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.86299999999999999}]}]},{"osd":10,"up_from":68,"seq":292057776156,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1671168,"data_stored":1123744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.65400000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.623,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68500000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42699999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48999999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.68500000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.496},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.59899999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.51100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.73099999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.01,"15min":1.01},"last":0.626},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95499999999999996,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97599999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.70099999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.71599999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.018,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.751},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65200000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.95699999999999996,"5min":1.147,"15min":1.147},"last":0.74099999999999999}]},{"osd":11,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.80000000000000004}]}]},{"osd":9,"up_from":63,"seq":270582939681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":1003520,"data_stored":478099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.706,"5min":3.706,"15min":3.706},"last":0.62},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.629,"15min":0.64700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":1.018,"15min":1.018},"last":0.70799999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.68300000000000005,"15min":0.69799999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.75900000000000001,"15min":0.76900000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.71999999999999997}]},{"osd":2,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.79400000000000004,"15min":0.81399999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2450000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.69499999999999995,"15min":0.70699999999999996},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77600000000000002,"5min":1.155,"15min":1.155},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.72799999999999998,"15min":0.75},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.872,"5min":1.294,"15min":1.294},"last":0.64300000000000002}]},{"osd":4,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.84499999999999997,"15min":0.86299999999999999},"min":{"1min":0.55600000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.93500000000000005,"15min":0.96499999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.105,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.747},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.89200000000000002,"15min":0.91100000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.276,"5min":1.47,"15min":1.47},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.98399999999999999,"15min":1.014},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0600000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.006,"15min":1.046},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3100000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.73499999999999999}]},{"osd":8,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.97999999999999998,"15min":1.0009999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.0489999999999999,"15min":1.0800000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.155,"5min":2.379,"15min":2.379},"last":1.0349999999999999}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.92700000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2430000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.96699999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.093},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.0780000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168613,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":995328,"data_stored":474381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66600000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.80500000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.747},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.78600000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.81799999999999995}]},{"osd":7,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.878}]},{"osd":9,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.495,"15min":0.495},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.76300000000000001,"15min":0.755},"min":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.85399999999999998}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.55200000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.90100000000000002}]}]},{"osd":7,"up_from":49,"seq":210453397542,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1593344,"data_stored":1071129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.371},{"interface":"front","average":{"1min":0.437,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.34899999999999998}]},{"osd":2,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.372,"15min":0.372},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51300000000000001}]},{"osd":3,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.624,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.438},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.41999999999999998}]},{"osd":5,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61199999999999999,"15min":0.60799999999999998},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86699999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.621},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63700000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.48299999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60999999999999999}]},{"osd":9,"last update":"Sat Mar 30 04:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.85499999999999998,"5min":1.05,"15min":1.05},"last":2.2120000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76500000000000001,"15min":0.76700000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.72999999999999998}]},{"osd":10,"last update":"Sat Mar 30 03:59:45 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.65400000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69399999999999995}]}]},{"osd":6,"up_from":44,"seq":188978561067,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.56999999999999995,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.48799999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.65100000000000002,"15min":0.63200000000000001},"min":{"1min":0.51200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.67100000000000004},"min":{"1min":0.55200000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.57299999999999995,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.69899999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.442,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.58299999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76700000000000002,"15min":0.749},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.61399999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.337,"5min":1.462,"15min":1.462},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.68300000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.1060000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.016},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.93999999999999995}]},{"osd":8,"last update":"Sat Mar 30 04:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.85999999999999999,"15min":0.85099999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1240000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.92400000000000004,"15min":0.91200000000000003},"min":{"1min":0.748,"5min":0.63,"15min":0.63},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.879,"15min":0.86599999999999999},"min":{"1min":0.61899999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.81499999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95199999999999996,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.075,"5min":0.97499999999999998,"15min":0.95899999999999996},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.1830000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.163},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.2210000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":1044480,"data_stored":533392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54000000000000004,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.51900000000000002,"15min":0.47099999999999997},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.54600000000000004,"15min":0.498},"min":{"1min":0.40100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.79500000000000004}]},{"osd":3,"last update":"Sat Mar 30 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.498},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.53800000000000003},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.871,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.32500000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.746},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92300000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.629}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84699999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.55200000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77300000000000002}]},{"osd":9,"last update":"Sat Mar 30 04:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.82399999999999995,"15min":0.80000000000000004},"min":{"1min":0.67800000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":3.2080000000000002,"5min":3.2080000000000002,"15min":3.2080000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.89700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.75900000000000001}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.91800000000000004}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.95499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":470915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.81000000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:00:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.59599999999999997,"15min":0.56000000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94699999999999995,"5min":0.999,"15min":0.999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.51000000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.47399999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.42399999999999999}]},{"osd":4,"last update":"Sat Mar 30 03:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85399999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68899999999999995,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90500000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.70699999999999996,"15min":0.67100000000000004},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69899999999999995,"15min":0.66800000000000004},"min":{"1min":0.51000000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67900000000000005,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.498},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.76200000000000001,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.90400000000000003}]},{"osd":7,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.54500000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.68999999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.1180000000000001,"5min":1.333,"15min":1.333},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.67500000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.038,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.623}]},{"osd":9,"last update":"Sat Mar 30 04:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.95999999999999996,"15min":0.96299999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.94699999999999995}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.71899999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":533171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.57899999999999996},"min":{"1min":0.44500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82899999999999996,"5min":1.133,"15min":1.133},"last":0.65600000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53200000000000003,"15min":0.50800000000000001},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53200000000000003,"15min":0.51500000000000001},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.755}]},{"osd":4,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.623,"15min":0.61099999999999999},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82599999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54600000000000004}]},{"osd":9,"last update":"Sat Mar 30 04:00:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.58799999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79200000000000004}]},{"osd":10,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.875},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71299999999999997,"15min":0.69999999999999996},"min":{"1min":0.58799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":0.93700000000000006,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.84199999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.73299999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":469215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46100000000000002,"15min":0.41499999999999998},"min":{"1min":0.36699999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.438},"min":{"1min":0.39000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66600000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":1.762,"15min":1.762},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.61699999999999999},"min":{"1min":0.52100000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.032,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.66000000000000003,"15min":0.61799999999999999},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98199999999999998,"5min":1.181,"15min":1.181},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.626},"min":{"1min":0.433,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.94299999999999995,"5min":1.296,"15min":1.296},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88200000000000001,"5min":1.113,"15min":1.113},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.70599999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.591,"15min":1.591},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70299999999999996,"15min":0.66500000000000004},"min":{"1min":0.61099999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93799999999999994,"5min":1.139,"15min":1.139},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.77000000000000002,"15min":0.72999999999999998},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0569999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.77200000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.75600000000000001},"min":{"1min":0.48399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1220000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.752,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0129999999999999,"5min":1.278,"15min":1.278},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.214,"15min":1.214},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1499999999999999,"5min":1.242,"15min":1.242},"last":0.96299999999999997}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.014,"15min":1.026},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.165,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.93999999999999995,"15min":0.95599999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.077,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.69499999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.02,"5min":0.99299999999999999,"15min":0.98899999999999999},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.006}]},{"osd":11,"last update":"Sat Mar 30 04:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.73899999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":983040,"data_stored":473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59999999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.307,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.52700000000000002},"min":{"1min":0.46000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.624},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83699999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Mar 30 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.51400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.88700000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.85299999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.52600000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.754},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.40000000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.92800000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72099999999999997}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93999999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.876},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.51900000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90000000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.89100000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.79500000000000004,"15min":0.79700000000000004},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.96299999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.91100000000000003}]},{"osd":10,"last update":"Sat Mar 30 04:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.749,"15min":0.75700000000000001},"min":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.746,"15min":0.754},"min":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.92300000000000004}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.94999999999999996}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712472,"statfs":{"total":95995035648,"available":95961571328,"internally_reserved":0,"allocated":1597440,"data_stored":1065690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.751,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":1.681}]},{"osd":1,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.45300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.42699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56899999999999995}]},{"osd":2,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.246,"15min":0.246},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42399999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":0.873},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":1.7070000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.625},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.73399999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.48499999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63400000000000001,"15min":0.629},"min":{"1min":0.54500000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.65800000000000003}]},{"osd":7,"last update":"Sat Mar 30 03:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.76500000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.58599999999999997}]},{"osd":9,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72699999999999998,"15min":0.71699999999999997},"min":{"1min":0.58399999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.67300000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.70799999999999996}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.82899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:00:53.592 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph pg dump --format=json 2024-03-30T04:00:53.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:53 smithi154 ceph-mon[79161]: pgmap v274: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-30T04:00:53.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:53 smithi193 ceph-mon[79581]: pgmap v274: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-30T04:00:53.961 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:54.628 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:54 smithi023 ceph-mon[38254]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:54 smithi154 ceph-mon[79161]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:54.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:54 smithi193 ceph-mon[79581]: from='client.15030 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.122+0000 7f31a3e00700 1 -- 172.21.15.23:0/1435351210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 msgr2=0x7f319c1012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.122+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1435351210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1012e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31880099c0 tx=0x7f3188009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 -- 172.21.15.23:0/1435351210 shutdown_connections 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1435351210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c101820 0x7f319c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1435351210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c106ef0 0x7f319c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1435351210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1012e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 -- 172.21.15.23:0/1435351210 >> 172.21.15.23:0/1435351210 conn(0x7f319c077fa0 msgr2=0x7f319c0783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 -- 172.21.15.23:0/1435351210 shutdown_connections 2024-03-30T04:00:55.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.123+0000 7f31a3e00700 1 -- 172.21.15.23:0/1435351210 wait complete. 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 Processor -- start 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 -- start start 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 0x7f319c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c106ef0 0x7f319c19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c19c0e0 con 0x7f319c100e60 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c19c250 con 0x7f319c101820 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a3e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c19c410 con 0x7f319c106ef0 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:55.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.124+0000 7f31a1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37150/0 (socket says 172.21.15.23:37150) 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a1400700 1 -- 172.21.15.23:0/1443563451 learned_addr learned my addr 172.21.15.23:0/1443563451 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 0x7f319c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a1e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c106ef0 0x7f319c19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 -- 172.21.15.23:0/1443563451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c106ef0 msgr2=0x7f319c19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c106ef0 0x7f319c19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 -- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 msgr2=0x7f319c1971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 -- 172.21.15.23:0/1443563451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31880095f0 con 0x7f319c101820 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a1400700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.125+0000 7f31a0a00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 0x7f319c197720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31880098e0 tx=0x7f318800b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:55.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.126+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318801d070 con 0x7f319c101820 2024-03-30T04:00:55.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.126+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c19c690 con 0x7f319c101820 2024-03-30T04:00:55.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.126+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f318800be00 con 0x7f319c101820 2024-03-30T04:00:55.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.126+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318800f9a0 con 0x7f319c101820 2024-03-30T04:00:55.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.126+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c1a3f50 con 0x7f319c101820 2024-03-30T04:00:55.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.127+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c1092f0 con 0x7f319c101820 2024-03-30T04:00:55.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.129+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f3188004c60 con 0x7f319c101820 2024-03-30T04:00:55.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.130+0000 7f319a000700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 0x7f31840a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:55.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.130+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31880cdcc0 con 0x7f319c101820 2024-03-30T04:00:55.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.131+0000 7f31a1400700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 0x7f31840a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:55.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.131+0000 7f31a1400700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 0x7f31840a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3190011550 tx=0x7f3190009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:55.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.133+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3188097010 con 0x7f319c101820 2024-03-30T04:00:55.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.342+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f319c04ea80 con 0x7f31840a11b0 2024-03-30T04:00:55.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.355+0000 7f319a000700 1 -- 172.21.15.23:0/1443563451 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426648 (secure 0 0 0) 0x7f319c04ea80 con 0x7f31840a11b0 2024-03-30T04:00:55.361 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 msgr2=0x7f31840a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 0x7f31840a3670 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3190011550 tx=0x7f3190009260 comp rx=0 tx=0).stop 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 msgr2=0x7f319c197720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 0x7f319c197720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f31880098e0 tx=0x7f318800b9c0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 shutdown_connections 2024-03-30T04:00:55.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f31840a11b0 0x7f31840a3670 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c106ef0 0x7f319c19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f319c100e60 0x7f319c1971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 --2- 172.21.15.23:0/1443563451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319c101820 0x7f319c197720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 >> 172.21.15.23:0/1443563451 conn(0x7f319c077fa0 msgr2=0x7f319c0ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 shutdown_connections 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:55.361+0000 7f31a3e00700 1 -- 172.21.15.23:0/1443563451 wait complete. 2024-03-30T04:00:55.365 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:00:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:55 smithi154 ceph-mon[79161]: pgmap v275: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:00:55.727 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:55 smithi023 ceph-mon[38254]: pgmap v275: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:00:55.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:55 smithi193 ceph-mon[79581]: pgmap v275: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:00:55.751 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-03-30T04:00:54.291653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660128,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":583,"num_read_kb":422,"num_write":418,"num_write_kb":3078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370576,"kb_used_data":13564,"kb_used_omap":18,"kb_used_meta":356845,"kb_avail":1124571248,"statfs":{"total":1151940427776,"available":1151560957952,"internally_reserved":0,"allocated":13889536,"data_stored":7637539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":365409605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":134,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003567+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T04:00:17.003567+0000","last_peered":"2024-03-30T04:00:17.003567+0000","last_clean":"2024-03-30T04:00:17.003567+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T04:00:17.003567+0000","last_undegraded":"2024-03-30T04:00:17.003567+0000","last_fullsized":"2024-03-30T04:00:17.003567+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009193+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T04:00:17.009193+0000","last_peered":"2024-03-30T04:00:17.009193+0000","last_clean":"2024-03-30T04:00:17.009193+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T04:00:17.009193+0000","last_undegraded":"2024-03-30T04:00:17.009193+0000","last_fullsized":"2024-03-30T04:00:17.009193+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009223+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T04:00:17.009223+0000","last_peered":"2024-03-30T04:00:17.009223+0000","last_clean":"2024-03-30T04:00:17.009223+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T04:00:17.009223+0000","last_undegraded":"2024-03-30T04:00:17.009223+0000","last_fullsized":"2024-03-30T04:00:17.009223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002560+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T04:00:17.002560+0000","last_peered":"2024-03-30T04:00:17.002560+0000","last_clean":"2024-03-30T04:00:17.002560+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T04:00:17.002560+0000","last_undegraded":"2024-03-30T04:00:17.002560+0000","last_fullsized":"2024-03-30T04:00:17.002560+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003129+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T04:00:17.003129+0000","last_peered":"2024-03-30T04:00:17.003129+0000","last_clean":"2024-03-30T04:00:17.003129+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T04:00:17.003129+0000","last_undegraded":"2024-03-30T04:00:17.003129+0000","last_fullsized":"2024-03-30T04:00:17.003129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007834+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T04:00:17.007834+0000","last_peered":"2024-03-30T04:00:17.007834+0000","last_clean":"2024-03-30T04:00:17.007834+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T04:00:17.007834+0000","last_undegraded":"2024-03-30T04:00:17.007834+0000","last_fullsized":"2024-03-30T04:00:17.007834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007464+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T04:00:17.007464+0000","last_peered":"2024-03-30T04:00:17.007464+0000","last_clean":"2024-03-30T04:00:17.007464+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T04:00:17.007464+0000","last_undegraded":"2024-03-30T04:00:17.007464+0000","last_fullsized":"2024-03-30T04:00:17.007464+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002761+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T04:00:17.002761+0000","last_peered":"2024-03-30T04:00:17.002761+0000","last_clean":"2024-03-30T04:00:17.002761+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T04:00:17.002761+0000","last_undegraded":"2024-03-30T04:00:17.002761+0000","last_fullsized":"2024-03-30T04:00:17.002761+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999414+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T04:00:16.999414+0000","last_peered":"2024-03-30T04:00:16.999414+0000","last_clean":"2024-03-30T04:00:16.999414+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T04:00:16.999414+0000","last_undegraded":"2024-03-30T04:00:16.999414+0000","last_fullsized":"2024-03-30T04:00:16.999414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008100+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T04:00:17.008100+0000","last_peered":"2024-03-30T04:00:17.008100+0000","last_clean":"2024-03-30T04:00:17.008100+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T04:00:17.008100+0000","last_undegraded":"2024-03-30T04:00:17.008100+0000","last_fullsized":"2024-03-30T04:00:17.008100+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007963+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T04:00:17.007963+0000","last_peered":"2024-03-30T04:00:17.007963+0000","last_clean":"2024-03-30T04:00:17.007963+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T04:00:17.007963+0000","last_undegraded":"2024-03-30T04:00:17.007963+0000","last_fullsized":"2024-03-30T04:00:17.007963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002431+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T04:00:17.002431+0000","last_peered":"2024-03-30T04:00:17.002431+0000","last_clean":"2024-03-30T04:00:17.002431+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T04:00:17.002431+0000","last_undegraded":"2024-03-30T04:00:17.002431+0000","last_fullsized":"2024-03-30T04:00:17.002431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998864+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T04:00:16.998864+0000","last_peered":"2024-03-30T04:00:16.998864+0000","last_clean":"2024-03-30T04:00:16.998864+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T04:00:16.998864+0000","last_undegraded":"2024-03-30T04:00:16.998864+0000","last_fullsized":"2024-03-30T04:00:16.998864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007636+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T04:00:17.007636+0000","last_peered":"2024-03-30T04:00:17.007636+0000","last_clean":"2024-03-30T04:00:17.007636+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T04:00:17.007636+0000","last_undegraded":"2024-03-30T04:00:17.007636+0000","last_fullsized":"2024-03-30T04:00:17.007636+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007543+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T04:00:17.007543+0000","last_peered":"2024-03-30T04:00:17.007543+0000","last_clean":"2024-03-30T04:00:17.007543+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T04:00:17.007543+0000","last_undegraded":"2024-03-30T04:00:17.007543+0000","last_fullsized":"2024-03-30T04:00:17.007543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998890+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T04:00:16.998890+0000","last_peered":"2024-03-30T04:00:16.998890+0000","last_clean":"2024-03-30T04:00:16.998890+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T04:00:16.998890+0000","last_undegraded":"2024-03-30T04:00:16.998890+0000","last_fullsized":"2024-03-30T04:00:16.998890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003437+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T04:00:17.003437+0000","last_peered":"2024-03-30T04:00:17.003437+0000","last_clean":"2024-03-30T04:00:17.003437+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T04:00:17.003437+0000","last_undegraded":"2024-03-30T04:00:17.003437+0000","last_fullsized":"2024-03-30T04:00:17.003437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005299+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T04:00:17.005299+0000","last_peered":"2024-03-30T04:00:17.005299+0000","last_clean":"2024-03-30T04:00:17.005299+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T04:00:17.005299+0000","last_undegraded":"2024-03-30T04:00:17.005299+0000","last_fullsized":"2024-03-30T04:00:17.005299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999974+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T04:00:16.999974+0000","last_peered":"2024-03-30T04:00:16.999974+0000","last_clean":"2024-03-30T04:00:16.999974+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T04:00:16.999974+0000","last_undegraded":"2024-03-30T04:00:16.999974+0000","last_fullsized":"2024-03-30T04:00:16.999974+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.000016+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T04:00:17.000016+0000","last_peered":"2024-03-30T04:00:17.000016+0000","last_clean":"2024-03-30T04:00:17.000016+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T04:00:17.000016+0000","last_undegraded":"2024-03-30T04:00:17.000016+0000","last_fullsized":"2024-03-30T04:00:17.000016+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'13","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:50.024355+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:00:50.024355+0000","last_peered":"2024-03-30T04:00:50.024355+0000","last_clean":"2024-03-30T04:00:50.024355+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:00:50.024355+0000","last_undegraded":"2024-03-30T04:00:50.024355+0000","last_fullsized":"2024-03-30T04:00:50.024355+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005839+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T04:00:17.005839+0000","last_peered":"2024-03-30T04:00:17.005839+0000","last_clean":"2024-03-30T04:00:17.005839+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T04:00:17.005839+0000","last_undegraded":"2024-03-30T04:00:17.005839+0000","last_fullsized":"2024-03-30T04:00:17.005839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002090+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T04:00:17.002090+0000","last_peered":"2024-03-30T04:00:17.002090+0000","last_clean":"2024-03-30T04:00:17.002090+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T04:00:17.002090+0000","last_undegraded":"2024-03-30T04:00:17.002090+0000","last_fullsized":"2024-03-30T04:00:17.002090+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005933+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T04:00:17.005933+0000","last_peered":"2024-03-30T04:00:17.005933+0000","last_clean":"2024-03-30T04:00:17.005933+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T04:00:17.005933+0000","last_undegraded":"2024-03-30T04:00:17.005933+0000","last_fullsized":"2024-03-30T04:00:17.005933+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009523+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T04:00:17.009523+0000","last_peered":"2024-03-30T04:00:17.009523+0000","last_clean":"2024-03-30T04:00:17.009523+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T04:00:17.009523+0000","last_undegraded":"2024-03-30T04:00:17.009523+0000","last_fullsized":"2024-03-30T04:00:17.009523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.019833+0000","last_change":"2024-03-30T04:00:18.019833+0000","last_active":"2024-03-30T04:00:18.019833+0000","last_peered":"2024-03-30T04:00:18.019833+0000","last_clean":"2024-03-30T04:00:18.019833+0000","last_became_active":"2024-03-30T04:00:18.019445+0000","last_became_peered":"2024-03-30T04:00:18.019445+0000","last_unstale":"2024-03-30T04:00:18.019833+0000","last_undegraded":"2024-03-30T04:00:18.019833+0000","last_fullsized":"2024-03-30T04:00:18.019833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:41:55.882417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.867086+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:00:17.867086+0000","last_peered":"2024-03-30T04:00:17.867086+0000","last_clean":"2024-03-30T04:00:17.867086+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:00:17.867086+0000","last_undegraded":"2024-03-30T04:00:17.867086+0000","last_fullsized":"2024-03-30T04:00:17.867086+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999282+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T04:00:16.999282+0000","last_peered":"2024-03-30T04:00:16.999282+0000","last_clean":"2024-03-30T04:00:16.999282+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T04:00:16.999282+0000","last_undegraded":"2024-03-30T04:00:16.999282+0000","last_fullsized":"2024-03-30T04:00:16.999282+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002020+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T04:00:17.002020+0000","last_peered":"2024-03-30T04:00:17.002020+0000","last_clean":"2024-03-30T04:00:17.002020+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T04:00:17.002020+0000","last_undegraded":"2024-03-30T04:00:17.002020+0000","last_fullsized":"2024-03-30T04:00:17.002020+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004349+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:00:17.004349+0000","last_peered":"2024-03-30T04:00:17.004349+0000","last_clean":"2024-03-30T04:00:17.004349+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:00:17.004349+0000","last_undegraded":"2024-03-30T04:00:17.004349+0000","last_fullsized":"2024-03-30T04:00:17.004349+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004142+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T04:00:17.004142+0000","last_peered":"2024-03-30T04:00:17.004142+0000","last_clean":"2024-03-30T04:00:17.004142+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T04:00:17.004142+0000","last_undegraded":"2024-03-30T04:00:17.004142+0000","last_fullsized":"2024-03-30T04:00:17.004142+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999784+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T04:00:16.999784+0000","last_peered":"2024-03-30T04:00:16.999784+0000","last_clean":"2024-03-30T04:00:16.999784+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T04:00:16.999784+0000","last_undegraded":"2024-03-30T04:00:16.999784+0000","last_fullsized":"2024-03-30T04:00:16.999784+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'5","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:51.996494+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:00:51.996494+0000","last_peered":"2024-03-30T04:00:51.996494+0000","last_clean":"2024-03-30T04:00:51.996494+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:00:51.996494+0000","last_undegraded":"2024-03-30T04:00:51.996494+0000","last_fullsized":"2024-03-30T04:00:51.996494+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003626+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T04:00:17.003626+0000","last_peered":"2024-03-30T04:00:17.003626+0000","last_clean":"2024-03-30T04:00:17.003626+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T04:00:17.003626+0000","last_undegraded":"2024-03-30T04:00:17.003626+0000","last_fullsized":"2024-03-30T04:00:17.003626+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.000058+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T04:00:17.000058+0000","last_peered":"2024-03-30T04:00:17.000058+0000","last_clean":"2024-03-30T04:00:17.000058+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T04:00:17.000058+0000","last_undegraded":"2024-03-30T04:00:17.000058+0000","last_fullsized":"2024-03-30T04:00:17.000058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007735+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:00:17.007735+0000","last_peered":"2024-03-30T04:00:17.007735+0000","last_clean":"2024-03-30T04:00:17.007735+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:00:17.007735+0000","last_undegraded":"2024-03-30T04:00:17.007735+0000","last_fullsized":"2024-03-30T04:00:17.007735+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003122+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T04:00:17.003122+0000","last_peered":"2024-03-30T04:00:17.003122+0000","last_clean":"2024-03-30T04:00:17.003122+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T04:00:17.003122+0000","last_undegraded":"2024-03-30T04:00:17.003122+0000","last_fullsized":"2024-03-30T04:00:17.003122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003360+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T04:00:17.003360+0000","last_peered":"2024-03-30T04:00:17.003360+0000","last_clean":"2024-03-30T04:00:17.003360+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T04:00:17.003360+0000","last_undegraded":"2024-03-30T04:00:17.003360+0000","last_fullsized":"2024-03-30T04:00:17.003360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998384+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:00:16.998384+0000","last_peered":"2024-03-30T04:00:16.998384+0000","last_clean":"2024-03-30T04:00:16.998384+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:00:16.998384+0000","last_undegraded":"2024-03-30T04:00:16.998384+0000","last_fullsized":"2024-03-30T04:00:16.998384+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003489+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T04:00:17.003489+0000","last_peered":"2024-03-30T04:00:17.003489+0000","last_clean":"2024-03-30T04:00:17.003489+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T04:00:17.003489+0000","last_undegraded":"2024-03-30T04:00:17.003489+0000","last_fullsized":"2024-03-30T04:00:17.003489+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009143+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T04:00:17.009143+0000","last_peered":"2024-03-30T04:00:17.009143+0000","last_clean":"2024-03-30T04:00:17.009143+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T04:00:17.009143+0000","last_undegraded":"2024-03-30T04:00:17.009143+0000","last_fullsized":"2024-03-30T04:00:17.009143+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005605+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:00:17.005605+0000","last_peered":"2024-03-30T04:00:17.005605+0000","last_clean":"2024-03-30T04:00:17.005605+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:00:17.005605+0000","last_undegraded":"2024-03-30T04:00:17.005605+0000","last_fullsized":"2024-03-30T04:00:17.005605+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002874+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T04:00:17.002874+0000","last_peered":"2024-03-30T04:00:17.002874+0000","last_clean":"2024-03-30T04:00:17.002874+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T04:00:17.002874+0000","last_undegraded":"2024-03-30T04:00:17.002874+0000","last_fullsized":"2024-03-30T04:00:17.002874+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003127+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T04:00:17.003127+0000","last_peered":"2024-03-30T04:00:17.003127+0000","last_clean":"2024-03-30T04:00:17.003127+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T04:00:17.003127+0000","last_undegraded":"2024-03-30T04:00:17.003127+0000","last_fullsized":"2024-03-30T04:00:17.003127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"84'4","reported_seq":107,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:53.998999+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:00:53.998999+0000","last_peered":"2024-03-30T04:00:53.998999+0000","last_clean":"2024-03-30T04:00:53.998999+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:00:53.998999+0000","last_undegraded":"2024-03-30T04:00:53.998999+0000","last_fullsized":"2024-03-30T04:00:53.998999+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":85,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999273+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T04:00:16.999273+0000","last_peered":"2024-03-30T04:00:16.999273+0000","last_clean":"2024-03-30T04:00:16.999273+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T04:00:16.999273+0000","last_undegraded":"2024-03-30T04:00:16.999273+0000","last_fullsized":"2024-03-30T04:00:16.999273+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007276+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T04:00:17.007276+0000","last_peered":"2024-03-30T04:00:17.007276+0000","last_clean":"2024-03-30T04:00:17.007276+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T04:00:17.007276+0000","last_undegraded":"2024-03-30T04:00:17.007276+0000","last_fullsized":"2024-03-30T04:00:17.007276+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007850+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:00:17.007850+0000","last_peered":"2024-03-30T04:00:17.007850+0000","last_clean":"2024-03-30T04:00:17.007850+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:00:17.007850+0000","last_undegraded":"2024-03-30T04:00:17.007850+0000","last_fullsized":"2024-03-30T04:00:17.007850+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.016147+0000","last_change":"2024-03-30T04:00:18.016147+0000","last_active":"2024-03-30T04:00:18.016147+0000","last_peered":"2024-03-30T04:00:18.016147+0000","last_clean":"2024-03-30T04:00:18.016147+0000","last_became_active":"2024-03-30T04:00:18.015780+0000","last_became_peered":"2024-03-30T04:00:18.015780+0000","last_unstale":"2024-03-30T04:00:18.016147+0000","last_undegraded":"2024-03-30T04:00:18.016147+0000","last_fullsized":"2024-03-30T04:00:18.016147+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:00:33.540122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002787+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T04:00:17.002787+0000","last_peered":"2024-03-30T04:00:17.002787+0000","last_clean":"2024-03-30T04:00:17.002787+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T04:00:17.002787+0000","last_undegraded":"2024-03-30T04:00:17.002787+0000","last_fullsized":"2024-03-30T04:00:17.002787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004047+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:00:17.004047+0000","last_peered":"2024-03-30T04:00:17.004047+0000","last_clean":"2024-03-30T04:00:17.004047+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:00:17.004047+0000","last_undegraded":"2024-03-30T04:00:17.004047+0000","last_fullsized":"2024-03-30T04:00:17.004047+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999454+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T04:00:16.999454+0000","last_peered":"2024-03-30T04:00:16.999454+0000","last_clean":"2024-03-30T04:00:16.999454+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T04:00:16.999454+0000","last_undegraded":"2024-03-30T04:00:16.999454+0000","last_fullsized":"2024-03-30T04:00:16.999454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002904+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T04:00:17.002904+0000","last_peered":"2024-03-30T04:00:17.002904+0000","last_clean":"2024-03-30T04:00:17.002904+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T04:00:17.002904+0000","last_undegraded":"2024-03-30T04:00:17.002904+0000","last_fullsized":"2024-03-30T04:00:17.002904+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'34","reported_seq":148,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:54.004035+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:00:54.004035+0000","last_peered":"2024-03-30T04:00:54.004035+0000","last_clean":"2024-03-30T04:00:54.004035+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:00:54.004035+0000","last_undegraded":"2024-03-30T04:00:54.004035+0000","last_fullsized":"2024-03-30T04:00:54.004035+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62510,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":96,"num_read_kb":110,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003952+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T04:00:17.003952+0000","last_peered":"2024-03-30T04:00:17.003952+0000","last_clean":"2024-03-30T04:00:17.003952+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T04:00:17.003952+0000","last_undegraded":"2024-03-30T04:00:17.003952+0000","last_fullsized":"2024-03-30T04:00:17.003952+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004359+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T04:00:17.004359+0000","last_peered":"2024-03-30T04:00:17.004359+0000","last_clean":"2024-03-30T04:00:17.004359+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T04:00:17.004359+0000","last_undegraded":"2024-03-30T04:00:17.004359+0000","last_fullsized":"2024-03-30T04:00:17.004359+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999656+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:00:16.999656+0000","last_peered":"2024-03-30T04:00:16.999656+0000","last_clean":"2024-03-30T04:00:16.999656+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:00:16.999656+0000","last_undegraded":"2024-03-30T04:00:16.999656+0000","last_fullsized":"2024-03-30T04:00:16.999656+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998526+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T04:00:16.998526+0000","last_peered":"2024-03-30T04:00:16.998526+0000","last_clean":"2024-03-30T04:00:16.998526+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T04:00:16.998526+0000","last_undegraded":"2024-03-30T04:00:16.998526+0000","last_fullsized":"2024-03-30T04:00:16.998526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003982+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T04:00:17.003982+0000","last_peered":"2024-03-30T04:00:17.003982+0000","last_clean":"2024-03-30T04:00:17.003982+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T04:00:17.003982+0000","last_undegraded":"2024-03-30T04:00:17.003982+0000","last_fullsized":"2024-03-30T04:00:17.003982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003013+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:00:17.003013+0000","last_peered":"2024-03-30T04:00:17.003013+0000","last_clean":"2024-03-30T04:00:17.003013+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:00:17.003013+0000","last_undegraded":"2024-03-30T04:00:17.003013+0000","last_fullsized":"2024-03-30T04:00:17.003013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007223+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T04:00:17.007223+0000","last_peered":"2024-03-30T04:00:17.007223+0000","last_clean":"2024-03-30T04:00:17.007223+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T04:00:17.007223+0000","last_undegraded":"2024-03-30T04:00:17.007223+0000","last_fullsized":"2024-03-30T04:00:17.007223+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.021325+0000","last_change":"2024-03-30T04:00:18.020869+0000","last_active":"2024-03-30T04:00:18.021325+0000","last_peered":"2024-03-30T04:00:18.021325+0000","last_clean":"2024-03-30T04:00:18.021325+0000","last_became_active":"2024-03-30T04:00:18.020410+0000","last_became_peered":"2024-03-30T04:00:18.020410+0000","last_unstale":"2024-03-30T04:00:18.021325+0000","last_undegraded":"2024-03-30T04:00:18.021325+0000","last_fullsized":"2024-03-30T04:00:18.021325+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:53.675022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"84'6","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:49.023400+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:00:49.023400+0000","last_peered":"2024-03-30T04:00:49.023400+0000","last_clean":"2024-03-30T04:00:49.023400+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:00:49.023400+0000","last_undegraded":"2024-03-30T04:00:49.023400+0000","last_fullsized":"2024-03-30T04:00:49.023400+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007454+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T04:00:17.007454+0000","last_peered":"2024-03-30T04:00:17.007454+0000","last_clean":"2024-03-30T04:00:17.007454+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T04:00:17.007454+0000","last_undegraded":"2024-03-30T04:00:17.007454+0000","last_fullsized":"2024-03-30T04:00:17.007454+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.288237+0000","last_change":"2024-03-30T04:00:18.288237+0000","last_active":"2024-03-30T04:00:18.288237+0000","last_peered":"2024-03-30T04:00:18.288237+0000","last_clean":"2024-03-30T04:00:18.288237+0000","last_became_active":"2024-03-30T04:00:18.287748+0000","last_became_peered":"2024-03-30T04:00:18.287748+0000","last_unstale":"2024-03-30T04:00:18.288237+0000","last_undegraded":"2024-03-30T04:00:18.288237+0000","last_fullsized":"2024-03-30T04:00:18.288237+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:50.893522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003013+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:00:17.003013+0000","last_peered":"2024-03-30T04:00:17.003013+0000","last_clean":"2024-03-30T04:00:17.003013+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:00:17.003013+0000","last_undegraded":"2024-03-30T04:00:17.003013+0000","last_fullsized":"2024-03-30T04:00:17.003013+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005288+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T04:00:17.005288+0000","last_peered":"2024-03-30T04:00:17.005288+0000","last_clean":"2024-03-30T04:00:17.005288+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T04:00:17.005288+0000","last_undegraded":"2024-03-30T04:00:17.005288+0000","last_fullsized":"2024-03-30T04:00:17.005288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005097+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T04:00:17.005097+0000","last_peered":"2024-03-30T04:00:17.005097+0000","last_clean":"2024-03-30T04:00:17.005097+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T04:00:17.005097+0000","last_undegraded":"2024-03-30T04:00:17.005097+0000","last_fullsized":"2024-03-30T04:00:17.005097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002818+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:00:17.002818+0000","last_peered":"2024-03-30T04:00:17.002818+0000","last_clean":"2024-03-30T04:00:17.002818+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:00:17.002818+0000","last_undegraded":"2024-03-30T04:00:17.002818+0000","last_fullsized":"2024-03-30T04:00:17.002818+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002909+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T04:00:17.002909+0000","last_peered":"2024-03-30T04:00:17.002909+0000","last_clean":"2024-03-30T04:00:17.002909+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T04:00:17.002909+0000","last_undegraded":"2024-03-30T04:00:17.002909+0000","last_fullsized":"2024-03-30T04:00:17.002909+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003547+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T04:00:17.003547+0000","last_peered":"2024-03-30T04:00:17.003547+0000","last_clean":"2024-03-30T04:00:17.003547+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T04:00:17.003547+0000","last_undegraded":"2024-03-30T04:00:17.003547+0000","last_fullsized":"2024-03-30T04:00:17.003547+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007033+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T04:00:17.007033+0000","last_peered":"2024-03-30T04:00:17.007033+0000","last_clean":"2024-03-30T04:00:17.007033+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T04:00:17.007033+0000","last_undegraded":"2024-03-30T04:00:17.007033+0000","last_fullsized":"2024-03-30T04:00:17.007033+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007538+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T04:00:17.007538+0000","last_peered":"2024-03-30T04:00:17.007538+0000","last_clean":"2024-03-30T04:00:17.007538+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T04:00:17.007538+0000","last_undegraded":"2024-03-30T04:00:17.007538+0000","last_fullsized":"2024-03-30T04:00:17.007538+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998771+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T04:00:16.998771+0000","last_peered":"2024-03-30T04:00:16.998771+0000","last_clean":"2024-03-30T04:00:16.998771+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T04:00:16.998771+0000","last_undegraded":"2024-03-30T04:00:16.998771+0000","last_fullsized":"2024-03-30T04:00:16.998771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"84'6","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:51.026000+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T04:00:51.026000+0000","last_peered":"2024-03-30T04:00:51.026000+0000","last_clean":"2024-03-30T04:00:51.026000+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T04:00:51.026000+0000","last_undegraded":"2024-03-30T04:00:51.026000+0000","last_fullsized":"2024-03-30T04:00:51.026000+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":933,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007339+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T04:00:17.007339+0000","last_peered":"2024-03-30T04:00:17.007339+0000","last_clean":"2024-03-30T04:00:17.007339+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T04:00:17.007339+0000","last_undegraded":"2024-03-30T04:00:17.007339+0000","last_fullsized":"2024-03-30T04:00:17.007339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998956+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T04:00:16.998956+0000","last_peered":"2024-03-30T04:00:16.998956+0000","last_clean":"2024-03-30T04:00:16.998956+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T04:00:16.998956+0000","last_undegraded":"2024-03-30T04:00:16.998956+0000","last_fullsized":"2024-03-30T04:00:16.998956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.018515+0000","last_change":"2024-03-30T04:00:18.018515+0000","last_active":"2024-03-30T04:00:18.018515+0000","last_peered":"2024-03-30T04:00:18.018515+0000","last_clean":"2024-03-30T04:00:18.018515+0000","last_became_active":"2024-03-30T04:00:18.017823+0000","last_became_peered":"2024-03-30T04:00:18.017823+0000","last_unstale":"2024-03-30T04:00:18.018515+0000","last_undegraded":"2024-03-30T04:00:18.018515+0000","last_fullsized":"2024-03-30T04:00:18.018515+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:15:54.957524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"84'5","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:52.026928+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:00:52.026928+0000","last_peered":"2024-03-30T04:00:52.026928+0000","last_clean":"2024-03-30T04:00:52.026928+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:00:52.026928+0000","last_undegraded":"2024-03-30T04:00:52.026928+0000","last_fullsized":"2024-03-30T04:00:52.026928+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007309+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T04:00:17.007309+0000","last_peered":"2024-03-30T04:00:17.007309+0000","last_clean":"2024-03-30T04:00:17.007309+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T04:00:17.007309+0000","last_undegraded":"2024-03-30T04:00:17.007309+0000","last_fullsized":"2024-03-30T04:00:17.007309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003405+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T04:00:17.003405+0000","last_peered":"2024-03-30T04:00:17.003405+0000","last_clean":"2024-03-30T04:00:17.003405+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T04:00:17.003405+0000","last_undegraded":"2024-03-30T04:00:17.003405+0000","last_fullsized":"2024-03-30T04:00:17.003405+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002659+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:00:17.002659+0000","last_peered":"2024-03-30T04:00:17.002659+0000","last_clean":"2024-03-30T04:00:17.002659+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:00:17.002659+0000","last_undegraded":"2024-03-30T04:00:17.002659+0000","last_fullsized":"2024-03-30T04:00:17.002659+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":337,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.006941+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T04:00:17.006941+0000","last_peered":"2024-03-30T04:00:17.006941+0000","last_clean":"2024-03-30T04:00:17.006941+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T04:00:17.006941+0000","last_undegraded":"2024-03-30T04:00:17.006941+0000","last_fullsized":"2024-03-30T04:00:17.006941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.018249+0000","last_change":"2024-03-30T04:00:18.017573+0000","last_active":"2024-03-30T04:00:18.018249+0000","last_peered":"2024-03-30T04:00:18.018249+0000","last_clean":"2024-03-30T04:00:18.018249+0000","last_became_active":"2024-03-30T04:00:18.016937+0000","last_became_peered":"2024-03-30T04:00:18.016937+0000","last_unstale":"2024-03-30T04:00:18.018249+0000","last_undegraded":"2024-03-30T04:00:18.018249+0000","last_fullsized":"2024-03-30T04:00:18.018249+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:26:20.863180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998323+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T04:00:16.998323+0000","last_peered":"2024-03-30T04:00:16.998323+0000","last_clean":"2024-03-30T04:00:16.998323+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T04:00:16.998323+0000","last_undegraded":"2024-03-30T04:00:16.998323+0000","last_fullsized":"2024-03-30T04:00:16.998323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007021+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:00:17.007021+0000","last_peered":"2024-03-30T04:00:17.007021+0000","last_clean":"2024-03-30T04:00:17.007021+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:00:17.007021+0000","last_undegraded":"2024-03-30T04:00:17.007021+0000","last_fullsized":"2024-03-30T04:00:17.007021+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002873+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T04:00:17.002873+0000","last_peered":"2024-03-30T04:00:17.002873+0000","last_clean":"2024-03-30T04:00:17.002873+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T04:00:17.002873+0000","last_undegraded":"2024-03-30T04:00:17.002873+0000","last_fullsized":"2024-03-30T04:00:17.002873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004064+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:00:17.004064+0000","last_peered":"2024-03-30T04:00:17.004064+0000","last_clean":"2024-03-30T04:00:17.004064+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:00:17.004064+0000","last_undegraded":"2024-03-30T04:00:17.004064+0000","last_fullsized":"2024-03-30T04:00:17.004064+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999003+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T04:00:16.999003+0000","last_peered":"2024-03-30T04:00:16.999003+0000","last_clean":"2024-03-30T04:00:16.999003+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T04:00:16.999003+0000","last_undegraded":"2024-03-30T04:00:16.999003+0000","last_fullsized":"2024-03-30T04:00:16.999003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002570+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T04:00:17.002570+0000","last_peered":"2024-03-30T04:00:17.002570+0000","last_clean":"2024-03-30T04:00:17.002570+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T04:00:17.002570+0000","last_undegraded":"2024-03-30T04:00:17.002570+0000","last_fullsized":"2024-03-30T04:00:17.002570+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":46,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.321082+0000","last_change":"2024-03-30T04:00:18.297312+0000","last_active":"2024-03-30T04:00:18.321082+0000","last_peered":"2024-03-30T04:00:18.321082+0000","last_clean":"2024-03-30T04:00:18.321082+0000","last_became_active":"2024-03-30T04:00:18.288640+0000","last_became_peered":"2024-03-30T04:00:18.288640+0000","last_unstale":"2024-03-30T04:00:18.321082+0000","last_undegraded":"2024-03-30T04:00:18.321082+0000","last_fullsized":"2024-03-30T04:00:18.321082+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:21.664643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.006942+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T04:00:17.006942+0000","last_peered":"2024-03-30T04:00:17.006942+0000","last_clean":"2024-03-30T04:00:17.006942+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T04:00:17.006942+0000","last_undegraded":"2024-03-30T04:00:17.006942+0000","last_fullsized":"2024-03-30T04:00:17.006942+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999181+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T04:00:16.999181+0000","last_peered":"2024-03-30T04:00:16.999181+0000","last_clean":"2024-03-30T04:00:16.999181+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T04:00:16.999181+0000","last_undegraded":"2024-03-30T04:00:16.999181+0000","last_fullsized":"2024-03-30T04:00:16.999181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002441+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:00:17.002441+0000","last_peered":"2024-03-30T04:00:17.002441+0000","last_clean":"2024-03-30T04:00:17.002441+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:00:17.002441+0000","last_undegraded":"2024-03-30T04:00:17.002441+0000","last_fullsized":"2024-03-30T04:00:17.002441+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009264+0000","last_change":"2024-03-30T03:59:40.464994+0000","last_active":"2024-03-30T04:00:17.009264+0000","last_peered":"2024-03-30T04:00:17.009264+0000","last_clean":"2024-03-30T04:00:17.009264+0000","last_became_active":"2024-03-30T03:59:40.462862+0000","last_became_peered":"2024-03-30T03:59:40.462862+0000","last_unstale":"2024-03-30T04:00:17.009264+0000","last_undegraded":"2024-03-30T04:00:17.009264+0000","last_fullsized":"2024-03-30T04:00:17.009264+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:29:48.413081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002280+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T04:00:17.002280+0000","last_peered":"2024-03-30T04:00:17.002280+0000","last_clean":"2024-03-30T04:00:17.002280+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T04:00:17.002280+0000","last_undegraded":"2024-03-30T04:00:17.002280+0000","last_fullsized":"2024-03-30T04:00:17.002280+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.291229+0000","last_change":"2024-03-30T04:00:18.291229+0000","last_active":"2024-03-30T04:00:18.291229+0000","last_peered":"2024-03-30T04:00:18.291229+0000","last_clean":"2024-03-30T04:00:18.291229+0000","last_became_active":"2024-03-30T04:00:18.287954+0000","last_became_peered":"2024-03-30T04:00:18.287954+0000","last_unstale":"2024-03-30T04:00:18.291229+0000","last_undegraded":"2024-03-30T04:00:18.291229+0000","last_fullsized":"2024-03-30T04:00:18.291229+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:54.678907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002590+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T04:00:17.002590+0000","last_peered":"2024-03-30T04:00:17.002590+0000","last_clean":"2024-03-30T04:00:17.002590+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T04:00:17.002590+0000","last_undegraded":"2024-03-30T04:00:17.002590+0000","last_fullsized":"2024-03-30T04:00:17.002590+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003702+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T04:00:17.003702+0000","last_peered":"2024-03-30T04:00:17.003702+0000","last_clean":"2024-03-30T04:00:17.003702+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T04:00:17.003702+0000","last_undegraded":"2024-03-30T04:00:17.003702+0000","last_fullsized":"2024-03-30T04:00:17.003702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002944+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:00:17.002944+0000","last_peered":"2024-03-30T04:00:17.002944+0000","last_clean":"2024-03-30T04:00:17.002944+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:00:17.002944+0000","last_undegraded":"2024-03-30T04:00:17.002944+0000","last_fullsized":"2024-03-30T04:00:17.002944+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003830+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T04:00:17.003830+0000","last_peered":"2024-03-30T04:00:17.003830+0000","last_clean":"2024-03-30T04:00:17.003830+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T04:00:17.003830+0000","last_undegraded":"2024-03-30T04:00:17.003830+0000","last_fullsized":"2024-03-30T04:00:17.003830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002261+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T04:00:17.002261+0000","last_peered":"2024-03-30T04:00:17.002261+0000","last_clean":"2024-03-30T04:00:17.002261+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T04:00:17.002261+0000","last_undegraded":"2024-03-30T04:00:17.002261+0000","last_fullsized":"2024-03-30T04:00:17.002261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999378+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T04:00:16.999378+0000","last_peered":"2024-03-30T04:00:16.999378+0000","last_clean":"2024-03-30T04:00:16.999378+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T04:00:16.999378+0000","last_undegraded":"2024-03-30T04:00:16.999378+0000","last_fullsized":"2024-03-30T04:00:16.999378+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004857+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T04:00:17.004857+0000","last_peered":"2024-03-30T04:00:17.004857+0000","last_clean":"2024-03-30T04:00:17.004857+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T04:00:17.004857+0000","last_undegraded":"2024-03-30T04:00:17.004857+0000","last_fullsized":"2024-03-30T04:00:17.004857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.002249+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T04:00:17.002249+0000","last_peered":"2024-03-30T04:00:17.002249+0000","last_clean":"2024-03-30T04:00:17.002249+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T04:00:17.002249+0000","last_undegraded":"2024-03-30T04:00:17.002249+0000","last_fullsized":"2024-03-30T04:00:17.002249+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003241+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:00:17.003241+0000","last_peered":"2024-03-30T04:00:17.003241+0000","last_clean":"2024-03-30T04:00:17.003241+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:00:17.003241+0000","last_undegraded":"2024-03-30T04:00:17.003241+0000","last_fullsized":"2024-03-30T04:00:17.003241+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007355+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T04:00:17.007355+0000","last_peered":"2024-03-30T04:00:17.007355+0000","last_clean":"2024-03-30T04:00:17.007355+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T04:00:17.007355+0000","last_undegraded":"2024-03-30T04:00:17.007355+0000","last_fullsized":"2024-03-30T04:00:17.007355+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998944+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T04:00:16.998944+0000","last_peered":"2024-03-30T04:00:16.998944+0000","last_clean":"2024-03-30T04:00:16.998944+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T04:00:16.998944+0000","last_undegraded":"2024-03-30T04:00:16.998944+0000","last_fullsized":"2024-03-30T04:00:16.998944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:51.026108+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:00:51.026108+0000","last_peered":"2024-03-30T04:00:51.026108+0000","last_clean":"2024-03-30T04:00:51.026108+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:00:51.026108+0000","last_undegraded":"2024-03-30T04:00:51.026108+0000","last_fullsized":"2024-03-30T04:00:51.026108+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008135+0000","last_change":"2024-03-30T03:59:40.462459+0000","last_active":"2024-03-30T04:00:17.008135+0000","last_peered":"2024-03-30T04:00:17.008135+0000","last_clean":"2024-03-30T04:00:17.008135+0000","last_became_active":"2024-03-30T03:59:40.461692+0000","last_became_peered":"2024-03-30T03:59:40.461692+0000","last_unstale":"2024-03-30T04:00:17.008135+0000","last_undegraded":"2024-03-30T04:00:17.008135+0000","last_fullsized":"2024-03-30T04:00:17.008135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:46.872671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.007945+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T04:00:17.007945+0000","last_peered":"2024-03-30T04:00:17.007945+0000","last_clean":"2024-03-30T04:00:17.007945+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T04:00:17.007945+0000","last_undegraded":"2024-03-30T04:00:17.007945+0000","last_fullsized":"2024-03-30T04:00:17.007945+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004251+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:00:17.004251+0000","last_peered":"2024-03-30T04:00:17.004251+0000","last_clean":"2024-03-30T04:00:17.004251+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:00:17.004251+0000","last_undegraded":"2024-03-30T04:00:17.004251+0000","last_fullsized":"2024-03-30T04:00:17.004251+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.016967+0000","last_change":"2024-03-30T04:00:18.016967+0000","last_active":"2024-03-30T04:00:18.016967+0000","last_peered":"2024-03-30T04:00:18.016967+0000","last_clean":"2024-03-30T04:00:18.016967+0000","last_became_active":"2024-03-30T04:00:18.016555+0000","last_became_peered":"2024-03-30T04:00:18.016555+0000","last_unstale":"2024-03-30T04:00:18.016967+0000","last_undegraded":"2024-03-30T04:00:18.016967+0000","last_fullsized":"2024-03-30T04:00:18.016967+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:34:50.538682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005262+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T04:00:17.005262+0000","last_peered":"2024-03-30T04:00:17.005262+0000","last_clean":"2024-03-30T04:00:17.005262+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T04:00:17.005262+0000","last_undegraded":"2024-03-30T04:00:17.005262+0000","last_fullsized":"2024-03-30T04:00:17.005262+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999106+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:00:16.999106+0000","last_peered":"2024-03-30T04:00:16.999106+0000","last_clean":"2024-03-30T04:00:16.999106+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:00:16.999106+0000","last_undegraded":"2024-03-30T04:00:16.999106+0000","last_fullsized":"2024-03-30T04:00:16.999106+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.009212+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T04:00:17.009212+0000","last_peered":"2024-03-30T04:00:17.009212+0000","last_clean":"2024-03-30T04:00:17.009212+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T04:00:17.009212+0000","last_undegraded":"2024-03-30T04:00:17.009212+0000","last_fullsized":"2024-03-30T04:00:17.009212+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003921+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T04:00:17.003921+0000","last_peered":"2024-03-30T04:00:17.003921+0000","last_clean":"2024-03-30T04:00:17.003921+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T04:00:17.003921+0000","last_undegraded":"2024-03-30T04:00:17.003921+0000","last_fullsized":"2024-03-30T04:00:17.003921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-30T04:00:18.019138+0000","last_change":"2024-03-30T04:00:18.019138+0000","last_active":"2024-03-30T04:00:18.019138+0000","last_peered":"2024-03-30T04:00:18.019138+0000","last_clean":"2024-03-30T04:00:18.019138+0000","last_became_active":"2024-03-30T04:00:18.018528+0000","last_became_peered":"2024-03-30T04:00:18.018528+0000","last_unstale":"2024-03-30T04:00:18.019138+0000","last_undegraded":"2024-03-30T04:00:18.019138+0000","last_fullsized":"2024-03-30T04:00:18.019138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:27.603607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999251+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T04:00:16.999251+0000","last_peered":"2024-03-30T04:00:16.999251+0000","last_clean":"2024-03-30T04:00:16.999251+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T04:00:16.999251+0000","last_undegraded":"2024-03-30T04:00:16.999251+0000","last_fullsized":"2024-03-30T04:00:16.999251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008104+0000","last_change":"2024-03-30T03:59:40.454510+0000","last_active":"2024-03-30T04:00:17.008104+0000","last_peered":"2024-03-30T04:00:17.008104+0000","last_clean":"2024-03-30T04:00:17.008104+0000","last_became_active":"2024-03-30T03:59:40.454119+0000","last_became_peered":"2024-03-30T03:59:40.454119+0000","last_unstale":"2024-03-30T04:00:17.008104+0000","last_undegraded":"2024-03-30T04:00:17.008104+0000","last_fullsized":"2024-03-30T04:00:17.008104+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:50:00.651542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998916+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T04:00:16.998916+0000","last_peered":"2024-03-30T04:00:16.998916+0000","last_clean":"2024-03-30T04:00:16.998916+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T04:00:16.998916+0000","last_undegraded":"2024-03-30T04:00:16.998916+0000","last_fullsized":"2024-03-30T04:00:16.998916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003214+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T04:00:17.003214+0000","last_peered":"2024-03-30T04:00:17.003214+0000","last_clean":"2024-03-30T04:00:17.003214+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T04:00:17.003214+0000","last_undegraded":"2024-03-30T04:00:17.003214+0000","last_fullsized":"2024-03-30T04:00:17.003214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999916+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T04:00:16.999916+0000","last_peered":"2024-03-30T04:00:16.999916+0000","last_clean":"2024-03-30T04:00:16.999916+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T04:00:16.999916+0000","last_undegraded":"2024-03-30T04:00:16.999916+0000","last_fullsized":"2024-03-30T04:00:16.999916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.008042+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T04:00:17.008042+0000","last_peered":"2024-03-30T04:00:17.008042+0000","last_clean":"2024-03-30T04:00:17.008042+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T04:00:17.008042+0000","last_undegraded":"2024-03-30T04:00:17.008042+0000","last_fullsized":"2024-03-30T04:00:17.008042+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.005659+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T04:00:17.005659+0000","last_peered":"2024-03-30T04:00:17.005659+0000","last_clean":"2024-03-30T04:00:17.005659+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T04:00:17.005659+0000","last_undegraded":"2024-03-30T04:00:17.005659+0000","last_fullsized":"2024-03-30T04:00:17.005659+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.003025+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T04:00:17.003025+0000","last_peered":"2024-03-30T04:00:17.003025+0000","last_clean":"2024-03-30T04:00:17.003025+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T04:00:17.003025+0000","last_undegraded":"2024-03-30T04:00:17.003025+0000","last_fullsized":"2024-03-30T04:00:17.003025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.999278+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T04:00:16.999278+0000","last_peered":"2024-03-30T04:00:16.999278+0000","last_clean":"2024-03-30T04:00:16.999278+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T04:00:16.999278+0000","last_undegraded":"2024-03-30T04:00:16.999278+0000","last_fullsized":"2024-03-30T04:00:16.999278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:16.998961+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T04:00:16.998961+0000","last_peered":"2024-03-30T04:00:16.998961+0000","last_clean":"2024-03-30T04:00:16.998961+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T04:00:16.998961+0000","last_undegraded":"2024-03-30T04:00:16.998961+0000","last_fullsized":"2024-03-30T04:00:16.998961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-30T04:00:17.004029+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T04:00:17.004029+0000","last_peered":"2024-03-30T04:00:17.004029+0000","last_clean":"2024-03-30T04:00:17.004029+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T04:00:17.004029+0000","last_undegraded":"2024-03-30T04:00:17.004029+0000","last_fullsized":"2024-03-30T04:00:17.004029+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69526,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":447,"num_read_kb":303,"num_write":235,"num_write_kb":138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":208578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547226,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":974848,"data_stored":474025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.504},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54600000000000004}]},{"osd":1,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.73799999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.629},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.69099999999999995}]},{"osd":5,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.9340000000000002,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.71699999999999997}]},{"osd":8,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.78900000000000003}]},{"osd":9,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.75},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.77400000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.83399999999999996}]}]},{"osd":10,"up_from":68,"seq":292057776157,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32744,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712408,"statfs":{"total":95995035648,"available":95961505792,"internally_reserved":0,"allocated":1671168,"data_stored":1123744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.748},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.623,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68500000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.35999999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48999999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53400000000000003}]},{"osd":3,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76200000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.42699999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.51100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.01,"15min":1.01},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95499999999999996,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97599999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.60299999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.71599999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.018,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65200000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.95699999999999996,"5min":1.147,"15min":1.147},"last":0.69599999999999995}]},{"osd":11,"last update":"Sat Mar 30 04:00:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.68600000000000005}]}]},{"osd":9,"up_from":63,"seq":270582939681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":1003520,"data_stored":478099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.706,"5min":3.706,"15min":3.706},"last":0.62},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.629,"15min":0.64700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93700000000000006,"5min":1.018,"15min":1.018},"last":0.70799999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.68300000000000005,"15min":0.69799999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.75900000000000001,"15min":0.76900000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.71999999999999997}]},{"osd":2,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.79400000000000004,"15min":0.81399999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2450000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.80900000000000005,"15min":0.82499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.67600000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.69499999999999995,"15min":0.70699999999999996},"min":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.77600000000000002,"5min":1.155,"15min":1.155},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.72799999999999998,"15min":0.75},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.872,"5min":1.294,"15min":1.294},"last":0.64300000000000002}]},{"osd":4,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.84499999999999997,"15min":0.86299999999999999},"min":{"1min":0.55600000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.86799999999999999,"15min":0.89400000000000002},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.91500000000000004,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.93500000000000005,"15min":0.96499999999999997},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.105,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.747},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.89200000000000002,"15min":0.91100000000000003},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.276,"5min":1.47,"15min":1.47},"last":0.69799999999999995}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.98399999999999999,"15min":1.014},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0600000000000001,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.006,"15min":1.046},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.3100000000000001,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.73499999999999999}]},{"osd":8,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.97999999999999998,"15min":1.0009999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.655,"5min":1.655,"15min":1.655},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":1.0489999999999999,"15min":1.0800000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.155,"5min":2.379,"15min":2.379},"last":1.0349999999999999}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.92700000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2430000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.96699999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":1.093},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":1.0780000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168613,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":995328,"data_stored":474381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.61699999999999999,"15min":0.629},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.74099999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53500000000000003,"15min":0.53200000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55000000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.66600000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.372,"15min":0.36199999999999999},"min":{"1min":0.29399999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63300000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.63900000000000001,"15min":0.64300000000000002},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56499999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76600000000000001,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.80500000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.54900000000000004,"15min":0.54200000000000004},"min":{"1min":0.42699999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80500000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.747},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.51600000000000001},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.78600000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.50900000000000001,"15min":0.503},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.51400000000000001,"15min":0.51100000000000001},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55800000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.71399999999999997,"15min":0.71999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90800000000000003,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.81799999999999995}]},{"osd":7,"last update":"Sat Mar 30 04:00:05 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.67200000000000004,"15min":0.66800000000000004},"min":{"1min":0.44700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.92600000000000005,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.48199999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.878}]},{"osd":9,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69399999999999995,"15min":0.67900000000000005},"min":{"1min":0.497,"5min":0.495,"15min":0.495},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.76300000000000001,"15min":0.755},"min":{"1min":0.56000000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.997,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.85399999999999998}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.55200000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.86599999999999999}]},{"osd":11,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.90100000000000002}]}]},{"osd":7,"up_from":49,"seq":210453397543,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28576,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716576,"statfs":{"total":95995035648,"available":95965773824,"internally_reserved":0,"allocated":1593344,"data_stored":1071129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.38900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.44500000000000001,"15min":0.44600000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.436,"5min":0.372,"15min":0.372},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.624},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56499999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.44,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55800000000000005}]},{"osd":4,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.502,"15min":0.502},"min":{"1min":0.34300000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.504},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.499,"15min":0.498},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44400000000000001}]},{"osd":5,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.61199999999999999,"15min":0.60799999999999998},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.621,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.64400000000000002}]},{"osd":6,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68999999999999995,"15min":0.69399999999999995},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86699999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.53300000000000003}]},{"osd":8,"last update":"Sat Mar 30 04:00:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.48299999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.67400000000000004}]},{"osd":9,"last update":"Sat Mar 30 04:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.77700000000000002,"15min":0.78600000000000003},"min":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.85499999999999998,"5min":1.05,"15min":1.05},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76500000000000001,"15min":0.76700000000000002},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.69199999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96599999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71999999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.74299999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561068,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":469852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.56999999999999995,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.48799999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.65100000000000002,"15min":0.63200000000000001},"min":{"1min":0.51200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.67100000000000004},"min":{"1min":0.55200000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.84299999999999997}]},{"osd":2,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.57299999999999995,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.69899999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.442,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.79700000000000004}]},{"osd":4,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.58299999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76700000000000002,"15min":0.749},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.61399999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.337,"5min":1.462,"15min":1.462},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.68300000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.1060000000000001}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79300000000000004,"15min":0.78700000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.016},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.82299999999999995,"15min":0.81899999999999995},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.93999999999999995}]},{"osd":8,"last update":"Sat Mar 30 04:00:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.85999999999999999,"15min":0.85099999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":0.86899999999999999,"15min":0.86099999999999999},"min":{"1min":0.59199999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":1.1240000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:31 2024","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":0.92400000000000004,"15min":0.91200000000000003},"min":{"1min":0.748,"5min":0.63,"15min":0.63},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.879,"15min":0.86599999999999999},"min":{"1min":0.61899999999999999,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.81499999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95199999999999996,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.075,"5min":0.97499999999999998,"15min":0.95899999999999996},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.1830000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.163},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":1.2210000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36228,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708924,"statfs":{"total":95995035648,"available":95957938176,"internally_reserved":0,"allocated":1044480,"data_stored":533392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.54000000000000004,"15min":0.51000000000000001},"min":{"1min":0.45900000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.54900000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.51900000000000002,"15min":0.47099999999999997},"min":{"1min":0.376,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77700000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.54600000000000004,"15min":0.498},"min":{"1min":0.40100000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.79500000000000004}]},{"osd":3,"last update":"Sat Mar 30 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.498},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.53800000000000003},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.871,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71699999999999997}]},{"osd":4,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.41799999999999998,"15min":0.39400000000000002},"min":{"1min":0.32500000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.436,"15min":0.41099999999999998},"min":{"1min":0.311,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.58499999999999996,"15min":0.56100000000000005},"min":{"1min":0.41099999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.746},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59099999999999997,"15min":0.56599999999999995},"min":{"1min":0.46000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92300000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Sat Mar 30 03:59:52 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.37,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51800000000000002,"15min":0.51000000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.629}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61899999999999999,"15min":0.61299999999999999},"min":{"1min":0.46000000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84699999999999998,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.746,"5min":0.70299999999999996,"15min":0.69599999999999995},"min":{"1min":0.55200000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77300000000000002}]},{"osd":9,"last update":"Sat Mar 30 04:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.82399999999999995,"15min":0.80000000000000004},"min":{"1min":0.67800000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":3.2080000000000002,"5min":3.2080000000000002,"15min":3.2080000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.60599999999999998,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.89700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.75900000000000001}]},{"osd":10,"last update":"Sat Mar 30 03:59:46 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.91800000000000004}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.95499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":470915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.81000000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.57899999999999996,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83799999999999997,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:00:18 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.59599999999999997,"15min":0.56000000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94699999999999995,"5min":0.999,"15min":0.999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.61799999999999999,"15min":0.58199999999999996},"min":{"1min":0.51000000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83199999999999996,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.47399999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.42399999999999999}]},{"osd":4,"last update":"Sat Mar 30 03:59:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85399999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.68899999999999995,"15min":0.66500000000000004},"min":{"1min":0.46000000000000002,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90500000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.82499999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:00:14 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.70699999999999996,"15min":0.67100000000000004},"min":{"1min":0.48799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94699999999999995,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69899999999999995,"15min":0.66800000000000004},"min":{"1min":0.51000000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67900000000000005,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.498},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.76200000000000001,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.90400000000000003}]},{"osd":7,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.54500000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:07 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.68999999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.1180000000000001,"5min":1.333,"15min":1.333},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86299999999999999},"min":{"1min":0.67500000000000004,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.038,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.623}]},{"osd":9,"last update":"Sat Mar 30 04:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.95999999999999996,"15min":0.96299999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.405,"5min":1.405,"15min":1.405},"last":0.94699999999999995}]},{"osd":10,"last update":"Sat Mar 30 03:59:48 2024","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.875,"15min":0.875},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:02 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.71899999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1064960,"data_stored":533171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85999999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56299999999999994,"15min":0.57899999999999996},"min":{"1min":0.44500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82899999999999996,"5min":1.133,"15min":1.133},"last":0.65600000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:17 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53200000000000003,"15min":0.50800000000000001},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53200000000000003,"15min":0.51500000000000001},"min":{"1min":0.32500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.755}]},{"osd":4,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.81599999999999995}]},{"osd":5,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.45100000000000001,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.441},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41299999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.623,"15min":0.61099999999999999},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.82599999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:00:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.55400000000000005,"15min":0.54800000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.53200000000000003,"15min":0.52000000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54600000000000004}]},{"osd":9,"last update":"Sat Mar 30 04:00:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.58799999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73199999999999998,"15min":0.72199999999999998},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.79200000000000004}]},{"osd":10,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.875},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71299999999999997,"15min":0.69999999999999996},"min":{"1min":0.58799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":0.93700000000000006,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.84199999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.73299999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":469215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.46100000000000002,"15min":0.41499999999999998},"min":{"1min":0.36699999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.48099999999999998,"15min":0.438},"min":{"1min":0.39000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.66600000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.43099999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.48499999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":1.762,"15min":1.762},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.68100000000000005,"15min":0.61699999999999999},"min":{"1min":0.52100000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.032,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.66000000000000003,"15min":0.61799999999999999},"min":{"1min":0.504,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98199999999999998,"5min":1.181,"15min":1.181},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64600000000000002,"15min":0.626},"min":{"1min":0.433,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.94299999999999995,"5min":1.296,"15min":1.296},"last":0.54300000000000004}]},{"osd":4,"last update":"Sat Mar 30 03:59:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64900000000000002,"15min":0.63900000000000001},"min":{"1min":0.501,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.88200000000000001,"5min":1.113,"15min":1.113},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.72199999999999998,"15min":0.70599999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.046,"5min":1.591,"15min":1.591},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.70299999999999996,"15min":0.66500000000000004},"min":{"1min":0.61099999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93799999999999994,"5min":1.139,"15min":1.139},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.77000000000000002,"15min":0.72999999999999998},"min":{"1min":0.52400000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.0569999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.77200000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.75600000000000001},"min":{"1min":0.48399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.1220000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.752,"15min":0.72599999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0129999999999999,"5min":1.278,"15min":1.278},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0649999999999999,"5min":1.214,"15min":1.214},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.88400000000000001,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.1499999999999999,"5min":1.242,"15min":1.242},"last":0.96299999999999997}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:34 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.014,"15min":1.026},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.165,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.93999999999999995,"15min":0.95599999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.077,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.69499999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.02,"5min":0.99299999999999999,"15min":0.98899999999999999},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":1.006}]},{"osd":11,"last update":"Sat Mar 30 04:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.73899999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":983040,"data_stored":473926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.30599999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.59999999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40200000000000002,"15min":0.38700000000000001},"min":{"1min":0.307,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55700000000000005}]},{"osd":2,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78900000000000003,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.52700000000000002},"min":{"1min":0.46000000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Mar 30 03:59:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.59499999999999997,"15min":0.57699999999999996},"min":{"1min":0.47599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.624},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83699999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Mar 30 04:00:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.46000000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78800000000000003,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.51400000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.88700000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.85299999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:33 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65500000000000003,"15min":0.64600000000000002},"min":{"1min":0.52600000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.754},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.40000000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.92800000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.69999999999999996}]},{"osd":7,"last update":"Sat Mar 30 03:59:51 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.54400000000000004,"15min":0.52600000000000002},"min":{"1min":0.46600000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.55500000000000005,"15min":0.54100000000000004},"min":{"1min":0.43099999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72099999999999997}]},{"osd":8,"last update":"Sat Mar 30 04:00:10 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.68100000000000005,"15min":0.67000000000000004},"min":{"1min":0.48899999999999999,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93999999999999995,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.876},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68300000000000005,"15min":0.67500000000000004},"min":{"1min":0.51900000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90000000000000002,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.89100000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:00:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.77100000000000002,"15min":0.76700000000000002},"min":{"1min":0.64800000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.79500000000000004,"15min":0.79700000000000004},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":0.96299999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.91100000000000003}]},{"osd":10,"last update":"Sat Mar 30 04:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.749,"15min":0.75700000000000001},"min":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.746,"15min":0.754},"min":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.92300000000000004}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.94999999999999996}]}]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32680,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712472,"statfs":{"total":95995035648,"available":95961571328,"internally_reserved":0,"allocated":1597440,"data_stored":1065690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.42699999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.751,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53200000000000003},"min":{"1min":0.44600000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.56899999999999995,"15min":0.55100000000000005},"min":{"1min":0.45300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.57599999999999996,"15min":0.56299999999999994},"min":{"1min":0.42699999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.57599999999999996}]},{"osd":2,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39600000000000002,"15min":0.38900000000000001},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.246,"15min":0.246},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.32300000000000001}]},{"osd":3,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.57899999999999996,"15min":0.58499999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.80800000000000005,"5min":0.873,"15min":0.873},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.58099999999999996},"min":{"1min":0.36799999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.379,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:00:32 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.48499999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.63400000000000001,"15min":0.629},"min":{"1min":0.54500000000000004,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.55200000000000005}]},{"osd":7,"last update":"Sat Mar 30 03:59:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.65300000000000002,"15min":0.65600000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.67800000000000005,"15min":0.68200000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.67100000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54600000000000004,"15min":0.54000000000000004},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.374},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58599999999999997,"15min":0.57499999999999996},"min":{"1min":0.34300000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.438}]},{"osd":9,"last update":"Sat Mar 30 04:00:27 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.72699999999999998,"15min":0.71699999999999997},"min":{"1min":0.58399999999999996,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.69999999999999996,"15min":0.68600000000000005},"min":{"1min":0.67300000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.496}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.84299999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90800000000000003,"5min":1.222,"15min":1.222},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":64386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":64591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:00:55.759 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-30T04:00:55.759 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T04:00:55.759 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-30T04:00:55.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph health --format=json 2024-03-30T04:00:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:00:56.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:56 smithi154 ceph-mon[79161]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:56.684 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:56 smithi023 ceph-mon[38254]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:56.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:56 smithi193 ceph-mon[79581]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 -- 172.21.15.23:0/2352667466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0106ef0 msgr2=0x7ff0f01072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 --2- 172.21.15.23:0/2352667466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0106ef0 0x7ff0f01072d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff0e40099c0 tx=0x7ff0e4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 -- 172.21.15.23:0/2352667466 shutdown_connections 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 --2- 172.21.15.23:0/2352667466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 0x7ff0f010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 --2- 172.21.15.23:0/2352667466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0106ef0 0x7ff0f01072d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 --2- 172.21.15.23:0/2352667466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0100e60 0x7ff0f01012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 -- 172.21.15.23:0/2352667466 >> 172.21.15.23:0/2352667466 conn(0x7ff0f0077fa0 msgr2=0x7ff0f00783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 -- 172.21.15.23:0/2352667466 shutdown_connections 2024-03-30T04:00:57.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.380+0000 7ff0f7600700 1 -- 172.21.15.23:0/2352667466 wait complete. 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.381+0000 7ff0f7600700 1 Processor -- start 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.381+0000 7ff0f7600700 1 -- start start 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 0x7ff0f0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0106ef0 0x7ff0f019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f019e320 con 0x7ff0f0100e60 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f019e490 con 0x7ff0f0106ef0 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f7600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f019e650 con 0x7ff0f0101820 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37176/0 (socket says 172.21.15.23:37176) 2024-03-30T04:00:57.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 -- 172.21.15.23:0/854454581 learned_addr learned my addr 172.21.15.23:0/854454581 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0efe00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 0x7ff0f0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f5600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0106ef0 0x7ff0f019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 -- 172.21.15.23:0/854454581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 msgr2=0x7ff0f0199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 0x7ff0f0199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 -- 172.21.15.23:0/854454581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0106ef0 msgr2=0x7ff0f019dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0106ef0 0x7ff0f019dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.382+0000 7ff0f4c00700 1 -- 172.21.15.23:0/854454581 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e40095f0 con 0x7ff0f0100e60 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0f4c00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400b580 tx=0x7ff0e400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e401c070 con 0x7ff0f0100e60 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0e400fd40 con 0x7ff0f0100e60 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f019e8d0 con 0x7ff0f0100e60 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4022b80 con 0x7ff0f0100e60 2024-03-30T04:00:57.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.383+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f01a5fc0 con 0x7ff0f0100e60 2024-03-30T04:00:57.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.384+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f0104180 con 0x7ff0f0100e60 2024-03-30T04:00:57.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.386+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7ff0e4020070 con 0x7ff0f0100e60 2024-03-30T04:00:57.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.387+0000 7ff0ed600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 0x7ff0d80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:57.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.387+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff0e40cdb60 con 0x7ff0f0100e60 2024-03-30T04:00:57.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.388+0000 7ff0efe00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 0x7ff0d80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:57.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.388+0000 7ff0efe00700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 0x7ff0d80a3670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff0f019a980 tx=0x7ff0dc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:57.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.392+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff0e4096eb0 con 0x7ff0f0100e60 2024-03-30T04:00:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:57 smithi023 ceph-mon[38254]: pgmap v276: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:57.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.648+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff0f019a730 con 0x7ff0f0100e60 2024-03-30T04:00:57.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.649+0000 7ff0ed600700 1 -- 172.21.15.23:0/854454581 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff0e409a4d0 con 0x7ff0f0100e60 2024-03-30T04:00:57.648 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:57.648 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 msgr2=0x7ff0d80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 0x7ff0d80a3670 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff0f019a980 tx=0x7ff0dc009260 comp rx=0 tx=0).stop 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 msgr2=0x7ff0f0199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400b580 tx=0x7ff0e400bfd0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 shutdown_connections 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0f0106ef0 0x7ff0f019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff0f0100e60 0x7ff0f0199420 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff0d80a11b0 0x7ff0d80a3670 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 --2- 172.21.15.23:0/854454581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f0101820 0x7ff0f0199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:57.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 >> 172.21.15.23:0/854454581 conn(0x7ff0f0077fa0 msgr2=0x7ff0f010c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:57.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 shutdown_connections 2024-03-30T04:00:57.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:57.651+0000 7ff0f7600700 1 -- 172.21.15.23:0/854454581 wait complete. 2024-03-30T04:00:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:57 smithi154 ceph-mon[79161]: pgmap v276: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:57.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:57 smithi193 ceph-mon[79581]: pgmap v276: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:00:58.025 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-30T04:00:58.026 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:00:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 -- 172.21.15.23:0/3799374587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 msgr2=0x7fd4e0103ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 --2- 172.21.15.23:0/3799374587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0103ad0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd4d0009990 tx=0x7fd4d0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 -- 172.21.15.23:0/3799374587 shutdown_connections 2024-03-30T04:00:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 --2- 172.21.15.23:0/3799374587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0106940 0x7fd4e0108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 --2- 172.21.15.23:0/3799374587 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0103ad0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 --2- 172.21.15.23:0/3799374587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0104010 0x7fd4e0106400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 -- 172.21.15.23:0/3799374587 >> 172.21.15.23:0/3799374587 conn(0x7fd4e00fb0d0 msgr2=0x7fd4e00fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:58.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 -- 172.21.15.23:0/3799374587 shutdown_connections 2024-03-30T04:00:58.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.231+0000 7fd4e6200700 1 -- 172.21.15.23:0/3799374587 wait complete. 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.232+0000 7fd4e6200700 1 Processor -- start 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.232+0000 7fd4e6200700 1 -- start start 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0104010 0x7fd4e0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0106940 0x7fd4e01a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a08e0 con 0x7fd4e01016e0 2024-03-30T04:00:58.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a0a50 con 0x7fd4e0104010 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4e6200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01a0c10 con 0x7fd4e0106940 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58068/0 (socket says 172.21.15.23:58068) 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 -- 172.21.15.23:0/1943287426 learned_addr learned my addr 172.21.15.23:0/1943287426 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4dea00700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0104010 0x7fd4e0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4dfe00700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0106940 0x7fd4e01a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 -- 172.21.15.23:0/1943287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0106940 msgr2=0x7fd4e01a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0106940 0x7fd4e01a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 -- 172.21.15.23:0/1943287426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0104010 msgr2=0x7fd4e0199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0104010 0x7fd4e0199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.233+0000 7fd4df400700 1 -- 172.21.15.23:0/1943287426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00095f0 con 0x7fd4e01016e0 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4df400700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4d000b680 tx=0x7fd4d0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:58.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d001d070 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd4d0022470 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e01a0e90 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0017650 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.234+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e01a1200 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.235+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e0108510 con 0x7fd4e01016e0 2024-03-30T04:00:58.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.237+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd4d00225e0 con 0x7fd4e01016e0 2024-03-30T04:00:58.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.239+0000 7fd4d7e00700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 0x7fd4c40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.239+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4d00ce610 con 0x7fd4e01016e0 2024-03-30T04:00:58.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.240+0000 7fd4dea00700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 0x7fd4c40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.240+0000 7fd4dea00700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 0x7fd4c40a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009c60 tx=0x7fd4cc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:58.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.242+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4d0097960 con 0x7fd4e01016e0 2024-03-30T04:00:58.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.503+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4e0067a10 con 0x7fd4e01016e0 2024-03-30T04:00:58.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.504+0000 7fd4d7e00700 1 -- 172.21.15.23:0/1943287426 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+24428 (secure 0 0 0) 0x7fd4d009af80 con 0x7fd4e01016e0 2024-03-30T04:00:58.503 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:00:58.504 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:17.995781+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:00:58.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 msgr2=0x7fd4c40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 0x7fd4c40a3670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc009c60 tx=0x7fd4cc009260 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 msgr2=0x7fd4e0198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd4d000b680 tx=0x7fd4d0009580 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 shutdown_connections 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd4c40a11b0 0x7fd4c40a3670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4e0106940 0x7fd4e01a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4e01016e0 0x7fd4e0198df0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 --2- 172.21.15.23:0/1943287426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e0104010 0x7fd4e0199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 >> 172.21.15.23:0/1943287426 conn(0x7fd4e00fb0d0 msgr2=0x7fd4e0100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 shutdown_connections 2024-03-30T04:00:58.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.507+0000 7fd4e6200700 1 -- 172.21.15.23:0/1943287426 wait complete. 2024-03-30T04:00:58.525 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:00:58.525 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:00:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:58 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/854454581' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T04:00:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 -- 172.21.15.23:0/999834399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc101f70 msgr2=0x7faecc1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 --2- 172.21.15.23:0/999834399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc101f70 0x7faecc1023d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faeb80099c0 tx=0x7faeb8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 -- 172.21.15.23:0/999834399 shutdown_connections 2024-03-30T04:00:58.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 --2- 172.21.15.23:0/999834399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 --2- 172.21.15.23:0/999834399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc100d80 0x7faecc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 --2- 172.21.15.23:0/999834399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc101f70 0x7faecc1023d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 -- 172.21.15.23:0/999834399 >> 172.21.15.23:0/999834399 conn(0x7faecc0fc2f0 msgr2=0x7faecc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:58.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 -- 172.21.15.23:0/999834399 shutdown_connections 2024-03-30T04:00:58.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.728+0000 7faed3000700 1 -- 172.21.15.23:0/999834399 wait complete. 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.729+0000 7faed3000700 1 Processor -- start 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.729+0000 7faed3000700 1 -- start start 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc101f70 0x7faecc197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc19e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19e840 con 0x7faecc100d80 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19e9b0 con 0x7faecc102910 2024-03-30T04:00:58.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed3000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc19eb70 con 0x7faecc101f70 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faecbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faecbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58096/0 (socket says 172.21.15.23:58096) 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faecbe00700 1 -- 172.21.15.23:0/1330576353 learned_addr learned my addr 172.21.15.23:0/1330576353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faecb400700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc101f70 0x7faecc197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.730+0000 7faed1000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc19e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 -- 172.21.15.23:0/1330576353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc101f70 msgr2=0x7faecc197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc101f70 0x7faecc197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 -- 172.21.15.23:0/1330576353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 msgr2=0x7faecc19e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc19e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 -- 172.21.15.23:0/1330576353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb80095f0 con 0x7faecc100d80 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faed1000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faecbe00700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faebc00e970 tx=0x7faebc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:58.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc009790 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faebc00eec0 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc003cf0 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc19ee50 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.731+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc19f270 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.732+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc108220 con 0x7faecc100d80 2024-03-30T04:00:58.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.734+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7faebc00ccd0 con 0x7faecc100d80 2024-03-30T04:00:58.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.735+0000 7faec8c00700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 0x7faeb00a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:58.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.736+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faebc014070 con 0x7faecc100d80 2024-03-30T04:00:58.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.736+0000 7faecb400700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 0x7faeb00a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:58.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.736+0000 7faecb400700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 0x7faeb00a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faeb8003730 tx=0x7faeb800b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:58.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.738+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7faebc0963b0 con 0x7faecc100d80 2024-03-30T04:00:58.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:58 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/854454581' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T04:00:58.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:58 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/854454581' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-30T04:00:58.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.953+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faecc10c5e0 con 0x7faecc100d80 2024-03-30T04:00:58.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.954+0000 7faec8c00700 1 -- 172.21.15.23:0/1330576353 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7faebc0999d0 con 0x7faecc100d80 2024-03-30T04:00:58.953 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 msgr2=0x7faeb00a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 0x7faeb00a36f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faeb8003730 tx=0x7faeb800b540 comp rx=0 tx=0).stop 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 msgr2=0x7faecc196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faebc00e970 tx=0x7faebc00ec80 comp rx=0 tx=0).stop 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 shutdown_connections 2024-03-30T04:00:58.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faecc102910 0x7faecc19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faecc100d80 0x7faecc196d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7faeb00a1230 0x7faeb00a36f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 --2- 172.21.15.23:0/1330576353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faecc101f70 0x7faecc197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.956+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 >> 172.21.15.23:0/1330576353 conn(0x7faecc0fc2f0 msgr2=0x7faecc105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.957+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 shutdown_connections 2024-03-30T04:00:58.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:58.957+0000 7faed3000700 1 -- 172.21.15.23:0/1330576353 wait complete. 2024-03-30T04:00:58.974 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:00:59.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 -- 172.21.15.23:0/1888158435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 msgr2=0x7f8d601023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 --2- 172.21.15.23:0/1888158435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d601023d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c009990 tx=0x7f8d4c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 -- 172.21.15.23:0/1888158435 shutdown_connections 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 --2- 172.21.15.23:0/1888158435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60102910 0x7f8d601091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 --2- 172.21.15.23:0/1888158435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60100d80 0x7f8d601011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 --2- 172.21.15.23:0/1888158435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d601023d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 -- 172.21.15.23:0/1888158435 >> 172.21.15.23:0/1888158435 conn(0x7f8d600fc2f0 msgr2=0x7f8d600fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 -- 172.21.15.23:0/1888158435 shutdown_connections 2024-03-30T04:00:59.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.178+0000 7f8d66200700 1 -- 172.21.15.23:0/1888158435 wait complete. 2024-03-30T04:00:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.179+0000 7f8d66200700 1 Processor -- start 2024-03-30T04:00:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 -- start start 2024-03-30T04:00:59.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d60199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 0x7f8d601a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a0a10 con 0x7f8d60100d80 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a0b50 con 0x7f8d60102910 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d66200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d601a0ce0 con 0x7f8d60101f70 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58110/0 (socket says 172.21.15.23:58110) 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 -- 172.21.15.23:0/2313680589 learned_addr learned my addr 172.21.15.23:0/2313680589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5ea00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d60199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5fe00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 0x7f8d601a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 -- 172.21.15.23:0/2313680589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 msgr2=0x7f8d60199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d60199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 -- 172.21.15.23:0/2313680589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 msgr2=0x7f8d601a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 0x7f8d601a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5f400700 1 -- 172.21.15.23:0/2313680589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4c0095f0 con 0x7f8d60100d80 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5ea00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d60199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.180+0000 7f8d5fe00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 0x7f8d601a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d5f400700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000e970 tx=0x7f8d5000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50009790 con 0x7f8d60100d80 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d500044e0 con 0x7f8d60100d80 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d601a0fc0 con 0x7f8d60100d80 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50003e30 con 0x7f8d60100d80 2024-03-30T04:00:59.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.181+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601a13e0 con 0x7f8d60100d80 2024-03-30T04:00:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.182+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60067af0 con 0x7f8d60100d80 2024-03-30T04:00:59.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.184+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f8d50004000 con 0x7f8d60100d80 2024-03-30T04:00:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.185+0000 7f8d57e00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 0x7f8d440a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.186+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8d50014070 con 0x7f8d60100d80 2024-03-30T04:00:59.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.186+0000 7f8d5ea00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 0x7f8d440a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.186+0000 7f8d5ea00700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 0x7f8d440a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b2e0 tx=0x7f8d4c00b770 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:59.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.190+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d500963a0 con 0x7f8d60100d80 2024-03-30T04:00:59.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.406+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d6010c810 con 0x7f8d60100d80 2024-03-30T04:00:59.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.407+0000 7f8d57e00700 1 -- 172.21.15.23:0/2313680589 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d500999c0 con 0x7f8d60100d80 2024-03-30T04:00:59.406 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 msgr2=0x7f8d440a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 0x7f8d440a3670 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b2e0 tx=0x7f8d4c00b770 comp rx=0 tx=0).stop 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 msgr2=0x7f8d60198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000e970 tx=0x7f8d5000ed30 comp rx=0 tx=0).stop 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 shutdown_connections 2024-03-30T04:00:59.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d60102910 0x7f8d601a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d60100d80 0x7f8d60198ee0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d440a11b0 0x7f8d440a3670 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 --2- 172.21.15.23:0/2313680589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d60101f70 0x7f8d60199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.409+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 >> 172.21.15.23:0/2313680589 conn(0x7f8d600fc2f0 msgr2=0x7f8d60072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.410+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 shutdown_connections 2024-03-30T04:00:59.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.410+0000 7f8d66200700 1 -- 172.21.15.23:0/2313680589 wait complete. 2024-03-30T04:00:59.427 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 -- 172.21.15.23:0/3109315573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0108c40 msgr2=0x7fe1e010b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/3109315573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0108c40 0x7fe1e010b0d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe1d800b0a0 tx=0x7fe1d800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 -- 172.21.15.23:0/3109315573 shutdown_connections 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/3109315573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0108c40 0x7fe1e010b0d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/3109315573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0074df0 0x7fe1e0073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/3109315573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0073810 0x7fe1e0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 -- 172.21.15.23:0/3109315573 >> 172.21.15.23:0/3109315573 conn(0x7fe1e00fc2f0 msgr2=0x7fe1e00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:59.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 -- 172.21.15.23:0/3109315573 shutdown_connections 2024-03-30T04:00:59.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.630+0000 7fe1e7a00700 1 -- 172.21.15.23:0/3109315573 wait complete. 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.631+0000 7fe1e7a00700 1 Processor -- start 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.631+0000 7fe1e7a00700 1 -- start start 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.631+0000 7fe1e7a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0073810 0x7fe1e0198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.631+0000 7fe1e7a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.631+0000 7fe1e7a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0108c40 0x7fe1e01a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1e7a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e01a0b00 con 0x7fe1e0074df0 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1e7a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e01a0c40 con 0x7fe1e0073810 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1e7a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e01a0dd0 con 0x7fe1e0108c40 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1e5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0073810 0x7fe1e0198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58126/0 (socket says 172.21.15.23:58126) 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 -- 172.21.15.23:0/114808997 learned_addr learned my addr 172.21.15.23:0/114808997 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1e5a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0108c40 0x7fe1e01a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 -- 172.21.15.23:0/114808997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0108c40 msgr2=0x7fe1e01a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0108c40 0x7fe1e01a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 -- 172.21.15.23:0/114808997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0073810 msgr2=0x7fe1e0198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0073810 0x7fe1e0198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 -- 172.21.15.23:0/114808997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8009d00 con 0x7fe1e0074df0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.632+0000 7fe1dfe00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00e970 tx=0x7fe1cc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.633+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc009790 con 0x7fe1e0074df0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.633+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1cc00eec0 con 0x7fe1e0074df0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.633+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc003cf0 con 0x7fe1e0074df0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.633+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e01a10b0 con 0x7fe1e0074df0 2024-03-30T04:00:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.633+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e01a1570 con 0x7fe1e0074df0 2024-03-30T04:00:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.634+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e0107a00 con 0x7fe1e0074df0 2024-03-30T04:00:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.636+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe1cc00ccd0 con 0x7fe1e0074df0 2024-03-30T04:00:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.637+0000 7fe1dd600700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 0x7fe1c80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:00:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.637+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe1cc014070 con 0x7fe1e0074df0 2024-03-30T04:00:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.637+0000 7fe1e5000700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 0x7fe1c80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:00:59.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.638+0000 7fe1e5000700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 0x7fe1c80a36f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009c60 tx=0x7fe1d4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:00:59.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.641+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1cc096280 con 0x7fe1e0074df0 2024-03-30T04:00:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:59 smithi154 ceph-mon[79161]: pgmap v277: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1943287426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:00:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1330576353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:00:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:59 smithi193 ceph-mon[79581]: pgmap v277: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1943287426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:00:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1330576353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:00:59.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:59 smithi023 ceph-mon[38254]: pgmap v277: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:00:59.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1943287426' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:00:59.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:00:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1330576353' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:00:59.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.854+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe1e010e7c0 con 0x7fe1e0074df0 2024-03-30T04:00:59.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.854+0000 7fe1dd600700 1 -- 172.21.15.23:0/114808997 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1cc0998a0 con 0x7fe1e0074df0 2024-03-30T04:00:59.853 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:00:59.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.856+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 msgr2=0x7fe1c80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.856+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 0x7fe1c80a36f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4009c60 tx=0x7fe1d4009260 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 msgr2=0x7fe1e0199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00e970 tx=0x7fe1cc00ec80 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 shutdown_connections 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe1c80a1230 0x7fe1c80a36f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1e0108c40 0x7fe1e01a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe1e0073810 0x7fe1e0198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 --2- 172.21.15.23:0/114808997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe1e0074df0 0x7fe1e0199510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 >> 172.21.15.23:0/114808997 conn(0x7fe1e00fc2f0 msgr2=0x7fe1e0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 shutdown_connections 2024-03-30T04:00:59.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:00:59.857+0000 7fe1e7a00700 1 -- 172.21.15.23:0/114808997 wait complete. 2024-03-30T04:00:59.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 -- 172.21.15.23:0/2914465711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24108c40 msgr2=0x7f0f2410b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 --2- 172.21.15.23:0/2914465711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24108c40 0x7f0f2410b0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00b0a0 tx=0x7f0f1c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 -- 172.21.15.23:0/2914465711 shutdown_connections 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 --2- 172.21.15.23:0/2914465711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24108c40 0x7f0f2410b0d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 --2- 172.21.15.23:0/2914465711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24074df0 0x7f0f24073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 --2- 172.21.15.23:0/2914465711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24073810 0x7f0f24073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.080+0000 7f0f2c000700 1 -- 172.21.15.23:0/2914465711 >> 172.21.15.23:0/2914465711 conn(0x7f0f240fc2f0 msgr2=0x7f0f240fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.081+0000 7f0f2c000700 1 -- 172.21.15.23:0/2914465711 shutdown_connections 2024-03-30T04:01:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.081+0000 7f0f2c000700 1 -- 172.21.15.23:0/2914465711 wait complete. 2024-03-30T04:01:00.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 Processor -- start 2024-03-30T04:01:00.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 -- start start 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24074df0 0x7f0f241995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24108c40 0x7f0f241a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f241a0b60 con 0x7f0f24073810 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f241a0ca0 con 0x7f0f24074df0 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2c000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f241a0e30 con 0x7f0f24108c40 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f29600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f29600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58158/0 (socket says 172.21.15.23:58158) 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f29600700 1 -- 172.21.15.23:0/1115118958 learned_addr learned my addr 172.21.15.23:0/1115118958 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f28c00700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24074df0 0x7f0f241995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.082+0000 7f0f2a000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24108c40 0x7f0f241a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 -- 172.21.15.23:0/1115118958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24108c40 msgr2=0x7f0f241a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24108c40 0x7f0f241a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 -- 172.21.15.23:0/1115118958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24074df0 msgr2=0x7f0f241995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24074df0 0x7f0f241995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 -- 172.21.15.23:0/1115118958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f18009520 con 0x7f0f24073810 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f29600700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800eb30 tx=0x7f0f1800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18009880 con 0x7f0f24073810 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f180044e0 con 0x7f0f24073810 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c009d00 con 0x7f0f24073810 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18003d80 con 0x7f0f24073810 2024-03-30T04:01:00.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.083+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f241a1490 con 0x7f0f24073810 2024-03-30T04:01:00.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.084+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f24107a00 con 0x7f0f24073810 2024-03-30T04:01:00.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.086+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f0f18003f20 con 0x7f0f24073810 2024-03-30T04:01:00.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.087+0000 7f0f22000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 0x7f0f0c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.088+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0f18014070 con 0x7f0f24073810 2024-03-30T04:01:00.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.088+0000 7f0f28c00700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 0x7f0f0c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.088+0000 7f0f28c00700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 0x7f0f0c0a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0f24074b10 tx=0x7f0f10009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:00.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.091+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0f18096610 con 0x7f0f24073810 2024-03-30T04:01:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.311+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f2410e9f0 con 0x7f0f24073810 2024-03-30T04:01:00.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.312+0000 7f0f22000700 1 -- 172.21.15.23:0/1115118958 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f18099c30 con 0x7f0f24073810 2024-03-30T04:01:00.310 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 msgr2=0x7f0f0c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 0x7f0f0c0a3670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0f24074b10 tx=0x7f0f10009260 comp rx=0 tx=0).stop 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 msgr2=0x7f0f24199080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800eb30 tx=0x7f0f1800ee40 comp rx=0 tx=0).stop 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 shutdown_connections 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0f0c0a11b0 0x7f0f0c0a3670 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0f24108c40 0x7f0f241a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f24073810 0x7f0f24199080 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 --2- 172.21.15.23:0/1115118958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f24074df0 0x7f0f241995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.314+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 >> 172.21.15.23:0/1115118958 conn(0x7f0f240fc2f0 msgr2=0x7f0f24104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.315+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 shutdown_connections 2024-03-30T04:01:00.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.315+0000 7f0f2c000700 1 -- 172.21.15.23:0/1115118958 wait complete. 2024-03-30T04:01:00.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:00.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 -- 172.21.15.23:0/1143296948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70108b40 msgr2=0x7fae7010afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 --2- 172.21.15.23:0/1143296948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70108b40 0x7fae7010afd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fae6800b0a0 tx=0x7fae6800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 -- 172.21.15.23:0/1143296948 shutdown_connections 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 --2- 172.21.15.23:0/1143296948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70108b40 0x7fae7010afd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 --2- 172.21.15.23:0/1143296948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70074df0 0x7fae70073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 --2- 172.21.15.23:0/1143296948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70073810 0x7fae70073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 -- 172.21.15.23:0/1143296948 >> 172.21.15.23:0/1143296948 conn(0x7fae700fc1f0 msgr2=0x7fae700fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 -- 172.21.15.23:0/1143296948 shutdown_connections 2024-03-30T04:01:00.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.538+0000 7fae77600700 1 -- 172.21.15.23:0/1143296948 wait complete. 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 Processor -- start 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 -- start start 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70073810 0x7fae70103f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 0x7fae70100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae70101280 con 0x7fae70073810 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.540+0000 7fae77600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701013f0 con 0x7fae70108b40 2024-03-30T04:01:00.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae77600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701015b0 con 0x7fae70074df0 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59106/0 (socket says 172.21.15.23:59106) 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 -- 172.21.15.23:0/2152922866 learned_addr learned my addr 172.21.15.23:0/2152922866 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae75600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 0x7fae70100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae74c00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70073810 0x7fae70103f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 -- 172.21.15.23:0/2152922866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 msgr2=0x7fae70100ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 0x7fae70100ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 -- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70073810 msgr2=0x7fae70103f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70073810 0x7fae70103f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae6fe00700 1 -- 172.21.15.23:0/2152922866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68009d00 con 0x7fae70074df0 2024-03-30T04:01:00.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.541+0000 7fae75600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 0x7fae70100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:00.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae6fe00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e970 tx=0x7fae5c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:00.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c009790 con 0x7fae70074df0 2024-03-30T04:01:00.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae701a5430 con 0x7fae70074df0 2024-03-30T04:01:00.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae5c00c7e0 con 0x7fae70074df0 2024-03-30T04:01:00.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c003c70 con 0x7fae70074df0 2024-03-30T04:01:00.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.542+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae701a5960 con 0x7fae70074df0 2024-03-30T04:01:00.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.543+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae70107900 con 0x7fae70074df0 2024-03-30T04:01:00.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.546+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fae5c00cc50 con 0x7fae70074df0 2024-03-30T04:01:00.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.547+0000 7fae6d600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 0x7fae580a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:00.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.547+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fae5c014070 con 0x7fae70074df0 2024-03-30T04:01:00.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.547+0000 7fae74c00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 0x7fae580a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:00.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.548+0000 7fae74c00700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 0x7fae580a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fae64009c60 tx=0x7fae64009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:00.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.551+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fae5c096280 con 0x7fae70074df0 2024-03-30T04:01:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2313680589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/114808997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1115118958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2313680589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/114808997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1115118958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2313680589' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/114808997' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1115118958' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:00.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.819+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae70067a10 con 0x7fae70074df0 2024-03-30T04:01:00.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.820+0000 7fae6d600700 1 -- 172.21.15.23:0/2152922866 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fae5c003dd0 con 0x7fae70074df0 2024-03-30T04:01:00.818 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:00.819 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:00.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 msgr2=0x7fae580a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 0x7fae580a3670 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fae64009c60 tx=0x7fae64009260 comp rx=0 tx=0).stop 2024-03-30T04:01:00.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 msgr2=0x7fae70104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:00.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e970 tx=0x7fae5c00ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:00.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 shutdown_connections 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae70108b40 0x7fae70100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae70073810 0x7fae70103f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fae580a11b0 0x7fae580a3670 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 --2- 172.21.15.23:0/2152922866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae70074df0 0x7fae70104490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.822+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 >> 172.21.15.23:0/2152922866 conn(0x7fae700fc1f0 msgr2=0x7fae70104a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.823+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 shutdown_connections 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:00.823+0000 7fae77600700 1 -- 172.21.15.23:0/2152922866 wait complete. 2024-03-30T04:01:00.822 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:00.840 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 -- 172.21.15.23:0/2749256090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c073810 msgr2=0x7f1d3c073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2749256090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c073810 0x7f1d3c073c90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009990 tx=0x7f1d28009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 -- 172.21.15.23:0/2749256090 shutdown_connections 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2749256090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c108b10 0x7f1d3c10afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2749256090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 0x7f1d3c073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2749256090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c073810 0x7f1d3c073c90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 -- 172.21.15.23:0/2749256090 >> 172.21.15.23:0/2749256090 conn(0x7f1d3c0fc1c0 msgr2=0x7f1d3c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 -- 172.21.15.23:0/2749256090 shutdown_connections 2024-03-30T04:01:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.044+0000 7f1d3be00700 1 -- 172.21.15.23:0/2749256090 wait complete. 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.045+0000 7f1d3be00700 1 Processor -- start 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.045+0000 7f1d3be00700 1 -- start start 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c073810 0x7f1d3c198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 0x7f1d3c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1a0960 con 0x7f1d3c108b10 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1a0aa0 con 0x7f1d3c073810 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3be00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c1a0c30 con 0x7f1d3c074df0 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58206/0 (socket says 172.21.15.23:58206) 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 -- 172.21.15.23:0/2892719399 learned_addr learned my addr 172.21.15.23:0/2892719399 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3aa00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c073810 0x7f1d3c198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3a000700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 0x7f1d3c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 -- 172.21.15.23:0/2892719399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 msgr2=0x7f1d3c199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 0x7f1d3c199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 -- 172.21.15.23:0/2892719399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c073810 msgr2=0x7f1d3c198e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c073810 0x7f1d3c198e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.046+0000 7f1d3b400700 1 -- 172.21.15.23:0/2892719399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d280095f0 con 0x7f1d3c108b10 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d3b400700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007ee0 tx=0x7f1d3400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3400dd20 con 0x7f1d3c108b10 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d3400f040 con 0x7f1d3c108b10 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c1a0f10 con 0x7f1d3c108b10 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d340048a0 con 0x7f1d3c108b10 2024-03-30T04:01:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.047+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c1a1340 con 0x7f1d3c108b10 2024-03-30T04:01:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.048+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c1078d0 con 0x7f1d3c108b10 2024-03-30T04:01:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.050+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f1d34003a00 con 0x7f1d3c108b10 2024-03-30T04:01:01.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.051+0000 7f1d40a00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 0x7f1d200a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.052+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1d340cc600 con 0x7f1d3c108b10 2024-03-30T04:01:01.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.052+0000 7f1d3aa00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 0x7f1d200a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.052+0000 7f1d3aa00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 0x7f1d200a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009c60 tx=0x7f1d2c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:01.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.056+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1d340958c0 con 0x7f1d3c108b10 2024-03-30T04:01:01.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.320+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d3c10e7a0 con 0x7f1d3c108b10 2024-03-30T04:01:01.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.322+0000 7f1d40a00700 1 -- 172.21.15.23:0/2892719399 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+24428 (secure 0 0 0) 0x7f1d34098ee0 con 0x7f1d3c108b10 2024-03-30T04:01:01.321 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:01.321 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:17.995781+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 msgr2=0x7f1d200a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 0x7f1d200a3670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009c60 tx=0x7f1d2c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 msgr2=0x7f1d3c1a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1d34007ee0 tx=0x7f1d3400d3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 shutdown_connections 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d3c108b10 0x7f1d3c1a0390 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d3c073810 0x7f1d3c198e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f1d200a11b0 0x7f1d200a3670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 --2- 172.21.15.23:0/2892719399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d3c074df0 0x7f1d3c199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 >> 172.21.15.23:0/2892719399 conn(0x7f1d3c0fc1c0 msgr2=0x7f1d3c104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 shutdown_connections 2024-03-30T04:01:01.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.325+0000 7f1d3be00700 1 -- 172.21.15.23:0/2892719399 wait complete. 2024-03-30T04:01:01.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:01.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:01 smithi154 ceph-mon[79161]: pgmap v278: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:01 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2152922866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:01 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2892719399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:01 smithi193 ceph-mon[79581]: pgmap v278: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:01 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2152922866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:01 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2892719399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 -- 172.21.15.23:0/3912848419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20074df0 msgr2=0x7f8d20073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 --2- 172.21.15.23:0/3912848419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20074df0 0x7f8d20073240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009990 tx=0x7f8d10009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 -- 172.21.15.23:0/3912848419 shutdown_connections 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 --2- 172.21.15.23:0/3912848419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20108c40 0x7f8d2010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 --2- 172.21.15.23:0/3912848419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20074df0 0x7f8d20073240 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 --2- 172.21.15.23:0/3912848419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 0x7f8d20073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 -- 172.21.15.23:0/3912848419 >> 172.21.15.23:0/3912848419 conn(0x7f8d200fc2f0 msgr2=0x7f8d200fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 -- 172.21.15.23:0/3912848419 shutdown_connections 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.549+0000 7f8d26600700 1 -- 172.21.15.23:0/3912848419 wait complete. 2024-03-30T04:01:01.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.551+0000 7f8d26600700 1 Processor -- start 2024-03-30T04:01:01.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 -- start start 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 0x7f8d20198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20074df0 0x7f8d201994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a0a60 con 0x7f8d20108c40 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a0ba0 con 0x7f8d20073810 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d26600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a0d30 con 0x7f8d20074df0 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d1fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d1fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58230/0 (socket says 172.21.15.23:58230) 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d1fe00700 1 -- 172.21.15.23:0/535015963 learned_addr learned my addr 172.21.15.23:0/535015963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d1f400700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 0x7f8d20198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.552+0000 7f8d1ea00700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20074df0 0x7f8d201994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 -- 172.21.15.23:0/535015963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20074df0 msgr2=0x7f8d201994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20074df0 0x7f8d201994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 -- 172.21.15.23:0/535015963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 msgr2=0x7f8d20198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 0x7f8d20198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 -- 172.21.15.23:0/535015963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d100095f0 con 0x7f8d20108c40 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d1fe00700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800ef30 tx=0x7f8d1800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18018070 con 0x7f8d20108c40 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d1800f040 con 0x7f8d20108c40 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d201a0fb0 con 0x7f8d20108c40 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.553+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1800cd00 con 0x7f8d20108c40 2024-03-30T04:01:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.554+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d201a1430 con 0x7f8d20108c40 2024-03-30T04:01:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.555+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d20107a00 con 0x7f8d20108c40 2024-03-30T04:01:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.556+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f8d18003a00 con 0x7f8d20108c40 2024-03-30T04:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.558+0000 7f8d17e00700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 0x7f8d040a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.559+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8d180cc8d0 con 0x7f8d20108c40 2024-03-30T04:01:01.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.559+0000 7f8d1f400700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 0x7f8d040a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:01.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.559+0000 7f8d1f400700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 0x7f8d040a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b2e0 tx=0x7f8d10005fd0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.563+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8d18095c20 con 0x7f8d20108c40 2024-03-30T04:01:01.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:01 smithi023 ceph-mon[38254]: pgmap v278: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:01 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2152922866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:01 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2892719399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:01.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.783+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8d2010e7c0 con 0x7f8d20108c40 2024-03-30T04:01:01.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.783+0000 7f8d17e00700 1 -- 172.21.15.23:0/535015963 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8d18099240 con 0x7f8d20108c40 2024-03-30T04:01:01.783 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:01.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.785+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 msgr2=0x7f8d040a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.785+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 0x7f8d040a3670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b2e0 tx=0x7f8d10005fd0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.785+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 msgr2=0x7f8d201a0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.785+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800ef30 tx=0x7f8d1800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 shutdown_connections 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d20108c40 0x7f8d201a0490 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8d20073810 0x7f8d20198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f8d040a11b0 0x7f8d040a3670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 --2- 172.21.15.23:0/535015963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20074df0 0x7f8d201994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 >> 172.21.15.23:0/535015963 conn(0x7f8d200fc2f0 msgr2=0x7f8d20104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 shutdown_connections 2024-03-30T04:01:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:01.786+0000 7f8d26600700 1 -- 172.21.15.23:0/535015963 wait complete. 2024-03-30T04:01:01.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 -- 172.21.15.23:0/1472037942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 msgr2=0x7f40fc108eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 --2- 172.21.15.23:0/1472037942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc108eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f40f400b0a0 tx=0x7f40f400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 -- 172.21.15.23:0/1472037942 shutdown_connections 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 --2- 172.21.15.23:0/1472037942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc108eb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 --2- 172.21.15.23:0/1472037942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 0x7f40fc103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 --2- 172.21.15.23:0/1472037942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 -- 172.21.15.23:0/1472037942 >> 172.21.15.23:0/1472037942 conn(0x7f40fc0fb1d0 msgr2=0x7f40fc0fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:02.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 -- 172.21.15.23:0/1472037942 shutdown_connections 2024-03-30T04:01:02.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.006+0000 7f4103600700 1 -- 172.21.15.23:0/1472037942 wait complete. 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.007+0000 7f4103600700 1 Processor -- start 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.007+0000 7f4103600700 1 -- start start 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 0x7f40fc101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc0ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc100470 con 0x7f40fc104110 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1005e0 con 0x7f40fc106a40 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4103600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1a0e20 con 0x7f40fc1017e0 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58248/0 (socket says 172.21.15.23:58248) 2024-03-30T04:01:02.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 -- 172.21.15.23:0/1575665504 learned_addr learned my addr 172.21.15.23:0/1575665504 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4100c00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 0x7f40fc101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f4101600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc0ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 -- 172.21.15.23:0/1575665504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 msgr2=0x7f40fc101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 0x7f40fc101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 -- 172.21.15.23:0/1575665504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 msgr2=0x7f40fc0ffcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc0ffcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.008+0000 7f40fbe00700 1 -- 172.21.15.23:0/1575665504 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40e8009520 con 0x7f40fc104110 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f4101600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f40fbe00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f40e800eb30 tx=0x7f40e800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8009880 con 0x7f40fc104110 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40e800ce30 con 0x7f40fc104110 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40e8003d10 con 0x7f40fc104110 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f4009d00 con 0x7f40fc104110 2024-03-30T04:01:02.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.009+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc1a13e0 con 0x7f40fc104110 2024-03-30T04:01:02.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.010+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40fc0fcd50 con 0x7f40fc104110 2024-03-30T04:01:02.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.012+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f40e800c970 con 0x7f40fc104110 2024-03-30T04:01:02.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.013+0000 7f40f9600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 0x7f40e40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.013+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f40e8014070 con 0x7f40fc104110 2024-03-30T04:01:02.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.014+0000 7f4100c00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 0x7f40e40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.014+0000 7f4100c00700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 0x7f40e40a36f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009c60 tx=0x7f40f0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.017+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f40e8096450 con 0x7f40fc104110 2024-03-30T04:01:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.235+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40fc10c590 con 0x7f40fc104110 2024-03-30T04:01:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.236+0000 7f40f9600700 1 -- 172.21.15.23:0/1575665504 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7f40e8099a70 con 0x7f40fc104110 2024-03-30T04:01:02.236 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.238+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 msgr2=0x7f40e40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.238+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 0x7f40e40a36f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009c60 tx=0x7f40f0009260 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.238+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 msgr2=0x7f40fc0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.238+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f40e800eb30 tx=0x7f40e800ee40 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 shutdown_connections 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40fc106a40 0x7f40fc0ffcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f40e40a1230 0x7f40e40a36f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40fc1017e0 0x7f40fc101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 --2- 172.21.15.23:0/1575665504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f40fc104110 0x7f40fc0ff790 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 >> 172.21.15.23:0/1575665504 conn(0x7f40fc0fb1d0 msgr2=0x7f40fc102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 shutdown_connections 2024-03-30T04:01:02.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.239+0000 7f4103600700 1 -- 172.21.15.23:0/1575665504 wait complete. 2024-03-30T04:01:02.258 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 -- 172.21.15.23:0/3314039332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0106a70 msgr2=0x7f93c0108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 --2- 172.21.15.23:0/3314039332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0106a70 0x7f93c0108ee0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f93b800b0a0 tx=0x7f93b800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 -- 172.21.15.23:0/3314039332 shutdown_connections 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 --2- 172.21.15.23:0/3314039332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0106a70 0x7f93c0108ee0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 --2- 172.21.15.23:0/3314039332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0101810 0x7f93c0103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 --2- 172.21.15.23:0/3314039332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 0x7f93c0106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 -- 172.21.15.23:0/3314039332 >> 172.21.15.23:0/3314039332 conn(0x7f93c00fb200 msgr2=0x7f93c00fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 -- 172.21.15.23:0/3314039332 shutdown_connections 2024-03-30T04:01:02.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.463+0000 7f93c7e00700 1 -- 172.21.15.23:0/3314039332 wait complete. 2024-03-30T04:01:02.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.464+0000 7f93c7e00700 1 Processor -- start 2024-03-30T04:01:02.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.464+0000 7f93c7e00700 1 -- start start 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 0x7f93c0199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 0x7f93c01a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c01a0b50 con 0x7f93c0101810 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c01a0c90 con 0x7f93c0106a70 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c7e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c01a0e20 con 0x7f93c0104140 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58256/0 (socket says 172.21.15.23:58256) 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 -- 172.21.15.23:0/1274321386 learned_addr learned my addr 172.21.15.23:0/1274321386 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c4a00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 0x7f93c0199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 0x7f93c01a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 -- 172.21.15.23:0/1274321386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 msgr2=0x7f93c0199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 0x7f93c0199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 -- 172.21.15.23:0/1274321386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 msgr2=0x7f93c01a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 0x7f93c01a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 -- 172.21.15.23:0/1274321386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b8009d00 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 0x7f93c01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.465+0000 7f93c5400700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f93b400e9a0 tx=0x7f93b400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.466+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4009790 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.466+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f93b400cbd0 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.466+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4018680 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.466+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c01a1100 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.466+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c01a15c0 con 0x7f93c0101810 2024-03-30T04:01:02.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.467+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c01085f0 con 0x7f93c0101810 2024-03-30T04:01:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.469+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f93b400cd40 con 0x7f93c0101810 2024-03-30T04:01:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.470+0000 7f93be000700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 0x7f93a80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.470+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f93b4014070 con 0x7f93c0101810 2024-03-30T04:01:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.470+0000 7f93c4a00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 0x7f93a80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.471+0000 7f93c4a00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 0x7f93a80a3670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009bf0 tx=0x7f93ac009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.475+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f93b40529f0 con 0x7f93c0101810 2024-03-30T04:01:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/535015963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1575665504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/535015963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1575665504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.694+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f93c0107fd0 con 0x7f93c0101810 2024-03-30T04:01:02.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.695+0000 7f93be000700 1 -- 172.21.15.23:0/1274321386 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7f93b4099e50 con 0x7f93c0101810 2024-03-30T04:01:02.695 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.697+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 msgr2=0x7f93a80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.697+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 0x7f93a80a3670 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009bf0 tx=0x7f93ac009260 comp rx=0 tx=0).stop 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.697+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 msgr2=0x7f93c0199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.697+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f93b400e9a0 tx=0x7f93b400ed60 comp rx=0 tx=0).stop 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 shutdown_connections 2024-03-30T04:01:02.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93c0106a70 0x7f93c01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f93c0101810 0x7f93c0199020 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f93a80a11b0 0x7f93a80a3670 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 --2- 172.21.15.23:0/1274321386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93c0104140 0x7f93c0199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 >> 172.21.15.23:0/1274321386 conn(0x7f93c00fb200 msgr2=0x7f93c0100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 shutdown_connections 2024-03-30T04:01:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.698+0000 7f93c7e00700 1 -- 172.21.15.23:0/1274321386 wait complete. 2024-03-30T04:01:02.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:02.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/535015963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1575665504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 -- 172.21.15.23:0/2256951056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 msgr2=0x7f74b81023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 --2- 172.21.15.23:0/2256951056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b81023d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f74a80099c0 tx=0x7f74a8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 -- 172.21.15.23:0/2256951056 shutdown_connections 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 --2- 172.21.15.23:0/2256951056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b81091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 --2- 172.21.15.23:0/2256951056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 0x7f74b81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 --2- 172.21.15.23:0/2256951056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b81023d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 -- 172.21.15.23:0/2256951056 >> 172.21.15.23:0/2256951056 conn(0x7f74b80fc2f0 msgr2=0x7f74b80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:02.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 -- 172.21.15.23:0/2256951056 shutdown_connections 2024-03-30T04:01:02.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.918+0000 7f74c0800700 1 -- 172.21.15.23:0/2256951056 wait complete. 2024-03-30T04:01:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.919+0000 7f74c0800700 1 Processor -- start 2024-03-30T04:01:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.919+0000 7f74c0800700 1 -- start start 2024-03-30T04:01:02.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.919+0000 7f74c0800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 0x7f74b8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.919+0000 7f74c0800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.919+0000 7f74c0800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74c0800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8073e80 con 0x7f74b8101f70 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74c0800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b8073ff0 con 0x7f74b8100d80 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74c0800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b81a0d80 con 0x7f74b8102910 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58278/0 (socket says 172.21.15.23:58278) 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 -- 172.21.15.23:0/2821081729 learned_addr learned my addr 172.21.15.23:0/2821081729 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bde00700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 0x7f74b8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74be800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 -- 172.21.15.23:0/2821081729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 msgr2=0x7f74b80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 -- 172.21.15.23:0/2821081729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 msgr2=0x7f74b8074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:02.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 0x7f74b8074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 -- 172.21.15.23:0/2821081729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a80095f0 con 0x7f74b8101f70 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74be800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.920+0000 7f74bd400700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006010 tx=0x7f74a800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.921+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a801d070 con 0x7f74b8101f70 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.921+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a8005030 con 0x7f74b8101f70 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.921+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a8004bb0 con 0x7f74b8101f70 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.921+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b81a0fb0 con 0x7f74b8101f70 2024-03-30T04:01:02.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.921+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b81a1320 con 0x7f74b8101f70 2024-03-30T04:01:02.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.922+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b8108200 con 0x7f74b8101f70 2024-03-30T04:01:02.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.924+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f74a80051a0 con 0x7f74b8101f70 2024-03-30T04:01:02.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.926+0000 7f74b2a00700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 0x7f74a00a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:02.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.926+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f74a80cdad0 con 0x7f74b8101f70 2024-03-30T04:01:02.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.926+0000 7f74bde00700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 0x7f74a00a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:02.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.927+0000 7f74bde00700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 0x7f74a00a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f74b8101dd0 tx=0x7f74ac009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:02.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:02.930+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f74a8096e80 con 0x7f74b8101f70 2024-03-30T04:01:03.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.147+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f74b810c5e0 con 0x7f74b8101f70 2024-03-30T04:01:03.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.148+0000 7f74b2a00700 1 -- 172.21.15.23:0/2821081729 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f74a809a4a0 con 0x7f74b8101f70 2024-03-30T04:01:03.148 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 msgr2=0x7f74a00a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 0x7f74a00a3670 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f74b8101dd0 tx=0x7f74ac009260 comp rx=0 tx=0).stop 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 msgr2=0x7f74b80731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f74a8006010 tx=0x7f74a800f690 comp rx=0 tx=0).stop 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 shutdown_connections 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f74a00a11b0 0x7f74a00a3670 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74b8102910 0x7f74b80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74b8100d80 0x7f74b8074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 --2- 172.21.15.23:0/2821081729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74b8101f70 0x7f74b80731a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 >> 172.21.15.23:0/2821081729 conn(0x7f74b80fc2f0 msgr2=0x7f74b8105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:03.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.150+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 shutdown_connections 2024-03-30T04:01:03.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.151+0000 7f74c0800700 1 -- 172.21.15.23:0/2821081729 wait complete. 2024-03-30T04:01:03.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 -- 172.21.15.23:0/1089966478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580101e70 msgr2=0x7fb5801022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 --2- 172.21.15.23:0/1089966478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580101e70 0x7fb5801022d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0099c0 tx=0x7fb56c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 -- 172.21.15.23:0/1089966478 shutdown_connections 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 --2- 172.21.15.23:0/1089966478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580102810 0x7fb5801090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 --2- 172.21.15.23:0/1089966478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580100c80 0x7fb5801010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 --2- 172.21.15.23:0/1089966478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580101e70 0x7fb5801022d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 -- 172.21.15.23:0/1089966478 >> 172.21.15.23:0/1089966478 conn(0x7fb5800fc1f0 msgr2=0x7fb5800fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:03.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 -- 172.21.15.23:0/1089966478 shutdown_connections 2024-03-30T04:01:03.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.368+0000 7fb586400700 1 -- 172.21.15.23:0/1089966478 wait complete. 2024-03-30T04:01:03.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.369+0000 7fb586400700 1 Processor -- start 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.369+0000 7fb586400700 1 -- start start 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580101e70 0x7fb5801970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580102810 0x7fb58019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58019e6c0 con 0x7fb580102810 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58019e800 con 0x7fb580100c80 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb586400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58019e990 con 0x7fb580101e70 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47082/0 (socket says 172.21.15.23:47082) 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580102810 0x7fb58019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 -- 172.21.15.23:0/3176809180 learned_addr learned my addr 172.21.15.23:0/3176809180 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57ea00700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580101e70 0x7fb5801970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 -- 172.21.15.23:0/3176809180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580101e70 msgr2=0x7fb5801970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580101e70 0x7fb5801970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 -- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580102810 msgr2=0x7fb58019e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.370+0000 7fb57f400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580102810 0x7fb58019e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb57f400700 1 -- 172.21.15.23:0/3176809180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0095f0 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb57f400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e970 tx=0x7fb57000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570009790 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58019ec70 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb57000c7e0 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570003c70 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.371+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58019f180 con 0x7fb580100c80 2024-03-30T04:01:03.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.372+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5801080d0 con 0x7fb580100c80 2024-03-30T04:01:03.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.374+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb57000cc50 con 0x7fb580100c80 2024-03-30T04:01:03.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.375+0000 7fb577e00700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 0x7fb5640a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.376+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb570014070 con 0x7fb580100c80 2024-03-30T04:01:03.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.376+0000 7fb57ea00700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 0x7fb5640a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.376+0000 7fb57ea00700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 0x7fb5640a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb56c003730 tx=0x7fb56c00b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:03.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.378+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb5700963d0 con 0x7fb580100c80 2024-03-30T04:01:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:03 smithi023 ceph-mon[38254]: pgmap v279: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:03 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1274321386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:03 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2821081729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.650+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb58010c760 con 0x7fb580100c80 2024-03-30T04:01:03.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.651+0000 7fb577e00700 1 -- 172.21.15.23:0/3176809180 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fb570003dd0 con 0x7fb580100c80 2024-03-30T04:01:03.651 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:03.651 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:03.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 msgr2=0x7fb5640a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 0x7fb5640a3670 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb56c003730 tx=0x7fb56c00b540 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 msgr2=0x7fb580196b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e970 tx=0x7fb57000ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 shutdown_connections 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb580102810 0x7fb58019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb580100c80 0x7fb580196b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb5640a11b0 0x7fb5640a3670 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 --2- 172.21.15.23:0/3176809180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb580101e70 0x7fb5801970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 >> 172.21.15.23:0/3176809180 conn(0x7fb5800fc1f0 msgr2=0x7fb580105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 shutdown_connections 2024-03-30T04:01:03.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.654+0000 7fb586400700 1 -- 172.21.15.23:0/3176809180 wait complete. 2024-03-30T04:01:03.655 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:03.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:03 smithi154 ceph-mon[79161]: pgmap v279: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:03 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1274321386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:03 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2821081729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:03 smithi193 ceph-mon[79581]: pgmap v279: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:03.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:03 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1274321386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:03 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2821081729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:03.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.871+0000 7f350aa00700 1 -- 172.21.15.23:0/2436809974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 msgr2=0x7f35041090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.871+0000 7f350aa00700 1 --2- 172.21.15.23:0/2436809974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35041090c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f34fc00b0d0 tx=0x7f34fc00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 -- 172.21.15.23:0/2436809974 shutdown_connections 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 --2- 172.21.15.23:0/2436809974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35041090c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 --2- 172.21.15.23:0/2436809974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 0x7f35041010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 --2- 172.21.15.23:0/2436809974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 0x7f35041022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 -- 172.21.15.23:0/2436809974 >> 172.21.15.23:0/2436809974 conn(0x7f35040fc1f0 msgr2=0x7f35040fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 -- 172.21.15.23:0/2436809974 shutdown_connections 2024-03-30T04:01:03.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.872+0000 7f350aa00700 1 -- 172.21.15.23:0/2436809974 wait complete. 2024-03-30T04:01:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 Processor -- start 2024-03-30T04:01:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 -- start start 2024-03-30T04:01:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 0x7f3504074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 0x7f35040731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504073e80 con 0x7f3504102810 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3504073ff0 con 0x7f3504101e70 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f350aa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35041a0d50 con 0x7f3504100c80 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.873+0000 7f3508a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58310/0 (socket says 172.21.15.23:58310) 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 -- 172.21.15.23:0/2717693336 learned_addr learned my addr 172.21.15.23:0/2717693336 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3503e00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 0x7f3504074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3503400700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 0x7f35040731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 -- 172.21.15.23:0/2717693336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 msgr2=0x7f3504074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 0x7f3504074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 -- 172.21.15.23:0/2717693336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 msgr2=0x7f35040731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 0x7f35040731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 -- 172.21.15.23:0/2717693336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc009d00 con 0x7f3504102810 2024-03-30T04:01:03.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3508a00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34fc007b60 tx=0x7f34fc008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc00e040 con 0x7f3504102810 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34fc01d930 con 0x7f3504102810 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc01b9f0 con 0x7f3504102810 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.874+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35041a0f80 con 0x7f3504102810 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.875+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35041a12f0 con 0x7f3504102810 2024-03-30T04:01:03.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.876+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35041080b0 con 0x7f3504102810 2024-03-30T04:01:03.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.877+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f34fc01daa0 con 0x7f3504102810 2024-03-30T04:01:03.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.879+0000 7f3500c00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 0x7f34e80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:03.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.879+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34fc0ce200 con 0x7f3504102810 2024-03-30T04:01:03.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.879+0000 7f3503e00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 0x7f34e80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:03.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.880+0000 7f3503e00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 0x7f34e80a36f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3504101cd0 tx=0x7f34f4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:03.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:03.883+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f34fc097550 con 0x7f3504102810 2024-03-30T04:01:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.137+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3504067a10 con 0x7f3504102810 2024-03-30T04:01:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.139+0000 7f3500c00700 1 -- 172.21.15.23:0/2717693336 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+24428 (secure 0 0 0) 0x7f34fc09ab70 con 0x7f3504102810 2024-03-30T04:01:04.139 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:04.139 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":86,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:00:17.995781+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 msgr2=0x7f34e80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 0x7f34e80a36f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3504101cd0 tx=0x7f34f4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 msgr2=0x7f35040736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34fc007b60 tx=0x7f34fc008e40 comp rx=0 tx=0).stop 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 shutdown_connections 2024-03-30T04:01:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3504102810 0x7f35040736e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f34e80a1230 0x7f34e80a36f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3504100c80 0x7f3504074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 --2- 172.21.15.23:0/2717693336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3504101e70 0x7f35040731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.141+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 >> 172.21.15.23:0/2717693336 conn(0x7f35040fc1f0 msgr2=0x7f35041051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.142+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 shutdown_connections 2024-03-30T04:01:04.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.142+0000 7f350aa00700 1 -- 172.21.15.23:0/2717693336 wait complete. 2024-03-30T04:01:04.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 -- 172.21.15.23:0/1433370183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470100c90 msgr2=0x7fe4701010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 --2- 172.21.15.23:0/1433370183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470100c90 0x7fe4701010b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe4600099c0 tx=0x7fe460009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 -- 172.21.15.23:0/1433370183 shutdown_connections 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 --2- 172.21.15.23:0/1433370183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470102820 0x7fe4701090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 --2- 172.21.15.23:0/1433370183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470100c90 0x7fe4701010b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 --2- 172.21.15.23:0/1433370183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe4701022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 -- 172.21.15.23:0/1433370183 >> 172.21.15.23:0/1433370183 conn(0x7fe4700fc200 msgr2=0x7fe4700fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 -- 172.21.15.23:0/1433370183 shutdown_connections 2024-03-30T04:01:04.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.360+0000 7fe476e00700 1 -- 172.21.15.23:0/1433370183 wait complete. 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.361+0000 7fe476e00700 1 Processor -- start 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.361+0000 7fe476e00700 1 -- start start 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.361+0000 7fe476e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 0x7fe470198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe476e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe476e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 0x7fe4701a03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe476e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701a0970 con 0x7fe470102820 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe476e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701a0ab0 con 0x7fe470100c90 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe476e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4701a0c40 con 0x7fe470101e80 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59266/0 (socket says 172.21.15.23:59266) 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 0x7fe4701a03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46f400700 1 -- 172.21.15.23:0/943299583 learned_addr learned my addr 172.21.15.23:0/943299583 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46fe00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 0x7fe470198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 -- 172.21.15.23:0/943299583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 msgr2=0x7fe470199380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 -- 172.21.15.23:0/943299583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 msgr2=0x7fe470198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 0x7fe470198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe474e00700 1 -- 172.21.15.23:0/943299583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4600095f0 con 0x7fe470102820 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46f400700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.362+0000 7fe46fe00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 0x7fe470198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe474e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 0x7fe4701a03a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe46800efc0 tx=0x7fe46800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468018070 con 0x7fe470102820 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe46800f040 con 0x7fe470102820 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46800cd40 con 0x7fe470102820 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4701a0ec0 con 0x7fe470102820 2024-03-30T04:01:04.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.363+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4701a1340 con 0x7fe470102820 2024-03-30T04:01:04.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.364+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe470067a10 con 0x7fe470102820 2024-03-30T04:01:04.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.366+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe468003a00 con 0x7fe470102820 2024-03-30T04:01:04.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.367+0000 7fe46cc00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 0x7fe4540a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.367+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe4680cc8c0 con 0x7fe470102820 2024-03-30T04:01:04.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.367+0000 7fe46fe00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 0x7fe4540a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.368+0000 7fe46fe00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 0x7fe4540a36f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe460003730 tx=0x7fe46000b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.371+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe468095c10 con 0x7fe470102820 2024-03-30T04:01:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:04 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3176809180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:04 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2717693336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:04 smithi023 ceph-mon[38254]: pgmap v280: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:04.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.645+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe47010c4f0 con 0x7fe470102820 2024-03-30T04:01:04.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.646+0000 7fe46cc00700 1 -- 172.21.15.23:0/943299583 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fe46801b320 con 0x7fe470102820 2024-03-30T04:01:04.646 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:04.646 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 msgr2=0x7fe4540a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 0x7fe4540a36f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe460003730 tx=0x7fe46000b540 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 msgr2=0x7fe4701a03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 0x7fe4701a03a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe46800efc0 tx=0x7fe46800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 shutdown_connections 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe470102820 0x7fe4701a03a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe470100c90 0x7fe470198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe4540a1230 0x7fe4540a36f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 --2- 172.21.15.23:0/943299583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe470101e80 0x7fe470199380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 >> 172.21.15.23:0/943299583 conn(0x7fe4700fc200 msgr2=0x7fe470072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 shutdown_connections 2024-03-30T04:01:04.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.649+0000 7fe476e00700 1 -- 172.21.15.23:0/943299583 wait complete. 2024-03-30T04:01:04.650 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:04.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-30T03:59:40.432862+0000', 'modified': '2024-03-30T04:00:13.957636+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24569, 'mds_1': 24533, 'mds_2': 24509, 'mds_3': 24508, 'mds_4': 24476}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14544': {'gid': 14544, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/667186519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 667186519}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 667186519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14556': {'gid': 14556, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6841/2954889723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 2954889723}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 2954889723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24457': {'gid': 24457, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/2731954066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 2731954066}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 2731954066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24476': {'gid': 24476, 'name': 'k', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.154:6833/3615295039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6832', 'nonce': 3615295039}, {'type': 'v1', 'addr': '172.21.15.154:6833', 'nonce': 3615295039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.193:6835/2492108108', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 2492108108}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 2492108108}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.154:6835/1153342637', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 1153342637}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 1153342637}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/1829293538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1829293538}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1829293538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6839/606583058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 606583058}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 606583058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.154:6837/3317012042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 3317012042}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 3317012042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/2911311868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2911311868}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2911311868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-30T04:01:04.668 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-30T04:01:04.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-30T04:01:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:04 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3176809180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:04 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2717693336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:04 smithi154 ceph-mon[79161]: pgmap v280: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:04.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:04 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3176809180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:04 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2717693336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:04.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:04 smithi193 ceph-mon[79581]: pgmap v280: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 -- 172.21.15.23:0/1256073169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 msgr2=0x7f6d18106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 --2- 172.21.15.23:0/1256073169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18106530 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009990 tx=0x7f6d04009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 -- 172.21.15.23:0/1256073169 shutdown_connections 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 --2- 172.21.15.23:0/1256073169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 0x7f6d18108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 --2- 172.21.15.23:0/1256073169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 --2- 172.21.15.23:0/1256073169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18106530 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 -- 172.21.15.23:0/1256073169 >> 172.21.15.23:0/1256073169 conn(0x7f6d180fb200 msgr2=0x7f6d180fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 -- 172.21.15.23:0/1256073169 shutdown_connections 2024-03-30T04:01:04.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.869+0000 7f6d1f000700 1 -- 172.21.15.23:0/1256073169 wait complete. 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.870+0000 7f6d1f000700 1 Processor -- start 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.870+0000 7f6d1f000700 1 -- start start 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.870+0000 7f6d1f000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.870+0000 7f6d1f000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.870+0000 7f6d1f000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 0x7f6d181a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d1f000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181a0a80 con 0x7f6d18104140 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d1f000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181a0bc0 con 0x7f6d18101810 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d1f000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d181a0d50 con 0x7f6d18106a70 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58344/0 (socket says 172.21.15.23:58344) 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 -- 172.21.15.23:0/587431638 learned_addr learned my addr 172.21.15.23:0/587431638 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17e00700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d1d000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 0x7f6d181a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 -- 172.21.15.23:0/587431638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 msgr2=0x7f6d181a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 0x7f6d181a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 -- 172.21.15.23:0/587431638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 msgr2=0x7f6d18198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 -- 172.21.15.23:0/587431638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040095f0 con 0x7f6d18104140 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17e00700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.871+0000 7f6d17400700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400b680 tx=0x7f6d04009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.872+0000 7f6d14c00700 1 -- 172.21.15.23:0/587431638 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0401d070 con 0x7f6d18104140 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.872+0000 7f6d14c00700 1 -- 172.21.15.23:0/587431638 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d04022470 con 0x7f6d18104140 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.872+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d181a0fd0 con 0x7f6d18104140 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.872+0000 7f6d14c00700 1 -- 172.21.15.23:0/587431638 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04017650 con 0x7f6d18104140 2024-03-30T04:01:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.872+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181a1390 con 0x7f6d18104140 2024-03-30T04:01:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.875+0000 7f6d14c00700 1 -- 172.21.15.23:0/587431638 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f6d040225e0 con 0x7f6d18104140 2024-03-30T04:01:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.876+0000 7f6d14c00700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 0x7f6cfc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.876+0000 7f6d32b21b80 1 Processor -- start 2024-03-30T04:01:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.876+0000 7f6d14c00700 1 -- 172.21.15.23:0/587431638 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6d040ce0a0 con 0x7f6d18104140 2024-03-30T04:01:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.876+0000 7f6d32b21b80 1 -- start start 2024-03-30T04:01:04.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.876+0000 7f6d17e00700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 0x7f6cfc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f4cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f4d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7ea4940 con 0x5563a7f4df80 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7ea2300 con 0x5563a7f4d0e0 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7e9dbf0 con 0x5563a7f4c780 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d17400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f4d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58346/0 (socket says 172.21.15.23:58346) 2024-03-30T04:01:04.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 -- 172.21.15.23:0/3452648935 learned_addr learned my addr 172.21.15.23:0/3452648935 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d17e00700 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f4cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d17e00700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 0x7f6cfc0a3670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d08011550 tx=0x7f6d08009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 -- 172.21.15.23:0/3452648935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 msgr2=0x5563a7f4cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f4cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 -- 172.21.15.23:0/3452648935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 msgr2=0x5563a7f4d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f4d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d1d000700 1 -- 172.21.15.23:0/3452648935 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563a7d1f360 con 0x5563a7f4df80 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.877+0000 7f6d17400700 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f4d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.878+0000 7f6d1d000700 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d10011680 tx=0x7f6d10009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241e9e8a1bfb1a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.878+0000 7f6d0d600700 1 -- 172.21.15.23:0/3452648935 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1000e040 con 0x5563a7f4df80 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.878+0000 7f6d0d600700 1 -- 172.21.15.23:0/3452648935 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d1000f040 con 0x5563a7f4df80 2024-03-30T04:01:04.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.878+0000 7f6d0d600700 1 -- 172.21.15.23:0/3452648935 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d10004860 con 0x5563a7f4df80 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3452648935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 msgr2=0x5563a7f4e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6d10011680 tx=0x7f6d10009920 comp rx=0 tx=0).stop 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3452648935 shutdown_connections 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e400 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f4cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3452648935 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f4d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3452648935 >> 172.21.15.23:0/3452648935 conn(0x5563a7eab650 msgr2=0x5563a7f47950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3452648935 shutdown_connections 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3452648935 wait complete. 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.879+0000 7f6d32b21b80 1 -- start start 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 Processor -- start 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 0x5563a7f56580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4df80 0x5563a7f5db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7e39d30 con 0x5563a7f4c780 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7e399d0 con 0x5563a7f4df80 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7d21650 con 0x5563a7f4d0e0 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d32b21b80 10 client.0 _ll_register_callbacks cb 0x5563a7e43420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 0x5563a7f56580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59294/0 (socket says 172.21.15.23:59294) 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d17400700 1 -- 172.21.15.23:0/1482187570 learned_addr learned my addr 172.21.15.23:0/1482187570 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.880+0000 7f6d1d000700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4df80 0x5563a7f5db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 -- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4df80 msgr2=0x5563a7f5db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4df80 0x5563a7f5db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 -- 172.21.15.23:0/1482187570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 msgr2=0x5563a7f56580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 0x5563a7f56580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 -- 172.21.15.23:0/1482187570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181a1390 con 0x5563a7f4d0e0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17e00700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 0x5563a7f56580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d17400700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6d040975b0 tx=0x7f6d04022bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe82b3d473d19020 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d02000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0409eee0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6d040a34a0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563a7e5e330 con 0x5563a7f4d0e0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.881+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d040a10b0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 10 client.25138 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f6d10011330 con 0x5563a7f4d0e0 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5563a7e5e330 con 0x5563a7f4d0e0 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d040a30a0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 10 client.25138 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 10 client.25138 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 20 client.25138 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7f61420 0x5563a7f66700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 4 client.25138 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.882+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563a7e966b0 con 0x5563a7f61420 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.883+0000 7f6d17e00700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7f61420 0x5563a7f66700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.883+0000 7f6d17e00700 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7f61420 0x5563a7f66700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1cc79d7077447331 server_cookie=c4be200da95f86b3 in_seq=0 out_seq=0 2024-03-30T04:01:04.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.883+0000 7f6d02000700 10 client.25138 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d02000700 1 -- 172.21.15.23:0/1482187570 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5563a7e966b0 con 0x5563a7f61420 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d02000700 10 client.25138 handle_command_reply: tid=0 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 client.25138 shutdown 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 msgr2=0x5563a7f56ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6d040975b0 tx=0x7f6d04022bc0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 shutdown_connections 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4df80 0x5563a7f5db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.884+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4c780 0x5563a7f56580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7f61420 0x5563a7f66700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/1482187570 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f56ac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 >> 172.21.15.23:0/1482187570 conn(0x5563a7eab650 msgr2=0x5563a7f4b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d02000700 0 client.25138 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 shutdown_connections 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 1 -- 172.21.15.23:0/1482187570 wait complete. 2024-03-30T04:01:04.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.885+0000 7f6d32b21b80 20 client.25138 trim_cache size 0 max 16384 2024-03-30T04:01:04.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 Processor -- start 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 -- start start 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f6ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7f43430 con 0x5563a7f4df80 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7f48610 con 0x5563a7f4d0e0 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.903+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7e966b0 con 0x5563a7f4c780 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58364/0 (socket says 172.21.15.23:58364) 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f6ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d1d000700 1 -- 172.21.15.23:0/3924032754 learned_addr learned my addr 172.21.15.23:0/3924032754 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 -- 172.21.15.23:0/3924032754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 msgr2=0x5563a7f6ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f6ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 -- 172.21.15.23:0/3924032754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 msgr2=0x5563a7f4e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 -- 172.21.15.23:0/3924032754 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563a7ccbb10 con 0x5563a7f4d0e0 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.904+0000 7f6d17400700 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5dac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005910 tx=0x7f6d040cbb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a1a5ecbaa2237cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d0e000700 1 -- 172.21.15.23:0/3924032754 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040a34a0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d0e000700 1 -- 172.21.15.23:0/3924032754 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d0409eee0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d0e000700 1 -- 172.21.15.23:0/3924032754 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d04096f40 con 0x5563a7f4d0e0 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3924032754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 msgr2=0x5563a7f5dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5dac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6d04005910 tx=0x7f6d040cbb40 comp rx=0 tx=0).stop 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3924032754 shutdown_connections 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f4e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4c780 0x5563a7f6ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/3924032754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5dac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3924032754 >> 172.21.15.23:0/3924032754 conn(0x5563a7eab650 msgr2=0x5563a7f6a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3924032754 shutdown_connections 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.905+0000 7f6d32b21b80 1 -- 172.21.15.23:0/3924032754 wait complete. 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.906+0000 7f6d32b21b80 1 -- start start 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.906+0000 7f6d32b21b80 1 Processor -- start 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7f5aff0 con 0x5563a7f4df80 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7f5b160 con 0x5563a7f4c780 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563a7fb5cf0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d32b21b80 10 client.0 _ll_register_callbacks cb 0x5563a7e43420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47176/0 (socket says 172.21.15.23:47176) 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59308/0 (socket says 172.21.15.23:59308) 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17e00700 1 -- 172.21.15.23:0/264607776 learned_addr learned my addr 172.21.15.23:0/264607776 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d1d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58368/0 (socket says 172.21.15.23:58368) 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 -- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 msgr2=0x5563a7f5d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 -- 172.21.15.23:0/264607776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 msgr2=0x5563a7f5aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.907+0000 7f6d17400700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d17400700 1 -- 172.21.15.23:0/264607776 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d10011330 con 0x5563a7f4d0e0 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d17e00700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d1d000700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:04.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d17400700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5563a7f62880 tx=0x7f6d040973f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39573ab236410d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d02000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040a34a0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6d040a3010 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d1000f040 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d040a31d0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d32b21b80 10 client.25144 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5563a7e63e50 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.908+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d1000f040 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6d0409a5a0 con 0x5563a7f4d0e0 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 10 client.25144 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 10 client.25144 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 20 client.25144 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:04.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7fba3f0 0x5563a7fb6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:04.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 4 client.25144 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:04.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5563a7cecca0 con 0x5563a7fba3f0 2024-03-30T04:01:04.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.909+0000 7f6d17e00700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7fba3f0 0x5563a7fb6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:04.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d02000700 10 client.25144 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:04.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d17e00700 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7fba3f0 0x5563a7fb6f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e9f5bee8e1be2c79 server_cookie=b4980736ffa60155 in_seq=0 out_seq=0 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d02000700 1 -- 172.21.15.23:0/264607776 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5563a7cecca0 con 0x5563a7fba3f0 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d02000700 10 client.25144 handle_command_reply: tid=0 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 client.25144 shutdown 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 msgr2=0x5563a7f5a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x5563a7f62880 tx=0x7f6d040973f0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 shutdown_connections 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5563a7f4df80 0x5563a7f5aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5563a7f4c780 0x5563a7f5d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5563a7fba3f0 0x5563a7fb6f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 --2- 172.21.15.23:0/264607776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5563a7f4d0e0 0x5563a7f5a570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 >> 172.21.15.23:0/264607776 conn(0x5563a7eab650 msgr2=0x5563a7f4b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d02000700 0 client.25144 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 shutdown_connections 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 1 -- 172.21.15.23:0/264607776 wait complete. 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.910+0000 7f6d32b21b80 20 client.25144 trim_cache size 0 max 16384 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:04.915 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24476, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 51.959335471000003, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 86, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 112.04992452400001 2024-03-30T04:01:04.916 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 msgr2=0x7f6cfc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 0x7f6cfc0a3670 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d08011550 tx=0x7f6d08009260 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 msgr2=0x7f6d18199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6d0400b680 tx=0x7f6d04009580 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 shutdown_connections 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6cfc0a11b0 0x7f6cfc0a3670 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d18106a70 0x7f6d181a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6d18101810 0x7f6d18198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 --2- 172.21.15.23:0/587431638 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d18104140 0x7f6d18199490 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:04.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 >> 172.21.15.23:0/587431638 conn(0x7f6d180fb200 msgr2=0x7f6d180ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 shutdown_connections 2024-03-30T04:01:04.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:04.911+0000 7f6d1f000700 1 -- 172.21.15.23:0/587431638 wait complete. 2024-03-30T04:01:04.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-30T04:01:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 -- 172.21.15.23:0/1179108402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678102910 msgr2=0x7fc6781091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 --2- 172.21.15.23:0/1179108402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678102910 0x7fc6781091c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc67000b0d0 tx=0x7fc67000b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 -- 172.21.15.23:0/1179108402 shutdown_connections 2024-03-30T04:01:05.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 --2- 172.21.15.23:0/1179108402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678102910 0x7fc6781091c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 --2- 172.21.15.23:0/1179108402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678100d80 0x7fc6781011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 --2- 172.21.15.23:0/1179108402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678101f70 0x7fc6781023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 -- 172.21.15.23:0/1179108402 >> 172.21.15.23:0/1179108402 conn(0x7fc6780fc2f0 msgr2=0x7fc6780fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 -- 172.21.15.23:0/1179108402 shutdown_connections 2024-03-30T04:01:05.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.131+0000 7fc67de00700 1 -- 172.21.15.23:0/1179108402 wait complete. 2024-03-30T04:01:05.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.132+0000 7fc67de00700 1 Processor -- start 2024-03-30T04:01:05.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.132+0000 7fc67de00700 1 -- start start 2024-03-30T04:01:05.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 0x7fc6781990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678102910 0x7fc6781a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781a0ba0 con 0x7fc678101f70 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781a0ce0 con 0x7fc678102910 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc67de00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6781a0e70 con 0x7fc678100d80 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 0x7fc6781990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58394/0 (socket says 172.21.15.23:58394) 2024-03-30T04:01:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 0x7fc6781990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59324/0 (socket says 172.21.15.23:59324) 2024-03-30T04:01:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 -- 172.21.15.23:0/1421636176 learned_addr learned my addr 172.21.15.23:0/1421636176 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc677400700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678102910 0x7fc6781a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 -- 172.21.15.23:0/1421636176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 msgr2=0x7fc6781990c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 0x7fc6781990c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 -- 172.21.15.23:0/1421636176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678102910 msgr2=0x7fc6781a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678102910 0x7fc6781a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.133+0000 7fc676000700 1 -- 172.21.15.23:0/1421636176 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc670009d00 con 0x7fc678101f70 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc676000700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e9a0 tx=0x7fc66400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc67ca00700 1 -- 172.21.15.23:0/1421636176 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664009790 con 0x7fc678101f70 2024-03-30T04:01:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc67ca00700 1 -- 172.21.15.23:0/1421636176 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc664010430 con 0x7fc678101f70 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6781a1150 con 0x7fc678101f70 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc67ca00700 1 -- 172.21.15.23:0/1421636176 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664018680 con 0x7fc678101f70 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.134+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6781a16b0 con 0x7fc678101f70 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.137+0000 7fc67ca00700 1 -- 172.21.15.23:0/1421636176 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc6640105a0 con 0x7fc678101f70 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.138+0000 7fc691956b80 1 Processor -- start 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.138+0000 7fc691956b80 1 -- start start 2024-03-30T04:01:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc67ca00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 0x7fc65c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc67ca00700 1 -- 172.21.15.23:0/1421636176 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc664014070 con 0x7fc678101f70 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc677400700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 0x7fc65c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919f939a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919f94400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919eea940 con 0x558919f92780 2024-03-30T04:01:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919ee8300 con 0x558919f93f80 2024-03-30T04:01:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc691956b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919ee3bf0 con 0x558919f930e0 2024-03-30T04:01:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc676a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc676a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58404/0 (socket says 172.21.15.23:58404) 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc676a00700 1 -- 172.21.15.23:0/490175318 learned_addr learned my addr 172.21.15.23:0/490175318 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.139+0000 7fc676000700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919f939a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc677400700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919f94400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 -- 172.21.15.23:0/490175318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 msgr2=0x558919f94400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919f94400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 -- 172.21.15.23:0/490175318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 msgr2=0x558919f92ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 -- 172.21.15.23:0/490175318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558919d65360 con 0x558919f930e0 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676a00700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc677400700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919f94400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc677400700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 0x7fc65c0a36f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc670009fd0 tx=0x7fc670009340 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc676000700 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919f939a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ca550 tx=0x7fc66409a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=773d1478b9239888 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc663e00700 1 -- 172.21.15.23:0/490175318 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66409aea0 con 0x558919f930e0 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc663e00700 1 -- 172.21.15.23:0/490175318 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc66409dcc0 con 0x558919f930e0 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.140+0000 7fc663e00700 1 -- 172.21.15.23:0/490175318 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664099580 con 0x558919f930e0 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 -- 172.21.15.23:0/490175318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 msgr2=0x558919f939a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919f939a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ca550 tx=0x7fc66409a590 comp rx=0 tx=0).stop 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 -- 172.21.15.23:0/490175318 shutdown_connections 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919f94400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919f92ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 --2- 172.21.15.23:0/490175318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919f939a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 -- 172.21.15.23:0/490175318 >> 172.21.15.23:0/490175318 conn(0x558919ef1650 msgr2=0x558919f8d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 -- 172.21.15.23:0/490175318 shutdown_connections 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.141+0000 7fc691956b80 1 -- 172.21.15.23:0/490175318 wait complete. 2024-03-30T04:01:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 -- start start 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 Processor -- start 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f9c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 0x558919fa3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919e7fd30 con 0x558919f93f80 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919e7f9d0 con 0x558919f930e0 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919d67650 con 0x558919f92780 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.142+0000 7fc691956b80 10 client.0 _ll_register_callbacks cb 0x558919e89420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47212/0 (socket says 172.21.15.23:47212) 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc677400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 0x558919fa3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 -- 172.21.15.23:0/1097296203 learned_addr learned my addr 172.21.15.23:0/1097296203 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676a00700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f9c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 -- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 msgr2=0x558919f9c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f9c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 -- 172.21.15.23:0/1097296203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 msgr2=0x558919fa3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 0x558919fa3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 -- 172.21.15.23:0/1097296203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc668009960 con 0x558919f930e0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc677400700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 0x558919fa3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc676000700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ca580 tx=0x7fc6640a2130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54be905f82f7f7a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.143+0000 7fc660c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66409aea0 con 0x558919f930e0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc664010880 con 0x558919f930e0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558919ea4330 con 0x558919f930e0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6640b82a0 con 0x558919f930e0 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 10 client.25166 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fc6781a16b0 con 0x558919f930e0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558919ea4330 con 0x558919f930e0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc664094aa0 con 0x558919f930e0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 10 client.25166 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 10 client.25166 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 20 client.25166 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558919fa7500 0x558919fac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 4 client.25166 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.144+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558919edc6b0 con 0x558919fa7500 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.145+0000 7fc676a00700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558919fa7500 0x558919fac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.145+0000 7fc676a00700 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558919fa7500 0x558919fac7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=985c59624c71a5cd server_cookie=2fe0cdc26046b622 in_seq=0 out_seq=0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.145+0000 7fc660c00700 10 client.25166 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc660c00700 1 -- 172.21.15.23:0/1097296203 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558919edc6b0 con 0x558919fa7500 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc660c00700 10 client.25166 handle_command_reply: tid=0 2024-03-30T04:01:05.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 client.25166 shutdown 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 msgr2=0x558919f9cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc6640ca580 tx=0x7fc6640a2130 comp rx=0 tx=0).stop 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 shutdown_connections 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558919fa7500 0x558919fac7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f93f80 0x558919fa3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f9c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 --2- 172.21.15.23:0/1097296203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f930e0 0x558919f9cbd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc660c00700 0 client.25166 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.146+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 >> 172.21.15.23:0/1097296203 conn(0x558919ef1650 msgr2=0x558919f91120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.147+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 shutdown_connections 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.147+0000 7fc691956b80 1 -- 172.21.15.23:0/1097296203 wait complete. 2024-03-30T04:01:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.147+0000 7fc691956b80 20 client.25166 trim_cache size 0 max 16384 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 Processor -- start 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 -- start start 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f92b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919ea4330 con 0x558919f930e0 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919f8e8c0 con 0x558919f93f80 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc691956b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919ea1ad0 con 0x558919f92780 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.165+0000 7fc676000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f92b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58410/0 (socket says 172.21.15.23:58410) 2024-03-30T04:01:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f92b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59358/0 (socket says 172.21.15.23:59358) 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 -- 172.21.15.23:0/4221528016 learned_addr learned my addr 172.21.15.23:0/4221528016 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc677400700 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 -- 172.21.15.23:0/4221528016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 msgr2=0x558919f92b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f92b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 -- 172.21.15.23:0/4221528016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 msgr2=0x558919fa3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 -- 172.21.15.23:0/4221528016 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558919d11b10 con 0x558919f930e0 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc676000700 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc664096390 tx=0x7fc664010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e5dac7bb670909 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc66d200700 1 -- 172.21.15.23:0/4221528016 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66409aea0 con 0x558919f930e0 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc66d200700 1 -- 172.21.15.23:0/4221528016 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc66409a590 con 0x558919f930e0 2024-03-30T04:01:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.166+0000 7fc66d200700 1 -- 172.21.15.23:0/4221528016 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664098df0 con 0x558919f930e0 2024-03-30T04:01:05.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 -- 172.21.15.23:0/4221528016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 msgr2=0x558919f9bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc664096390 tx=0x7fc664010c80 comp rx=0 tx=0).stop 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 -- 172.21.15.23:0/4221528016 shutdown_connections 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f92780 0x558919f92b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 --2- 172.21.15.23:0/4221528016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f930e0 0x558919f9bba0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 -- 172.21.15.23:0/4221528016 >> 172.21.15.23:0/4221528016 conn(0x558919ef1650 msgr2=0x558919f9a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 -- 172.21.15.23:0/4221528016 shutdown_connections 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.167+0000 7fc691956b80 1 -- 172.21.15.23:0/4221528016 wait complete. 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 -- start start 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 Processor -- start 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919fa0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919fa0ef0 con 0x558919f92780 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919fa1030 con 0x558919f93f80 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558919ffbd90 con 0x558919f930e0 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc691956b80 10 client.0 _ll_register_callbacks cb 0x558919e89420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc676a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc676a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58426/0 (socket says 172.21.15.23:58426) 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.168+0000 7fc676a00700 1 -- 172.21.15.23:0/3045611606 learned_addr learned my addr 172.21.15.23:0/3045611606 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676000700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919fa0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc677400700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 -- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 msgr2=0x558919fa0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919fa0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 -- 172.21.15.23:0/3045611606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 msgr2=0x558919fa09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 -- 172.21.15.23:0/3045611606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6781a16b0 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc676a00700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x558919f9d220 tx=0x7fc668004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be6a35396409726 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc660c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668017040 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.169+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc66409dcc0 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc668021370 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc668021530 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 10 client.15162 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558919ea9e50 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc66409dcc0 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc6680109a0 con 0x558919f92780 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 10 client.15162 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 10 client.15162 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 20 client.15162 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55891a000c20 0x558919ffd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 4 client.15162 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.170+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558919d32ca0 con 0x55891a000c20 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc676000700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55891a000c20 0x558919ffd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc676000700 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55891a000c20 0x558919ffd590 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=47cee571213875eb server_cookie=2abb9316567db723 in_seq=0 out_seq=0 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc660c00700 10 client.15162 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc660c00700 1 -- 172.21.15.23:0/3045611606 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x558919d32ca0 con 0x55891a000c20 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc660c00700 10 client.15162 handle_command_reply: tid=0 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc691956b80 1 client.15162 shutdown 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 msgr2=0x558919fa3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.171+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x558919f9d220 tx=0x7fc668004690 comp rx=0 tx=0).stop 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 shutdown_connections 2024-03-30T04:01:05.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55891a000c20 0x558919ffd590 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558919f93f80 0x558919fa09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558919f92780 0x558919fa3980 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 --2- 172.21.15.23:0/3045611606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558919f930e0 0x558919fa0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 >> 172.21.15.23:0/3045611606 conn(0x558919ef1650 msgr2=0x558919f9a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc660c00700 0 client.15162 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 shutdown_connections 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 1 -- 172.21.15.23:0/3045611606 wait complete. 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc691956b80 20 client.15162 trim_cache size 0 max 16384 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:05.176 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24508, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 53.228754125999998, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 86, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 103.98482836399999 2024-03-30T04:01:05.177 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 msgr2=0x7fc65c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 0x7fc65c0a36f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc670009fd0 tx=0x7fc670009340 comp rx=0 tx=0).stop 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 msgr2=0x7fc678199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.172+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc66400e9a0 tx=0x7fc66400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 shutdown_connections 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc678102910 0x7fc6781a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc65c0a1230 0x7fc65c0a36f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc678100d80 0x7fc6781990c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 --2- 172.21.15.23:0/1421636176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc678101f70 0x7fc678199600 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 >> 172.21.15.23:0/1421636176 conn(0x7fc6780fc2f0 msgr2=0x7fc678072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 shutdown_connections 2024-03-30T04:01:05.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.173+0000 7fc67de00700 1 -- 172.21.15.23:0/1421636176 wait complete. 2024-03-30T04:01:05.190 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-30T04:01:05.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 -- 172.21.15.23:0/3256353643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 msgr2=0x7f5e041023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/3256353643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e041023d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5df00099c0 tx=0x7f5df0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 -- 172.21.15.23:0/3256353643 shutdown_connections 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/3256353643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04102910 0x7f5e041091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/3256353643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04100d80 0x7f5e041011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/3256353643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e041023d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 -- 172.21.15.23:0/3256353643 >> 172.21.15.23:0/3256353643 conn(0x7f5e040fc2f0 msgr2=0x7f5e040fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 -- 172.21.15.23:0/3256353643 shutdown_connections 2024-03-30T04:01:05.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.392+0000 7f5e0bc00700 1 -- 172.21.15.23:0/3256353643 wait complete. 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 Processor -- start 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 -- start start 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 0x7f5e04074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04102910 0x7f5e040736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04073e80 con 0x7f5e04101f70 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e04073ff0 con 0x7f5e04102910 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.393+0000 7f5e0bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e041a0d80 con 0x7f5e04100d80 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 0x7f5e04074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58448/0 (socket says 172.21.15.23:58448) 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 -- 172.21.15.23:0/2862959143 learned_addr learned my addr 172.21.15.23:0/2862959143 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e09200700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 0x7f5e04074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59386/0 (socket says 172.21.15.23:59386) 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e09c00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04102910 0x7f5e040736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 -- 172.21.15.23:0/2862959143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 msgr2=0x7f5e04074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 0x7f5e04074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 -- 172.21.15.23:0/2862959143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04102910 msgr2=0x7f5e040736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04102910 0x7f5e040736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 -- 172.21.15.23:0/2862959143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df00095f0 con 0x7f5e04101f70 2024-03-30T04:01:05.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e03e00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006010 tx=0x7f5df000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.394+0000 7f5e01600700 1 -- 172.21.15.23:0/2862959143 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df001d070 con 0x7f5e04101f70 2024-03-30T04:01:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.395+0000 7f5e01600700 1 -- 172.21.15.23:0/2862959143 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5df0005030 con 0x7f5e04101f70 2024-03-30T04:01:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.395+0000 7f5e01600700 1 -- 172.21.15.23:0/2862959143 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0004b10 con 0x7f5e04101f70 2024-03-30T04:01:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.395+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e041a0fb0 con 0x7f5e04101f70 2024-03-30T04:01:05.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.395+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e041a1320 con 0x7f5e04101f70 2024-03-30T04:01:05.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.398+0000 7f5e01600700 1 -- 172.21.15.23:0/2862959143 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5df00051a0 con 0x7f5e04101f70 2024-03-30T04:01:05.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.399+0000 7f5e01600700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 0x7f5dec0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.399+0000 7f5e01600700 1 -- 172.21.15.23:0/2862959143 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5df00cda40 con 0x7f5e04101f70 2024-03-30T04:01:05.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.399+0000 7f5e1f7b8b80 1 Processor -- start 2024-03-30T04:01:05.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.399+0000 7f5e09200700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 0x7f5dec0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.399+0000 7f5e1f7b8b80 1 -- start start 2024-03-30T04:01:05.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e09776aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976c2940 con 0x55e09776bf80 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976c0300 con 0x55e09776b0e0 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976bbbf0 con 0x55e09776a780 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e09200700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 0x7f5dec0a36f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5e04101dd0 tx=0x7f5df8009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e09776aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47250/0 (socket says 172.21.15.23:47250) 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 -- 172.21.15.23:0/3695304996 learned_addr learned my addr 172.21.15.23:0/3695304996 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e09c00700 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 -- 172.21.15.23:0/3695304996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 msgr2=0x55e09776aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e09776aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 -- 172.21.15.23:0/3695304996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 msgr2=0x55e09776c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.400+0000 7f5e03e00700 1 -- 172.21.15.23:0/3695304996 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e09753d360 con 0x55e09776b0e0 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.401+0000 7f5e03e00700 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5df009a9e0 tx=0x7f5df009a450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97da2b8e89a09e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.401+0000 7f5df6000700 1 -- 172.21.15.23:0/3695304996 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df009e880 con 0x55e09776b0e0 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.401+0000 7f5df6000700 1 -- 172.21.15.23:0/3695304996 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5df00a2e40 con 0x55e09776b0e0 2024-03-30T04:01:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.401+0000 7f5df6000700 1 -- 172.21.15.23:0/3695304996 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df009af10 con 0x55e09776b0e0 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3695304996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 msgr2=0x55e09776b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5df009a9e0 tx=0x7f5df009a450 comp rx=0 tx=0).stop 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3695304996 shutdown_connections 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e09776aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3695304996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e09776b9a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3695304996 >> 172.21.15.23:0/3695304996 conn(0x55e0976c9650 msgr2=0x55e097765950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3695304996 shutdown_connections 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3695304996 wait complete. 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 -- start start 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.402+0000 7f5e1f7b8b80 1 Processor -- start 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e097774300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 0x55e097774840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e097657d30 con 0x55e09776b0e0 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976579d0 con 0x55e09776a780 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e09753f650 con 0x55e09776bf80 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e1f7b8b80 10 client.0 _ll_register_callbacks cb 0x55e097661420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 0x55e097774840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e09c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e09c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59406/0 (socket says 172.21.15.23:59406) 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 0x55e097774840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58466/0 (socket says 172.21.15.23:58466) 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e09c00700 1 -- 172.21.15.23:0/3865121444 learned_addr learned my addr 172.21.15.23:0/3865121444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.403+0000 7f5e09200700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e097774300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 -- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 msgr2=0x55e097774300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e097774300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 -- 172.21.15.23:0/3865121444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 msgr2=0x55e097774840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 0x55e097774840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 -- 172.21.15.23:0/3865121444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e041a1320 con 0x55e09776bf80 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e09c00700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc012e20 tx=0x7f5dfc014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314917a6a63c013 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5deaa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00e040 con 0x55e09776bf80 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.404+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e09767c330 con 0x55e09776bf80 2024-03-30T04:01:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.405+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5dfc014ae0 con 0x55e09776bf80 2024-03-30T04:01:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.408+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dfc014ca0 con 0x55e09776bf80 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.408+0000 7f5e1f7b8b80 10 client.25174 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e0974e9b10 con 0x55e09776bf80 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e09767c330 con 0x55e09776bf80 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5dfc01c020 con 0x55e09776bf80 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 10 client.25174 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 10 client.25174 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 20 client.25174 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e09777f550 0x55e097784830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 4 client.25174 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:05.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.409+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0976b46b0 con 0x55e09777f550 2024-03-30T04:01:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.410+0000 7f5e09200700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e09777f550 0x55e097784830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.410+0000 7f5e09200700 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e09777f550 0x55e097784830 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5b7ec2c9d05310d7 server_cookie=369a9c4c66ad0eec in_seq=0 out_seq=0 2024-03-30T04:01:05.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.410+0000 7f5deaa00700 10 client.25174 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:05.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.411+0000 7f5deaa00700 1 -- 172.21.15.23:0/3865121444 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e0976b46b0 con 0x55e09777f550 2024-03-30T04:01:05.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.411+0000 7f5deaa00700 10 client.25174 handle_command_reply: tid=0 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.411+0000 7f5e1f7b8b80 1 client.25174 shutdown 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 msgr2=0x55e09777b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc012e20 tx=0x7f5dfc014600 comp rx=0 tx=0).stop 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 shutdown_connections 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e09777f550 0x55e097784830 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776bf80 0x55e09777b880 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e097774300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5deaa00700 0 client.25174 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/3865121444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776b0e0 0x55e097774840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 >> 172.21.15.23:0/3865121444 conn(0x55e0976c9650 msgr2=0x55e097769120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 shutdown_connections 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/3865121444 wait complete. 2024-03-30T04:01:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.412+0000 7f5e1f7b8b80 20 client.25174 trim_cache size 0 max 16384 2024-03-30T04:01:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.430+0000 7f5e1f7b8b80 1 Processor -- start 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.430+0000 7f5e1f7b8b80 1 -- start start 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e09776ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776b0e0 0x55e09777b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976b46e0 con 0x55e09776bf80 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e09767c360 con 0x55e09776a780 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0976b4990 con 0x55e09776b0e0 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e09776ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58482/0 (socket says 172.21.15.23:58482) 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 -- 172.21.15.23:0/1610046038 learned_addr learned my addr 172.21.15.23:0/1610046038 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e03e00700 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776b0e0 0x55e09777b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 -- 172.21.15.23:0/1610046038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776b0e0 msgr2=0x55e09777b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776b0e0 0x55e09777b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 -- 172.21.15.23:0/1610046038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 msgr2=0x55e09776ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e09776ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09c00700 1 -- 172.21.15.23:0/1610046038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e097681e50 con 0x55e09776bf80 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.431+0000 7f5e09200700 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e09776ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.432+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00b010 tx=0x7f5dfc009e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7fa2ba032e0be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.432+0000 7f5df6a00700 1 -- 172.21.15.23:0/1610046038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00e040 con 0x55e09776bf80 2024-03-30T04:01:05.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.432+0000 7f5df6a00700 1 -- 172.21.15.23:0/1610046038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5dfc00f040 con 0x55e09776bf80 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.432+0000 7f5df6a00700 1 -- 172.21.15.23:0/1610046038 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc004000 con 0x55e09776bf80 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1610046038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 msgr2=0x55e09776c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00b010 tx=0x7f5dfc009e50 comp rx=0 tx=0).stop 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1610046038 shutdown_connections 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e09776c360 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776a780 0x55e09776ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1610046038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776b0e0 0x55e09777b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1610046038 >> 172.21.15.23:0/1610046038 conn(0x55e0976c9650 msgr2=0x55e097772530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1610046038 shutdown_connections 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1610046038 wait complete. 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.433+0000 7f5e1f7b8b80 1 -- start start 2024-03-30T04:01:05.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 Processor -- start 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e097778bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e0977d3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e09777ba50 con 0x55e09776bf80 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e09777bbc0 con 0x55e09776b0e0 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0977d4360 con 0x55e09776a780 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e1f7b8b80 10 client.0 _ll_register_callbacks cb 0x55e097661420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e09c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e0977d3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59418/0 (socket says 172.21.15.23:59418) 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.434+0000 7f5e09200700 1 -- 172.21.15.23:0/1516411128 learned_addr learned my addr 172.21.15.23:0/1516411128 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e03e00700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e097778bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 -- 172.21.15.23:0/1516411128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 msgr2=0x55e097778680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 -- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 msgr2=0x55e097778bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e097778bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 -- 172.21.15.23:0/1516411128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df8017070 con 0x55e09776bf80 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09200700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e09c00700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e0977d3e20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009ec0 tx=0x7f5dfc003a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1e9d5acf5f1bca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5deaa00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00e040 con 0x55e09776bf80 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5dfc014ce0 con 0x55e09776bf80 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.435+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5df00a2e40 con 0x55e09776bf80 2024-03-30T04:01:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dfc009420 con 0x55e09776bf80 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 10 client.15180 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e0974e9b10 con 0x55e09776bf80 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5df00a2e40 con 0x55e09776bf80 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5dfc032070 con 0x55e09776bf80 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 10 client.15180 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 10 client.15180 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 20 client.15180 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e0977d8b70 0x55e0977d55a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 4 client.15180 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.436+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e09750aca0 con 0x55e0977d8b70 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.437+0000 7f5e09200700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e0977d8b70 0x55e0977d55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.437+0000 7f5deaa00700 10 client.15180 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.437+0000 7f5e09200700 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e0977d8b70 0x55e0977d55a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=592b5cb7f6fc5740 server_cookie=9e2b9e578aa53b3a in_seq=0 out_seq=0 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5deaa00700 1 -- 172.21.15.23:0/1516411128 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e09750aca0 con 0x55e0977d8b70 2024-03-30T04:01:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5deaa00700 10 client.15180 handle_command_reply: tid=0 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 client.15180 shutdown 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 msgr2=0x55e0977d3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e0977d3e20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc009ec0 tx=0x7f5dfc003a00 comp rx=0 tx=0).stop 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 shutdown_connections 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55e0977d8b70 0x55e0977d55a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e09776bf80 0x55e0977d3e20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e09776a780 0x55e097778680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 --2- 172.21.15.23:0/1516411128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e09776b0e0 0x55e097778bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5deaa00700 0 client.15180 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 >> 172.21.15.23:0/1516411128 conn(0x55e0976c9650 msgr2=0x55e097773810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 shutdown_connections 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 1 -- 172.21.15.23:0/1516411128 wait complete. 2024-03-30T04:01:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.438+0000 7f5e1f7b8b80 20 client.15180 trim_cache size 0 max 16384 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24509, 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 54.503696683000001, 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 86, 2024-03-30T04:01:05.442 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:05.443 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 105.995011729 2024-03-30T04:01:05.443 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:05.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 msgr2=0x7f5dec0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 0x7f5dec0a36f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5e04101dd0 tx=0x7f5df8009260 comp rx=0 tx=0).stop 2024-03-30T04:01:05.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 msgr2=0x7f5e040731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5df0006010 tx=0x7f5df000f690 comp rx=0 tx=0).stop 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 shutdown_connections 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e04102910 0x7f5e040736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5dec0a1230 0x7f5dec0a36f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e04100d80 0x7f5e04074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 --2- 172.21.15.23:0/2862959143 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e04101f70 0x7f5e040731a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 >> 172.21.15.23:0/2862959143 conn(0x7f5e040fc2f0 msgr2=0x7f5e041052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 shutdown_connections 2024-03-30T04:01:05.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.439+0000 7f5e0bc00700 1 -- 172.21.15.23:0/2862959143 wait complete. 2024-03-30T04:01:05.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:05 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/943299583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:05.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-30T04:01:05.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 -- 172.21.15.23:0/1889471132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 msgr2=0x7f88e81091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 --2- 172.21.15.23:0/1889471132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81091c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b0a0 tx=0x7f88e000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 -- 172.21.15.23:0/1889471132 shutdown_connections 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 --2- 172.21.15.23:0/1889471132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81091c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 --2- 172.21.15.23:0/1889471132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8100d80 0x7f88e81011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 --2- 172.21.15.23:0/1889471132 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8101f70 0x7f88e81023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 -- 172.21.15.23:0/1889471132 >> 172.21.15.23:0/1889471132 conn(0x7f88e80fc2f0 msgr2=0x7f88e80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 -- 172.21.15.23:0/1889471132 shutdown_connections 2024-03-30T04:01:05.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.657+0000 7f88ee600700 1 -- 172.21.15.23:0/1889471132 wait complete. 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.658+0000 7f88ee600700 1 Processor -- start 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.658+0000 7f88ee600700 1 -- start start 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8101f70 0x7f88e8199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e81a0b60 con 0x7f88e8100d80 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e81a0ca0 con 0x7f88e8101f70 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88ee600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e81a0e30 con 0x7f88e8102910 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58506/0 (socket says 172.21.15.23:58506) 2024-03-30T04:01:05.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7400700 1 -- 172.21.15.23:0/4116771116 learned_addr learned my addr 172.21.15.23:0/4116771116 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e6a00700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8101f70 0x7f88e8199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7e00700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7400700 1 -- 172.21.15.23:0/4116771116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 msgr2=0x7f88e81a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.659+0000 7f88e7400700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88e7400700 1 -- 172.21.15.23:0/4116771116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8101f70 msgr2=0x7f88e8199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88e7400700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8101f70 0x7f88e8199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88e7400700 1 -- 172.21.15.23:0/4116771116 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e0009d00 con 0x7f88e8100d80 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88e7400700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e970 tx=0x7f88d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88dfe00700 1 -- 172.21.15.23:0/4116771116 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8009790 con 0x7f88e8100d80 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88dfe00700 1 -- 172.21.15.23:0/4116771116 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88d800c7e0 con 0x7f88e8100d80 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e81a1110 con 0x7f88e8100d80 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88dfe00700 1 -- 172.21.15.23:0/4116771116 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8003c70 con 0x7f88e8100d80 2024-03-30T04:01:05.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.660+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e81a1620 con 0x7f88e8100d80 2024-03-30T04:01:05.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.663+0000 7f88dfe00700 1 -- 172.21.15.23:0/4116771116 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f88d800cc50 con 0x7f88e8100d80 2024-03-30T04:01:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.664+0000 7f88dfe00700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 0x7f88cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 Processor -- start 2024-03-30T04:01:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f88dfe00700 1 -- 172.21.15.23:0/4116771116 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88d8014070 con 0x7f88e8100d80 2024-03-30T04:01:05.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f88e6a00700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 0x7f88cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 -- start start 2024-03-30T04:01:05.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec50e0 0x55c16dec59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16dec6400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16de1c940 con 0x55c16dec5f80 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16de1a300 con 0x55c16dec50e0 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.665+0000 7f8902096b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16de15bf0 con 0x55c16dec4780 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e6a00700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 0x7f88cc0a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88d4009c60 tx=0x7f88d4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16dec6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59450/0 (socket says 172.21.15.23:59450) 2024-03-30T04:01:05.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7400700 1 -- 172.21.15.23:0/901662012 learned_addr learned my addr 172.21.15.23:0/901662012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e6a00700 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec50e0 0x55c16dec59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 -- 172.21.15.23:0/901662012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 msgr2=0x55c16dec4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 -- 172.21.15.23:0/901662012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec50e0 msgr2=0x55c16dec59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec50e0 0x55c16dec59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 -- 172.21.15.23:0/901662012 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c16dc97360 con 0x55c16dec5f80 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7400700 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.666+0000 7f88e7e00700 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16dec6400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f88e0007a90 tx=0x7f88e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a134f3aab20d6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.667+0000 7f88dcc00700 1 -- 172.21.15.23:0/901662012 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000e040 con 0x55c16dec5f80 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.667+0000 7f88dcc00700 1 -- 172.21.15.23:0/901662012 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e000bab0 con 0x55c16dec5f80 2024-03-30T04:01:05.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.667+0000 7f88dcc00700 1 -- 172.21.15.23:0/901662012 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e001c440 con 0x55c16dec5f80 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.667+0000 7f8902096b80 1 -- 172.21.15.23:0/901662012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 msgr2=0x55c16dec6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.667+0000 7f8902096b80 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16dec6400 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f88e0007a90 tx=0x7f88e0009c00 comp rx=0 tx=0).stop 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 -- 172.21.15.23:0/901662012 shutdown_connections 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16dec6400 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec4780 0x55c16dec4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 --2- 172.21.15.23:0/901662012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec50e0 0x55c16dec59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 -- 172.21.15.23:0/901662012 >> 172.21.15.23:0/901662012 conn(0x55c16de23650 msgr2=0x55c16debf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 -- 172.21.15.23:0/901662012 shutdown_connections 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 -- 172.21.15.23:0/901662012 wait complete. 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 -- start start 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.668+0000 7f8902096b80 1 Processor -- start 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16deceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec5f80 0x55c16ded5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16ddb1d30 con 0x55c16dec4780 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16ddb19d0 con 0x55c16dec5f80 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16dc99650 con 0x55c16dec50e0 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f8902096b80 10 client.0 _ll_register_callbacks cb 0x55c16ddbb420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f88e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec5f80 0x55c16ded5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f88e7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58520/0 (socket says 172.21.15.23:58520) 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f88e7400700 1 -- 172.21.15.23:0/2861901208 learned_addr learned my addr 172.21.15.23:0/2861901208 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.669+0000 7f88e6a00700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16deceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 -- 172.21.15.23:0/2861901208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 msgr2=0x55c16deceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16deceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 -- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec5f80 msgr2=0x55c16ded5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec5f80 0x55c16ded5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 -- 172.21.15.23:0/2861901208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e81a1620 con 0x55c16dec4780 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88e7400700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f88d800efd0 tx=0x7f88d80999a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c32a612b1a2786 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88d1600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d809a3a0 con 0x55c16dec4780 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88d8099f40 con 0x55c16dec4780 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c16ddd6330 con 0x55c16dec4780 2024-03-30T04:01:05.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88d809dd70 con 0x55c16dec4780 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f8902096b80 10 client.15192 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.670+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f88e8109b10 con 0x55c16dec4780 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88d807c080 con 0x55c16dec4780 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f88d80a1f30 con 0x55c16dec4780 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 10 client.15192 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 10 client.15192 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 20 client.15192 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16ded9770 0x55c16dedea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 4 client.15192 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c16de0e6b0 con 0x55c16ded9770 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.671+0000 7f88e6a00700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16ded9770 0x55c16dedea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.672+0000 7f88e6a00700 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16ded9770 0x55c16dedea70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fa6b0fb12c15453 server_cookie=4db2c433701467af in_seq=0 out_seq=0 2024-03-30T04:01:05.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.672+0000 7f88d1600700 10 client.15192 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f88d1600700 1 -- 172.21.15.23:0/2861901208 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c16de0e6b0 con 0x55c16ded9770 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f88d1600700 10 client.15192 handle_command_reply: tid=0 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 client.15192 shutdown 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 msgr2=0x55c16dece5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f88d800efd0 tx=0x7f88d80999a0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 shutdown_connections 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec5f80 0x55c16ded5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec4780 0x55c16dece5a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16ded9770 0x55c16dedea70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 --2- 172.21.15.23:0/2861901208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16deceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 >> 172.21.15.23:0/2861901208 conn(0x55c16de23650 msgr2=0x55c16dec3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f88d1600700 0 client.15192 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 shutdown_connections 2024-03-30T04:01:05.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.673+0000 7f8902096b80 1 -- 172.21.15.23:0/2861901208 wait complete. 2024-03-30T04:01:05.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.674+0000 7f8902096b80 20 client.15192 trim_cache size 0 max 16384 2024-03-30T04:01:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:05 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/943299583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 Processor -- start 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 -- start start 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16debeec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16ded5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16ddb4de0 con 0x55c16dec5f80 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16dec0460 con 0x55c16dec4780 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f8902096b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16ded78b0 con 0x55c16dec50e0 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58524/0 (socket says 172.21.15.23:58524) 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e7e00700 1 -- 172.21.15.23:0/2730546190 learned_addr learned my addr 172.21.15.23:0/2730546190 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e7400700 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16debeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e6a00700 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16ded5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.693+0000 7f88e7e00700 1 -- 172.21.15.23:0/2730546190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 msgr2=0x55c16ded5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88e7e00700 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16ded5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88e7e00700 1 -- 172.21.15.23:0/2730546190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 msgr2=0x55c16debeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88e7e00700 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16debeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88e7e00700 1 -- 172.21.15.23:0/2730546190 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c16dc43b10 con 0x55c16dec5f80 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88e7e00700 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f88e0007a90 tx=0x7f88e000b010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93f0507dbcaa21b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88dd600700 1 -- 172.21.15.23:0/2730546190 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e000e040 con 0x55c16dec5f80 2024-03-30T04:01:05.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88dd600700 1 -- 172.21.15.23:0/2730546190 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e00094a0 con 0x55c16dec5f80 2024-03-30T04:01:05.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.694+0000 7f88dd600700 1 -- 172.21.15.23:0/2730546190 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0005920 con 0x55c16dec5f80 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- 172.21.15.23:0/2730546190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 msgr2=0x55c16deb8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f88e0007a90 tx=0x7f88e000b010 comp rx=0 tx=0).stop 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- 172.21.15.23:0/2730546190 shutdown_connections 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16deb8550 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16debeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 --2- 172.21.15.23:0/2730546190 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16ded5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- 172.21.15.23:0/2730546190 >> 172.21.15.23:0/2730546190 conn(0x55c16de23650 msgr2=0x55c16decd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- 172.21.15.23:0/2730546190 shutdown_connections 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- 172.21.15.23:0/2730546190 wait complete. 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 -- start start 2024-03-30T04:01:05.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.695+0000 7f8902096b80 1 Processor -- start 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16decb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16decb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16decbac0 con 0x55c16dec5f80 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16decbc30 con 0x55c16dec4780 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c16df2e5a0 con 0x55c16dec50e0 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f8902096b80 10 client.0 _ll_register_callbacks cb 0x55c16ddbb420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58540/0 (socket says 172.21.15.23:58540) 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 -- 172.21.15.23:0/885159509 learned_addr learned my addr 172.21.15.23:0/885159509 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7400700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16decb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e6a00700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16decb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 -- 172.21.15.23:0/885159509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 msgr2=0x55c16decb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16decb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 -- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 msgr2=0x55c16decb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16decb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.696+0000 7f88e7e00700 1 -- 172.21.15.23:0/885159509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c16dc97360 con 0x55c16dec5f80 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88e7e00700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b3f0 tx=0x7f88e0005430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25baa9e76ce73d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88d1600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e00058a0 con 0x55c16dec5f80 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f88e0005a00 con 0x55c16dec5f80 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88e000bab0 con 0x55c16dec5f80 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88e002d450 con 0x55c16dec5f80 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f8902096b80 10 client.15198 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55c16dddbe50 con 0x55c16dec5f80 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.697+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88e000bab0 con 0x55c16dec5f80 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f88e0019030 con 0x55c16dec5f80 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 10 client.15198 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 10 client.15198 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 20 client.15198 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16df32d80 0x55c16df2f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 4 client.15198 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c16dc64ca0 con 0x55c16df32d80 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f88e7400700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16df32d80 0x55c16df2f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f88e7400700 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16df32d80 0x55c16df2f790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b48de050cad09ac server_cookie=550550224b8dfae in_seq=0 out_seq=0 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.698+0000 7f88d1600700 10 client.15198 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f88d1600700 1 -- 172.21.15.23:0/885159509 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c16dc64ca0 con 0x55c16df32d80 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f88d1600700 10 client.15198 handle_command_reply: tid=0 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 client.15198 shutdown 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 msgr2=0x55c16df2e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f88e000b3f0 tx=0x7f88e0005430 comp rx=0 tx=0).stop 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 shutdown_connections 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c16dec5f80 0x55c16df2e060 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c16dec4780 0x55c16decb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c16df32d80 0x55c16df2f790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 --2- 172.21.15.23:0/885159509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c16dec50e0 0x55c16decb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 >> 172.21.15.23:0/885159509 conn(0x55c16de23650 msgr2=0x55c16decc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f88d1600700 0 client.15198 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:05.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 shutdown_connections 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 1 -- 172.21.15.23:0/885159509 wait complete. 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.699+0000 7f8902096b80 20 client.15198 trim_cache size 0 max 16384 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24533, 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:05.705 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 55.727977903999999, 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 86, 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 101.065546711 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:05.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 msgr2=0x7f88cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 0x7f88cc0a3670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f88d4009c60 tx=0x7f88d4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 msgr2=0x7f88e8199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e970 tx=0x7f88d800ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 shutdown_connections 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f88cc0a11b0 0x7f88cc0a3670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88e8102910 0x7f88e81a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f88e8100d80 0x7f88e8199030 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 --2- 172.21.15.23:0/4116771116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88e8101f70 0x7f88e8199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 >> 172.21.15.23:0/4116771116 conn(0x7f88e80fc2f0 msgr2=0x7f88e8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 shutdown_connections 2024-03-30T04:01:05.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.700+0000 7f88ee600700 1 -- 172.21.15.23:0/4116771116 wait complete. 2024-03-30T04:01:05.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T04:01:05.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:05 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/943299583' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:05.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.917+0000 7efef6800700 1 -- 172.21.15.23:0/4127096273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0100d80 msgr2=0x7efef01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.917+0000 7efef6800700 1 --2- 172.21.15.23:0/4127096273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0100d80 0x7efef01011a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efee00099c0 tx=0x7efee0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 -- 172.21.15.23:0/4127096273 shutdown_connections 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 --2- 172.21.15.23:0/4127096273 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 0x7efef01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 --2- 172.21.15.23:0/4127096273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0100d80 0x7efef01011a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 --2- 172.21.15.23:0/4127096273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0101f70 0x7efef01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 -- 172.21.15.23:0/4127096273 >> 172.21.15.23:0/4127096273 conn(0x7efef00fc2f0 msgr2=0x7efef00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 -- 172.21.15.23:0/4127096273 shutdown_connections 2024-03-30T04:01:05.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.918+0000 7efef6800700 1 -- 172.21.15.23:0/4127096273 wait complete. 2024-03-30T04:01:05.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 Processor -- start 2024-03-30T04:01:05.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 -- start start 2024-03-30T04:01:05.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0100d80 0x7efef0198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 0x7efef01a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01a0a90 con 0x7efef0102910 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01a0bd0 con 0x7efef0101f70 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.919+0000 7efef6800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef01a0d60 con 0x7efef0100d80 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47360/0 (socket says 172.21.15.23:47360) 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeefe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 0x7efef01a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeef400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0100d80 0x7efef0198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 -- 172.21.15.23:0/1767958535 learned_addr learned my addr 172.21.15.23:0/1767958535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 -- 172.21.15.23:0/1767958535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0100d80 msgr2=0x7efef0198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0100d80 0x7efef0198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 -- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 msgr2=0x7efef01a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 0x7efef01a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 -- 172.21.15.23:0/1767958535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee00095f0 con 0x7efef0101f70 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efeeea00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efedc00e9a0 tx=0x7efedc00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.920+0000 7efee7e00700 1 -- 172.21.15.23:0/1767958535 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc009790 con 0x7efef0101f70 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.921+0000 7efee7e00700 1 -- 172.21.15.23:0/1767958535 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efedc010430 con 0x7efef0101f70 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.921+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef01a0fe0 con 0x7efef0101f70 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.921+0000 7efee7e00700 1 -- 172.21.15.23:0/1767958535 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc018680 con 0x7efef0101f70 2024-03-30T04:01:05.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.921+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef01a1510 con 0x7efef0101f70 2024-03-30T04:01:05.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.925+0000 7efee7e00700 1 -- 172.21.15.23:0/1767958535 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7efedc0105a0 con 0x7efef0101f70 2024-03-30T04:01:05.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 Processor -- start 2024-03-30T04:01:05.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 -- start start 2024-03-30T04:01:05.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aed780 0x55cda5aedba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5aee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5a45940 con 0x55cda5aeef80 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.926+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5a43300 con 0x55cda5aee0e0 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5a3ebf0 con 0x55cda5aed780 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeef400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5aee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58572/0 (socket says 172.21.15.23:58572) 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 -- 172.21.15.23:0/2052917457 learned_addr learned my addr 172.21.15.23:0/2052917457 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeefe00700 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aed780 0x55cda5aedba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efee7e00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 0x7efed40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 -- 172.21.15.23:0/2052917457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aed780 msgr2=0x55cda5aedba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aed780 0x55cda5aedba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 -- 172.21.15.23:0/2052917457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 msgr2=0x55cda5aee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5aee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efee7e00700 1 -- 172.21.15.23:0/1767958535 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efedc014070 con 0x7efef0101f70 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeeea00700 1 -- 172.21.15.23:0/2052917457 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda58c0360 con 0x55cda5aeef80 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeef400700 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5aee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.927+0000 7efeefe00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 0x7efed40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.928+0000 7efeeea00700 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efedc0ca580 tx=0x7efedc010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba14e9c27dd4a55 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.928+0000 7efee4c00700 1 -- 172.21.15.23:0/2052917457 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0999d0 con 0x55cda5aeef80 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.928+0000 7efee4c00700 1 -- 172.21.15.23:0/2052917457 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efedc0997f0 con 0x55cda5aeef80 2024-03-30T04:01:05.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.928+0000 7efee4c00700 1 -- 172.21.15.23:0/2052917457 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc0d02f0 con 0x55cda5aeef80 2024-03-30T04:01:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2052917457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 msgr2=0x55cda5aef400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7efeefe00700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 0x7efed40a36f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efee8009ec0 tx=0x7efee8009040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efedc0ca580 tx=0x7efedc010c80 comp rx=0 tx=0).stop 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2052917457 shutdown_connections 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aeef80 0x55cda5aef400 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aed780 0x55cda5aedba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2052917457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5aee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2052917457 >> 172.21.15.23:0/2052917457 conn(0x55cda5a4c650 msgr2=0x55cda5ae8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2052917457 shutdown_connections 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2052917457 wait complete. 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.929+0000 7eff0a34cb80 1 -- start start 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 Processor -- start 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5af7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5afeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda59dad30 con 0x55cda5aed780 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda59da9d0 con 0x55cda5aee0e0 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda58c2650 con 0x55cda5aeef80 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7eff0a34cb80 10 client.0 _ll_register_callbacks cb 0x55cda59e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7efeef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7efeef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58588/0 (socket says 172.21.15.23:58588) 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7efeef400700 1 -- 172.21.15.23:0/2629883064 learned_addr learned my addr 172.21.15.23:0/2629883064 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7efeeea00700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5af7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.930+0000 7efeefe00700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5afeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 -- 172.21.15.23:0/2629883064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 msgr2=0x55cda5afeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5afeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 -- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 msgr2=0x55cda5af7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5af7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 -- 172.21.15.23:0/2629883064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0109b10 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeefe00700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5afeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efeef400700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efee0006010 tx=0x7efee000bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d343f0f0d5f261dc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efed9600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0003d90 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cda59ff330 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efee0005040 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.931+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efee0005200 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 10 client.15210 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7efef01a1510 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cda59ff330 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efee000fc50 con 0x55cda5aed780 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 10 client.15210 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 10 client.15210 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T04:01:05.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 20 client.15210 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b02730 0x55cda5b07a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 4 client.15210 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cda5a376b0 con 0x55cda5b02730 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.932+0000 7efeeea00700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b02730 0x55cda5b07a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.933+0000 7efeeea00700 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b02730 0x55cda5b07a30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b04e2718749b989e server_cookie=e0ba2cc83ac04ac in_seq=0 out_seq=0 2024-03-30T04:01:05.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.933+0000 7efed9600700 10 client.15210 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.933+0000 7efed9600700 1 -- 172.21.15.23:0/2629883064 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cda5a376b0 con 0x55cda5b02730 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7efed9600700 10 client.15210 handle_command_reply: tid=0 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 client.15210 shutdown 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 msgr2=0x55cda5af7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efee0006010 tx=0x7efee000bee0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 shutdown_connections 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b02730 0x55cda5b07a30 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5afeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5af7570 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/2629883064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5af7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7efed9600700 0 client.15210 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 >> 172.21.15.23:0/2629883064 conn(0x55cda5a4c650 msgr2=0x55cda5aec120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 shutdown_connections 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 1 -- 172.21.15.23:0/2629883064 wait complete. 2024-03-30T04:01:05.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.934+0000 7eff0a34cb80 20 client.15210 trim_cache size 0 max 16384 2024-03-30T04:01:05.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.954+0000 7eff0a34cb80 1 Processor -- start 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.954+0000 7eff0a34cb80 1 -- start start 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5afeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5aef360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda59fcad0 con 0x55cda5aed780 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5a37990 con 0x55cda5aee0e0 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5a3b850 con 0x55cda5aeef80 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58604/0 (socket says 172.21.15.23:58604) 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 -- 172.21.15.23:0/1234591263 learned_addr learned my addr 172.21.15.23:0/1234591263 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeeea00700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5afeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeefe00700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5aef360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 -- 172.21.15.23:0/1234591263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 msgr2=0x55cda5aef360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5aef360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 -- 172.21.15.23:0/1234591263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 msgr2=0x55cda5afeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5afeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeef400700 1 -- 172.21.15.23:0/1234591263 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda586cb10 con 0x55cda5aed780 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.955+0000 7efeefe00700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5aef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.956+0000 7efeef400700 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efee0032980 tx=0x7efee0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244be92e564499d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.956+0000 7efee5600700 1 -- 172.21.15.23:0/1234591263 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0017630 con 0x55cda5aed780 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.956+0000 7efee5600700 1 -- 172.21.15.23:0/1234591263 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee001e040 con 0x55cda5aed780 2024-03-30T04:01:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.956+0000 7efee5600700 1 -- 172.21.15.23:0/1234591263 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0017870 con 0x55cda5aed780 2024-03-30T04:01:05.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1234591263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 msgr2=0x55cda5aedb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7efee0032980 tx=0x7efee0009310 comp rx=0 tx=0).stop 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1234591263 shutdown_connections 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5aef360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aed780 0x55cda5aedb60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1234591263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aee0e0 0x55cda5afeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1234591263 >> 172.21.15.23:0/1234591263 conn(0x55cda5a4c650 msgr2=0x55cda5ae1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1234591263 shutdown_connections 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1234591263 wait complete. 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 -- start start 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.957+0000 7eff0a34cb80 1 Processor -- start 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aed780 0x55cda5af4010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5b57020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5af4ba0 con 0x55cda5aee0e0 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5afed10 con 0x55cda5aed780 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda5afeed0 con 0x55cda5aeef80 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7eff0a34cb80 10 client.0 _ll_register_callbacks cb 0x55cda59e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58612/0 (socket says 172.21.15.23:58612) 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 -- 172.21.15.23:0/1174839685 learned_addr learned my addr 172.21.15.23:0/1174839685 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeef400700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aed780 0x55cda5af4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeefe00700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5b57020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 -- 172.21.15.23:0/1174839685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 msgr2=0x55cda5b57020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5b57020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 -- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aed780 msgr2=0x55cda5af4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aed780 0x55cda5af4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.958+0000 7efeeea00700 1 -- 172.21.15.23:0/1174839685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef01a1510 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efeeea00700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efedc0981e0 tx=0x7efedc010f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea302f7ada3d6ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc09aea0 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efedc09a590 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efedc0997f0 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7efedc099e90 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7eff0a34cb80 10 client.15216 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cda5a04e50 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efedc0997f0 con 0x55cda5aee0e0 2024-03-30T04:01:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.959+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efedc0d0380 con 0x55cda5aee0e0 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 10 client.15216 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 10 client.15216 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 20 client.15216 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b5beb0 0x55cda5b58800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 4 client.15216 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cda588dca0 con 0x55cda5b5beb0 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.960+0000 7efeef400700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b5beb0 0x55cda5b58800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7efeef400700 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b5beb0 0x55cda5b58800 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ecbf4e3e56eadb server_cookie=4669d78c83e747f in_seq=0 out_seq=0 2024-03-30T04:01:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7efed9600700 10 client.15216 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7efed9600700 1 -- 172.21.15.23:0/1174839685 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55cda588dca0 con 0x55cda5b5beb0 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7efed9600700 10 client.15216 handle_command_reply: tid=0 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 client.15216 shutdown 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 msgr2=0x55cda5af4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7efedc0981e0 tx=0x7efedc010f80 comp rx=0 tx=0).stop 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 shutdown_connections 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cda5b5beb0 0x55cda5b58800 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cda5aeef80 0x55cda5b57020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cda5aed780 0x55cda5af4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7efed9600700 0 client.15216 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 --2- 172.21.15.23:0/1174839685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cda5aee0e0 0x55cda5af4550 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 >> 172.21.15.23:0/1174839685 conn(0x55cda5a4c650 msgr2=0x55cda5af6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 shutdown_connections 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 1 -- 172.21.15.23:0/1174839685 wait complete. 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.961+0000 7eff0a34cb80 20 client.15216 trim_cache size 0 max 16384 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24569, 2024-03-30T04:01:05.967 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 85.526372434999999, 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 86, 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 92.646209597999999 2024-03-30T04:01:05.968 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 msgr2=0x7efed40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 0x7efed40a36f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efee8009ec0 tx=0x7efee8009040 comp rx=0 tx=0).stop 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 msgr2=0x7efef01994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efedc00e9a0 tx=0x7efedc00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 shutdown_connections 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efef0102910 0x7efef01a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7efed40a1230 0x7efed40a36f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efef0100d80 0x7efef0198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 --2- 172.21.15.23:0/1767958535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efef0101f70 0x7efef01994f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:05.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 >> 172.21.15.23:0/1767958535 conn(0x7efef00fc2f0 msgr2=0x7efef0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:05.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 shutdown_connections 2024-03-30T04:01:05.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:05.962+0000 7efef6800700 1 -- 172.21.15.23:0/1767958535 wait complete. 2024-03-30T04:01:05.981 INFO:tasks.cephadm:Setup complete, yielding 2024-03-30T04:01:05.981 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:01:05.993 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:01:05.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-03-30T04:01:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:07.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:06 smithi193 ceph-mon[79581]: pgmap v281: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:07.253 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:06 smithi023 ceph-mon[38254]: pgmap v281: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:06 smithi154 ceph-mon[79161]: pgmap v281: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 -- 172.21.15.23:0/3856658253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 msgr2=0x7fe6a0101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 --2- 172.21.15.23:0/3856658253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0101220 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe690009990 tx=0x7fe690009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 -- 172.21.15.23:0/3856658253 shutdown_connections 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 --2- 172.21.15.23:0/3856658253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 0x7fe6a0109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 --2- 172.21.15.23:0/3856658253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0101220 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 --2- 172.21.15.23:0/3856658253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 -- 172.21.15.23:0/3856658253 >> 172.21.15.23:0/3856658253 conn(0x7fe6a00fc3c0 msgr2=0x7fe6a00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 -- 172.21.15.23:0/3856658253 shutdown_connections 2024-03-30T04:01:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.488+0000 7fe6a7200700 1 -- 172.21.15.23:0/3856658253 wait complete. 2024-03-30T04:01:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.489+0000 7fe6a7200700 1 Processor -- start 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.489+0000 7fe6a7200700 1 -- start start 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a0107070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 0x7fe6a01036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a0103e50 con 0x7fe6a0100e00 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a0103fc0 con 0x7fe6a0102920 2024-03-30T04:01:07.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a7200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a0104180 con 0x7fe6a0101f60 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58638/0 (socket says 172.21.15.23:58638) 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 -- 172.21.15.23:0/640263127 learned_addr learned my addr 172.21.15.23:0/640263127 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69f400700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a0107070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe6a5200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 0x7fe6a01036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 -- 172.21.15.23:0/640263127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 msgr2=0x7fe6a0107070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:07.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a0107070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 -- 172.21.15.23:0/640263127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 msgr2=0x7fe6a01036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 0x7fe6a01036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.490+0000 7fe69fe00700 1 -- 172.21.15.23:0/640263127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6900095f0 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe69f400700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a0107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe69fe00700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe69000b620 tx=0x7fe690009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69001d070 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe690022470 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe690017650 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a01a5470 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.491+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a01a5880 con 0x7fe6a0100e00 2024-03-30T04:01:07.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.492+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a004ea80 con 0x7fe6a0100e00 2024-03-30T04:01:07.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.494+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe690022870 con 0x7fe6a0100e00 2024-03-30T04:01:07.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.496+0000 7fe69cc00700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 0x7fe6840a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:07.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.496+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe6900ce8e0 con 0x7fe6a0100e00 2024-03-30T04:01:07.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.496+0000 7fe69f400700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 0x7fe6840a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:07.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.496+0000 7fe69f400700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 0x7fe6840a36f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009cb0 tx=0x7fe68c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:07.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.498+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe690097c30 con 0x7fe6a0100e00 2024-03-30T04:01:07.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.731+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6a0074700 con 0x7fe6840a1230 2024-03-30T04:01:07.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.732+0000 7fe69cc00700 1 -- 172.21.15.23:0/640263127 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe6a0074700 con 0x7fe6840a1230 2024-03-30T04:01:07.732 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-30T04:01:07.732 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-30T04:01:07.732 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-30T04:01:07.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 msgr2=0x7fe6840a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:07.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 0x7fe6840a36f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe68c009cb0 tx=0x7fe68c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:07.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 msgr2=0x7fe6a0106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:07.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe69000b620 tx=0x7fe690009510 comp rx=0 tx=0).stop 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 shutdown_connections 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6a0102920 0x7fe6a01036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.734+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a0100e00 0x7fe6a0106b30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.735+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6840a1230 0x7fe6840a36f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.735+0000 7fe6a7200700 1 --2- 172.21.15.23:0/640263127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6a0101f60 0x7fe6a0107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.735+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 >> 172.21.15.23:0/640263127 conn(0x7fe6a00fc3c0 msgr2=0x7fe6a0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.735+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 shutdown_connections 2024-03-30T04:01:07.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:07.735+0000 7fe6a7200700 1 -- 172.21.15.23:0/640263127 wait complete. 2024-03-30T04:01:08.131 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-03-30T04:01:08.506 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:09 smithi154 ceph-mon[79161]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:09 smithi154 ceph-mon[79161]: pgmap v282: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:09.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 -- 172.21.15.23:0/2165893260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784101f60 msgr2=0x7f97841023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 --2- 172.21.15.23:0/2165893260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784101f60 0x7f97841023e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9770009990 tx=0x7f9770009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 -- 172.21.15.23:0/2165893260 shutdown_connections 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 --2- 172.21.15.23:0/2165893260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784102920 0x7f9784109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 --2- 172.21.15.23:0/2165893260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 --2- 172.21.15.23:0/2165893260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784101f60 0x7f97841023e0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 -- 172.21.15.23:0/2165893260 >> 172.21.15.23:0/2165893260 conn(0x7f97840fc3c0 msgr2=0x7f97840fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 -- 172.21.15.23:0/2165893260 shutdown_connections 2024-03-30T04:01:09.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.737+0000 7f978ae00700 1 -- 172.21.15.23:0/2165893260 wait complete. 2024-03-30T04:01:09.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.738+0000 7f978ae00700 1 Processor -- start 2024-03-30T04:01:09.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.738+0000 7f978ae00700 1 -- start start 2024-03-30T04:01:09.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784101f60 0x7f97841994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784102920 0x7f97841a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97841a0aa0 con 0x7f9784102920 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97841a0be0 con 0x7f9784100e00 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f978ae00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97841a0d70 con 0x7f9784101f60 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9788e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784102920 0x7f97841a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52468/0 (socket says 172.21.15.23:52468) 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 -- 172.21.15.23:0/4260342047 learned_addr learned my addr 172.21.15.23:0/4260342047 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783400700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784101f60 0x7f97841994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 -- 172.21.15.23:0/4260342047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784101f60 msgr2=0x7f97841994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784101f60 0x7f97841994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 -- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784102920 msgr2=0x7f97841a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784102920 0x7f97841a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.739+0000 7f9783e00700 1 -- 172.21.15.23:0/4260342047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97700095f0 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f9783e00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f977400e9a0 tx=0x7f977400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774009790 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9774010430 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97841a0ff0 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774018680 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.740+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97841a1460 con 0x7f9784100e00 2024-03-30T04:01:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.741+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978404ea80 con 0x7f9784100e00 2024-03-30T04:01:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.743+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f97740105a0 con 0x7f9784100e00 2024-03-30T04:01:09.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:09 smithi193 ceph-mon[79581]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:09.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:09 smithi193 ceph-mon[79581]: pgmap v282: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:09.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.744+0000 7f9780c00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 0x7f97680a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:09.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.744+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9774014070 con 0x7f9784100e00 2024-03-30T04:01:09.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.745+0000 7f9783400700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 0x7f97680a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:09.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.745+0000 7f9783400700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 0x7f97680a3670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f977000b2e0 tx=0x7f977000b6f0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:09.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.747+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f97740961d0 con 0x7f9784100e00 2024-03-30T04:01:09.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:09 smithi023 ceph-mon[38254]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:09.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:09 smithi023 ceph-mon[38254]: pgmap v282: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:09.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.969+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9784074700 con 0x7f97680a11b0 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.983+0000 7f9780c00700 1 -- 172.21.15.23:0/4260342047 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f9784074700 con 0x7f97680a11b0 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (98s) 19s ago 98s 16.1M - 18.2.2-1018-g470100a4 e7c444a08cf7 99643adfed19 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi154 running (96s) 30s ago 96s 30.4M - 18.2.2-1018-g470100a4 e7c444a08cf7 f303fea401f7 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi193 running (107s) 30s ago 107s 19.6M - 18.2.2-1018-g470100a4 e7c444a08cf7 a0df9def68eb 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (112s) 19s ago 112s 13.8M - 18.2.2-1018-g470100a4 e7c444a08cf7 207612e9ad0d 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi154 running (110s) 30s ago 110s 18.6M - 18.2.2-1018-g470100a4 e7c444a08cf7 6551cb6a691f 2024-03-30T04:01:09.983 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi193 running (109s) 30s ago 108s 20.5M - 18.2.2-1018-g470100a4 e7c444a08cf7 d97e8fe86f09 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (101s) 19s ago 100s 16.1M - 18.2.2-1018-g470100a4 e7c444a08cf7 6a98f76e7b4d 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi154 running (105s) 30s ago 105s 20.5M - 18.2.2-1018-g470100a4 e7c444a08cf7 bf0cc1692e78 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi193 running (103s) 30s ago 103s 17.6M - 18.2.2-1018-g470100a4 e7c444a08cf7 7de756b6354f 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (114s) 19s ago 114s 13.8M - 18.2.2-1018-g470100a4 e7c444a08cf7 db07187900b0 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi154 running (117s) 30s ago 117s 18.3M - 18.2.2-1018-g470100a4 e7c444a08cf7 2c26fe29f57e 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi193 running (119s) 30s ago 118s 17.0M - 18.2.2-1018-g470100a4 e7c444a08cf7 d316f730517c 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (7m) 19s ago 7m 536M - 18.2.2-1018-g470100a4 e7c444a08cf7 5412540cb7c9 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi154 *:8443,8765 running (5m) 30s ago 5m 459M - 18.2.2-1018-g470100a4 e7c444a08cf7 584510d83dca 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi193 *:8443,8765 running (6m) 30s ago 6m 457M - 18.2.2-1018-g470100a4 e7c444a08cf7 c639521c9107 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (7m) 19s ago 7m 89.2M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 8d24ba58831e 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi154 running (6m) 30s ago 6m 60.7M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 de6155bac50b 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi193 running (6m) 30s ago 6m 61.8M 2048M 18.2.2-1018-g470100a4 e7c444a08cf7 1654bc87aa63 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (5m) 19s ago 5m 70.1M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 92cdf1465b99 2024-03-30T04:01:09.984 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi154 running (5m) 30s ago 5m 152M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 5c30d119dfbd 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi193 running (5m) 30s ago 5m 75.5M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 92325cbd66b8 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (4m) 19s ago 4m 64.7M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 136149aaa3cc 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi154 running (4m) 30s ago 4m 139M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 d213bd442d2b 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi193 running (4m) 30s ago 4m 143M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 3ccc649e8918 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (3m) 19s ago 3m 59.6M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 6fa083c3e64d 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi154 running (3m) 30s ago 3m 65.8M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 f8f6292a89d1 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi193 running (3m) 30s ago 3m 137M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 9fc682e9089c 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (2m) 19s ago 2m 133M 4317M 18.2.2-1018-g470100a4 e7c444a08cf7 1063d6a87aa7 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi154 running (2m) 30s ago 2m 137M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 a744d213cece 2024-03-30T04:01:09.985 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi193 running (2m) 30s ago 2m 130M 4278M 18.2.2-1018-g470100a4 e7c444a08cf7 0acdb557180b 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 msgr2=0x7f97680a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 0x7f97680a3670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f977000b2e0 tx=0x7f977000b6f0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 msgr2=0x7f9784198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f977400e9a0 tx=0x7f977400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 shutdown_connections 2024-03-30T04:01:09.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9784102920 0x7f97841a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9784100e00 0x7f9784198f60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f97680a11b0 0x7f97680a3670 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 --2- 172.21.15.23:0/4260342047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9784101f60 0x7f97841994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.985+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 >> 172.21.15.23:0/4260342047 conn(0x7f97840fc3c0 msgr2=0x7f9784072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.986+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 shutdown_connections 2024-03-30T04:01:09.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:09.986+0000 7f978ae00700 1 -- 172.21.15.23:0/4260342047 wait complete. 2024-03-30T04:01:10.422 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-03-30T04:01:10.790 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:11.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:11 smithi023 ceph-mon[38254]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:11.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:11 smithi023 ceph-mon[38254]: pgmap v283: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:11.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:11 smithi154 ceph-mon[79161]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:11.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:11 smithi154 ceph-mon[79161]: pgmap v283: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:11.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:11 smithi193 ceph-mon[79581]: from='client.25238 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:11.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:11 smithi193 ceph-mon[79581]: pgmap v283: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 -- 172.21.15.23:0/2519960363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4102920 msgr2=0x7f15f4109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 --2- 172.21.15.23:0/2519960363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4102920 0x7f15f4109240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f15ec009f90 tx=0x7f15ec00b330 comp rx=0 tx=0).stop 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 -- 172.21.15.23:0/2519960363 shutdown_connections 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 --2- 172.21.15.23:0/2519960363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4102920 0x7f15f4109240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 --2- 172.21.15.23:0/2519960363 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4100e00 0x7f15f4101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:11.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 --2- 172.21.15.23:0/2519960363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4101f60 0x7f15f41023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 -- 172.21.15.23:0/2519960363 >> 172.21.15.23:0/2519960363 conn(0x7f15f40fc3c0 msgr2=0x7f15f40fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 -- 172.21.15.23:0/2519960363 shutdown_connections 2024-03-30T04:01:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.937+0000 7f15fb200700 1 -- 172.21.15.23:0/2519960363 wait complete. 2024-03-30T04:01:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 Processor -- start 2024-03-30T04:01:11.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 -- start start 2024-03-30T04:01:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 0x7f15f4074b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 0x7f15f40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:11.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4073e80 con 0x7f15f4101f60 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4073ff0 con 0x7f15f4100e00 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.939+0000 7f15fb200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f41a0e50 con 0x7f15f4102920 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 0x7f15f4074b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37600/0 (socket says 172.21.15.23:37600) 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3400700 1 -- 172.21.15.23:0/1485348606 learned_addr learned my addr 172.21.15.23:0/1485348606 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:11.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f9200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 0x7f15f40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 -- 172.21.15.23:0/1485348606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 msgr2=0x7f15f40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 0x7f15f40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:11.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 -- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 msgr2=0x7f15f40731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 -- 172.21.15.23:0/1485348606 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec009c40 con 0x7f15f4100e00 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3400700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f9200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 0x7f15f40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.940+0000 7f15f3e00700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 0x7f15f4074b00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9c0 tx=0x7f15e400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.941+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e400c9c0 con 0x7f15f4100e00 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.941+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f15e4010430 con 0x7f15f4100e00 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.941+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f41a10e0 con 0x7f15f4100e00 2024-03-30T04:01:11.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.941+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4018680 con 0x7f15f4100e00 2024-03-30T04:01:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.941+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f41a15e0 con 0x7f15f4100e00 2024-03-30T04:01:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.942+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f4108280 con 0x7f15f4100e00 2024-03-30T04:01:11.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.944+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f15e4010860 con 0x7f15f4100e00 2024-03-30T04:01:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.945+0000 7f15f0c00700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 0x7f15d80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.945+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15e4014070 con 0x7f15f4100e00 2024-03-30T04:01:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.945+0000 7f15f3400700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 0x7f15d80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:11.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.946+0000 7f15f3400700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 0x7f15d80a3670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f15e0011550 tx=0x7f15e0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:11.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:11.947+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f15e40962d0 con 0x7f15f4100e00 2024-03-30T04:01:12.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.170+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f15f40611f0 con 0x7f15d80a11b0 2024-03-30T04:01:12.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.176+0000 7f15f0c00700 1 -- 172.21.15.23:0/1485348606 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f15f40611f0 con 0x7f15d80a11b0 2024-03-30T04:01:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-30T04:01:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 33s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi154=b;smithi154=e;smithi154=h;smithi154=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-30T04:01:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 33s ago 6m smithi023=x;smithi154=y;smithi193=z;count:3 2024-03-30T04:01:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 33s ago 6m smithi023:172.21.15.23=a;smithi154:172.21.15.154=b;smithi193:172.21.15.193=c;count:3 2024-03-30T04:01:12.176 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 33s ago - 2024-03-30T04:01:12.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.178+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 msgr2=0x7f15d80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.178+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 0x7f15d80a3670 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f15e0011550 tx=0x7f15e0009260 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.178+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 msgr2=0x7f15f4074b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.178+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 0x7f15f4074b00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f15e400e9c0 tx=0x7f15e400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 shutdown_connections 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f15d80a11b0 0x7f15d80a3670 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15f4102920 0x7f15f40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f15f4100e00 0x7f15f4074b00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 --2- 172.21.15.23:0/1485348606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f15f4101f60 0x7f15f40731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 >> 172.21.15.23:0/1485348606 conn(0x7f15f40fc3c0 msgr2=0x7f15f41053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 shutdown_connections 2024-03-30T04:01:12.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:12.179+0000 7f15fb200700 1 -- 172.21.15.23:0/1485348606 wait complete. 2024-03-30T04:01:12.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-03-30T04:01:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:13 smithi154 ceph-mon[79161]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:13 smithi154 ceph-mon[79161]: pgmap v284: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:13.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:13 smithi193 ceph-mon[79581]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:13.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:13 smithi193 ceph-mon[79581]: pgmap v284: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:13.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:13 smithi023 ceph-mon[38254]: from='client.25244 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:13.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:13 smithi023 ceph-mon[38254]: pgmap v284: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 -- 172.21.15.23:0/2769579501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0102920 msgr2=0x7fc0a0109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 --2- 172.21.15.23:0/2769579501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0102920 0x7fc0a0109240 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc09800b0a0 tx=0x7fc09800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 -- 172.21.15.23:0/2769579501 shutdown_connections 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 --2- 172.21.15.23:0/2769579501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0102920 0x7fc0a0109240 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 --2- 172.21.15.23:0/2769579501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0100e00 0x7fc0a0101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 --2- 172.21.15.23:0/2769579501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 0x7fc0a01023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 -- 172.21.15.23:0/2769579501 >> 172.21.15.23:0/2769579501 conn(0x7fc0a00fc3c0 msgr2=0x7fc0a00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 -- 172.21.15.23:0/2769579501 shutdown_connections 2024-03-30T04:01:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.140+0000 7fc0a7400700 1 -- 172.21.15.23:0/2769579501 wait complete. 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.141+0000 7fc0a7400700 1 Processor -- start 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.141+0000 7fc0a7400700 1 -- start start 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 0x7fc0a00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0102920 0x7fc0a00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:14.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0073dc0 con 0x7fc0a0100e00 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a0073f30 con 0x7fc0a0102920 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a7400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0a01a0ee0 con 0x7fc0a0101f60 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a4a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a4a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37612/0 (socket says 172.21.15.23:37612) 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a4a00700 1 -- 172.21.15.23:0/1378278912 learned_addr learned my addr 172.21.15.23:0/1378278912 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc09fe00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 0x7fc0a00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.142+0000 7fc0a5400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0102920 0x7fc0a00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 -- 172.21.15.23:0/1378278912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 msgr2=0x7fc0a00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 0x7fc0a00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 -- 172.21.15.23:0/1378278912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0102920 msgr2=0x7fc0a00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0102920 0x7fc0a00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 -- 172.21.15.23:0/1378278912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098009d00 con 0x7fc0a0100e00 2024-03-30T04:01:14.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a4a00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc09400e9a0 tx=0x7fc09400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094009790 con 0x7fc0a0100e00 2024-03-30T04:01:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc094010430 con 0x7fc0a0100e00 2024-03-30T04:01:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc094018680 con 0x7fc0a0100e00 2024-03-30T04:01:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0a01a1140 con 0x7fc0a0100e00 2024-03-30T04:01:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.143+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a01a15c0 con 0x7fc0a0100e00 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.144+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0a01081b0 con 0x7fc0a0100e00 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.146+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc0940105a0 con 0x7fc0a0100e00 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.147+0000 7fc09d600700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 0x7fc0880a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.148+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc094014070 con 0x7fc0a0100e00 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.148+0000 7fc09fe00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 0x7fc0880a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:14.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.149+0000 7fc09fe00700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 0x7fc0880a3670 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009c60 tx=0x7fc08c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:14.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.152+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc0940529f0 con 0x7fc0a0100e00 2024-03-30T04:01:14.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.374+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0a002cd20 con 0x7fc0880a11b0 2024-03-30T04:01:14.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.375+0000 7fc09d600700 1 -- 172.21.15.23:0/1378278912 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fc0a002cd20 con 0x7fc0880a11b0 2024-03-30T04:01:14.375 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-30T04:01:14.375 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-30T04:01:14.376 INFO:teuthology.orchestra.run.smithi023.stdout:smithi154 172.21.15.154 2024-03-30T04:01:14.376 INFO:teuthology.orchestra.run.smithi023.stdout:smithi193 172.21.15.193 2024-03-30T04:01:14.376 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-30T04:01:14.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 msgr2=0x7fc0880a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 0x7fc0880a3670 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc08c009c60 tx=0x7fc08c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 msgr2=0x7fc0a0074ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc09400e9a0 tx=0x7fc09400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 shutdown_connections 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a0102920 0x7fc0a00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0a0100e00 0x7fc0a0074ab0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc0880a11b0 0x7fc0880a3670 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 --2- 172.21.15.23:0/1378278912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0a0101f60 0x7fc0a00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:14.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 >> 172.21.15.23:0/1378278912 conn(0x7fc0a00fc3c0 msgr2=0x7fc0a01052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:14.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 shutdown_connections 2024-03-30T04:01:14.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:14.378+0000 7fc0a7400700 1 -- 172.21.15.23:0/1378278912 wait complete. 2024-03-30T04:01:14.764 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-03-30T04:01:15.135 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:15.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:15 smithi023 ceph-mon[38254]: pgmap v285: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:15.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:15 smithi023 ceph-mon[38254]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:15 smithi154 ceph-mon[79161]: pgmap v285: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:15 smithi154 ceph-mon[79161]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:15 smithi193 ceph-mon[79581]: pgmap v285: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-30T04:01:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:15 smithi193 ceph-mon[79581]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 -- 172.21.15.23:0/3538818149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4102920 msgr2=0x7f18d4109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3538818149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4102920 0x7f18d4109240 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00b0a0 tx=0x7f18cc00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 -- 172.21.15.23:0/3538818149 shutdown_connections 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3538818149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4102920 0x7f18d4109240 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3538818149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4100e00 0x7f18d4101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3538818149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 -- 172.21.15.23:0/3538818149 >> 172.21.15.23:0/3538818149 conn(0x7f18d40fc3c0 msgr2=0x7f18d40fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:16.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 -- 172.21.15.23:0/3538818149 shutdown_connections 2024-03-30T04:01:16.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.363+0000 7f18d8e00700 1 -- 172.21.15.23:0/3538818149 wait complete. 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.364+0000 7f18d8e00700 1 Processor -- start 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.364+0000 7f18d8e00700 1 -- start start 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4102920 0x7f18d41a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41a0ba0 con 0x7f18d4100e00 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41a0ce0 con 0x7f18d4101f60 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d8e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41a0e70 con 0x7f18d4102920 2024-03-30T04:01:16.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37640/0 (socket says 172.21.15.23:37640) 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 -- 172.21.15.23:0/3382482054 learned_addr learned my addr 172.21.15.23:0/3382482054 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1400700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d2800700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4102920 0x7f18d41a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 -- 172.21.15.23:0/3382482054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4102920 msgr2=0x7f18d41a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4102920 0x7f18d41a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 -- 172.21.15.23:0/3382482054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 msgr2=0x7f18d41995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1e00700 1 -- 172.21.15.23:0/3382482054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc009d00 con 0x7f18d4100e00 2024-03-30T04:01:16.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.365+0000 7f18d1400700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:16.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18d1e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18c400e9f0 tx=0x7f18c400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:16.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40097b0 con 0x7f18d4100e00 2024-03-30T04:01:16.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c400ee90 con 0x7f18d4100e00 2024-03-30T04:01:16.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d41a1150 con 0x7f18d4100e00 2024-03-30T04:01:16.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d41a1650 con 0x7f18d4100e00 2024-03-30T04:01:16.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.366+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4005520 con 0x7f18d4100e00 2024-03-30T04:01:16.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.367+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4005320 con 0x7f18d4100e00 2024-03-30T04:01:16.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.369+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f18c400cbd0 con 0x7f18d4100e00 2024-03-30T04:01:16.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.370+0000 7f18caa00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 0x7f18b80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:16.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.371+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f18c4014070 con 0x7f18d4100e00 2024-03-30T04:01:16.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.371+0000 7f18d1400700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 0x7f18b80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:16.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.371+0000 7f18d1400700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 0x7f18b80a36f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18c0011490 tx=0x7f18c0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:16.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.375+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f18c40961f0 con 0x7f18d4100e00 2024-03-30T04:01:16.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.599+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b4000bf0 con 0x7f18b80a1230 2024-03-30T04:01:16.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.601+0000 7f18caa00700 1 -- 172.21.15.23:0/3382482054 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f18b4000bf0 con 0x7f18b80a1230 2024-03-30T04:01:16.601 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-30T04:01:16.601 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 2m ago Has a FileSystem, LVM detected 2024-03-30T04:01:16.601 INFO:teuthology.orchestra.run.smithi023.stdout:smithi154 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233003B400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T04:01:16.601 INFO:teuthology.orchestra.run.smithi023.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-30T04:01:16.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 msgr2=0x7f18b80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:16.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 0x7f18b80a36f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18c0011490 tx=0x7f18c0009260 comp rx=0 tx=0).stop 2024-03-30T04:01:16.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 msgr2=0x7f18d4199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f18c400e9f0 tx=0x7f18c400ed00 comp rx=0 tx=0).stop 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 shutdown_connections 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f18b80a1230 0x7f18b80a36f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18d4102920 0x7f18d41a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18d4100e00 0x7f18d4199060 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 --2- 172.21.15.23:0/3382482054 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f18d4101f60 0x7f18d41995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 >> 172.21.15.23:0/3382482054 conn(0x7f18d40fc3c0 msgr2=0x7f18d4072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 shutdown_connections 2024-03-30T04:01:16.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:16.604+0000 7f18d8e00700 1 -- 172.21.15.23:0/3382482054 wait complete. 2024-03-30T04:01:17.014 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-30T04:01:17.024 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:01:17.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph fs dump' 2024-03-30T04:01:17.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:16 smithi193 ceph-mon[79581]: pgmap v286: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:17.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:16 smithi023 ceph-mon[38254]: pgmap v286: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:17.392 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:16 smithi154 ceph-mon[79161]: pgmap v286: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:01:18.245 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:18.245 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T04:01:18.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-03-30T04:01:18.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2024-03-30T04:01:18.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-30T04:01:18.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-30T04:01:18.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:17 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-30T04:01:18.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:17 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-30T04:01:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: mgrmap e18: x(active, since 7m), standbys: z, y 2024-03-30T04:01:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]: dispatch 2024-03-30T04:01:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2024-03-30T04:01:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]: dispatch 2024-03-30T04:01:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-30T04:01:18.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:17 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2024-03-30T04:01:18.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 -- 172.21.15.23:0/110559323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4102920 msgr2=0x7fa8a4109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/110559323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4102920 0x7fa8a4109240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa89c00b0a0 tx=0x7fa89c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 -- 172.21.15.23:0/110559323 shutdown_connections 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/110559323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4102920 0x7fa8a4109240 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/110559323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4100e00 0x7fa8a4101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/110559323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4101f60 0x7fa8a41023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 -- 172.21.15.23:0/110559323 >> 172.21.15.23:0/110559323 conn(0x7fa8a40fc3c0 msgr2=0x7fa8a40fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 -- 172.21.15.23:0/110559323 shutdown_connections 2024-03-30T04:01:18.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.650+0000 7fa8aaa00700 1 -- 172.21.15.23:0/110559323 wait complete. 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.651+0000 7fa8aaa00700 1 Processor -- start 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.651+0000 7fa8aaa00700 1 -- start start 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4100e00 0x7fa8a419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4102920 0x7fa8a41a49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a41a4fb0 con 0x7fa8a4101f60 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a41a50f0 con 0x7fa8a4100e00 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8aaa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a41a5280 con 0x7fa8a4102920 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36850/0 (socket says 172.21.15.23:36850) 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 -- 172.21.15.23:0/4020999491 learned_addr learned my addr 172.21.15.23:0/4020999491 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:18.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3e00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4100e00 0x7fa8a419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a8a00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4102920 0x7fa8a41a49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 -- 172.21.15.23:0/4020999491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4102920 msgr2=0x7fa8a41a49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4102920 0x7fa8a41a49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 -- 172.21.15.23:0/4020999491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4100e00 msgr2=0x7fa8a419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.652+0000 7fa8a3400700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4100e00 0x7fa8a419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8a3400700 1 -- 172.21.15.23:0/4020999491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89c009d00 con 0x7fa8a4101f60 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8a3400700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa89000e9d0 tx=0x7fa89000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890009790 con 0x7fa8a4101f60 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa89000ee70 con 0x7fa8a4101f60 2024-03-30T04:01:18.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a41a5560 con 0x7fa8a4101f60 2024-03-30T04:01:18.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890010ea0 con 0x7fa8a4101f60 2024-03-30T04:01:18.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.653+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a41a5ab0 con 0x7fa8a4101f60 2024-03-30T04:01:18.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.654+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a41061a0 con 0x7fa8a4101f60 2024-03-30T04:01:18.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.656+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa89000cbd0 con 0x7fa8a4101f60 2024-03-30T04:01:18.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.657+0000 7fa8a0c00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 0x7fa8880a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:18.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.658+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fa890014070 con 0x7fa8a4101f60 2024-03-30T04:01:18.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.658+0000 7fa8a3e00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 0x7fa8880a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:18.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.658+0000 7fa8a3e00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 0x7fa8880a3660 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa894009c60 tx=0x7fa894009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:18.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.662+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa8900529f0 con 0x7fa8a4101f60 2024-03-30T04:01:18.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.921+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa8a404ea80 con 0x7fa8a4101f60 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.921+0000 7fa8a0c00700 1 -- 172.21.15.23:0/4020999491 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+2933 (secure 0 0 0) 0x7fa8900961a0 con 0x7fa8a4101f60 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout:e27 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:18.921 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 27 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-30T03:59:40.432862+0000 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-30T04:00:13.957636+0000 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2024-03-30T04:01:18.922 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 5 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1,2,3,4 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24569,1=24533,2=24509,3=24508,4=24476} 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3,4] 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{0:24569} state up:active seq 3 addr [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.923 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.193:6838/606583058,v1:172.21.15.193:6839/606583058] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{1:24533} state up:active seq 13 addr [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{1:24514} state up:standby-replay seq 1 addr [v2:172.21.15.193:6836/1829293538,v1:172.21.15.193:6837/1829293538] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{2:24509} state up:active seq 14 addr [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{2:24457} state up:standby-replay seq 1 addr [v2:172.21.15.193:6832/2731954066,v1:172.21.15.193:6833/2731954066] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{3:24508} state up:active seq 14 addr [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{3:14556} state up:standby-replay seq 1 addr [v2:172.21.15.23:6840/2954889723,v1:172.21.15.23:6841/2954889723] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{4:24476} state up:active seq 17 addr [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{4:14544} state up:standby-replay seq 1 addr [v2:172.21.15.23:6838/667186519,v1:172.21.15.23:6839/667186519] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14484} state up:standby seq 1 addr [v2:172.21.15.23:6834/1931657906,v1:172.21.15.23:6835/1931657906] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.924 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14496} state up:standby seq 1 addr [v2:172.21.15.23:6836/180927964,v1:172.21.15.23:6837/180927964] compat {c=[1],r=[1],i=[7ff]}] 2024-03-30T04:01:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.923+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 msgr2=0x7fa8880a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.923+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 0x7fa8880a3660 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa894009c60 tx=0x7fa894009260 comp rx=0 tx=0).stop 2024-03-30T04:01:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.923+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 msgr2=0x7fa8a419d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.923+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa89000e9d0 tx=0x7fa89000ece0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 shutdown_connections 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8880a11a0 0x7fa8880a3660 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa8a4102920 0x7fa8a41a49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a4100e00 0x7fa8a419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 --2- 172.21.15.23:0/4020999491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa8a4101f60 0x7fa8a419d9b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 >> 172.21.15.23:0/4020999491 conn(0x7fa8a40fc3c0 msgr2=0x7fa8a41032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 shutdown_connections 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:18.924+0000 7fa8aaa00700 1 -- 172.21.15.23:0/4020999491 wait complete. 2024-03-30T04:01:18.926 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: osdmap e87: 12 total, 12 up, 12 in 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: pgmap v288: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-30T04:01:19.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:18 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4020999491' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: osdmap e87: 12 total, 12 up, 12 in 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: pgmap v288: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-30T04:01:19.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:18 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4020999491' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:01:19.355 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- bash -c 'ceph osd dump' 2024-03-30T04:01:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 6]}]': finished 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7, 5, 8]}]': finished 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='mgr.14154 172.21.15.23:0/122927390' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: osdmap e87: 12 total, 12 up, 12 in 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: pgmap v288: 129 pgs: 129 active+clean; 645 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-30T04:01:19.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:18 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4020999491' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-30T04:01:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:20.223 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:19 smithi023 ceph-mon[38254]: osdmap e88: 12 total, 12 up, 12 in 2024-03-30T04:01:20.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:19 smithi193 ceph-mon[79581]: osdmap e88: 12 total, 12 up, 12 in 2024-03-30T04:01:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:19 smithi154 ceph-mon[79161]: osdmap e88: 12 total, 12 up, 12 in 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 -- 172.21.15.23:0/911989873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388101f60 msgr2=0x7f33881023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 --2- 172.21.15.23:0/911989873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388101f60 0x7f33881023e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3374009990 tx=0x7f3374009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 -- 172.21.15.23:0/911989873 shutdown_connections 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 --2- 172.21.15.23:0/911989873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388102920 0x7f3388109240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 --2- 172.21.15.23:0/911989873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 0x7f3388101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 --2- 172.21.15.23:0/911989873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388101f60 0x7f33881023e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 -- 172.21.15.23:0/911989873 >> 172.21.15.23:0/911989873 conn(0x7f33880fc3c0 msgr2=0x7f33880fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 -- 172.21.15.23:0/911989873 shutdown_connections 2024-03-30T04:01:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.936+0000 7f338d400700 1 -- 172.21.15.23:0/911989873 wait complete. 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.937+0000 7f338d400700 1 Processor -- start 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.937+0000 7f338d400700 1 -- start start 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.937+0000 7f338d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 0x7f3388198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.937+0000 7f338d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f338d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388102920 0x7f33881a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f338d400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a0a50 con 0x7f3388102920 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f338d400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a0b90 con 0x7f3388101f60 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f338d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33881a0d20 con 0x7f3388100e00 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3387200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388102920 0x7f33881a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:45186/0 (socket says 172.21.15.23:45186) 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 -- 172.21.15.23:0/104680864 learned_addr learned my addr 172.21.15.23:0/104680864 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3386800700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 0x7f3388198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 -- 172.21.15.23:0/104680864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 msgr2=0x7f3388198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 0x7f3388198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 -- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388102920 msgr2=0x7f33881a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388102920 0x7f33881a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:20.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.938+0000 7f3385e00700 1 -- 172.21.15.23:0/104680864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740095f0 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f3385e00700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f337400b620 tx=0x7f3374004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337401d070 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33881a0fa0 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3374022470 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374017650 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.939+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33881a1350 con 0x7f3388101f60 2024-03-30T04:01:20.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.940+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338804ea80 con 0x7f3388101f60 2024-03-30T04:01:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.942+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f33740225e0 con 0x7f3388101f60 2024-03-30T04:01:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.944+0000 7f337f400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.944+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f33740ce330 con 0x7f3388101f60 2024-03-30T04:01:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.944+0000 7f3386800700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:20.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.944+0000 7f3386800700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 0x7f336c0a35e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3378009c60 tx=0x7f3378009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:20.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:20.946+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3374097870 con 0x7f3388101f60 2024-03-30T04:01:21.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.161+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f338810c800 con 0x7f3388101f60 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.162+0000 7f337f400700 1 -- 172.21.15.23:0/104680864 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5682 (secure 0 0 0) 0x7f337409ae90 con 0x7f3388101f60 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 88 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-30T03:53:14.698271+0000 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-30T04:01:18.983496+0000 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 27 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-30T04:01:21.162 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release reef 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/2819496083,v1:172.21.15.23:6803/2819496083] [v2:172.21.15.23:6804/2819496083,v1:172.21.15.23:6805/2819496083] exists,up 60b581f8-6bf1-4e71-8ad6-974da9b9db58 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6800/1863895073,v1:172.21.15.154:6801/1863895073] [v2:172.21.15.154:6802/1863895073,v1:172.21.15.154:6803/1863895073] exists,up 4e38bd8a-9f96-4585-ad8f-a364a12543df 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/304660967,v1:172.21.15.193:6801/304660967] [v2:172.21.15.193:6802/304660967,v1:172.21.15.193:6803/304660967] exists,up e2244e3c-4dd3-4c06-a796-07187e9a77a0 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 25 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/1020396955,v1:172.21.15.23:6811/1020396955] [v2:172.21.15.23:6812/1020396955,v1:172.21.15.23:6813/1020396955] exists,up f3b063ec-ce2c-4142-a8c8-5bca5f8252bf 2024-03-30T04:01:21.163 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 32 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6808/2815560551,v1:172.21.15.154:6809/2815560551] [v2:172.21.15.154:6810/2815560551,v1:172.21.15.154:6811/2815560551] exists,up c90bc547-0492-4582-be42-0b7cdd2f2fb1 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 37 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/1684695578,v1:172.21.15.193:6809/1684695578] [v2:172.21.15.193:6810/1684695578,v1:172.21.15.193:6811/1684695578] exists,up 93e66232-688d-4fc5-a935-1c4e147cd6d9 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 44 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/4279410626,v1:172.21.15.23:6819/4279410626] [v2:172.21.15.23:6820/4279410626,v1:172.21.15.23:6821/4279410626] exists,up 5c236d02-e504-41f1-be6d-acc36f957d54 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 49 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6816/4282776176,v1:172.21.15.154:6817/4282776176] [v2:172.21.15.154:6818/4282776176,v1:172.21.15.154:6819/4282776176] exists,up dca94a7d-3a01-47d2-aac6-b633e302111d 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 56 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/54947619,v1:172.21.15.193:6817/54947619] [v2:172.21.15.193:6818/54947619,v1:172.21.15.193:6819/54947619] exists,up 81abadd3-48ab-4d42-abd2-f687f5122279 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 63 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/2836554643,v1:172.21.15.23:6827/2836554643] [v2:172.21.15.23:6828/2836554643,v1:172.21.15.23:6829/2836554643] exists,up 89bd59bf-4452-4a10-b466-f00fab24b496 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 68 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.154:6824/1797593825,v1:172.21.15.154:6825/1797593825] [v2:172.21.15.154:6826/1797593825,v1:172.21.15.154:6827/1797593825] exists,up 0803d869-d43c-46f0-9725-be71809628b6 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 75 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/2276508968,v1:172.21.15.193:6825/2276508968] [v2:172.21.15.193:6826/2276508968,v1:172.21.15.193:6827/2276508968] exists,up 7746428c-6c3a-4801-922d-7dd9000dfe7d 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.6 [8,11] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.8 [2,11] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.0 [4,6] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.6 [3,6] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.c [3,9] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.d [1,9] 2024-03-30T04:01:21.164 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.17 [4,7,5,8] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.21 [4,8,5,9] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.23 [4,7] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.6 [5,8] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.16 [9,8] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.17 [9,10] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1b [9,8] 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1161676542 expires 2024-03-31T03:54:16.197829+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/443135190 expires 2024-03-31T03:54:16.197829+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2173844842 expires 2024-03-31T03:53:51.749812+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2471664699 expires 2024-03-31T03:53:51.749812+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/783324330 expires 2024-03-31T03:53:51.749812+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1219945839 expires 2024-03-31T03:54:16.197829+0000 2024-03-30T04:01:21.165 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/604121731 expires 2024-03-31T03:54:16.197829+0000 2024-03-30T04:01:21.166 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/2151815679 expires 2024-03-31T03:53:51.749812+0000 2024-03-30T04:01:21.166 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/604121731 expires 2024-03-31T03:54:16.197829+0000 2024-03-30T04:01:21.166 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/2151815679 expires 2024-03-31T03:53:51.749812+0000 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 msgr2=0x7f336c0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 0x7f336c0a35e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3378009c60 tx=0x7f3378009260 comp rx=0 tx=0).stop 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 msgr2=0x7f3388199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f337400b620 tx=0x7f3374004730 comp rx=0 tx=0).stop 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 shutdown_connections 2024-03-30T04:01:21.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3388102920 0x7f33881a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f336c0a1120 0x7f336c0a35e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3388100e00 0x7f3388198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 --2- 172.21.15.23:0/104680864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3388101f60 0x7f3388199450 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.164+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 >> 172.21.15.23:0/104680864 conn(0x7f33880fc3c0 msgr2=0x7f3388072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.165+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 shutdown_connections 2024-03-30T04:01:21.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.165+0000 7f338d400700 1 -- 172.21.15.23:0/104680864 wait complete. 2024-03-30T04:01:21.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:20 smithi023 ceph-mon[38254]: pgmap v290: 129 pgs: 129 active+clean; 645 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T04:01:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:20 smithi154 ceph-mon[79161]: pgmap v290: 129 pgs: 129 active+clean; 645 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T04:01:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:20 smithi193 ceph-mon[79581]: pgmap v290: 129 pgs: 129 active+clean; 645 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-30T04:01:21.613 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-30T04:01:21.624 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 -- 172.21.15.23:0/3777237591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 msgr2=0x7f56c81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 --2- 172.21.15.23:0/3777237591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c81022d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56b40099c0 tx=0x7f56b4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 -- 172.21.15.23:0/3777237591 shutdown_connections 2024-03-30T04:01:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 --2- 172.21.15.23:0/3777237591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8102810 0x7f56c81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 --2- 172.21.15.23:0/3777237591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8100c80 0x7f56c81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 --2- 172.21.15.23:0/3777237591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c81022d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 -- 172.21.15.23:0/3777237591 >> 172.21.15.23:0/3777237591 conn(0x7f56c80fc1f0 msgr2=0x7f56c80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 -- 172.21.15.23:0/3777237591 shutdown_connections 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.828+0000 7f56cd400700 1 -- 172.21.15.23:0/3777237591 wait complete. 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.829+0000 7f56cd400700 1 Processor -- start 2024-03-30T04:01:21.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.829+0000 7f56cd400700 1 -- start start 2024-03-30T04:01:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.829+0000 7f56cd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:21.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56cd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c8197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56cd400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 0x7f56c819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56cd400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819e730 con 0x7f56c8100c80 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56cd400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819e870 con 0x7f56c8101e70 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56cd400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c819ea00 con 0x7f56c8102810 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36880/0 (socket says 172.21.15.23:36880) 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 -- 172.21.15.23:0/1172585195 learned_addr learned my addr 172.21.15.23:0/1172585195 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c5e00700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c8197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c7200700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 0x7f56c819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 -- 172.21.15.23:0/1172585195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 msgr2=0x7f56c819e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 0x7f56c819e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 -- 172.21.15.23:0/1172585195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 msgr2=0x7f56c8197140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c8197140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c6800700 1 -- 172.21.15.23:0/1172585195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40095f0 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c7200700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 0x7f56c819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.830+0000 7f56c5e00700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c8197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56c6800700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e970 tx=0x7f56b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8009790 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56b80044e0 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c819ece0 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8003e30 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.831+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c819f1a0 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.832+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c8108100 con 0x7f56c8100c80 2024-03-30T04:01:21.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.834+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f56b8003f90 con 0x7f56c8100c80 2024-03-30T04:01:21.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.836+0000 7f56bf400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 0x7f56ac0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:21.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.836+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f56b8014070 con 0x7f56c8100c80 2024-03-30T04:01:21.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.836+0000 7f56c5e00700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 0x7f56ac0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:21.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.837+0000 7f56c5e00700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 0x7f56ac0a35e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56b400b580 tx=0x7f56b401a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:21.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:21.840+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f56b80963d0 con 0x7f56c8100c80 2024-03-30T04:01:22.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.082+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c8067a10 con 0x7f56c8100c80 2024-03-30T04:01:22.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.084+0000 7f56bf400700 1 -- 172.21.15.23:0/1172585195 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f56b80999f0 con 0x7f56c8100c80 2024-03-30T04:01:22.084 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:22.084 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:22.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.086+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 msgr2=0x7f56ac0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.086+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 0x7f56ac0a35e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f56b400b580 tx=0x7f56b401a040 comp rx=0 tx=0).stop 2024-03-30T04:01:22.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.086+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 msgr2=0x7f56c8196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.086+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e970 tx=0x7f56b800ed30 comp rx=0 tx=0).stop 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 shutdown_connections 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56ac0a1120 0x7f56ac0a35e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56c8102810 0x7f56c819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56c8100c80 0x7f56c8196c00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 --2- 172.21.15.23:0/1172585195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c8101e70 0x7f56c8197140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 >> 172.21.15.23:0/1172585195 conn(0x7f56c80fc1f0 msgr2=0x7f56c8105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 shutdown_connections 2024-03-30T04:01:22.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.087+0000 7f56cd400700 1 -- 172.21.15.23:0/1172585195 wait complete. 2024-03-30T04:01:22.106 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:22.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:22.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:21 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/104680864' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T04:01:22.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.310+0000 7f6a37e00700 1 -- 172.21.15.23:0/4211753632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30101f70 msgr2=0x7f6a301023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.310+0000 7f6a37e00700 1 --2- 172.21.15.23:0/4211753632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30101f70 0x7f6a301023d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c0099c0 tx=0x7f6a1c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 -- 172.21.15.23:0/4211753632 shutdown_connections 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 --2- 172.21.15.23:0/4211753632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30102910 0x7f6a301091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 --2- 172.21.15.23:0/4211753632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30100d80 0x7f6a301011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 --2- 172.21.15.23:0/4211753632 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30101f70 0x7f6a301023d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 -- 172.21.15.23:0/4211753632 >> 172.21.15.23:0/4211753632 conn(0x7f6a300fc2f0 msgr2=0x7f6a300fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 -- 172.21.15.23:0/4211753632 shutdown_connections 2024-03-30T04:01:22.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.311+0000 7f6a37e00700 1 -- 172.21.15.23:0/4211753632 wait complete. 2024-03-30T04:01:22.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 Processor -- start 2024-03-30T04:01:22.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 -- start start 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30102910 0x7f6a301a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0a60 con 0x7f6a30102910 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0ba0 con 0x7f6a30100d80 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.312+0000 7f6a37e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a301a0d30 con 0x7f6a30101f70 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a35400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a35400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:45222/0 (socket says 172.21.15.23:45222) 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52546/0 (socket says 172.21.15.23:52546) 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a35400700 1 -- 172.21.15.23:0/1770007897 learned_addr learned my addr 172.21.15.23:0/1770007897 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:22.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a35e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30102910 0x7f6a301a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 -- 172.21.15.23:0/1770007897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 msgr2=0x7f6a30198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 -- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30102910 msgr2=0x7f6a301a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30102910 0x7f6a301a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 -- 172.21.15.23:0/1770007897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1c0095f0 con 0x7f6a30101f70 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a35400700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.313+0000 7f6a34a00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009990 tx=0x7f6a1c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.314+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c01d070 con 0x7f6a30101f70 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.314+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a1c004050 con 0x7f6a30101f70 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.314+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a301a0fb0 con 0x7f6a30101f70 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.314+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c005280 con 0x7f6a30101f70 2024-03-30T04:01:22.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.314+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a301a1320 con 0x7f6a30101f70 2024-03-30T04:01:22.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.315+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a30067af0 con 0x7f6a30101f70 2024-03-30T04:01:22.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.317+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6a1c004870 con 0x7f6a30101f70 2024-03-30T04:01:22.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.318+0000 7f6a2e000700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 0x7f6a180a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.318+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f6a1c0cda20 con 0x7f6a30101f70 2024-03-30T04:01:22.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.318+0000 7f6a35400700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 0x7f6a180a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.319+0000 7f6a35400700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 0x7f6a180a35e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6a240105c0 tx=0x7f6a2400b4a0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:22.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.321+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a1c096e60 con 0x7f6a30101f70 2024-03-30T04:01:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:21 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/104680864' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T04:01:22.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:21 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/104680864' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-30T04:01:22.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.538+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a3010c790 con 0x7f6a30101f70 2024-03-30T04:01:22.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.539+0000 7f6a2e000700 1 -- 172.21.15.23:0/1770007897 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a1c09a480 con 0x7f6a30101f70 2024-03-30T04:01:22.538 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 msgr2=0x7f6a180a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 0x7f6a180a35e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6a240105c0 tx=0x7f6a2400b4a0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 msgr2=0x7f6a30199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009990 tx=0x7f6a1c004730 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 shutdown_connections 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a30102910 0x7f6a301a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a30100d80 0x7f6a30198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a180a1120 0x7f6a180a35e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 --2- 172.21.15.23:0/1770007897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a30101f70 0x7f6a30199470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 >> 172.21.15.23:0/1770007897 conn(0x7f6a300fc2f0 msgr2=0x7f6a30072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:22.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 shutdown_connections 2024-03-30T04:01:22.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.541+0000 7f6a37e00700 1 -- 172.21.15.23:0/1770007897 wait complete. 2024-03-30T04:01:22.559 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 -- 172.21.15.23:0/1435309321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 msgr2=0x7feabc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 --2- 172.21.15.23:0/1435309321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc1091c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feab400b0d0 tx=0x7feab400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 -- 172.21.15.23:0/1435309321 shutdown_connections 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 --2- 172.21.15.23:0/1435309321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc1091c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 --2- 172.21.15.23:0/1435309321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc100d80 0x7feabc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 --2- 172.21.15.23:0/1435309321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc101f70 0x7feabc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 -- 172.21.15.23:0/1435309321 >> 172.21.15.23:0/1435309321 conn(0x7feabc0fc2f0 msgr2=0x7feabc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 -- 172.21.15.23:0/1435309321 shutdown_connections 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.763+0000 7feac1e00700 1 -- 172.21.15.23:0/1435309321 wait complete. 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.765+0000 7feac1e00700 1 Processor -- start 2024-03-30T04:01:22.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.765+0000 7feac1e00700 1 -- start start 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.765+0000 7feac1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feac1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 0x7feabc197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feac1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feac1e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19e980 con 0x7feabc102910 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feac1e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19eac0 con 0x7feabc101f70 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feac1e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc19ec50 con 0x7feabc100d80 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feaba000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 0x7feabc197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52566/0 (socket says 172.21.15.23:52566) 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 -- 172.21.15.23:0/832549666 learned_addr learned my addr 172.21.15.23:0/832549666 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 -- 172.21.15.23:0/832549666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 msgr2=0x7feabc197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 0x7feabc197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 -- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 msgr2=0x7feabc19e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc19e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabaa00700 1 -- 172.21.15.23:0/832549666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4009d00 con 0x7feabc100d80 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feabb400700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.766+0000 7feaba000700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 0x7feabc197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:22.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feabaa00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e970 tx=0x7feaac00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:22.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac009790 con 0x7feabc100d80 2024-03-30T04:01:22.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feaac00c7e0 con 0x7feabc100d80 2024-03-30T04:01:22.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc19ef30 con 0x7feabc100d80 2024-03-30T04:01:22.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac003c70 con 0x7feabc100d80 2024-03-30T04:01:22.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.767+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc19f350 con 0x7feabc100d80 2024-03-30T04:01:22.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.768+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc108180 con 0x7feabc100d80 2024-03-30T04:01:22.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.770+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7feaac00cc50 con 0x7feabc100d80 2024-03-30T04:01:22.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.771+0000 7feac0a00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 0x7feaa00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.771+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7feaac014070 con 0x7feabc100d80 2024-03-30T04:01:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.772+0000 7feaba000700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 0x7feaa00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:22.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.772+0000 7feaba000700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 0x7feaa00a35e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feaa8011490 tx=0x7feaa8009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:22.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.774+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feaac096310 con 0x7feabc100d80 2024-03-30T04:01:22.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:22.997+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feabc10c700 con 0x7feabc100d80 2024-03-30T04:01:23.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.003+0000 7feac0a00700 1 -- 172.21.15.23:0/832549666 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7feaac003dd0 con 0x7feabc100d80 2024-03-30T04:01:23.003 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 msgr2=0x7feaa00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 0x7feaa00a35e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feaa8011490 tx=0x7feaa8009260 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 msgr2=0x7feabc196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7feaac00e970 tx=0x7feaac00ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 shutdown_connections 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feabc102910 0x7feabc19e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7feaa00a1120 0x7feaa00a35e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feabc100d80 0x7feabc196e50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 --2- 172.21.15.23:0/832549666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feabc101f70 0x7feabc197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 >> 172.21.15.23:0/832549666 conn(0x7feabc0fc2f0 msgr2=0x7feabc1052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:23.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 shutdown_connections 2024-03-30T04:01:23.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.006+0000 7feac1e00700 1 -- 172.21.15.23:0/832549666 wait complete. 2024-03-30T04:01:23.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 -- 172.21.15.23:0/432273754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 msgr2=0x7ff680073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 --2- 172.21.15.23:0/432273754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680073c90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff66c009990 tx=0x7ff66c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 -- 172.21.15.23:0/432273754 shutdown_connections 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 --2- 172.21.15.23:0/432273754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680108c40 0x7ff68010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 --2- 172.21.15.23:0/432273754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680074df0 0x7ff680073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 --2- 172.21.15.23:0/432273754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680073c90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 -- 172.21.15.23:0/432273754 >> 172.21.15.23:0/432273754 conn(0x7ff6800fc2f0 msgr2=0x7ff6800fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 -- 172.21.15.23:0/432273754 shutdown_connections 2024-03-30T04:01:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.227+0000 7ff688600700 1 -- 172.21.15.23:0/432273754 wait complete. 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 Processor -- start 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 -- start start 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 0x7ff6801994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680108c40 0x7ff6801a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0a90 con 0x7ff680073810 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0bd0 con 0x7ff680108c40 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.228+0000 7ff688600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6801a0d60 con 0x7ff680074df0 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36932/0 (socket says 172.21.15.23:36932) 2024-03-30T04:01:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 -- 172.21.15.23:0/1409642397 learned_addr learned my addr 172.21.15.23:0/1409642397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff686600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680108c40 0x7ff6801a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685200700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 0x7ff6801994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 -- 172.21.15.23:0/1409642397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 msgr2=0x7ff6801994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 0x7ff6801994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 -- 172.21.15.23:0/1409642397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680108c40 msgr2=0x7ff6801a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680108c40 0x7ff6801a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 -- 172.21.15.23:0/1409642397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0095f0 con 0x7ff680073810 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685200700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 0x7ff6801994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.229+0000 7ff685c00700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff67400e9a0 tx=0x7ff67400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:23.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.230+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674009790 con 0x7ff680073810 2024-03-30T04:01:23.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.230+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff67400cbd0 con 0x7ff680073810 2024-03-30T04:01:23.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.230+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674018680 con 0x7ff680073810 2024-03-30T04:01:23.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.230+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6801a0fe0 con 0x7ff680073810 2024-03-30T04:01:23.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.230+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6801a1410 con 0x7ff680073810 2024-03-30T04:01:23.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.231+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff680107a00 con 0x7ff680073810 2024-03-30T04:01:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.233+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ff67400cd40 con 0x7ff680073810 2024-03-30T04:01:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.234+0000 7ff67e000700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 0x7ff6680a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.234+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7ff674014070 con 0x7ff680073810 2024-03-30T04:01:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.234+0000 7ff685200700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 0x7ff6680a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.235+0000 7ff685200700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 0x7ff6680a35e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00b7b0 tx=0x7ff66c005fb0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:23.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.238+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff6740d2020 con 0x7ff680073810 2024-03-30T04:01:23.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1172585195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:23 smithi023 ceph-mon[38254]: pgmap v291: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 9 op/s 2024-03-30T04:01:23.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1770007897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:23 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1172585195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:23 smithi154 ceph-mon[79161]: pgmap v291: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 9 op/s 2024-03-30T04:01:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:23 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1770007897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:23.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.454+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff68002cd50 con 0x7ff680073810 2024-03-30T04:01:23.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.454+0000 7ff67e000700 1 -- 172.21.15.23:0/1409642397 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff674096280 con 0x7ff680073810 2024-03-30T04:01:23.454 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:23.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 msgr2=0x7ff6680a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 0x7ff6680a35e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00b7b0 tx=0x7ff66c005fb0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 msgr2=0x7ff680198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff67400e9a0 tx=0x7ff67400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 shutdown_connections 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff680108c40 0x7ff6801a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff680073810 0x7ff680198fb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff6680a1120 0x7ff6680a35e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 --2- 172.21.15.23:0/1409642397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff680074df0 0x7ff6801994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 >> 172.21.15.23:0/1409642397 conn(0x7ff6800fc2f0 msgr2=0x7ff680104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 shutdown_connections 2024-03-30T04:01:23.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.457+0000 7ff688600700 1 -- 172.21.15.23:0/1409642397 wait complete. 2024-03-30T04:01:23.476 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:23 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1172585195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:23.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:23 smithi193 ceph-mon[79581]: pgmap v291: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 895 B/s wr, 9 op/s 2024-03-30T04:01:23.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:23 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1770007897' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 -- 172.21.15.23:0/2733643080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474074df0 msgr2=0x7fa474073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 --2- 172.21.15.23:0/2733643080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474074df0 0x7fa474073240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa464009990 tx=0x7fa464009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 -- 172.21.15.23:0/2733643080 shutdown_connections 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 --2- 172.21.15.23:0/2733643080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474108c40 0x7fa47410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 --2- 172.21.15.23:0/2733643080 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474074df0 0x7fa474073240 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 --2- 172.21.15.23:0/2733643080 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474073810 0x7fa474073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 -- 172.21.15.23:0/2733643080 >> 172.21.15.23:0/2733643080 conn(0x7fa4740fc2f0 msgr2=0x7fa4740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 -- 172.21.15.23:0/2733643080 shutdown_connections 2024-03-30T04:01:23.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.678+0000 7fa47a600700 1 -- 172.21.15.23:0/2733643080 wait complete. 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 Processor -- start 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 -- start start 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 0x7fa4741a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4741a0a70 con 0x7fa474108c40 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4741a0bb0 con 0x7fa474073810 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa47a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4741a0d40 con 0x7fa474074df0 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.679+0000 7fa473e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 0x7fa4741a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa472a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:45270/0 (socket says 172.21.15.23:45270) 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 -- 172.21.15.23:0/3690262427 learned_addr learned my addr 172.21.15.23:0/3690262427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa472a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52600/0 (socket says 172.21.15.23:52600) 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 0x7fa4741a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36950/0 (socket says 172.21.15.23:36950) 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 -- 172.21.15.23:0/3690262427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 msgr2=0x7fa474199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 -- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 msgr2=0x7fa4741a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 0x7fa4741a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 -- 172.21.15.23:0/3690262427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4640095f0 con 0x7fa474073810 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa472a00700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.680+0000 7fa473400700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa4640095c0 tx=0x7fa46400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.681+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46401d070 con 0x7fa474073810 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.681+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa464005030 con 0x7fa474073810 2024-03-30T04:01:23.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.681+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4741a0fc0 con 0x7fa474073810 2024-03-30T04:01:23.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.681+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464004a50 con 0x7fa474073810 2024-03-30T04:01:23.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.681+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4741a1330 con 0x7fa474073810 2024-03-30T04:01:23.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.682+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa474107a00 con 0x7fa474073810 2024-03-30T04:01:23.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.684+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa464005450 con 0x7fa474073810 2024-03-30T04:01:23.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.685+0000 7fa46be00700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 0x7fa4580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:23.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.685+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fa4640cdec0 con 0x7fa474073810 2024-03-30T04:01:23.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.685+0000 7fa472a00700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 0x7fa4580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:23.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.686+0000 7fa472a00700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 0x7fa4580a35e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa474074b10 tx=0x7fa46000b410 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:23.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.688+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa464097300 con 0x7fa474073810 2024-03-30T04:01:23.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.905+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa474067af0 con 0x7fa474073810 2024-03-30T04:01:23.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.906+0000 7fa46be00700 1 -- 172.21.15.23:0/3690262427 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa46409a920 con 0x7fa474073810 2024-03-30T04:01:23.906 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 msgr2=0x7fa4580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 0x7fa4580a35e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa474074b10 tx=0x7fa46000b410 comp rx=0 tx=0).stop 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 msgr2=0x7fa474198f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa4640095c0 tx=0x7fa46400f690 comp rx=0 tx=0).stop 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 shutdown_connections 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa474108c40 0x7fa4741a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa474073810 0x7fa474198f40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4580a1120 0x7fa4580a35e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 --2- 172.21.15.23:0/3690262427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa474074df0 0x7fa474199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 >> 172.21.15.23:0/3690262427 conn(0x7fa4740fc2f0 msgr2=0x7fa474104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 shutdown_connections 2024-03-30T04:01:23.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:23.908+0000 7fa47a600700 1 -- 172.21.15.23:0/3690262427 wait complete. 2024-03-30T04:01:23.927 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:24.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.127+0000 7f65eda00700 1 -- 172.21.15.23:0/4277941841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8101e70 msgr2=0x7f65e81022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.127+0000 7f65eda00700 1 --2- 172.21.15.23:0/4277941841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8101e70 0x7f65e81022d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f65d40099c0 tx=0x7f65d4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 -- 172.21.15.23:0/4277941841 shutdown_connections 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 --2- 172.21.15.23:0/4277941841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8102810 0x7f65e81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 --2- 172.21.15.23:0/4277941841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8100c80 0x7f65e81010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 --2- 172.21.15.23:0/4277941841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8101e70 0x7f65e81022d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 -- 172.21.15.23:0/4277941841 >> 172.21.15.23:0/4277941841 conn(0x7f65e80fc1f0 msgr2=0x7f65e80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 -- 172.21.15.23:0/4277941841 shutdown_connections 2024-03-30T04:01:24.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.128+0000 7f65eda00700 1 -- 172.21.15.23:0/4277941841 wait complete. 2024-03-30T04:01:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.129+0000 7f65eda00700 1 Processor -- start 2024-03-30T04:01:24.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.129+0000 7f65eda00700 1 -- start start 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8101e70 0x7f65e8199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8102810 0x7f65e81a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e81a08e0 con 0x7f65e8100c80 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e81a0a50 con 0x7f65e8102810 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65eda00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e81a0c10 con 0x7f65e8101e70 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36968/0 (socket says 172.21.15.23:36968) 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 -- 172.21.15.23:0/23049910 learned_addr learned my addr 172.21.15.23:0/23049910 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6000700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8101e70 0x7f65e8199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e7400700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8102810 0x7f65e81a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 -- 172.21.15.23:0/23049910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8101e70 msgr2=0x7f65e8199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8101e70 0x7f65e8199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.130+0000 7f65e6a00700 1 -- 172.21.15.23:0/23049910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8102810 msgr2=0x7f65e81a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65e6a00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8102810 0x7f65e81a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65e6a00700 1 -- 172.21.15.23:0/23049910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40095f0 con 0x7f65e8100c80 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65e6a00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65d800e970 tx=0x7f65d800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8009790 con 0x7f65e8100c80 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65d800c7e0 con 0x7f65e8100c80 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e81a0ef0 con 0x7f65e8100c80 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d8003c70 con 0x7f65e8100c80 2024-03-30T04:01:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.131+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e81a1320 con 0x7f65e8100c80 2024-03-30T04:01:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.132+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e8067a10 con 0x7f65e8100c80 2024-03-30T04:01:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.134+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f65d800cc50 con 0x7f65e8100c80 2024-03-30T04:01:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.135+0000 7f65df400700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 0x7f65cc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.136+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f65d8014070 con 0x7f65e8100c80 2024-03-30T04:01:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.136+0000 7f65e6000700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 0x7f65cc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.136+0000 7f65e6000700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 0x7f65cc0a3660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009990 tx=0x7f65d4000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.139+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f65d8096190 con 0x7f65e8100c80 2024-03-30T04:01:24.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:24 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/832549666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:24 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1409642397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:24 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3690262427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.404+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65e810c490 con 0x7f65e8100c80 2024-03-30T04:01:24.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.405+0000 7f65df400700 1 -- 172.21.15.23:0/23049910 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f65d8003dd0 con 0x7f65e8100c80 2024-03-30T04:01:24.405 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:24.405 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.407+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 msgr2=0x7f65cc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 0x7f65cc0a3660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f65d4009990 tx=0x7f65d4000bc0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 msgr2=0x7f65e8198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f65d800e970 tx=0x7f65d800ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 shutdown_connections 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65e8102810 0x7f65e81a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65e8100c80 0x7f65e8198df0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f65cc0a11a0 0x7f65cc0a3660 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 --2- 172.21.15.23:0/23049910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65e8101e70 0x7f65e8199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 >> 172.21.15.23:0/23049910 conn(0x7f65e80fc1f0 msgr2=0x7f65e8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 shutdown_connections 2024-03-30T04:01:24.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.408+0000 7f65eda00700 1 -- 172.21.15.23:0/23049910 wait complete. 2024-03-30T04:01:24.409 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:24.426 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:24 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/832549666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:24 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1409642397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:24 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3690262427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:24 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/832549666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:24 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1409642397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:24 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3690262427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 -- 172.21.15.23:0/1497899501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678100c80 msgr2=0x7f06781010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 --2- 172.21.15.23:0/1497899501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678100c80 0x7f06781010a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f066c0099c0 tx=0x7f066c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 -- 172.21.15.23:0/1497899501 shutdown_connections 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 --2- 172.21.15.23:0/1497899501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678102810 0x7f06781090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 --2- 172.21.15.23:0/1497899501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678100c80 0x7f06781010a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 --2- 172.21.15.23:0/1497899501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678101e70 0x7f06781022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 -- 172.21.15.23:0/1497899501 >> 172.21.15.23:0/1497899501 conn(0x7f06780fc1f0 msgr2=0x7f06780fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 -- 172.21.15.23:0/1497899501 shutdown_connections 2024-03-30T04:01:24.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.628+0000 7f067f600700 1 -- 172.21.15.23:0/1497899501 wait complete. 2024-03-30T04:01:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 Processor -- start 2024-03-30T04:01:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 -- start start 2024-03-30T04:01:24.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 0x7f06781993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678102810 0x7f06781a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06781a0990 con 0x7f0678101e70 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06781a0ad0 con 0x7f0678102810 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.629+0000 7f067f600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06781a0c60 con 0x7f0678100c80 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 0x7f06781993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f067cc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f067cc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52644/0 (socket says 172.21.15.23:52644) 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f067cc00700 1 -- 172.21.15.23:0/1149446139 learned_addr learned my addr 172.21.15.23:0/1149446139 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:24.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f067d600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678102810 0x7f06781a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 -- 172.21.15.23:0/1149446139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 msgr2=0x7f0678198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 -- 172.21.15.23:0/1149446139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678102810 msgr2=0x7f06781a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678102810 0x7f06781a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f0677e00700 1 -- 172.21.15.23:0/1149446139 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c0095f0 con 0x7f0678101e70 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.630+0000 7f067cc00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f0677e00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 0x7f06781993f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f066400e9f0 tx=0x7f066400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066400c9c0 con 0x7f0678101e70 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0664010430 con 0x7f0678101e70 2024-03-30T04:01:24.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06781a0f40 con 0x7f0678101e70 2024-03-30T04:01:24.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664018680 con 0x7f0678101e70 2024-03-30T04:01:24.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.631+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06781a13c0 con 0x7f0678101e70 2024-03-30T04:01:24.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.632+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678067a10 con 0x7f0678101e70 2024-03-30T04:01:24.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.634+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f0664010860 con 0x7f0678101e70 2024-03-30T04:01:24.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.635+0000 7f0675600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 0x7f06600a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:24.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.635+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f0664014070 con 0x7f0678101e70 2024-03-30T04:01:24.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.635+0000 7f067cc00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 0x7f06600a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:24.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.636+0000 7f067cc00700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 0x7f06600a35e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f066c000c00 tx=0x7f066c01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:24.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.639+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f06640529f0 con 0x7f0678101e70 2024-03-30T04:01:24.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.892+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f067810c4e0 con 0x7f0678101e70 2024-03-30T04:01:24.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.894+0000 7f0675600700 1 -- 172.21.15.23:0/1149446139 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f0664096170 con 0x7f0678101e70 2024-03-30T04:01:24.894 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:24.894 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 msgr2=0x7f06600a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 0x7f06600a35e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f066c000c00 tx=0x7f066c01a040 comp rx=0 tx=0).stop 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 msgr2=0x7f06781993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 0x7f06781993f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f066400e9f0 tx=0x7f066400ed00 comp rx=0 tx=0).stop 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 shutdown_connections 2024-03-30T04:01:24.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678102810 0x7f06781a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f06600a1120 0x7f06600a35e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0678100c80 0x7f0678198eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 --2- 172.21.15.23:0/1149446139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0678101e70 0x7f06781993f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.896+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 >> 172.21.15.23:0/1149446139 conn(0x7f06780fc1f0 msgr2=0x7f0678072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.897+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 shutdown_connections 2024-03-30T04:01:24.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:24.897+0000 7f067f600700 1 -- 172.21.15.23:0/1149446139 wait complete. 2024-03-30T04:01:24.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:24.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 -- 172.21.15.23:0/1544128051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 msgr2=0x7f57dc1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 --2- 172.21.15.23:0/1544128051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1091c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f57d400b0d0 tx=0x7f57d400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 -- 172.21.15.23:0/1544128051 shutdown_connections 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 --2- 172.21.15.23:0/1544128051 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1091c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 --2- 172.21.15.23:0/1544128051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 0x7f57dc1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 --2- 172.21.15.23:0/1544128051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 0x7f57dc1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 -- 172.21.15.23:0/1544128051 >> 172.21.15.23:0/1544128051 conn(0x7f57dc0fc2f0 msgr2=0x7f57dc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 -- 172.21.15.23:0/1544128051 shutdown_connections 2024-03-30T04:01:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.122+0000 7f57e2200700 1 -- 172.21.15.23:0/1544128051 wait complete. 2024-03-30T04:01:25.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.123+0000 7f57e2200700 1 Processor -- start 2024-03-30T04:01:25.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 -- start start 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 0x7f57dc1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 0x7f57dc199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1a0bc0 con 0x7f57dc102910 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1a0d00 con 0x7f57dc100d80 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57e2200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc1a0e90 con 0x7f57dc101f70 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57dbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36990/0 (socket says 172.21.15.23:36990) 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57dbe00700 1 -- 172.21.15.23:0/4145052496 learned_addr learned my addr 172.21.15.23:0/4145052496 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57db400700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 0x7f57dc1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.124+0000 7f57daa00700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 0x7f57dc199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 -- 172.21.15.23:0/4145052496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 msgr2=0x7f57dc199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 0x7f57dc199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 -- 172.21.15.23:0/4145052496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 msgr2=0x7f57dc1990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 0x7f57dc1990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 -- 172.21.15.23:0/4145052496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d4009d00 con 0x7f57dc102910 2024-03-30T04:01:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57dbe00700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f57d4007b60 tx=0x7f57d4008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400e040 con 0x7f57dc102910 2024-03-30T04:01:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57d401d930 con 0x7f57dc102910 2024-03-30T04:01:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d401b9f0 con 0x7f57dc102910 2024-03-30T04:01:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.125+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57dc1a1110 con 0x7f57dc102910 2024-03-30T04:01:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.126+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57dc1a1540 con 0x7f57dc102910 2024-03-30T04:01:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.127+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57dc067af0 con 0x7f57dc102910 2024-03-30T04:01:25.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.129+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f57d401daa0 con 0x7f57dc102910 2024-03-30T04:01:25.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.130+0000 7f57d3e00700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 0x7f57c00a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.130+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f57d40ce860 con 0x7f57dc102910 2024-03-30T04:01:25.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.130+0000 7f57db400700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 0x7f57c00a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.131+0000 7f57db400700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 0x7f57c00a35e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f57dc101dd0 tx=0x7f57cc009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:25.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.132+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f57d4097ca0 con 0x7f57dc102910 2024-03-30T04:01:25.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:25 smithi023 ceph-mon[38254]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s 2024-03-30T04:01:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:25 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/23049910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:25 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1149446139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.348+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f57dc1a17a0 con 0x7f57dc102910 2024-03-30T04:01:25.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.349+0000 7f57d3e00700 1 -- 172.21.15.23:0/4145052496 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f57d409b2c0 con 0x7f57dc102910 2024-03-30T04:01:25.349 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 msgr2=0x7f57c00a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 0x7f57c00a35e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f57dc101dd0 tx=0x7f57cc009260 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 msgr2=0x7f57dc1a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f57d4007b60 tx=0x7f57d4008e40 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 shutdown_connections 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57dc102910 0x7f57dc1a05f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57dc100d80 0x7f57dc1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f57c00a1120 0x7f57c00a35e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 --2- 172.21.15.23:0/4145052496 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f57dc101f70 0x7f57dc199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 >> 172.21.15.23:0/4145052496 conn(0x7f57dc0fc2f0 msgr2=0x7f57dc072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:25.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 shutdown_connections 2024-03-30T04:01:25.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.351+0000 7f57e2200700 1 -- 172.21.15.23:0/4145052496 wait complete. 2024-03-30T04:01:25.370 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:25 smithi154 ceph-mon[79161]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s 2024-03-30T04:01:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:25 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/23049910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:25 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1149446139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:25 smithi193 ceph-mon[79581]: pgmap v292: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 895 B/s wr, 13 op/s 2024-03-30T04:01:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:25 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/23049910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:25 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1149446139' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 -- 172.21.15.23:0/4005043336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c100d80 msgr2=0x7fd57c1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 --2- 172.21.15.23:0/4005043336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c100d80 0x7fd57c1011a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd5700099c0 tx=0x7fd570009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 -- 172.21.15.23:0/4005043336 shutdown_connections 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 --2- 172.21.15.23:0/4005043336 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c102910 0x7fd57c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 --2- 172.21.15.23:0/4005043336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c100d80 0x7fd57c1011a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 --2- 172.21.15.23:0/4005043336 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.572+0000 7fd583e00700 1 -- 172.21.15.23:0/4005043336 >> 172.21.15.23:0/4005043336 conn(0x7fd57c0fc2f0 msgr2=0x7fd57c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.573+0000 7fd583e00700 1 -- 172.21.15.23:0/4005043336 shutdown_connections 2024-03-30T04:01:25.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.573+0000 7fd583e00700 1 -- 172.21.15.23:0/4005043336 wait complete. 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 Processor -- start 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 -- start start 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c102910 0x7fd57c1a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1a0ad0 con 0x7fd57c100d80 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1a0c10 con 0x7fd57c102910 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd583e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1a0da0 con 0x7fd57c101f70 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd581400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd581400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36996/0 (socket says 172.21.15.23:36996) 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd581400700 1 -- 172.21.15.23:0/2225941445 learned_addr learned my addr 172.21.15.23:0/2225941445 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd580a00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.574+0000 7fd581e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c102910 0x7fd57c1a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 -- 172.21.15.23:0/2225941445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 msgr2=0x7fd57c199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 -- 172.21.15.23:0/2225941445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c102910 msgr2=0x7fd57c1a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c102910 0x7fd57c1a0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 -- 172.21.15.23:0/2225941445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5700095f0 con 0x7fd57c100d80 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd580a00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd581400700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd570003730 tx=0x7fd57000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57001d070 con 0x7fd57c100d80 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c1a1020 con 0x7fd57c100d80 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd570005050 con 0x7fd57c100d80 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570004af0 con 0x7fd57c100d80 2024-03-30T04:01:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.575+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c1a14a0 con 0x7fd57c100d80 2024-03-30T04:01:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.576+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57c067af0 con 0x7fd57c100d80 2024-03-30T04:01:25.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.578+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fd5700051c0 con 0x7fd57c100d80 2024-03-30T04:01:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.579+0000 7fd57a000700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 0x7fd5640a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.580+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fd5700cda40 con 0x7fd57c100d80 2024-03-30T04:01:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.580+0000 7fd580a00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 0x7fd5640a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.580+0000 7fd580a00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 0x7fd5640a3660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd568009cb0 tx=0x7fd568009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:25.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.583+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd570096e80 con 0x7fd57c100d80 2024-03-30T04:01:25.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.800+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd57c1a1760 con 0x7fd57c100d80 2024-03-30T04:01:25.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.800+0000 7fd57a000700 1 -- 172.21.15.23:0/2225941445 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd57009a4a0 con 0x7fd57c100d80 2024-03-30T04:01:25.800 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 msgr2=0x7fd5640a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 0x7fd5640a3660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd568009cb0 tx=0x7fd568009260 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 msgr2=0x7fd57c198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd570003730 tx=0x7fd57000f690 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 shutdown_connections 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd57c102910 0x7fd57c1a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd57c100d80 0x7fd57c198ff0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5640a11a0 0x7fd5640a3660 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 --2- 172.21.15.23:0/2225941445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd57c101f70 0x7fd57c199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 >> 172.21.15.23:0/2225941445 conn(0x7fd57c0fc2f0 msgr2=0x7fd57c107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 shutdown_connections 2024-03-30T04:01:25.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:25.803+0000 7fd583e00700 1 -- 172.21.15.23:0/2225941445 wait complete. 2024-03-30T04:01:25.823 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 -- 172.21.15.23:0/3181956177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c101f70 msgr2=0x7ff78c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 --2- 172.21.15.23:0/3181956177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c101f70 0x7ff78c1023d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff7780099c0 tx=0x7ff778009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 -- 172.21.15.23:0/3181956177 shutdown_connections 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 --2- 172.21.15.23:0/3181956177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c102910 0x7ff78c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 --2- 172.21.15.23:0/3181956177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 0x7ff78c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 --2- 172.21.15.23:0/3181956177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c101f70 0x7ff78c1023d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 -- 172.21.15.23:0/3181956177 >> 172.21.15.23:0/3181956177 conn(0x7ff78c0fc2f0 msgr2=0x7ff78c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 -- 172.21.15.23:0/3181956177 shutdown_connections 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.025+0000 7ff792c00700 1 -- 172.21.15.23:0/3181956177 wait complete. 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.026+0000 7ff792c00700 1 Processor -- start 2024-03-30T04:01:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.026+0000 7ff792c00700 1 -- start start 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 0x7ff78c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c101f70 0x7ff78c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c073e80 con 0x7ff78c102910 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c073ff0 con 0x7ff78c100d80 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff792c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c1a0d10 con 0x7ff78c101f70 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37016/0 (socket says 172.21.15.23:37016) 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 -- 172.21.15.23:0/141186776 learned_addr learned my addr 172.21.15.23:0/141186776 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff78be00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 0x7ff78c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff78b400700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c101f70 0x7ff78c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 -- 172.21.15.23:0/141186776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c101f70 msgr2=0x7ff78c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c101f70 0x7ff78c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 -- 172.21.15.23:0/141186776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 msgr2=0x7ff78c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 0x7ff78c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.027+0000 7ff790c00700 1 -- 172.21.15.23:0/141186776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780095f0 con 0x7ff78c102910 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff790c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ef30 tx=0x7ff78400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff784018070 con 0x7ff78c102910 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff78400f040 con 0x7ff78c102910 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c1a0f40 con 0x7ff78c102910 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78400cca0 con 0x7ff78c102910 2024-03-30T04:01:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.028+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c1a13c0 con 0x7ff78c102910 2024-03-30T04:01:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.029+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c108200 con 0x7ff78c102910 2024-03-30T04:01:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.031+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ff784003a00 con 0x7ff78c102910 2024-03-30T04:01:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.033+0000 7ff788c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 0x7ff7700a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.033+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7ff7840cc7e0 con 0x7ff78c102910 2024-03-30T04:01:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.033+0000 7ff78be00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 0x7ff7700a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.034+0000 7ff78be00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 0x7ff7700a35e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff78c101dd0 tx=0x7ff77c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.038+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff784095c20 con 0x7ff78c102910 2024-03-30T04:01:26.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.259+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff78c02ccb0 con 0x7ff78c102910 2024-03-30T04:01:26.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.260+0000 7ff788c00700 1 -- 172.21.15.23:0/141186776 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff784099240 con 0x7ff78c102910 2024-03-30T04:01:26.260 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.262+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 msgr2=0x7ff7700a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.262+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 0x7ff7700a35e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff78c101dd0 tx=0x7ff77c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.262+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 msgr2=0x7ff78c0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.262+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff78400ef30 tx=0x7ff78400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 shutdown_connections 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff78c102910 0x7ff78c0736e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c100d80 0x7ff78c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ff7700a1120 0x7ff7700a35e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 --2- 172.21.15.23:0/141186776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff78c101f70 0x7ff78c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 >> 172.21.15.23:0/141186776 conn(0x7ff78c0fc2f0 msgr2=0x7ff78c105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 shutdown_connections 2024-03-30T04:01:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.263+0000 7ff792c00700 1 -- 172.21.15.23:0/141186776 wait complete. 2024-03-30T04:01:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:26 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4145052496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:26 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2225941445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:26 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4145052496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:26 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2225941445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 -- 172.21.15.23:0/2595541055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674100d80 msgr2=0x7fa6741011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 --2- 172.21.15.23:0/2595541055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674100d80 0x7fa6741011a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa6640099c0 tx=0x7fa664009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 -- 172.21.15.23:0/2595541055 shutdown_connections 2024-03-30T04:01:26.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 --2- 172.21.15.23:0/2595541055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674102910 0x7fa6741091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 --2- 172.21.15.23:0/2595541055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674100d80 0x7fa6741011a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 --2- 172.21.15.23:0/2595541055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 0x7fa6741023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 -- 172.21.15.23:0/2595541055 >> 172.21.15.23:0/2595541055 conn(0x7fa6740fc2f0 msgr2=0x7fa6740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 -- 172.21.15.23:0/2595541055 shutdown_connections 2024-03-30T04:01:26.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.483+0000 7fa67ac00700 1 -- 172.21.15.23:0/2595541055 wait complete. 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.484+0000 7fa67ac00700 1 Processor -- start 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 -- start start 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674100d80 0x7fa674074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 0x7fa6740731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674073de0 con 0x7fa674102910 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa674073f50 con 0x7fa674100d80 2024-03-30T04:01:26.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa67ac00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741a0dc0 con 0x7fa674101f70 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa678c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa678c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37030/0 (socket says 172.21.15.23:37030) 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa678c00700 1 -- 172.21.15.23:0/4009127549 learned_addr learned my addr 172.21.15.23:0/4009127549 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa673400700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 0x7fa6740731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.485+0000 7fa673e00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674100d80 0x7fa674074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 -- 172.21.15.23:0/4009127549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 msgr2=0x7fa6740731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 0x7fa6740731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 -- 172.21.15.23:0/4009127549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674100d80 msgr2=0x7fa674074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674100d80 0x7fa674074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 -- 172.21.15.23:0/4009127549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640095f0 con 0x7fa674102910 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa678c00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00ef30 tx=0x7fa66c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c018070 con 0x7fa674102910 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa66c00f040 con 0x7fa674102910 2024-03-30T04:01:26.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6741a0fc0 con 0x7fa674102910 2024-03-30T04:01:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c00cd00 con 0x7fa674102910 2024-03-30T04:01:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.486+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6741a1490 con 0x7fa674102910 2024-03-30T04:01:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.487+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa674108250 con 0x7fa674102910 2024-03-30T04:01:26.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.489+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fa66c003a00 con 0x7fa674102910 2024-03-30T04:01:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.490+0000 7fa670c00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 0x7fa6580a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.491+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fa66c0cc730 con 0x7fa674102910 2024-03-30T04:01:26.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.491+0000 7fa673e00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 0x7fa6580a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.491+0000 7fa673e00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 0x7fa6580a3660 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa664009fd0 tx=0x7fa664019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:26 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4145052496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:26 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2225941445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:26.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.495+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa66c095b70 con 0x7fa674102910 2024-03-30T04:01:26.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.717+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa67402ccb0 con 0x7fa674102910 2024-03-30T04:01:26.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.718+0000 7fa670c00700 1 -- 172.21.15.23:0/4009127549 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa66c099190 con 0x7fa674102910 2024-03-30T04:01:26.717 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 msgr2=0x7fa6580a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 0x7fa6580a3660 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa664009fd0 tx=0x7fa664019040 comp rx=0 tx=0).stop 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 msgr2=0x7fa6740736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa66c00ef30 tx=0x7fa66c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 shutdown_connections 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa674102910 0x7fa6740736e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa674100d80 0x7fa674074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa6580a11a0 0x7fa6580a3660 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 --2- 172.21.15.23:0/4009127549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa674101f70 0x7fa6740731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 >> 172.21.15.23:0/4009127549 conn(0x7fa6740fc2f0 msgr2=0x7fa674105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:26.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 shutdown_connections 2024-03-30T04:01:26.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.720+0000 7fa67ac00700 1 -- 172.21.15.23:0/4009127549 wait complete. 2024-03-30T04:01:26.740 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 -- 172.21.15.23:0/415240134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 msgr2=0x7fc1841090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 --2- 172.21.15.23:0/415240134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1841090c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc17c00b0d0 tx=0x7fc17c00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 -- 172.21.15.23:0/415240134 shutdown_connections 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 --2- 172.21.15.23:0/415240134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1841090c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 --2- 172.21.15.23:0/415240134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 0x7fc1841010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 --2- 172.21.15.23:0/415240134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 0x7fc1841022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 -- 172.21.15.23:0/415240134 >> 172.21.15.23:0/415240134 conn(0x7fc1840fc1f0 msgr2=0x7fc1840fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:26.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 -- 172.21.15.23:0/415240134 shutdown_connections 2024-03-30T04:01:26.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.939+0000 7fc18a600700 1 -- 172.21.15.23:0/415240134 wait complete. 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.940+0000 7fc18a600700 1 Processor -- start 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 -- start start 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 0x7fc184074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 0x7fc1840731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184073e80 con 0x7fc184102810 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc184073ff0 con 0x7fc184101e70 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc18a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1841a0d50 con 0x7fc184100c80 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc183e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc183400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 0x7fc184074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc183e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37062/0 (socket says 172.21.15.23:37062) 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc183e00700 1 -- 172.21.15.23:0/3138229870 learned_addr learned my addr 172.21.15.23:0/3138229870 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.941+0000 7fc182a00700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 0x7fc1840731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 -- 172.21.15.23:0/3138229870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 msgr2=0x7fc184074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 0x7fc184074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 -- 172.21.15.23:0/3138229870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 msgr2=0x7fc1840731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 0x7fc1840731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 -- 172.21.15.23:0/3138229870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c009d00 con 0x7fc184102810 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc183e00700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007b60 tx=0x7fc17c008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c00e040 con 0x7fc184102810 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc17c01d930 con 0x7fc184102810 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1841a0f80 con 0x7fc184102810 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c01b9f0 con 0x7fc184102810 2024-03-30T04:01:26.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.942+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841a1340 con 0x7fc184102810 2024-03-30T04:01:26.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.943+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1841080b0 con 0x7fc184102810 2024-03-30T04:01:26.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.947+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fc17c01daa0 con 0x7fc184102810 2024-03-30T04:01:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.948+0000 7fc17be00700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 0x7fc1680a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.948+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fc17c0ce6c0 con 0x7fc184102810 2024-03-30T04:01:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.948+0000 7fc183400700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 0x7fc1680a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:26.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.949+0000 7fc183400700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 0x7fc1680a3660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc184101cd0 tx=0x7fc174009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:26.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:26.950+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc17c097b60 con 0x7fc184102810 2024-03-30T04:01:27.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.221+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc184067a10 con 0x7fc184102810 2024-03-30T04:01:27.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.222+0000 7fc17be00700 1 -- 172.21.15.23:0/3138229870 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fc17c09b180 con 0x7fc184102810 2024-03-30T04:01:27.221 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:27.221 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:00:13.957636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:27.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 msgr2=0x7fc1680a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 0x7fc1680a3660 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc184101cd0 tx=0x7fc174009260 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 msgr2=0x7fc1840736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc17c007b60 tx=0x7fc17c008e40 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 shutdown_connections 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc184102810 0x7fc1840736e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc1680a11a0 0x7fc1680a3660 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.224+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc184100c80 0x7fc184074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.225+0000 7fc18a600700 1 --2- 172.21.15.23:0/3138229870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc184101e70 0x7fc1840731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.225+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 >> 172.21.15.23:0/3138229870 conn(0x7fc1840fc1f0 msgr2=0x7fc1841051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.225+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 shutdown_connections 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.225+0000 7fc18a600700 1 -- 172.21.15.23:0/3138229870 wait complete. 2024-03-30T04:01:27.225 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 27 2024-03-30T04:01:27.243 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:27.301 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:27 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/141186776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.301 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:27 smithi023 ceph-mon[38254]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 615 B/s wr, 11 op/s 2024-03-30T04:01:27.301 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:27 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4009127549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:27 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/141186776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:27 smithi154 ceph-mon[79161]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 615 B/s wr, 11 op/s 2024-03-30T04:01:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:27 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4009127549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 -- 172.21.15.23:0/1948764812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930101e70 msgr2=0x7f79301022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 --2- 172.21.15.23:0/1948764812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930101e70 0x7f79301022d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f791c009920 tx=0x7f791c009c30 comp rx=0 tx=0).stop 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 -- 172.21.15.23:0/1948764812 shutdown_connections 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 --2- 172.21.15.23:0/1948764812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930102810 0x7f79301090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 --2- 172.21.15.23:0/1948764812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930100c80 0x7f79301010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 --2- 172.21.15.23:0/1948764812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930101e70 0x7f79301022d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 -- 172.21.15.23:0/1948764812 >> 172.21.15.23:0/1948764812 conn(0x7f79300fc1f0 msgr2=0x7f79300fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 -- 172.21.15.23:0/1948764812 shutdown_connections 2024-03-30T04:01:27.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.443+0000 7f7937800700 1 -- 172.21.15.23:0/1948764812 wait complete. 2024-03-30T04:01:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 Processor -- start 2024-03-30T04:01:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 -- start start 2024-03-30T04:01:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930100c80 0x7f7930074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930102810 0x7f79300736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7930073e80 con 0x7f7930101e70 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7930073ff0 con 0x7f7930102810 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.444+0000 7f7937800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79301a0ce0 con 0x7f7930100c80 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f7935800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930102810 0x7f79300736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37088/0 (socket says 172.21.15.23:37088) 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 -- 172.21.15.23:0/126018420 learned_addr learned my addr 172.21.15.23:0/126018420 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f7934e00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930100c80 0x7f7930074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 -- 172.21.15.23:0/126018420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930100c80 msgr2=0x7f7930074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930100c80 0x7f7930074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 -- 172.21.15.23:0/126018420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930102810 msgr2=0x7f79300736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930102810 0x7f79300736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 -- 172.21.15.23:0/126018420 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c009520 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792fe00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f791c009840 tx=0x7f791c00b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.445+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c01c070 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.446+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79301a0f10 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.446+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f791c00f7d0 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.446+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c003b60 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.446+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79301a12d0 con 0x7f7930101e70 2024-03-30T04:01:27.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.447+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79301081a0 con 0x7f7930101e70 2024-03-30T04:01:27.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.449+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f791c003cc0 con 0x7f7930101e70 2024-03-30T04:01:27.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.450+0000 7f792d600700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 0x7f79180a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.450+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f791c0cd980 con 0x7f7930101e70 2024-03-30T04:01:27.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.450+0000 7f7934e00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 0x7f79180a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.451+0000 7f7934e00700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 0x7f79180a35e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7930101cd0 tx=0x7f7924009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.454+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f791c096dc0 con 0x7f7930101e70 2024-03-30T04:01:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:27 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/141186776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:27 smithi193 ceph-mon[79581]: pgmap v293: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 615 B/s wr, 11 op/s 2024-03-30T04:01:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:27 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4009127549' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:27.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.710+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7930067a10 con 0x7f7930101e70 2024-03-30T04:01:27.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.712+0000 7f792d600700 1 -- 172.21.15.23:0/126018420 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f791c09a3e0 con 0x7f7930101e70 2024-03-30T04:01:27.711 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:27.712 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 msgr2=0x7f79180a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 0x7f79180a35e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7930101cd0 tx=0x7f7924009310 comp rx=0 tx=0).stop 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 msgr2=0x7f79300731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f791c009840 tx=0x7f791c00b790 comp rx=0 tx=0).stop 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 shutdown_connections 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7930102810 0x7f79300736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f79180a1120 0x7f79180a35e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7930100c80 0x7f7930074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 --2- 172.21.15.23:0/126018420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7930101e70 0x7f79300731a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 >> 172.21.15.23:0/126018420 conn(0x7f79300fc1f0 msgr2=0x7f79301052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.714+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 shutdown_connections 2024-03-30T04:01:27.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.715+0000 7f7937800700 1 -- 172.21.15.23:0/126018420 wait complete. 2024-03-30T04:01:27.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-30T03:59:40.432862+0000', 'modified': '2024-03-30T04:00:13.957636+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24569, 'mds_1': 24533, 'mds_2': 24509, 'mds_3': 24508, 'mds_4': 24476}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14544': {'gid': 14544, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/667186519', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 667186519}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 667186519}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14556': {'gid': 14556, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6841/2954889723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 2954889723}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 2954889723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24457': {'gid': 24457, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/2731954066', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 2731954066}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 2731954066}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24476': {'gid': 24476, 'name': 'k', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.154:6833/3615295039', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6832', 'nonce': 3615295039}, {'type': 'v1', 'addr': '172.21.15.154:6833', 'nonce': 3615295039}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.193:6835/2492108108', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 2492108108}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 2492108108}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24509': {'gid': 24509, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.154:6835/1153342637', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6834', 'nonce': 1153342637}, {'type': 'v1', 'addr': '172.21.15.154:6835', 'nonce': 1153342637}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24514': {'gid': 24514, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6837/1829293538', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1829293538}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1829293538}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6839/606583058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 606583058}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 606583058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.154:6837/3317012042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6836', 'nonce': 3317012042}, {'type': 'v1', 'addr': '172.21.15.154:6837', 'nonce': 3317012042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.154:6839/2911311868', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6838', 'nonce': 2911311868}, {'type': 'v1', 'addr': '172.21.15.154:6839', 'nonce': 2911311868}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-30T04:01:27.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-30T04:01:27.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-30T04:01:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 -- 172.21.15.23:0/3098544765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4102910 msgr2=0x7f01c41091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 --2- 172.21.15.23:0/3098544765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4102910 0x7f01c41091c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00f560 tx=0x7f01bc00f870 comp rx=0 tx=0).stop 2024-03-30T04:01:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 -- 172.21.15.23:0/3098544765 shutdown_connections 2024-03-30T04:01:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 --2- 172.21.15.23:0/3098544765 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4102910 0x7f01c41091c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 --2- 172.21.15.23:0/3098544765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 0x7f01c41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 --2- 172.21.15.23:0/3098544765 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4101f70 0x7f01c41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 -- 172.21.15.23:0/3098544765 >> 172.21.15.23:0/3098544765 conn(0x7f01c40fc2f0 msgr2=0x7f01c40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 -- 172.21.15.23:0/3098544765 shutdown_connections 2024-03-30T04:01:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.935+0000 7f01c9200700 1 -- 172.21.15.23:0/3098544765 wait complete. 2024-03-30T04:01:27.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.936+0000 7f01c9200700 1 Processor -- start 2024-03-30T04:01:27.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.936+0000 7f01c9200700 1 -- start start 2024-03-30T04:01:27.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 0x7f01c4194c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4102910 0x7f01c419c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419c760 con 0x7f01c4101f70 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419c8d0 con 0x7f01c4102910 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c9200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419ca90 con 0x7f01c4100d80 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37106/0 (socket says 172.21.15.23:37106) 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 -- 172.21.15.23:0/4112611590 learned_addr learned my addr 172.21.15.23:0/4112611590 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1e00700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 0x7f01c4194c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c2800700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4102910 0x7f01c419c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 -- 172.21.15.23:0/4112611590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 msgr2=0x7f01c4194c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 0x7f01c4194c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 -- 172.21.15.23:0/4112611590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4102910 msgr2=0x7f01c419c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4102910 0x7f01c419c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.937+0000 7f01c1400700 1 -- 172.21.15.23:0/4112611590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01bc009d00 con 0x7f01c4101f70 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01c1400700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f01b000e9a0 tx=0x7f01b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01baa00700 1 -- 172.21.15.23:0/4112611590 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0009790 con 0x7f01c4101f70 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01baa00700 1 -- 172.21.15.23:0/4112611590 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b0010430 con 0x7f01c4101f70 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01c419cd70 con 0x7f01c4101f70 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01baa00700 1 -- 172.21.15.23:0/4112611590 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0018680 con 0x7f01c4101f70 2024-03-30T04:01:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.938+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c419d1d0 con 0x7f01c4101f70 2024-03-30T04:01:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.941+0000 7f01baa00700 1 -- 172.21.15.23:0/4112611590 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f01b00105a0 con 0x7f01c4101f70 2024-03-30T04:01:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.942+0000 7f01dcc70b80 1 Processor -- start 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01baa00700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 0x7f01a80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01baa00700 1 -- 172.21.15.23:0/4112611590 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f01b0014070 con 0x7f01c4101f70 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 -- start start 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01c2800700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 0x7f01a80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f269a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f27400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e7d940 con 0x561f90f25780 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e7b300 con 0x561f90f26f80 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e76bf0 con 0x561f90f260e0 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.943+0000 7f01c1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37108/0 (socket says 172.21.15.23:37108) 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 -- 172.21.15.23:0/1333947867 learned_addr learned my addr 172.21.15.23:0/1333947867 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c2800700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 0x7f01a80a35e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00f530 tx=0x7f01bc006000 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1400700 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f269a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c2800700 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f27400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 -- 172.21.15.23:0/1333947867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 msgr2=0x561f90f269a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f269a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 -- 172.21.15.23:0/1333947867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 msgr2=0x561f90f27400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f27400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 -- 172.21.15.23:0/1333947867 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f90cf8360 con 0x561f90f25780 2024-03-30T04:01:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01c1e00700 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cf0 tx=0x7f01b4011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80485deb9cfcf7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.944+0000 7f01af400700 1 -- 172.21.15.23:0/1333947867 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4011d80 con 0x561f90f25780 2024-03-30T04:01:27.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01af400700 1 -- 172.21.15.23:0/1333947867 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b40044e0 con 0x561f90f25780 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01af400700 1 -- 172.21.15.23:0/1333947867 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b4005280 con 0x561f90f25780 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 -- 172.21.15.23:0/1333947867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 msgr2=0x561f90f25ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f01b4009cf0 tx=0x7f01b4011440 comp rx=0 tx=0).stop 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 -- 172.21.15.23:0/1333947867 shutdown_connections 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f27400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f25780 0x561f90f25ba0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/1333947867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f269a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 -- 172.21.15.23:0/1333947867 >> 172.21.15.23:0/1333947867 conn(0x561f90e84650 msgr2=0x561f90f20950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 -- 172.21.15.23:0/1333947867 shutdown_connections 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.945+0000 7f01dcc70b80 1 -- 172.21.15.23:0/1333947867 wait complete. 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.946+0000 7f01dcc70b80 1 -- start start 2024-03-30T04:01:27.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.946+0000 7f01dcc70b80 1 Processor -- start 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.946+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.946+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f2fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.946+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f26f80 0x561f90f36b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e12d30 con 0x561f90f26f80 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e129d0 con 0x561f90f25780 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90cfa650 con 0x561f90f260e0 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01dcc70b80 10 client.0 _ll_register_callbacks cb 0x561f90e1c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:45444/0 (socket says 172.21.15.23:45444) 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 -- 172.21.15.23:0/2844775221 learned_addr learned my addr 172.21.15.23:0/2844775221 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c2800700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f26f80 0x561f90f36b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1400700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f2fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 -- 172.21.15.23:0/2844775221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 msgr2=0x561f90f2fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f2fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 -- 172.21.15.23:0/2844775221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f26f80 msgr2=0x561f90f36b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f26f80 0x561f90f36b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 -- 172.21.15.23:0/2844775221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b4009920 con 0x561f90f25780 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1400700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f2fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01c1e00700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011db0 tx=0x7f01b4011de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c379be7fcdd246f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.947+0000 7f01a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b40050e0 con 0x561f90f25780 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f90e37330 con 0x561f90f25780 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f01b401f040 con 0x561f90f25780 2024-03-30T04:01:27.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b4005240 con 0x561f90f25780 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01dcc70b80 10 client.25358 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.948+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561f90ca4b10 con 0x561f90f25780 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561f90e37330 con 0x561f90f25780 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f01b4024cf0 con 0x561f90f25780 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 10 client.25358 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 10 client.25358 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 20 client.25358 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f3a770 0x561f90f3fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 4 client.25358 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f90e6f6b0 con 0x561f90f3a770 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.949+0000 7f01c1400700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f3a770 0x561f90f3fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.950+0000 7f01c1400700 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f3a770 0x561f90f3fa90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3c8b9f3372d420ca server_cookie=d5befa86836cf90 in_seq=0 out_seq=0 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.950+0000 7f01a3e00700 10 client.25358 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.950+0000 7f01a3e00700 1 -- 172.21.15.23:0/2844775221 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561f90e6f6b0 con 0x561f90f3a770 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.950+0000 7f01a3e00700 10 client.25358 handle_command_reply: tid=0 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.950+0000 7f01dcc70b80 1 client.25358 shutdown 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 msgr2=0x561f90f2f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f01b4011db0 tx=0x7f01b4011de0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 shutdown_connections 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f26f80 0x561f90f36b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f2f630 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f3a770 0x561f90f3fa90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/2844775221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f260e0 0x561f90f2fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 >> 172.21.15.23:0/2844775221 conn(0x561f90e84650 msgr2=0x561f90f24120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01a3e00700 0 client.25358 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 shutdown_connections 2024-03-30T04:01:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 1 -- 172.21.15.23:0/2844775221 wait complete. 2024-03-30T04:01:27.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.951+0000 7f01dcc70b80 20 client.25358 trim_cache size 0 max 16384 2024-03-30T04:01:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.969+0000 7f01dcc70b80 1 Processor -- start 2024-03-30T04:01:27.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.969+0000 7f01dcc70b80 1 -- start start 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 0x561f90f25b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f36ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e73850 con 0x561f90f260e0 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90e15de0 con 0x561f90f26f80 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90f37c90 con 0x561f90f25780 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37126/0 (socket says 172.21.15.23:37126) 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 -- 172.21.15.23:0/4232548924 learned_addr learned my addr 172.21.15.23:0/4232548924 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c2800700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f36ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1e00700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 0x561f90f25b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 -- 172.21.15.23:0/4232548924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 msgr2=0x561f90f25b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 0x561f90f25b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 -- 172.21.15.23:0/4232548924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 msgr2=0x561f90f36ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f36ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1400700 1 -- 172.21.15.23:0/4232548924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f90e3ce50 con 0x561f90f260e0 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.970+0000 7f01c1e00700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 0x561f90f25b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01c1400700 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01b00963e0 tx=0x7f01b00cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7923dedbdf6ac419 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01afe00700 1 -- 172.21.15.23:0/4232548924 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00a2360 con 0x561f90f260e0 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01afe00700 1 -- 172.21.15.23:0/4232548924 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b00ca6a0 con 0x561f90f260e0 2024-03-30T04:01:27.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01afe00700 1 -- 172.21.15.23:0/4232548924 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b0099630 con 0x561f90f260e0 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01dcc70b80 1 -- 172.21.15.23:0/4232548924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 msgr2=0x561f90f371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.971+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01b00963e0 tx=0x7f01b00cc040 comp rx=0 tx=0).stop 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 -- 172.21.15.23:0/4232548924 shutdown_connections 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f26f80 0x561f90f36ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f25780 0x561f90f25b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/4232548924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f371c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 -- 172.21.15.23:0/4232548924 >> 172.21.15.23:0/4232548924 conn(0x561f90e84650 msgr2=0x561f90f2eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 -- 172.21.15.23:0/4232548924 shutdown_connections 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 -- 172.21.15.23:0/4232548924 wait complete. 2024-03-30T04:01:27.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 -- start start 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.972+0000 7f01dcc70b80 1 Processor -- start 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f33cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 0x561f90f8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90f36c00 con 0x561f90f260e0 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90f36d70 con 0x561f90f25780 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f90f36ee0 con 0x561f90f26f80 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01dcc70b80 10 client.0 _ll_register_callbacks cb 0x561f90e1c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37134/0 (socket says 172.21.15.23:37134) 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 -- 172.21.15.23:0/720644090 learned_addr learned my addr 172.21.15.23:0/720644090 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:27.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1e00700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f33cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c2800700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 0x561f90f8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 -- 172.21.15.23:0/720644090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 msgr2=0x561f90f8f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 0x561f90f8f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 -- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 msgr2=0x561f90f33cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f33cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c1400700 1 -- 172.21.15.23:0/720644090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c419d1d0 con 0x561f90f260e0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.973+0000 7f01c2800700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 0x561f90f8f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01c1400700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01b0099b90 tx=0x7f01b00092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98cec4affaa1e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01a3e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00993d0 con 0x561f90f260e0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f01b0099530 con 0x561f90f260e0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01b40044e0 con 0x561f90f260e0 2024-03-30T04:01:27.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01b00a1ed0 con 0x561f90f260e0 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01dcc70b80 10 client.15354 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561f90cfcc80 con 0x561f90f260e0 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.974+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01b40044e0 con 0x561f90f260e0 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f01b0098e40 con 0x561f90f260e0 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 10 client.15354 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 10 client.15354 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 20 client.15354 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f93f00 0x561f90f908d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 4 client.15354 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561f90cc5ca0 con 0x561f90f93f00 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.975+0000 7f01c1e00700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f93f00 0x561f90f908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:27.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01a3e00700 10 client.15354 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01c1e00700 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f93f00 0x561f90f908d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a996f2bc43db197d server_cookie=53dad177b672514d in_seq=0 out_seq=0 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01a3e00700 1 -- 172.21.15.23:0/720644090 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x561f90cc5ca0 con 0x561f90f93f00 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01a3e00700 10 client.15354 handle_command_reply: tid=0 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 client.15354 shutdown 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 msgr2=0x561f90f341f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f01b0099b90 tx=0x7f01b00092d0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 shutdown_connections 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561f90f93f00 0x561f90f908d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f90f26f80 0x561f90f8f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f90f25780 0x561f90f33cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 --2- 172.21.15.23:0/720644090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f90f260e0 0x561f90f341f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 >> 172.21.15.23:0/720644090 conn(0x561f90e84650 msgr2=0x561f90f2a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01a3e00700 0 client.15354 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 shutdown_connections 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 1 -- 172.21.15.23:0/720644090 wait complete. 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.976+0000 7f01dcc70b80 20 client.15354 trim_cache size 0 max 16384 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24476, 2024-03-30T04:01:27.983 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 75.025275789000005, 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 135.115865077 2024-03-30T04:01:27.984 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 msgr2=0x7f01a80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 0x7f01a80a35e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f01bc00f530 tx=0x7f01bc006000 comp rx=0 tx=0).stop 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 msgr2=0x7f01c41951b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f01b000e9a0 tx=0x7f01b000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 shutdown_connections 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f01c4102910 0x7f01c419c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f01a80a1120 0x7f01a80a35e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f01c4100d80 0x7f01c4194c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 --2- 172.21.15.23:0/4112611590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01c4101f70 0x7f01c41951b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:27.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 >> 172.21.15.23:0/4112611590 conn(0x7f01c40fc2f0 msgr2=0x7f01c41052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:27.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 shutdown_connections 2024-03-30T04:01:27.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:27.977+0000 7f01c9200700 1 -- 172.21.15.23:0/4112611590 wait complete. 2024-03-30T04:01:27.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-30T04:01:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 -- 172.21.15.23:0/4033281837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0100d80 msgr2=0x7facd01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 --2- 172.21.15.23:0/4033281837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0100d80 0x7facd01011a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7facc4009920 tx=0x7facc4009c30 comp rx=0 tx=0).stop 2024-03-30T04:01:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 -- 172.21.15.23:0/4033281837 shutdown_connections 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 --2- 172.21.15.23:0/4033281837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0102910 0x7facd01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 --2- 172.21.15.23:0/4033281837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0100d80 0x7facd01011a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 --2- 172.21.15.23:0/4033281837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0101f70 0x7facd01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.197+0000 7facd7800700 1 -- 172.21.15.23:0/4033281837 >> 172.21.15.23:0/4033281837 conn(0x7facd00fc2f0 msgr2=0x7facd00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.198+0000 7facd7800700 1 -- 172.21.15.23:0/4033281837 shutdown_connections 2024-03-30T04:01:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.198+0000 7facd7800700 1 -- 172.21.15.23:0/4033281837 wait complete. 2024-03-30T04:01:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.198+0000 7facd7800700 1 Processor -- start 2024-03-30T04:01:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 -- start start 2024-03-30T04:01:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0101f70 0x7facd0199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 0x7facd01a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd01a0b50 con 0x7facd0100d80 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd01a0c90 con 0x7facd0102910 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd7800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd01a0e20 con 0x7facd0101f70 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49384/0 (socket says 172.21.15.23:49384) 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd4e00700 1 -- 172.21.15.23:0/3994481364 learned_addr learned my addr 172.21.15.23:0/3994481364 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7faccfe00700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0101f70 0x7facd0199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.199+0000 7facd5800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 0x7facd01a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 -- 172.21.15.23:0/3994481364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0101f70 msgr2=0x7facd0199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0101f70 0x7facd0199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 -- 172.21.15.23:0/3994481364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 msgr2=0x7facd01a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 0x7facd01a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 -- 172.21.15.23:0/3994481364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc4009520 con 0x7facd0100d80 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd5800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 0x7facd01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd4e00700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7facc4009230 tx=0x7facc400bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7faccd600700 1 -- 172.21.15.23:0/3994481364 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc401c070 con 0x7facd0100d80 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7faccd600700 1 -- 172.21.15.23:0/3994481364 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facc4005040 con 0x7facd0100d80 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd01a10a0 con 0x7facd0100d80 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7faccd600700 1 -- 172.21.15.23:0/3994481364 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40049e0 con 0x7facd0100d80 2024-03-30T04:01:28.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.200+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd01a1470 con 0x7facd0100d80 2024-03-30T04:01:28.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.205+0000 7faccd600700 1 -- 172.21.15.23:0/3994481364 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7facc4020070 con 0x7facd0100d80 2024-03-30T04:01:28.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 Processor -- start 2024-03-30T04:01:28.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 -- start start 2024-03-30T04:01:28.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faccd600700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 0x7facb80a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x5640181899a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x56401818a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640180e0940 con 0x5640181890e0 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640180de300 con 0x564018189f80 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faceb21fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640180d9bf0 con 0x564018188780 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.206+0000 7faccd600700 1 -- 172.21.15.23:0/3994481364 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7facc40cd900 con 0x7facd0100d80 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7faccfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x5640181899a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:58436/0 (socket says 172.21.15.23:58436) 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 -- 172.21.15.23:0/285465182 learned_addr learned my addr 172.21.15.23:0/285465182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd5800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 0x7facb80a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd5800700 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x56401818a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 -- 172.21.15.23:0/285465182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 msgr2=0x56401818a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x56401818a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 -- 172.21.15.23:0/285465182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 msgr2=0x5640181899a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x5640181899a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 -- 172.21.15.23:0/285465182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564017f5b360 con 0x564018188780 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7faccfe00700 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x5640181899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.207+0000 7facd4e00700 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7facc409a700 tx=0x7facc402a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbae2ff362e64497 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7facd5800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 0x7facb80a35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x564018189760 tx=0x7facc8009920 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7facc2000700 1 -- 172.21.15.23:0/285465182 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc409b100 con 0x564018188780 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7facc2000700 1 -- 172.21.15.23:0/285465182 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facc400e3e0 con 0x564018188780 2024-03-30T04:01:28.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7facc2000700 1 -- 172.21.15.23:0/285465182 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc409e8c0 con 0x564018188780 2024-03-30T04:01:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 -- 172.21.15.23:0/285465182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 msgr2=0x564018188ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7facc409a700 tx=0x7facc402a000 comp rx=0 tx=0).stop 2024-03-30T04:01:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 -- 172.21.15.23:0/285465182 shutdown_connections 2024-03-30T04:01:28.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x56401818a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x564018188ba0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 --2- 172.21.15.23:0/285465182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x5640181899a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 -- 172.21.15.23:0/285465182 >> 172.21.15.23:0/285465182 conn(0x5640180e7650 msgr2=0x564018183950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.208+0000 7faceb21fb80 1 -- 172.21.15.23:0/285465182 shutdown_connections 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.209+0000 7faceb21fb80 1 -- 172.21.15.23:0/285465182 wait complete. 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.209+0000 7faceb21fb80 1 -- start start 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.209+0000 7faceb21fb80 1 Processor -- start 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x5640181927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x5640181997f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564018075d30 con 0x564018188780 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640180759d0 con 0x564018189f80 2024-03-30T04:01:28.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564017f5d650 con 0x5640181890e0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faceb21fb80 10 client.0 _ll_register_callbacks cb 0x56401807f420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49408/0 (socket says 172.21.15.23:49408) 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 -- 172.21.15.23:0/700010137 learned_addr learned my addr 172.21.15.23:0/700010137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7faccfe00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x5640181927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd5800700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x5640181997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 -- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 msgr2=0x5640181927b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x5640181927b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 -- 172.21.15.23:0/700010137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 msgr2=0x5640181997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x5640181997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd4e00700 1 -- 172.21.15.23:0/700010137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc009920 con 0x564018188780 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.210+0000 7facd5800700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x5640181997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facd4e00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7facc400a010 tx=0x7facc409b1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd363ca3f2d9576 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facb6a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:28.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40051b0 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7facc4005490 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56401809a330 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7facc4005310 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7faceb21fb80 10 client.15366 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7facd01a1470 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.211+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56401809a330 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7facc409ad80 con 0x564018188780 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 10 client.15366 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 10 client.15366 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 20 client.15366 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56401819cf20 0x5640181a21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 4 client.15366 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640180d26b0 con 0x56401819cf20 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.212+0000 7faccfe00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56401819cf20 0x5640181a21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.213+0000 7faccfe00700 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56401819cf20 0x5640181a21e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=31917ad945eb8584 server_cookie=d5e9f917901c7334 in_seq=0 out_seq=0 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.213+0000 7facb6a00700 10 client.15366 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7facb6a00700 1 -- 172.21.15.23:0/700010137 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5640180d26b0 con 0x56401819cf20 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7facb6a00700 10 client.15366 handle_command_reply: tid=0 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 client.15366 shutdown 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 msgr2=0x564018192270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7facc400a010 tx=0x7facc409b1f0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 shutdown_connections 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56401819cf20 0x5640181a21e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x5640181997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018188780 0x564018192270 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7facb6a00700 0 client.15366 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 --2- 172.21.15.23:0/700010137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x5640181927b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 >> 172.21.15.23:0/700010137 conn(0x5640180e7650 msgr2=0x564018187120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 shutdown_connections 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 1 -- 172.21.15.23:0/700010137 wait complete. 2024-03-30T04:01:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.214+0000 7faceb21fb80 20 client.15366 trim_cache size 0 max 16384 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 Processor -- start 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 -- start start 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018188780 0x564018188b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x564018199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56401819e970 con 0x564018189f80 2024-03-30T04:01:28.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56401819b500 con 0x564018188780 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7faceb21fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640180e0940 con 0x5640181890e0 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.233+0000 7facd5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7facd5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49414/0 (socket says 172.21.15.23:49414) 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7facd5800700 1 -- 172.21.15.23:0/1280896060 learned_addr learned my addr 172.21.15.23:0/1280896060 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x564018199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7facd4e00700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018188780 0x564018188b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 -- 172.21.15.23:0/1280896060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018188780 msgr2=0x564018188b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018188780 0x564018188b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 -- 172.21.15.23:0/1280896060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 msgr2=0x564018199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 -- 172.21.15.23:0/1280896060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564017f07b10 con 0x5640181890e0 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7facd5800700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7faccfe00700 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x564018199ea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56401819ef30 tx=0x7facbc005900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e3f861b5ffe8fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.234+0000 7facc2a00700 1 -- 172.21.15.23:0/1280896060 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc004230 con 0x5640181890e0 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7facc2a00700 1 -- 172.21.15.23:0/1280896060 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facbc018040 con 0x5640181890e0 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7facc2a00700 1 -- 172.21.15.23:0/1280896060 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc011dc0 con 0x5640181890e0 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 -- 172.21.15.23:0/1280896060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 msgr2=0x564018199ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x564018199ea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x56401819ef30 tx=0x7facbc005900 comp rx=0 tx=0).stop 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 -- 172.21.15.23:0/1280896060 shutdown_connections 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564018189f80 0x564018199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018188780 0x564018188b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1280896060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5640181890e0 0x564018199ea0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 -- 172.21.15.23:0/1280896060 >> 172.21.15.23:0/1280896060 conn(0x5640180e7650 msgr2=0x56401817c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 -- 172.21.15.23:0/1280896060 shutdown_connections 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.235+0000 7faceb21fb80 1 -- 172.21.15.23:0/1280896060 wait complete. 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.236+0000 7faceb21fb80 1 -- start start 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.236+0000 7faceb21fb80 1 Processor -- start 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.236+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x5640181996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.236+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faceb21fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x564018196760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faceb21fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564018196d80 con 0x5640181890e0 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faceb21fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640181f17d0 con 0x564018189f80 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faceb21fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640181f1910 con 0x564018188780 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faceb21fb80 10 client.0 _ll_register_callbacks cb 0x56401807f420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49430/0 (socket says 172.21.15.23:49430) 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 -- 172.21.15.23:0/1247723820 learned_addr learned my addr 172.21.15.23:0/1247723820 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7facd5800700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x564018196760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7facd4e00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x5640181996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 -- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 msgr2=0x5640181996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x5640181996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 -- 172.21.15.23:0/1247723820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 msgr2=0x564018196760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x564018196760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.237+0000 7faccfe00700 1 -- 172.21.15.23:0/1247723820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc80090d0 con 0x5640181890e0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7faccfe00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7facbc022040 tx=0x7facbc011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66c141c898b2e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facbc004030 con 0x5640181890e0 2024-03-30T04:01:28.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7facc400e3e0 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7facbc004190 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7facbc004350 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7faceb21fb80 10 client.15372 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7facbc01f9f0 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7facc400e3e0 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.238+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7facbc02e030 con 0x5640181890e0 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 10 client.15372 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 10 client.15372 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 20 client.15372 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5640181f6640 0x5640181f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 4 client.15372 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564017f28ca0 con 0x5640181f6640 2024-03-30T04:01:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7facd4e00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5640181f6640 0x5640181f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7facd4e00700 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5640181f6640 0x5640181f2fb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5a69a0beafcb978d server_cookie=2bb53f5ce1d19804 in_seq=0 out_seq=0 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.239+0000 7facb6a00700 10 client.15372 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7facb6a00700 1 -- 172.21.15.23:0/1247723820 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564017f28ca0 con 0x5640181f6640 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7facb6a00700 10 client.15372 handle_command_reply: tid=0 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 client.15372 shutdown 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 msgr2=0x564018196220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7facbc022040 tx=0x7facbc011b40 comp rx=0 tx=0).stop 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 shutdown_connections 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5640181f6640 0x5640181f2fb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564018189f80 0x564018196760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564018188780 0x5640181996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 --2- 172.21.15.23:0/1247723820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5640181890e0 0x564018196220 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7facb6a00700 0 client.15372 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 >> 172.21.15.23:0/1247723820 conn(0x5640180e7650 msgr2=0x564018191780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 shutdown_connections 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 1 -- 172.21.15.23:0/1247723820 wait complete. 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.240+0000 7faceb21fb80 20 client.15372 trim_cache size 0 max 16384 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24508, 2024-03-30T04:01:28.245 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 76.297072233999998, 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 127.053148701 2024-03-30T04:01:28.246 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 msgr2=0x7facb80a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 0x7facb80a35e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x564018189760 tx=0x7facc8009920 comp rx=0 tx=0).stop 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 msgr2=0x7facd0199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7facc4009230 tx=0x7facc400bfa0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 shutdown_connections 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facd0102910 0x7facd01a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7facd0100d80 0x7facd0199020 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7facb80a1120 0x7facb80a35e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 --2- 172.21.15.23:0/3994481364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7facd0101f70 0x7facd0199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 >> 172.21.15.23:0/3994481364 conn(0x7facd00fc2f0 msgr2=0x7facd01077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 shutdown_connections 2024-03-30T04:01:28.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.241+0000 7facd7800700 1 -- 172.21.15.23:0/3994481364 wait complete. 2024-03-30T04:01:28.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:28 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3138229870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.256 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:28 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/126018420' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-30T04:01:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:28 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3138229870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:28 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/126018420' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 -- 172.21.15.23:0/1499919331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0100d80 msgr2=0x7f48f01011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1499919331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0100d80 0x7f48f01011a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009990 tx=0x7f48e0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 -- 172.21.15.23:0/1499919331 shutdown_connections 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1499919331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0102910 0x7f48f01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1499919331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0100d80 0x7f48f01011a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1499919331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0101f70 0x7f48f01023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.458+0000 7f48f4c00700 1 -- 172.21.15.23:0/1499919331 >> 172.21.15.23:0/1499919331 conn(0x7f48f00fc2f0 msgr2=0x7f48f00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.459+0000 7f48f4c00700 1 -- 172.21.15.23:0/1499919331 shutdown_connections 2024-03-30T04:01:28.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.459+0000 7f48f4c00700 1 -- 172.21.15.23:0/1499919331 wait complete. 2024-03-30T04:01:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.459+0000 7f48f4c00700 1 Processor -- start 2024-03-30T04:01:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 -- start start 2024-03-30T04:01:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0101f70 0x7f48f00731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0102910 0x7f48f00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0073e80 con 0x7f48f0100d80 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f0073ff0 con 0x7f48f0102910 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48f4c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f01a0db0 con 0x7f48f0101f70 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48ede00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48ede00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49452/0 (socket says 172.21.15.23:49452) 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48ede00700 1 -- 172.21.15.23:0/1789244200 learned_addr learned my addr 172.21.15.23:0/1789244200 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48ed400700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0101f70 0x7f48f00731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.460+0000 7f48ee800700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0102910 0x7f48f00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 -- 172.21.15.23:0/1789244200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0101f70 msgr2=0x7f48f00731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0101f70 0x7f48f00731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 -- 172.21.15.23:0/1789244200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0102910 msgr2=0x7f48f00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0102910 0x7f48f00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 -- 172.21.15.23:0/1789244200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e00095f0 con 0x7f48f0100d80 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48ede00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f48e000b680 tx=0x7f48e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48e6a00700 1 -- 172.21.15.23:0/1789244200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e001d070 con 0x7f48f0100d80 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48e6a00700 1 -- 172.21.15.23:0/1789244200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48e0021410 con 0x7f48f0100d80 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f01a0fe0 con 0x7f48f0100d80 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48e6a00700 1 -- 172.21.15.23:0/1789244200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0017650 con 0x7f48f0100d80 2024-03-30T04:01:28.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.461+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f01a1350 con 0x7f48f0100d80 2024-03-30T04:01:28.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.464+0000 7f48e6a00700 1 -- 172.21.15.23:0/1789244200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f48e0021580 con 0x7f48f0100d80 2024-03-30T04:01:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 Processor -- start 2024-03-30T04:01:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 -- start start 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f48e6a00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 0x7f48d40a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f48e6a00700 1 -- 172.21.15.23:0/1789244200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f48e00ce130 con 0x7f48f0100d80 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b68400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62abe940 con 0x55ab62b670e0 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62abc300 con 0x55ab62b66780 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.466+0000 7f4908689b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62ab7bf0 con 0x55ab62b67f80 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ee800700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 0x7f48d40a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:36892/0 (socket says 172.21.15.23:36892) 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 -- 172.21.15.23:0/3387674623 learned_addr learned my addr 172.21.15.23:0/3387674623 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ee800700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b68400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 -- 172.21.15.23:0/3387674623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 msgr2=0x55ab62b68400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b68400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ed400700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 -- 172.21.15.23:0/3387674623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 msgr2=0x55ab62b679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b679a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 -- 172.21.15.23:0/3387674623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab62939360 con 0x55ab62b66780 2024-03-30T04:01:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ed400700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:01:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ee800700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 0x7f48d40a35e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007b50 tx=0x7f48e8009200 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.467+0000 7f48ede00700 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48e009b1c0 tx=0x7f48e00976a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee79e34dce90654d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.468+0000 7f48db000700 1 -- 172.21.15.23:0/3387674623 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e009b230 con 0x55ab62b66780 2024-03-30T04:01:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.468+0000 7f48db000700 1 -- 172.21.15.23:0/3387674623 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48e009f060 con 0x55ab62b66780 2024-03-30T04:01:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.468+0000 7f48db000700 1 -- 172.21.15.23:0/3387674623 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e009b490 con 0x55ab62b66780 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.468+0000 7f4908689b80 1 -- 172.21.15.23:0/3387674623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 msgr2=0x55ab62b66ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.468+0000 7f4908689b80 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f48e009b1c0 tx=0x7f48e00976a0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 -- 172.21.15.23:0/3387674623 shutdown_connections 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b68400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b66ba0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 --2- 172.21.15.23:0/3387674623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b679a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 -- 172.21.15.23:0/3387674623 >> 172.21.15.23:0/3387674623 conn(0x55ab62ac5650 msgr2=0x55ab62b61950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 -- 172.21.15.23:0/3387674623 shutdown_connections 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 -- 172.21.15.23:0/3387674623 wait complete. 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.469+0000 7f4908689b80 1 -- start start 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 Processor -- start 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b67f80 0x55ab62b77830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62a53d30 con 0x55ab62b67f80 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62a539d0 con 0x55ab62b670e0 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6293b650 con 0x55ab62b66780 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f4908689b80 10 client.0 _ll_register_callbacks cb 0x55ab62a5d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f48ede00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f48ede00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:58512/0 (socket says 172.21.15.23:58512) 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f48ede00700 1 -- 172.21.15.23:0/3682108680 learned_addr learned my addr 172.21.15.23:0/3682108680 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f48ee800700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b67f80 0x55ab62b77830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.470+0000 7f48ed400700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 -- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 msgr2=0x55ab62b707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 -- 172.21.15.23:0/3682108680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b67f80 msgr2=0x55ab62b77830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b67f80 0x55ab62b77830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 -- 172.21.15.23:0/3682108680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f01a1350 con 0x55ab62b66780 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48ede00700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009ce0 tx=0x7f48e009aab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e8399c349ba3006 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48cf400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e009b690 con 0x55ab62b66780 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f48e0095e40 con 0x55ab62b66780 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.471+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab62a78330 con 0x55ab62b66780 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.472+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48e0062070 con 0x55ab62b66780 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.472+0000 7f4908689b80 10 client.25381 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.472+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ab628e5b10 con 0x55ab62b66780 2024-03-30T04:01:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.472+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ab62a78330 con 0x55ab62b66780 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.472+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48e0097020 con 0x55ab62b66780 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 10 client.25381 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 10 client.25381 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 20 client.25381 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62b7afa0 0x55ab62b80240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 4 client.25381 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab62ab06b0 con 0x55ab62b7afa0 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.473+0000 7f48ed400700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62b7afa0 0x55ab62b80240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.474+0000 7f48ed400700 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62b7afa0 0x55ab62b80240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=577c6fbdf727e742 server_cookie=cc5baa7afabf9aaa in_seq=0 out_seq=0 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.474+0000 7f48cf400700 10 client.25381 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.474+0000 7f48cf400700 1 -- 172.21.15.23:0/3682108680 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ab62ab06b0 con 0x55ab62b7afa0 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.474+0000 7f48cf400700 10 client.25381 handle_command_reply: tid=0 2024-03-30T04:01:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 client.25381 shutdown 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 msgr2=0x55ab62b702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009ce0 tx=0x7f48e009aab0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 shutdown_connections 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62b7afa0 0x55ab62b80240 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b67f80 0x55ab62b77830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b66780 0x55ab62b702b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 --2- 172.21.15.23:0/3682108680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f48cf400700 0 client.25381 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 >> 172.21.15.23:0/3682108680 conn(0x55ab62ac5650 msgr2=0x55ab62b65120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 shutdown_connections 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 1 -- 172.21.15.23:0/3682108680 wait complete. 2024-03-30T04:01:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.475+0000 7f4908689b80 20 client.25381 trim_cache size 0 max 16384 2024-03-30T04:01:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:28 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3138229870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:28 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/126018420' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 Processor -- start 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 -- start start 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b84260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62b711b0 con 0x55ab62b670e0 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62b61ed0 con 0x55ab62b66780 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f4908689b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62ab06b0 con 0x55ab62b67f80 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f48ed400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f48ed400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49470/0 (socket says 172.21.15.23:49470) 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f48ed400700 1 -- 172.21.15.23:0/995757948 learned_addr learned my addr 172.21.15.23:0/995757948 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.494+0000 7f48ee800700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ede00700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b84260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 -- 172.21.15.23:0/995757948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 msgr2=0x55ab62b847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 -- 172.21.15.23:0/995757948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 msgr2=0x55ab62b84260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b84260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 -- 172.21.15.23:0/995757948 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab62b5ab00 con 0x55ab62b670e0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ede00700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b84260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48ed400700 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55ab62b7c710 tx=0x7f48dc01dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f88c7089b63859a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48dba00700 1 -- 172.21.15.23:0/995757948 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc018040 con 0x55ab62b670e0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48dba00700 1 -- 172.21.15.23:0/995757948 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48dc021020 con 0x55ab62b670e0 2024-03-30T04:01:28.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.495+0000 7f48dba00700 1 -- 172.21.15.23:0/995757948 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48dc003c50 con 0x55ab62b670e0 2024-03-30T04:01:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 -- 172.21.15.23:0/995757948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 msgr2=0x55ab62b777f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55ab62b7c710 tx=0x7f48dc01dbe0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 -- 172.21.15.23:0/995757948 shutdown_connections 2024-03-30T04:01:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62b847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b66780 0x55ab62b84260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 --2- 172.21.15.23:0/995757948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b670e0 0x55ab62b777f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 -- 172.21.15.23:0/995757948 >> 172.21.15.23:0/995757948 conn(0x55ab62ac5650 msgr2=0x55ab62b6f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 -- 172.21.15.23:0/995757948 shutdown_connections 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.496+0000 7f4908689b80 1 -- 172.21.15.23:0/995757948 wait complete. 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 -- start start 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 Processor -- start 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b74e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62bcf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62b75520 con 0x55ab62b66780 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62bcfd70 con 0x55ab62b670e0 2024-03-30T04:01:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62bcff30 con 0x55ab62b67f80 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.497+0000 7f4908689b80 10 client.0 _ll_register_callbacks cb 0x55ab62a5d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49486/0 (socket says 172.21.15.23:49486) 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 -- 172.21.15.23:0/205432356 learned_addr learned my addr 172.21.15.23:0/205432356 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ee800700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62bcf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ed400700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b74e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 -- 172.21.15.23:0/205432356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 msgr2=0x55ab62bcf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62bcf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 -- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 msgr2=0x55ab62b74e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b74e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 -- 172.21.15.23:0/205432356 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48dc009be0 con 0x55ab62b66780 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48ede00700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f48e00974f0 tx=0x7f48e0021c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8bbdcc8e87e2dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48cf400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.498+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0062070 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f48e0095e40 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48e009f060 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48e00a10b0 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 10 client.15387 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ab62a7de50 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48e009f060 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f48e00a3190 con 0x55ab62b66780 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 10 client.15387 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 10 client.15387 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 20 client.15387 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62bd47d0 0x55ab62bd1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 4 client.15387 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:28.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.499+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ab62906ca0 con 0x55ab62bd47d0 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f48ed400700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62bd47d0 0x55ab62bd1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f48ed400700 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62bd47d0 0x55ab62bd1220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a9ca3ce791bbacb server_cookie=c2b856449d7090a in_seq=0 out_seq=0 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f48cf400700 10 client.15387 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f48cf400700 1 -- 172.21.15.23:0/205432356 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ab62906ca0 con 0x55ab62bd47d0 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f48cf400700 10 client.15387 handle_command_reply: tid=0 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.500+0000 7f4908689b80 1 client.15387 shutdown 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 msgr2=0x55ab62b748d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f48e00974f0 tx=0x7f48e0021c20 comp rx=0 tx=0).stop 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 shutdown_connections 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab62bd47d0 0x55ab62bd1220 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab62b67f80 0x55ab62bcf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab62b66780 0x55ab62b748d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 --2- 172.21.15.23:0/205432356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab62b670e0 0x55ab62b74e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f48cf400700 0 client.15387 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 >> 172.21.15.23:0/205432356 conn(0x55ab62ac5650 msgr2=0x55ab62b84260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 shutdown_connections 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 1 -- 172.21.15.23:0/205432356 wait complete. 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f4908689b80 20 client.15387 trim_cache size 0 max 16384 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24509, 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:28.505 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 77.566493450999999, 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 129.057810846 2024-03-30T04:01:28.506 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 msgr2=0x7f48d40a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 0x7f48d40a35e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007b50 tx=0x7f48e8009200 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 msgr2=0x7f48f0074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.501+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f48e000b680 tx=0x7f48e0004730 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 shutdown_connections 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48f0102910 0x7f48f00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f48f0100d80 0x7f48f0074b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f48d40a1120 0x7f48d40a35e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 --2- 172.21.15.23:0/1789244200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f48f0101f70 0x7f48f00731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 >> 172.21.15.23:0/1789244200 conn(0x7f48f00fc2f0 msgr2=0x7f48f01053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 shutdown_connections 2024-03-30T04:01:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.502+0000 7f48f4c00700 1 -- 172.21.15.23:0/1789244200 wait complete. 2024-03-30T04:01:28.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 -- 172.21.15.23:0/1228109800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4100d80 msgr2=0x7f56b41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 --2- 172.21.15.23:0/1228109800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4100d80 0x7f56b41011a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f56a40099c0 tx=0x7f56a4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 -- 172.21.15.23:0/1228109800 shutdown_connections 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 --2- 172.21.15.23:0/1228109800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4102910 0x7f56b41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 --2- 172.21.15.23:0/1228109800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4100d80 0x7f56b41011a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 --2- 172.21.15.23:0/1228109800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 0x7f56b41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 -- 172.21.15.23:0/1228109800 >> 172.21.15.23:0/1228109800 conn(0x7f56b40fc2f0 msgr2=0x7f56b40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 -- 172.21.15.23:0/1228109800 shutdown_connections 2024-03-30T04:01:28.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.722+0000 7f56b9800700 1 -- 172.21.15.23:0/1228109800 wait complete. 2024-03-30T04:01:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 Processor -- start 2024-03-30T04:01:28.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 -- start start 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 0x7f56b4198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 0x7f56b41994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41a0a90 con 0x7f56b4102910 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41a0bd0 con 0x7f56b4101f70 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.723+0000 7f56b9800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b41a0d60 con 0x7f56b4100d80 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 0x7f56b4198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49504/0 (socket says 172.21.15.23:49504) 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 -- 172.21.15.23:0/1184605000 learned_addr learned my addr 172.21.15.23:0/1184605000 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b2000700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 0x7f56b41994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 -- 172.21.15.23:0/1184605000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 msgr2=0x7f56b4198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 0x7f56b4198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 -- 172.21.15.23:0/1184605000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 msgr2=0x7f56b41994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 0x7f56b41994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 -- 172.21.15.23:0/1184605000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a40095f0 con 0x7f56b4102910 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 0x7f56b4198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.724+0000 7f56b3400700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00ef30 tx=0x7f56ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.725+0000 7f56ab400700 1 -- 172.21.15.23:0/1184605000 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac018070 con 0x7f56b4102910 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.725+0000 7f56ab400700 1 -- 172.21.15.23:0/1184605000 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ac00f040 con 0x7f56b4102910 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.725+0000 7f56ab400700 1 -- 172.21.15.23:0/1184605000 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac00cd00 con 0x7f56b4102910 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.725+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b41a0fe0 con 0x7f56b4102910 2024-03-30T04:01:28.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.725+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41a1410 con 0x7f56b4102910 2024-03-30T04:01:28.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.728+0000 7f56ab400700 1 -- 172.21.15.23:0/1184605000 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f56ac003a00 con 0x7f56b4102910 2024-03-30T04:01:28.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 Processor -- start 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56ab400700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 0x7f56980a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 -- start start 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56ab400700 1 -- 172.21.15.23:0/1184605000 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f56ac0cc7b0 con 0x7f56b4102910 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56b2000700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 0x7f56980a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641949940 con 0x55e6419f2f80 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641947300 con 0x55e6419f1780 2024-03-30T04:01:28.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.730+0000 7f56cd309b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641942bf0 con 0x55e6419f20e0 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:36938/0 (socket says 172.21.15.23:36938) 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 -- 172.21.15.23:0/1606111101 learned_addr learned my addr 172.21.15.23:0/1606111101 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b3400700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2000700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 -- 172.21.15.23:0/1606111101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 msgr2=0x55e6419f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 -- 172.21.15.23:0/1606111101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 msgr2=0x55e6419f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2000700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 0x7f56980a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009c70 tx=0x7f56a0011720 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 -- 172.21.15.23:0/1606111101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6417c4360 con 0x55e6419f1780 2024-03-30T04:01:28.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b3400700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.731+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56a4000c00 tx=0x7f56a4005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93eea35d609674d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.732+0000 7f569fe00700 1 -- 172.21.15.23:0/1606111101 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a400bdc0 con 0x55e6419f1780 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.732+0000 7f569fe00700 1 -- 172.21.15.23:0/1606111101 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56a401e040 con 0x55e6419f1780 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.732+0000 7f569fe00700 1 -- 172.21.15.23:0/1606111101 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a400f460 con 0x55e6419f1780 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- 172.21.15.23:0/1606111101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 msgr2=0x55e6419f1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f56a4000c00 tx=0x7f56a4005be0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- 172.21.15.23:0/1606111101 shutdown_connections 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f1780 0x55e6419f1ba0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1606111101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- 172.21.15.23:0/1606111101 >> 172.21.15.23:0/1606111101 conn(0x55e641950650 msgr2=0x55e6419ec950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- 172.21.15.23:0/1606111101 shutdown_connections 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- 172.21.15.23:0/1606111101 wait complete. 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 -- start start 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.733+0000 7f56cd309b80 1 Processor -- start 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e6419fbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f2f80 0x55e641a02b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6418ded30 con 0x55e6419f1780 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6418de9d0 con 0x55e6419f20e0 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6417c6650 con 0x55e6419f2f80 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56cd309b80 10 client.0 _ll_register_callbacks cb 0x55e6418e8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49522/0 (socket says 172.21.15.23:49522) 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b2a00700 1 -- 172.21.15.23:0/2998040357 learned_addr learned my addr 172.21.15.23:0/2998040357 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b2000700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e6419fbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b3400700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f2f80 0x55e641a02b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.734+0000 7f56b2a00700 1 -- 172.21.15.23:0/2998040357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f2f80 msgr2=0x55e641a02b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56b2a00700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f2f80 0x55e641a02b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56b2a00700 1 -- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 msgr2=0x55e6419fbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56b2a00700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e6419fbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56b2a00700 1 -- 172.21.15.23:0/2998040357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41a1410 con 0x55e6419f1780 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56b2a00700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009d10 tx=0x7f56a4005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3a5d8bf2bf119e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f569cc00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:28.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a4005710 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f56a400f460 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e641903330 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a400f620 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56cd309b80 10 client.15399 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.735+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f56b4109b10 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e641903330 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56a402b900 con 0x55e6419f1780 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 10 client.15399 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 10 client.15399 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 20 client.15399 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a062d0 0x55e641a0b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 4 client.15399 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e64193b6b0 con 0x55e641a062d0 2024-03-30T04:01:28.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.736+0000 7f56b2000700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a062d0 0x55e641a0b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.737+0000 7f56b2000700 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a062d0 0x55e641a0b5d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=367674895be0c769 server_cookie=f3031cdf56b08283 in_seq=0 out_seq=0 2024-03-30T04:01:28.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.737+0000 7f569cc00700 10 client.15399 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f569cc00700 1 -- 172.21.15.23:0/2998040357 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e64193b6b0 con 0x55e641a062d0 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f569cc00700 10 client.15399 handle_command_reply: tid=0 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 client.15399 shutdown 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 msgr2=0x55e6419fb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f56a4009d10 tx=0x7f56a4005050 comp rx=0 tx=0).stop 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 shutdown_connections 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a062d0 0x55e641a0b5d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f2f80 0x55e641a02b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e6419fb5d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 --2- 172.21.15.23:0/2998040357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e6419fbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f569cc00700 0 client.15399 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 >> 172.21.15.23:0/2998040357 conn(0x55e641950650 msgr2=0x55e6419f0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.738+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 shutdown_connections 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.739+0000 7f56cd309b80 1 -- 172.21.15.23:0/2998040357 wait complete. 2024-03-30T04:01:28.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.739+0000 7f56cd309b80 20 client.15399 trim_cache size 0 max 16384 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 Processor -- start 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 -- start start 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 0x55e641a03200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e641a02b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641a07b80 con 0x55e6419f2f80 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6419ed610 con 0x55e6419f20e0 2024-03-30T04:01:28.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.757+0000 7f56cd309b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6419fc4d0 con 0x55e6419f1780 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 0x55e641a03200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e641a02b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49526/0 (socket says 172.21.15.23:49526) 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 0x55e641a03200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:58560/0 (socket says 172.21.15.23:58560) 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 -- 172.21.15.23:0/3971711155 learned_addr learned my addr 172.21.15.23:0/3971711155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 -- 172.21.15.23:0/3971711155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 msgr2=0x55e641a03200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 0x55e641a03200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 -- 172.21.15.23:0/3971711155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 msgr2=0x55e641a02b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e641a02b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 -- 172.21.15.23:0/3971711155 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e641770b10 con 0x55e6419f2f80 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56b3400700 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c6f0 tx=0x7f56ac0092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ff2afedac654a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56a8c00700 1 -- 172.21.15.23:0/3971711155 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac01b320 con 0x55e6419f2f80 2024-03-30T04:01:28.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56a8c00700 1 -- 172.21.15.23:0/3971711155 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56ac099fb0 con 0x55e6419f2f80 2024-03-30T04:01:28.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.758+0000 7f56a8c00700 1 -- 172.21.15.23:0/3971711155 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0cc410 con 0x55e6419f2f80 2024-03-30T04:01:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 -- 172.21.15.23:0/3971711155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 msgr2=0x55e6419f3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00c6f0 tx=0x7f56ac0092e0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 -- 172.21.15.23:0/3971711155 shutdown_connections 2024-03-30T04:01:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f2f80 0x55e6419f3360 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f1780 0x55e641a03200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 --2- 172.21.15.23:0/3971711155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f20e0 0x55e641a02b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 -- 172.21.15.23:0/3971711155 >> 172.21.15.23:0/3971711155 conn(0x55e641950650 msgr2=0x55e6419f6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 -- 172.21.15.23:0/3971711155 shutdown_connections 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.759+0000 7f56cd309b80 1 -- 172.21.15.23:0/3971711155 wait complete. 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 -- start start 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 Processor -- start 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f2f80 0x55e6419ffbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641a00260 con 0x55e6419f1780 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641a5abc0 con 0x55e6419f2f80 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e641a5ad30 con 0x55e6419f20e0 2024-03-30T04:01:28.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56cd309b80 10 client.0 _ll_register_callbacks cb 0x55e6418e8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56b2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f2f80 0x55e6419ffbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:58564/0 (socket says 172.21.15.23:58564) 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 -- 172.21.15.23:0/1348027060 learned_addr learned my addr 172.21.15.23:0/1348027060 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.760+0000 7f56b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49540/0 (socket says 172.21.15.23:49540) 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 -- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f2f80 msgr2=0x55e6419ffbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f2f80 0x55e6419ffbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 -- 172.21.15.23:0/1348027060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 msgr2=0x55e641a02aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 -- 172.21.15.23:0/1348027060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b41a0fe0 con 0x55e6419f20e0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f56b2000700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55e6419fc630 tx=0x7f56a00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b98bab301afc1909 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f569cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.761+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56a0023040 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56a401e040 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f56a001e2c0 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56a002a3c0 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 10 client.25405 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e641908e50 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56a401e040 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f56a001d6b0 con 0x55e6419f20e0 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 10 client.25405 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 10 client.25405 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 20 client.25405 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a5fc40 0x55e641a5c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.762+0000 7f56cd309b80 4 client.25405 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.763+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e641791ca0 con 0x55e641a5fc40 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.763+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a5fc40 0x55e641a5c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f569cc00700 10 client.25405 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56b2a00700 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a5fc40 0x55e641a5c570 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=550221b6a9df800c server_cookie=545f6662b4f99ec3 in_seq=0 out_seq=0 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f569cc00700 1 -- 172.21.15.23:0/1348027060 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55e641791ca0 con 0x55e641a5fc40 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f569cc00700 10 client.25405 handle_command_reply: tid=0 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 client.25405 shutdown 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 msgr2=0x55e6419ff680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55e6419fc630 tx=0x7f56a00047c0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 shutdown_connections 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6419f2f80 0x55e6419ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6419f1780 0x55e641a02aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e641a5fc40 0x55e641a5c570 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 --2- 172.21.15.23:0/1348027060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6419f20e0 0x55e6419ff680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f569cc00700 0 client.25405 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 >> 172.21.15.23:0/1348027060 conn(0x55e641950650 msgr2=0x55e6419fa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 shutdown_connections 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 1 -- 172.21.15.23:0/1348027060 wait complete. 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.764+0000 7f56cd309b80 20 client.25405 trim_cache size 0 max 16384 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24533, 2024-03-30T04:01:28.770 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 78.793118304000004, 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 124.13068674 2024-03-30T04:01:28.771 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 msgr2=0x7f56980a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 0x7f56980a35e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009c70 tx=0x7f56a0011720 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 msgr2=0x7f56b41a04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f56ac00ef30 tx=0x7f56ac00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 shutdown_connections 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56b4102910 0x7f56b41a04c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f56980a1120 0x7f56980a35e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56b4100d80 0x7f56b4198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 --2- 172.21.15.23:0/1184605000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56b4101f70 0x7f56b41994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 >> 172.21.15.23:0/1184605000 conn(0x7f56b40fc2f0 msgr2=0x7f56b4107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 shutdown_connections 2024-03-30T04:01:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.765+0000 7f56b9800700 1 -- 172.21.15.23:0/1184605000 wait complete. 2024-03-30T04:01:28.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-30T04:01:28.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 -- 172.21.15.23:0/1329314008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 msgr2=0x7ffb181011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/1329314008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb181011a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffb080099c0 tx=0x7ffb08009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 -- 172.21.15.23:0/1329314008 shutdown_connections 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/1329314008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 0x7ffb181091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/1329314008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb181011a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/1329314008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 0x7ffb181023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 -- 172.21.15.23:0/1329314008 >> 172.21.15.23:0/1329314008 conn(0x7ffb180fc2f0 msgr2=0x7ffb180fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 -- 172.21.15.23:0/1329314008 shutdown_connections 2024-03-30T04:01:28.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.982+0000 7ffb1ea00700 1 -- 172.21.15.23:0/1329314008 wait complete. 2024-03-30T04:01:28.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.983+0000 7ffb1ea00700 1 Processor -- start 2024-03-30T04:01:28.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.983+0000 7ffb1ea00700 1 -- start start 2024-03-30T04:01:28.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 0x7ffb181994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 0x7ffb181a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181a0ab0 con 0x7ffb18100d80 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181a0bf0 con 0x7ffb18101f70 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181a0d80 con 0x7ffb18102910 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49552/0 (socket says 172.21.15.23:49552) 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 -- 172.21.15.23:0/2247116187 learned_addr learned my addr 172.21.15.23:0/2247116187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 0x7ffb181a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17400700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 0x7ffb181994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 -- 172.21.15.23:0/2247116187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 msgr2=0x7ffb181a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 0x7ffb181a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 -- 172.21.15.23:0/2247116187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 msgr2=0x7ffb181994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 0x7ffb181994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.984+0000 7ffb17e00700 1 -- 172.21.15.23:0/2247116187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb080095f0 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffb08006010 tx=0x7ffb0800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb14c00700 1 -- 172.21.15.23:0/2247116187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0801d070 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb14c00700 1 -- 172.21.15.23:0/2247116187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb08005020 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb14c00700 1 -- 172.21.15.23:0/2247116187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08004bb0 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb181a1000 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.985+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb181a1370 con 0x7ffb18100d80 2024-03-30T04:01:28.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.988+0000 7ffb14c00700 1 -- 172.21.15.23:0/2247116187 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7ffb08005190 con 0x7ffb18100d80 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.989+0000 7ffb325cfb80 1 Processor -- start 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.989+0000 7ffb14c00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 0x7ffafc0a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.989+0000 7ffb14c00700 1 -- 172.21.15.23:0/2247116187 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7ffb080cd750 con 0x7ffb18100d80 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 -- start start 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 0x7ffafc0a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d160bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d160d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d1563940 con 0x55e6d160cf80 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d1561300 con 0x55e6d160b780 2024-03-30T04:01:28.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d155cbf0 con 0x55e6d160c0e0 2024-03-30T04:01:28.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d160bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb17400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:58600/0 (socket says 172.21.15.23:58600) 2024-03-30T04:01:28.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.990+0000 7ffb17400700 1 -- 172.21.15.23:0/4081911548 learned_addr learned my addr 172.21.15.23:0/4081911548 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 -- 172.21.15.23:0/4081911548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 msgr2=0x55e6d160bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d160bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 -- 172.21.15.23:0/4081911548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 msgr2=0x55e6d160d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d160d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d160d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 -- 172.21.15.23:0/4081911548 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6d13de360 con 0x55e6d160c0e0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d160d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 0x7ffafc0a35e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000f590 tx=0x7ffb10009920 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb17400700 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009d10 tx=0x7ffb040116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6232751834bd042 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb0d200700 1 -- 172.21.15.23:0/4081911548 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb040097e0 con 0x55e6d160c0e0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb0d200700 1 -- 172.21.15.23:0/4081911548 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb04009e60 con 0x55e6d160c0e0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.991+0000 7ffb0d200700 1 -- 172.21.15.23:0/4081911548 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0400f800 con 0x55e6d160c0e0 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 -- 172.21.15.23:0/4081911548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 msgr2=0x55e6d160c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009d10 tx=0x7ffb040116c0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 -- 172.21.15.23:0/4081911548 shutdown_connections 2024-03-30T04:01:28.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d160d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d160bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/4081911548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d160c9a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 -- 172.21.15.23:0/4081911548 >> 172.21.15.23:0/4081911548 conn(0x55e6d156a650 msgr2=0x55e6d1606950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 -- 172.21.15.23:0/4081911548 shutdown_connections 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.992+0000 7ffb325cfb80 1 -- 172.21.15.23:0/4081911548 wait complete. 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.993+0000 7ffb325cfb80 1 -- start start 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.993+0000 7ffb325cfb80 1 Processor -- start 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.993+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.993+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d161cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d14f8d30 con 0x55e6d160cf80 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d14f89d0 con 0x55e6d160b780 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d13e0650 con 0x55e6d160c0e0 2024-03-30T04:01:28.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb325cfb80 10 client.0 _ll_register_callbacks cb 0x55e6d1502420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37006/0 (socket says 172.21.15.23:37006) 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 -- 172.21.15.23:0/2429557510 learned_addr learned my addr 172.21.15.23:0/2429557510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d161cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17400700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 -- 172.21.15.23:0/2429557510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 msgr2=0x55e6d1615b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 -- 172.21.15.23:0/2429557510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 msgr2=0x55e6d161cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d161cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb17e00700 1 -- 172.21.15.23:0/2429557510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb04009940 con 0x55e6d160b780 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.994+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d161cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb17e00700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb08096d00 tx=0x7ffb0809afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a3432e02921ae0e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:28.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080a2c40 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7ffb080cd410 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6d151d330 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb0809aab0 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb325cfb80 10 client.25430 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e6d138ab10 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e6d151d330 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb08099dd0 con 0x55e6d160b780 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb325cfb80 10 client.25430 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.995+0000 7ffb325cfb80 10 client.25430 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb325cfb80 20 client.25430 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1620360 0x55e6d1625660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb325cfb80 4 client.25430 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:28.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6d15556b0 con 0x55e6d1620360 2024-03-30T04:01:29.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb17400700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1620360 0x55e6d1625660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb17400700 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1620360 0x55e6d1625660 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d64d58f9870d89e server_cookie=d5f61cafc9d5c6d6 in_seq=0 out_seq=0 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.996+0000 7ffb01600700 10 client.25430 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb01600700 1 -- 172.21.15.23:0/2429557510 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e6d15556b0 con 0x55e6d1620360 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb01600700 10 client.25430 handle_command_reply: tid=0 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 client.25430 shutdown 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 msgr2=0x55e6d16155d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ffb08096d00 tx=0x7ffb0809afd0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 shutdown_connections 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160cf80 0x55e6d161cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1620360 0x55e6d1625660 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/2429557510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 >> 172.21.15.23:0/2429557510 conn(0x55e6d156a650 msgr2=0x55e6d160a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.997+0000 7ffb01600700 0 client.25430 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.998+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 shutdown_connections 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.998+0000 7ffb325cfb80 1 -- 172.21.15.23:0/2429557510 wait complete. 2024-03-30T04:01:29.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:28.998+0000 7ffb325cfb80 20 client.25430 trim_cache size 0 max 16384 2024-03-30T04:01:29.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.016+0000 7ffb325cfb80 1 Processor -- start 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 -- start start 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d161cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 0x55e6d1613490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d151aad0 con 0x55e6d160b780 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d1555990 con 0x55e6d160cf80 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d1559850 con 0x55e6d160c0e0 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49578/0 (socket says 172.21.15.23:49578) 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb17e00700 1 -- 172.21.15.23:0/3342425480 learned_addr learned my addr 172.21.15.23:0/3342425480 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 0x55e6d1613490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb17400700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d161cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.017+0000 7ffb17e00700 1 -- 172.21.15.23:0/3342425480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 msgr2=0x55e6d161cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d161cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb17e00700 1 -- 172.21.15.23:0/3342425480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 msgr2=0x55e6d1613490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 0x55e6d1613490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb17e00700 1 -- 172.21.15.23:0/3342425480 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6d1522e50 con 0x55e6d160b780 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 0x55e6d1613490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800b580 tx=0x7ffb08098e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e85b911c090668 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb0dc00700 1 -- 172.21.15.23:0/3342425480 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08097f40 con 0x55e6d160b780 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb0dc00700 1 -- 172.21.15.23:0/3342425480 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb080a2970 con 0x55e6d160b780 2024-03-30T04:01:29.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.018+0000 7ffb0dc00700 1 -- 172.21.15.23:0/3342425480 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0809ad10 con 0x55e6d160b780 2024-03-30T04:01:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3342425480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 msgr2=0x55e6d1614ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:29.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ffb0800b580 tx=0x7ffb08098e00 comp rx=0 tx=0).stop 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3342425480 shutdown_connections 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160cf80 0x55e6d1613490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d1614ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3342425480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160c0e0 0x55e6d161cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3342425480 >> 172.21.15.23:0/3342425480 conn(0x55e6d156a650 msgr2=0x55e6d15ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3342425480 shutdown_connections 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3342425480 wait complete. 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 -- start start 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.019+0000 7ffb325cfb80 1 Processor -- start 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160cf80 0x55e6d161be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d161c510 con 0x55e6d160b780 2024-03-30T04:01:29.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d161c680 con 0x55e6d160c0e0 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6d161c840 con 0x55e6d160cf80 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb325cfb80 10 client.0 _ll_register_callbacks cb 0x55e6d1502420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49580/0 (socket says 172.21.15.23:49580) 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 -- 172.21.15.23:0/3138488561 learned_addr learned my addr 172.21.15.23:0/3138488561 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17400700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb1ca00700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160cf80 0x55e6d161be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 -- 172.21.15.23:0/3138488561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160cf80 msgr2=0x55e6d161be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160cf80 0x55e6d161be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 -- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160c0e0 msgr2=0x55e6d1615b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.020+0000 7ffb17e00700 1 -- 172.21.15.23:0/3138488561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb10019820 con 0x55e6d160b780 2024-03-30T04:01:29.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb17e00700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffb080972c0 tx=0x7ffb080056a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b11766642a41124 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb01600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb08097f40 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7ffb0800fb00 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb04009e60 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb0809afd0 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb325cfb80 10 client.15420 fetch_fsmap learned FSMap version 27 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e6d138ab10 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.021+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb080a6020 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb080998d0 con 0x55e6d160b780 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 10 client.15420 fetch_fsmap finished waiting for FSMap version 27 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 10 client.15420 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 20 client.15420 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:29.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1678a80 0x55e6d16756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 4 client.15420 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e6d13abca0 con 0x55e6d1678a80 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.022+0000 7ffb17400700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1678a80 0x55e6d16756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb17400700 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1678a80 0x55e6d16756b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a537c6494c9962c3 server_cookie=bff68dbaa7ce7b27 in_seq=0 out_seq=0 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb01600700 10 client.15420 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb01600700 1 -- 172.21.15.23:0/3138488561 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55e6d13abca0 con 0x55e6d1678a80 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb01600700 10 client.15420 handle_command_reply: tid=0 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 client.15420 shutdown 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 msgr2=0x55e6d16155d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ffb080972c0 tx=0x7ffb080056a0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 shutdown_connections 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6d1678a80 0x55e6d16756b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6d160cf80 0x55e6d161be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6d160b780 0x55e6d16155d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb01600700 0 client.15420 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 --2- 172.21.15.23:0/3138488561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6d160c0e0 0x55e6d1615b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 >> 172.21.15.23:0/3138488561 conn(0x55e6d156a650 msgr2=0x55e6d1613450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 shutdown_connections 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.023+0000 7ffb325cfb80 1 -- 172.21.15.23:0/3138488561 wait complete. 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb325cfb80 20 client.15420 trim_cache size 0 max 16384 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "eac7e298-ee48-11ee-b647-cb9ed24678a4", 2024-03-30T04:01:29.028 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24569, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 108.588550188, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 27, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 88, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 84, 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 115.708385209 2024-03-30T04:01:29.029 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 msgr2=0x7ffafc0a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 0x7ffafc0a35e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000f590 tx=0x7ffb10009920 comp rx=0 tx=0).stop 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 msgr2=0x7ffb18198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffb08006010 tx=0x7ffb0800f690 comp rx=0 tx=0).stop 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 shutdown_connections 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7ffafc0a1120 0x7ffafc0a35e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb18102910 0x7ffb181a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb18100d80 0x7ffb18198f80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 --2- 172.21.15.23:0/2247116187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb18101f70 0x7ffb181994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 >> 172.21.15.23:0/2247116187 conn(0x7ffb180fc2f0 msgr2=0x7ffb18072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 shutdown_connections 2024-03-30T04:01:29.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:29.024+0000 7ffb1ea00700 1 -- 172.21.15.23:0/2247116187 wait complete. 2024-03-30T04:01:29.042 INFO:teuthology.run_tasks:Running task kclient... 2024-03-30T04:01:29.055 INFO:tasks.kclient:Mounting kernel clients... 2024-03-30T04:01:29.055 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-30T04:01:29.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:29.055 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-30T04:01:29.097 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-30T04:01:29.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:29.099 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-30T04:01:29.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:29.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-30T04:01:29.309 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:29 smithi023 ceph-mon[38254]: pgmap v294: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-30T04:01:29.324 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-30T04:01:29.324 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-30T04:01:29.324 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:29.324 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph fs ls 2024-03-30T04:01:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:29 smithi154 ceph-mon[79161]: pgmap v294: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-30T04:01:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:29 smithi193 ceph-mon[79581]: pgmap v294: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-30T04:01:29.693 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 -- 172.21.15.23:0/2934112751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 msgr2=0x7f04f810dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 --2- 172.21.15.23:0/2934112751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f810dc10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f04f000b0d0 tx=0x7f04f000b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 -- 172.21.15.23:0/2934112751 shutdown_connections 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 --2- 172.21.15.23:0/2934112751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f810dc10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 --2- 172.21.15.23:0/2934112751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8106ef0 0x7f04f81072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 --2- 172.21.15.23:0/2934112751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8100e60 0x7f04f81012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 -- 172.21.15.23:0/2934112751 >> 172.21.15.23:0/2934112751 conn(0x7f04f8077fa0 msgr2=0x7f04f80783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:30.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.932+0000 7f04fe600700 1 -- 172.21.15.23:0/2934112751 shutdown_connections 2024-03-30T04:01:30.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.933+0000 7f04fe600700 1 -- 172.21.15.23:0/2934112751 wait complete. 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.934+0000 7f04fe600700 1 Processor -- start 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.934+0000 7f04fe600700 1 -- start start 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 0x7f04f81972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8106ef0 0x7f04f819bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f819c1c0 con 0x7f04f8101820 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f819c330 con 0x7f04f8106ef0 2024-03-30T04:01:30.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04fe600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f819c4f0 con 0x7f04f8100e60 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04efe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04efe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49586/0 (socket says 172.21.15.23:49586) 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04efe00700 1 -- 172.21.15.23:0/567476335 learned_addr learned my addr 172.21.15.23:0/567476335 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04f7400700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 0x7f04f81972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.935+0000 7f04f7e00700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8106ef0 0x7f04f819bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 -- 172.21.15.23:0/567476335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 msgr2=0x7f04f81972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 0x7f04f81972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 -- 172.21.15.23:0/567476335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8106ef0 msgr2=0x7f04f819bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8106ef0 0x7f04f819bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 -- 172.21.15.23:0/567476335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0009d00 con 0x7f04f8101820 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04f7400700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 0x7f04f81972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04efe00700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9a0 tx=0x7f04e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:30.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0009790 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f04e0010430 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f819c7d0 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.936+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e0018680 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.937+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f81a4030 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.938+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d40052f0 con 0x7f04f8101820 2024-03-30T04:01:30.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.940+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f04e000ca10 con 0x7f04f8101820 2024-03-30T04:01:30.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.942+0000 7f04f4c00700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 0x7f04dc0a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:30.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.942+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f04e0014070 con 0x7f04f8101820 2024-03-30T04:01:30.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.942+0000 7f04f7400700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 0x7f04dc0a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:30.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.942+0000 7f04f7400700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 0x7f04dc0a3660 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f04e8011490 tx=0x7f04e8009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:30.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:30.946+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f04e00529f0 con 0x7f04f8101820 2024-03-30T04:01:31.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.204+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f04d4005db0 con 0x7f04f8101820 2024-03-30T04:01:31.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.205+0000 7f04f4c00700 1 -- 172.21.15.23:0/567476335 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+88 (secure 0 0 0) 0x7f04e0096090 con 0x7f04f8101820 2024-03-30T04:01:31.205 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 msgr2=0x7f04dc0a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 0x7f04dc0a3660 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f04e8011490 tx=0x7f04e8009260 comp rx=0 tx=0).stop 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 msgr2=0x7f04f8197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f04e000e9a0 tx=0x7f04e000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 shutdown_connections 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f04f8106ef0 0x7f04f819bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f04dc0a11a0 0x7f04dc0a3660 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:31.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f04f8100e60 0x7f04f81972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:31.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 --2- 172.21.15.23:0/567476335 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f04f8101820 0x7f04f8197800 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 >> 172.21.15.23:0/567476335 conn(0x7f04f8077fa0 msgr2=0x7f04f80ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 shutdown_connections 2024-03-30T04:01:31.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:31.207+0000 7f04fe600700 1 -- 172.21.15.23:0/567476335 wait complete. 2024-03-30T04:01:31.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:31 smithi023 ceph-mon[38254]: pgmap v295: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 362 B/s wr, 11 op/s 2024-03-30T04:01:31.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:31 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/567476335' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:01:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:31 smithi154 ceph-mon[79161]: pgmap v295: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 362 B/s wr, 11 op/s 2024-03-30T04:01:31.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:31 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/567476335' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:01:31.687 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-30T04:01:31.687 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-30T04:01:31.688 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-30T04:01:31.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:31.688 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-30T04:01:31.732 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 41705sec preferred_lft 41705sec 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-30T04:01:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-30T04:01:31.734 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-30T04:01:31.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:31.734 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T04:01:31.734 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T04:01:31.734 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-30T04:01:31.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-30T04:01:31.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-30T04:01:31.735 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-30T04:01:31.735 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T04:01:31.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:31 smithi193 ceph-mon[79581]: pgmap v295: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 362 B/s wr, 11 op/s 2024-03-30T04:01:31.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:31 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/567476335' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-30T04:01:31.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:31.979 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-30T04:01:32.047 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-30T04:01:32.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.050 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-30T04:01:32.095 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-30T04:01:32.095 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-30T04:01:32.095 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-30T04:01:32.095 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-30T04:01:32.095 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-30T04:01:32.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-30T04:01:32.096 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T04:01:32.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.302 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-30T04:01:32.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.345 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-30T04:01:32.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.429 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T04:01:32.429 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T04:01:32.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-30T04:01:32.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-30T04:01:32.429 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T04:01:32.610 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-30T04:01:32.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-30T04:01:32.611 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T04:01:32.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:32.860 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-30T04:01:32.860 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-30T04:01:32.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-30T04:01:32.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-30T04:01:32.860 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-30T04:01:33.029 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:01:33.029 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:33.071 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-30T04:01:33.071 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-30T04:01:33.071 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:33.152 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-30T04:01:33.152 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-30T04:01:33.152 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/51da4228-0048-4ff1-b829-59e3acc5f4e7 2024-03-30T04:01:33.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:33.153 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/51da4228-0048-4ff1-b829-59e3acc5f4e7 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-30T04:01:33.294 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-30T04:01:33.294 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-30T04:01:33.294 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2024-03-30T04:01:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:33 smithi154 ceph-mon[79161]: pgmap v296: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:01:33.726 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:33 smithi023 ceph-mon[38254]: pgmap v296: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:01:33.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:33 smithi193 ceph-mon[79581]: pgmap v296: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:01:33.937 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-30T04:01:33.937 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-30T04:01:33.937 INFO:teuthology.orchestra.run.smithi023.stdout:invalid new device string format 2024-03-30T04:01:33.937 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.154:3300,172.21.15.193:3300" 2024-03-30T04:01:33.938 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.23:3300,172.21.15.154:3300,172.21.15.193:3300:/volumes/qa/sv_1/51da4228-0048-4ff1-b829-59e3acc5f4e7 2024-03-30T04:01:33.938 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=eac7e298-ee48-11ee-b647-cb9ed24678a4" will pass to kernel 2024-03-30T04:01:33.945 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-30T04:01:33.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:33.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:34.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:34.019 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T04:01:34.101 INFO:teuthology.orchestra.run.smithi023.stdout:client25432 2024-03-30T04:01:34.102 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T04:01:34.102 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T04:01:34.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:34.145 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T04:01:34.266 INFO:teuthology.orchestra.run.smithi023.stdout:client25432 2024-03-30T04:01:34.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:34.266 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-30T04:01:34.354 INFO:teuthology.orchestra.run.smithi023.stdout:client25432 2024-03-30T04:01:34.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:01:34.355 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4.client25432/status') 2024-03-30T04:01:34.459 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.25432 (3)172.21.15.23:0/35642103 2024-03-30T04:01:34.459 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2024-03-30T04:01:34.459 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2024-03-30T04:01:34.459 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2024-03-30T04:01:34.459 INFO:teuthology.orchestra.run.smithi023.stderr:69 bytes copied, 5.5076e-05 s, 1.3 MB/s 2024-03-30T04:01:34.462 INFO:teuthology.run_tasks:Running task exec... 2024-03-30T04:01:34.473 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T04:01:34.473 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:01:34.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-30T04:01:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 -- 172.21.15.23:0/1825628126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30101f00 msgr2=0x7f0e30102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 --2- 172.21.15.23:0/1825628126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30101f00 0x7f0e30102360 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0099c0 tx=0x7f0e1c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 -- 172.21.15.23:0/1825628126 shutdown_connections 2024-03-30T04:01:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 --2- 172.21.15.23:0/1825628126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e30109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 --2- 172.21.15.23:0/1825628126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30100d10 0x7f0e30101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 --2- 172.21.15.23:0/1825628126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30101f00 0x7f0e30102360 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 -- 172.21.15.23:0/1825628126 >> 172.21.15.23:0/1825628126 conn(0x7f0e300fc280 msgr2=0x7f0e300fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 -- 172.21.15.23:0/1825628126 shutdown_connections 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.669+0000 7f0e36c00700 1 -- 172.21.15.23:0/1825628126 wait complete. 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.671+0000 7f0e36c00700 1 Processor -- start 2024-03-30T04:01:34.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.671+0000 7f0e36c00700 1 -- start start 2024-03-30T04:01:34.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.671+0000 7f0e36c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30100d10 0x7f0e30196d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:34.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.671+0000 7f0e36c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30101f00 0x7f0e30197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:34.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.671+0000 7f0e36c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e36c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019e7e0 con 0x7f0e301028a0 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e36c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019e950 con 0x7f0e30100d10 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e36c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019eb10 con 0x7f0e30101f00 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e2fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30100d10 0x7f0e30196d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e2f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30101f00 0x7f0e30197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49598/0 (socket says 172.21.15.23:49598) 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 -- 172.21.15.23:0/2221297299 learned_addr learned my addr 172.21.15.23:0/2221297299 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 -- 172.21.15.23:0/2221297299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30101f00 msgr2=0x7f0e30197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30101f00 0x7f0e30197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 -- 172.21.15.23:0/2221297299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30100d10 msgr2=0x7f0e30196d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30100d10 0x7f0e30196d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 -- 172.21.15.23:0/2221297299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0095f0 con 0x7f0e301028a0 2024-03-30T04:01:34.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.672+0000 7f0e34c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800ef30 tx=0x7f0e2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.673+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28018070 con 0x7f0e301028a0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.673+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e30072fe0 con 0x7f0e301028a0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.673+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e2800f040 con 0x7f0e301028a0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.673+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2800cca0 con 0x7f0e301028a0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.673+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e300734c0 con 0x7f0e301028a0 2024-03-30T04:01:34.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.674+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e30108200 con 0x7f0e301028a0 2024-03-30T04:01:34.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.676+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f0e28003a00 con 0x7f0e301028a0 2024-03-30T04:01:34.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.677+0000 7f0e2cc00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 0x7f0e140a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:34.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.677+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f0e280cc770 con 0x7f0e301028a0 2024-03-30T04:01:34.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.677+0000 7f0e2fe00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 0x7f0e140a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:34.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.678+0000 7f0e2fe00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 0x7f0e140a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0e300fe670 tx=0x7f0e20012040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:34.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.682+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f0e28095bb0 con 0x7f0e301028a0 2024-03-30T04:01:34.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:34.952+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f0e30067c30 con 0x7f0e301028a0 2024-03-30T04:01:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.368+0000 7f0e2cc00700 1 -- 172.21.15.23:0/2221297299 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0e280991d0 con 0x7f0e301028a0 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 msgr2=0x7f0e140a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 0x7f0e140a35e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0e300fe670 tx=0x7f0e20012040 comp rx=0 tx=0).stop 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 msgr2=0x7f0e3019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800ef30 tx=0x7f0e2800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 shutdown_connections 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e301028a0 0x7f0e3019e210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e30100d10 0x7f0e30196d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f0e140a1120 0x7f0e140a35e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 --2- 172.21.15.23:0/2221297299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e30101f00 0x7f0e30197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 >> 172.21.15.23:0/2221297299 conn(0x7f0e300fc280 msgr2=0x7f0e30105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.371+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 shutdown_connections 2024-03-30T04:01:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.372+0000 7f0e36c00700 1 -- 172.21.15.23:0/2221297299 wait complete. 2024-03-30T04:01:35.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.587+0000 7f7390400700 1 -- 172.21.15.23:0/3540461131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388100d10 msgr2=0x7f7388101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.587+0000 7f7390400700 1 --2- 172.21.15.23:0/3540461131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388100d10 0x7f7388101130 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f737c0099c0 tx=0x7f737c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 -- 172.21.15.23:0/3540461131 shutdown_connections 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 --2- 172.21.15.23:0/3540461131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73881028a0 0x7f7388109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 --2- 172.21.15.23:0/3540461131 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388100d10 0x7f7388101130 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 --2- 172.21.15.23:0/3540461131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7388101f00 0x7f7388102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 -- 172.21.15.23:0/3540461131 >> 172.21.15.23:0/3540461131 conn(0x7f73880fc280 msgr2=0x7f73880fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 -- 172.21.15.23:0/3540461131 shutdown_connections 2024-03-30T04:01:35.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.588+0000 7f7390400700 1 -- 172.21.15.23:0/3540461131 wait complete. 2024-03-30T04:01:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 Processor -- start 2024-03-30T04:01:35.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 -- start start 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 0x7f7388198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73881028a0 0x7f73881a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73881a0a10 con 0x7f7388101f00 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73881a0b50 con 0x7f73881028a0 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f7390400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73881a0ce0 con 0x7f7388100d10 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.589+0000 7f738d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49620/0 (socket says 172.21.15.23:49620) 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 -- 172.21.15.23:0/588768960 learned_addr learned my addr 172.21.15.23:0/588768960 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738da00700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 0x7f7388198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738e400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73881028a0 0x7f73881a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:35.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 -- 172.21.15.23:0/588768960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 msgr2=0x7f7388198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 0x7f7388198ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 -- 172.21.15.23:0/588768960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73881028a0 msgr2=0x7f73881a0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73881028a0 0x7f73881a0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 -- 172.21.15.23:0/588768960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c0095f0 con 0x7f7388101f00 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738da00700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 0x7f7388198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.590+0000 7f738d000700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f737400e9a0 tx=0x7f737400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.591+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374009790 con 0x7f7388101f00 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.591+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7374010430 con 0x7f7388101f00 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.591+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374018680 con 0x7f7388101f00 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.591+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73881a0fc0 con 0x7f7388101f00 2024-03-30T04:01:35.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.591+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73881a13f0 con 0x7f7388101f00 2024-03-30T04:01:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.592+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388067c30 con 0x7f7388101f00 2024-03-30T04:01:35.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.594+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f73740105a0 con 0x7f7388101f00 2024-03-30T04:01:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.595+0000 7f7386000700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 0x7f73700a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.595+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f7374014070 con 0x7f7388101f00 2024-03-30T04:01:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.595+0000 7f738da00700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 0x7f73700a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:35.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.596+0000 7f738da00700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 0x7f73700a3660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f737c009990 tx=0x7f737c01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:35.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.599+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f73740529f0 con 0x7f7388101f00 2024-03-30T04:01:35.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:35 smithi154 ceph-mon[79161]: pgmap v297: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-30T04:01:35.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:35 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-30T04:01:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:35 smithi193 ceph-mon[79581]: pgmap v297: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-30T04:01:35.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:35 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-30T04:01:35.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-mon[38254]: pgmap v297: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-30T04:01:35.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-30T04:01:35.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.857+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f73880747a0 con 0x7f73700a11a0 2024-03-30T04:01:35.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.881+0000 7f7386000700 1 -- 172.21.15.23:0/588768960 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f73880747a0 con 0x7f73700a11a0 2024-03-30T04:01:35.881 INFO:teuthology.orchestra.run.smithi023.stdout:{} 2024-03-30T04:01:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 msgr2=0x7f73700a3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 0x7f73700a3660 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f737c009990 tx=0x7f737c01a040 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 msgr2=0x7f7388199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f737400e9a0 tx=0x7f737400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 shutdown_connections 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73881028a0 0x7f73881a0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f73700a11a0 0x7f73700a3660 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7388100d10 0x7f7388198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 --2- 172.21.15.23:0/588768960 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7388101f00 0x7f7388199420 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.883+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 >> 172.21.15.23:0/588768960 conn(0x7f73880fc280 msgr2=0x7f7388072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.884+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 shutdown_connections 2024-03-30T04:01:35.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:35.884+0000 7f7390400700 1 -- 172.21.15.23:0/588768960 wait complete. 2024-03-30T04:01:35.901 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-30T04:01:35.913 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-30T04:01:35.937 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 -- 172.21.15.23:0/1273171059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70100c80 msgr2=0x7f6a701010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 --2- 172.21.15.23:0/1273171059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70100c80 0x7f6a701010a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6a640099c0 tx=0x7f6a64009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 -- 172.21.15.23:0/1273171059 shutdown_connections 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 --2- 172.21.15.23:0/1273171059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 --2- 172.21.15.23:0/1273171059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70100c80 0x7f6a701010a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 --2- 172.21.15.23:0/1273171059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70101e70 0x7f6a701022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 -- 172.21.15.23:0/1273171059 >> 172.21.15.23:0/1273171059 conn(0x7f6a700fc1f0 msgr2=0x7f6a700fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 -- 172.21.15.23:0/1273171059 shutdown_connections 2024-03-30T04:01:36.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.139+0000 7f6a77600700 1 -- 172.21.15.23:0/1273171059 wait complete. 2024-03-30T04:01:36.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 Processor -- start 2024-03-30T04:01:36.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 -- start start 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70100c80 0x7f6a70198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701a09c0 con 0x7f6a70102810 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701a0b00 con 0x7f6a70101e70 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.140+0000 7f6a77600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a701a0c90 con 0x7f6a70100c80 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a75600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37128/0 (socket says 172.21.15.23:37128) 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 -- 172.21.15.23:0/1431538701 learned_addr learned my addr 172.21.15.23:0/1431538701 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a74c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70100c80 0x7f6a70198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 -- 172.21.15.23:0/1431538701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70100c80 msgr2=0x7f6a70198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70100c80 0x7f6a70198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 -- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 msgr2=0x7f6a701a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 -- 172.21.15.23:0/1431538701 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5c009520 con 0x7f6a70101e70 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a75600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.141+0000 7f6a6fe00700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00eb30 tx=0x7f6a5c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.142+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c009880 con 0x7f6a70101e70 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.142+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a5c0044e0 con 0x7f6a70101e70 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.142+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a640095f0 con 0x7f6a70101e70 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.142+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c003d80 con 0x7f6a70101e70 2024-03-30T04:01:36.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.142+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a701a1260 con 0x7f6a70101e70 2024-03-30T04:01:36.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.143+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70067a80 con 0x7f6a70101e70 2024-03-30T04:01:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.145+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7f6a5c004000 con 0x7f6a70101e70 2024-03-30T04:01:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.146+0000 7f6a6d600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 0x7f6a580a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.146+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f6a5c014070 con 0x7f6a70101e70 2024-03-30T04:01:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.146+0000 7f6a74c00700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 0x7f6a580a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.147+0000 7f6a74c00700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 0x7f6a580a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a64003730 tx=0x7f6a64000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:36.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.149+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a5c096ab0 con 0x7f6a70101e70 2024-03-30T04:01:36.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:35.863+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:36.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:35.863+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:36.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:35.863+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:36.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:35.863+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:36.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:35 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:35.863+0000 7f451c800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.405+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a7010c4e0 con 0x7f6a70101e70 2024-03-30T04:01:36.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.407+0000 7f6a6d600700 1 -- 172.21.15.23:0/1431538701 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f6a5c09a0d0 con 0x7f6a70101e70 2024-03-30T04:01:36.407 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:36.407 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:36.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 msgr2=0x7f6a580a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 0x7f6a580a35e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6a64003730 tx=0x7f6a64000bc0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 msgr2=0x7f6a701993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00eb30 tx=0x7f6a5c00ee40 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 shutdown_connections 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a70102810 0x7f6a701a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6a580a1120 0x7f6a580a35e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a70100c80 0x7f6a70198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 --2- 172.21.15.23:0/1431538701 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a70101e70 0x7f6a701993d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 >> 172.21.15.23:0/1431538701 conn(0x7f6a700fc1f0 msgr2=0x7f6a70072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 shutdown_connections 2024-03-30T04:01:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.410+0000 7f6a77600700 1 -- 172.21.15.23:0/1431538701 wait complete. 2024-03-30T04:01:36.428 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:36.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 -- 172.21.15.23:0/2098443426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 msgr2=0x7fe27c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 --2- 172.21.15.23:0/2098443426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1091c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe27400b0d0 tx=0x7fe27400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 -- 172.21.15.23:0/2098443426 shutdown_connections 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 --2- 172.21.15.23:0/2098443426 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1091c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 --2- 172.21.15.23:0/2098443426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c100d80 0x7fe27c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 --2- 172.21.15.23:0/2098443426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c101f70 0x7fe27c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 -- 172.21.15.23:0/2098443426 >> 172.21.15.23:0/2098443426 conn(0x7fe27c0fc2f0 msgr2=0x7fe27c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:36.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 -- 172.21.15.23:0/2098443426 shutdown_connections 2024-03-30T04:01:36.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.634+0000 7fe284400700 1 -- 172.21.15.23:0/2098443426 wait complete. 2024-03-30T04:01:36.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.635+0000 7fe284400700 1 Processor -- start 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.635+0000 7fe284400700 1 -- start start 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.635+0000 7fe284400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe284400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c101f70 0x7fe27c199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe284400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe284400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c1a0bc0 con 0x7fe27c102910 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe284400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c1a0d00 con 0x7fe27c100d80 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe284400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27c1a0e90 con 0x7fe27c101f70 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe282400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c101f70 0x7fe27c199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37152/0 (socket says 172.21.15.23:37152) 2024-03-30T04:01:36.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 -- 172.21.15.23:0/3220299320 learned_addr learned my addr 172.21.15.23:0/3220299320 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe282400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49676/0 (socket says 172.21.15.23:49676) 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 -- 172.21.15.23:0/3220299320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c101f70 msgr2=0x7fe27c199620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c101f70 0x7fe27c199620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 -- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 msgr2=0x7fe27c1a05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe281a00700 1 -- 172.21.15.23:0/3220299320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274009d00 con 0x7fe27c100d80 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.636+0000 7fe282400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe281a00700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe27000e970 tx=0x7fe27000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270009790 con 0x7fe27c100d80 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe27000c7e0 con 0x7fe27c100d80 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe27c1a1170 con 0x7fe27c100d80 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270003c70 con 0x7fe27c100d80 2024-03-30T04:01:36.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.637+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27c1a1680 con 0x7fe27c100d80 2024-03-30T04:01:36.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.638+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe27c067af0 con 0x7fe27c100d80 2024-03-30T04:01:36.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.640+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 18) v1 ==== 135865+0+0 (secure 0 0 0) 0x7fe27000cc50 con 0x7fe27c100d80 2024-03-30T04:01:36.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.641+0000 7fe27a000700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 0x7fe2640a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:36.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.641+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fe270014070 con 0x7fe27c100d80 2024-03-30T04:01:36.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.641+0000 7fe281000700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 0x7fe2640a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:36.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.642+0000 7fe281000700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 0x7fe2640a35e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe268009c60 tx=0x7fe268009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:36.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.643+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe270096350 con 0x7fe27c100d80 2024-03-30T04:01:36.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:36 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-30T04:01:36.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:36 smithi154 ceph-mon[79161]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:01:36.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:36 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-30T04:01:36.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:36 smithi193 ceph-mon[79581]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:01:36.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:36 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2221297299' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-30T04:01:36.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:36 smithi023 ceph-mon[38254]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:01:36.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.864+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe27c1a1a00 con 0x7fe27c100d80 2024-03-30T04:01:36.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.864+0000 7fe27a000700 1 -- 172.21.15.23:0/3220299320 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe270099970 con 0x7fe27c100d80 2024-03-30T04:01:36.864 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 msgr2=0x7fe2640a35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 0x7fe2640a35e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe268009c60 tx=0x7fe268009260 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 msgr2=0x7fe27c1990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe27000e970 tx=0x7fe27000ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 shutdown_connections 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe27c102910 0x7fe27c1a05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe27c100d80 0x7fe27c1990e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe2640a1120 0x7fe2640a35e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 --2- 172.21.15.23:0/3220299320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe27c101f70 0x7fe27c199620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 >> 172.21.15.23:0/3220299320 conn(0x7fe27c0fc2f0 msgr2=0x7fe27c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:36.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 shutdown_connections 2024-03-30T04:01:36.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:36.867+0000 7fe284400700 1 -- 172.21.15.23:0/3220299320 wait complete. 2024-03-30T04:01:36.885 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:37.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 -- 172.21.15.23:0/2992472592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110100d80 msgr2=0x7f31101011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 --2- 172.21.15.23:0/2992472592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110100d80 0x7f31101011a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f31000099c0 tx=0x7f3100009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 -- 172.21.15.23:0/2992472592 shutdown_connections 2024-03-30T04:01:37.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 --2- 172.21.15.23:0/2992472592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110102910 0x7f31101091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 --2- 172.21.15.23:0/2992472592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110100d80 0x7f31101011a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 --2- 172.21.15.23:0/2992472592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110101f70 0x7f31101023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 -- 172.21.15.23:0/2992472592 >> 172.21.15.23:0/2992472592 conn(0x7f31100fc2f0 msgr2=0x7f31100fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 -- 172.21.15.23:0/2992472592 shutdown_connections 2024-03-30T04:01:37.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.095+0000 7f3116600700 1 -- 172.21.15.23:0/2992472592 wait complete. 2024-03-30T04:01:37.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.096+0000 7f3116600700 1 Processor -- start 2024-03-30T04:01:37.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.096+0000 7f3116600700 1 -- start start 2024-03-30T04:01:37.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110100d80 0x7f3110074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110102910 0x7f31100736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110073de0 con 0x7f3110101f70 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110073f50 con 0x7f3110102910 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f3116600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101a0dc0 con 0x7f3110100d80 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49694/0 (socket says 172.21.15.23:49694) 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 -- 172.21.15.23:0/1559733847 learned_addr learned my addr 172.21.15.23:0/1559733847 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310f400700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110100d80 0x7f3110074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310fe00700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110102910 0x7f31100736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 -- 172.21.15.23:0/1559733847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110100d80 msgr2=0x7f3110074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110100d80 0x7f3110074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 -- 172.21.15.23:0/1559733847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110102910 msgr2=0x7f31100736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110102910 0x7f31100736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.097+0000 7f310ea00700 1 -- 172.21.15.23:0/1559733847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31000095f0 con 0x7f3110101f70 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f310ea00700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00e9a0 tx=0x7f30fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc009790 con 0x7f3110101f70 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30fc00cbd0 con 0x7f3110101f70 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31101a0fc0 con 0x7f3110101f70 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc018680 con 0x7f3110101f70 2024-03-30T04:01:37.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.098+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31101a14f0 con 0x7f3110101f70 2024-03-30T04:01:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.099+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110108250 con 0x7f3110101f70 2024-03-30T04:01:37.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.101+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f30fc00cd40 con 0x7f3110101f70 2024-03-30T04:01:37.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.102+0000 7f3107e00700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 0x7f30f40a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.102+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f30fc014070 con 0x7f3110101f70 2024-03-30T04:01:37.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.103+0000 7f310f400700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 0x7f30f40a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.103+0000 7f310f400700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 0x7f30f40a36f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3100009fd0 tx=0x7f3100019040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:37.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.106+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f30fc0529f0 con 0x7f3110101f70 2024-03-30T04:01:37.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.328+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f311010c5e0 con 0x7f3110101f70 2024-03-30T04:01:37.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.329+0000 7f3107e00700 1 -- 172.21.15.23:0/1559733847 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30fc096270 con 0x7f3110101f70 2024-03-30T04:01:37.329 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:37.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 msgr2=0x7f30f40a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 0x7f30f40a36f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f3100009fd0 tx=0x7f3100019040 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 msgr2=0x7f31100731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f30fc00e9a0 tx=0x7f30fc00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 shutdown_connections 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3110102910 0x7f31100736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f30f40a1230 0x7f30f40a36f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3110100d80 0x7f3110074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 --2- 172.21.15.23:0/1559733847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3110101f70 0x7f31100731a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 >> 172.21.15.23:0/1559733847 conn(0x7f31100fc2f0 msgr2=0x7f3110105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 shutdown_connections 2024-03-30T04:01:37.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.332+0000 7f3116600700 1 -- 172.21.15.23:0/1559733847 wait complete. 2024-03-30T04:01:37.350 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 -- 172.21.15.23:0/4205243161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8108c40 msgr2=0x7fd5e810b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 --2- 172.21.15.23:0/4205243161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8108c40 0x7fd5e810b0d0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000b0a0 tx=0x7fd5e000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 -- 172.21.15.23:0/4205243161 shutdown_connections 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 --2- 172.21.15.23:0/4205243161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8108c40 0x7fd5e810b0d0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 --2- 172.21.15.23:0/4205243161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8074df0 0x7fd5e8073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 --2- 172.21.15.23:0/4205243161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 -- 172.21.15.23:0/4205243161 >> 172.21.15.23:0/4205243161 conn(0x7fd5e80fc2f0 msgr2=0x7fd5e80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:37.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 -- 172.21.15.23:0/4205243161 shutdown_connections 2024-03-30T04:01:37.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.552+0000 7fd5ee000700 1 -- 172.21.15.23:0/4205243161 wait complete. 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.553+0000 7fd5ee000700 1 Processor -- start 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.553+0000 7fd5ee000700 1 -- start start 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.553+0000 7fd5ee000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.553+0000 7fd5ee000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 0x7fd5e8199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5ee000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8108c40 0x7fd5e81a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5ee000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e81a0b30 con 0x7fd5e8074df0 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5ee000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e81a0c70 con 0x7fd5e8073810 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5ee000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e81a0e00 con 0x7fd5e8108c40 2024-03-30T04:01:37.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 0x7fd5e8199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37192/0 (socket says 172.21.15.23:37192) 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 -- 172.21.15.23:0/567379020 learned_addr learned my addr 172.21.15.23:0/567379020 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7e00700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8108c40 0x7fd5e81a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 -- 172.21.15.23:0/567379020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8108c40 msgr2=0x7fd5e81a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8108c40 0x7fd5e81a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 -- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 msgr2=0x7fd5e8199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 0x7fd5e8199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e7400700 1 -- 172.21.15.23:0/567379020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e0009d00 con 0x7fd5e8073810 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.554+0000 7fd5e6a00700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 0x7fd5e8199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5e7400700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800e9a0 tx=0x7fd5d800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8009790 con 0x7fd5e8073810 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5d8010430 con 0x7fd5e8073810 2024-03-30T04:01:37.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e81a10e0 con 0x7fd5e8073810 2024-03-30T04:01:37.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8018680 con 0x7fd5e8073810 2024-03-30T04:01:37.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.555+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e81a15f0 con 0x7fd5e8073810 2024-03-30T04:01:37.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.556+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e8107a00 con 0x7fd5e8073810 2024-03-30T04:01:37.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.558+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd5d80105a0 con 0x7fd5e8073810 2024-03-30T04:01:37.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.559+0000 7fd5dfe00700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 0x7fd5cc0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:37.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.559+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fd5d8014070 con 0x7fd5e8073810 2024-03-30T04:01:37.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.559+0000 7fd5e6a00700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 0x7fd5cc0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:37.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.560+0000 7fd5e6a00700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 0x7fd5cc0a3670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8074b10 tx=0x7fd5d4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:37.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.563+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd5d80529f0 con 0x7fd5e8073810 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: pgmap v298: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1431538701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3220299320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-30T04:01:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:37 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1559733847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: pgmap v298: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1431538701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3220299320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-30T04:01:37.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:37 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1559733847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: pgmap v298: 129 pgs: 129 active+clean; 645 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:01:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1431538701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3220299320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: mgrmap e19: x(active, since 7m), standbys: z, y 2024-03-30T04:01:37.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:37 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1559733847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:37.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.781+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd5e810e9f0 con 0x7fd5e8073810 2024-03-30T04:01:37.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.781+0000 7fd5dfe00700 1 -- 172.21.15.23:0/567379020 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd5d8096250 con 0x7fd5e8073810 2024-03-30T04:01:37.781 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:37.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 msgr2=0x7fd5cc0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 0x7fd5cc0a3670 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8074b10 tx=0x7fd5d4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 msgr2=0x7fd5e8199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800e9a0 tx=0x7fd5d800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 shutdown_connections 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd5cc0a11b0 0x7fd5cc0a3670 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd5e8108c40 0x7fd5e81a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e8073810 0x7fd5e8199000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 --2- 172.21.15.23:0/567379020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd5e8074df0 0x7fd5e8199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 >> 172.21.15.23:0/567379020 conn(0x7fd5e80fc2f0 msgr2=0x7fd5e8104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 shutdown_connections 2024-03-30T04:01:37.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:37.784+0000 7fd5ee000700 1 -- 172.21.15.23:0/567379020 wait complete. 2024-03-30T04:01:37.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 -- 172.21.15.23:0/2286999223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 msgr2=0x7f62f41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 --2- 172.21.15.23:0/2286999223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f41011a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f62e40099c0 tx=0x7f62e4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 -- 172.21.15.23:0/2286999223 shutdown_connections 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 --2- 172.21.15.23:0/2286999223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 --2- 172.21.15.23:0/2286999223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f41011a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 --2- 172.21.15.23:0/2286999223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 -- 172.21.15.23:0/2286999223 >> 172.21.15.23:0/2286999223 conn(0x7f62f40fc2f0 msgr2=0x7f62f40fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 -- 172.21.15.23:0/2286999223 shutdown_connections 2024-03-30T04:01:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.006+0000 7f62f9200700 1 -- 172.21.15.23:0/2286999223 wait complete. 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 Processor -- start 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 -- start start 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a0a40 con 0x7f62f4100d80 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a0b80 con 0x7f62f4101f70 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.007+0000 7f62f9200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f41a0d10 con 0x7f62f4102910 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49738/0 (socket says 172.21.15.23:49738) 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 -- 172.21.15.23:0/474752188 learned_addr learned my addr 172.21.15.23:0/474752188 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37208/0 (socket says 172.21.15.23:37208) 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 -- 172.21.15.23:0/474752188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 msgr2=0x7f62f41a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 -- 172.21.15.23:0/474752188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 msgr2=0x7f62f41994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 -- 172.21.15.23:0/474752188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e40095f0 con 0x7f62f4100d80 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1400700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f2800700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.008+0000 7f62f1e00700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f62e4003730 tx=0x7f62e400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.009+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e401d070 con 0x7f62f4100d80 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.009+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62e400bda0 con 0x7f62f4100d80 2024-03-30T04:01:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.009+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e40173f0 con 0x7f62f4100d80 2024-03-30T04:01:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.009+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f41a0f90 con 0x7f62f4100d80 2024-03-30T04:01:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.009+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f41a1300 con 0x7f62f4100d80 2024-03-30T04:01:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.010+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f4067af0 con 0x7f62f4100d80 2024-03-30T04:01:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.012+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f62e4022470 con 0x7f62f4100d80 2024-03-30T04:01:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.013+0000 7f62eaa00700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 0x7f62d80a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.013+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f62e40cdeb0 con 0x7f62f4100d80 2024-03-30T04:01:38.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.013+0000 7f62f1400700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 0x7f62d80a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.014+0000 7f62f1400700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 0x7f62d80a3670 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62e0010420 tx=0x7f62e00073d0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.018+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f62e4097190 con 0x7f62f4100d80 2024-03-30T04:01:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.232+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f62f41a15c0 con 0x7f62f4100d80 2024-03-30T04:01:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.233+0000 7f62eaa00700 1 -- 172.21.15.23:0/474752188 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f62e409a7b0 con 0x7f62f4100d80 2024-03-30T04:01:38.233 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.235+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 msgr2=0x7f62d80a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.235+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 0x7f62d80a3670 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62e0010420 tx=0x7f62e00073d0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.235+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 msgr2=0x7f62f4198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.235+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f62e4003730 tx=0x7f62e400fa00 comp rx=0 tx=0).stop 2024-03-30T04:01:38.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 shutdown_connections 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f62d80a11b0 0x7f62d80a3670 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f62f4102910 0x7f62f41a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f62f4100d80 0x7f62f4198f60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 --2- 172.21.15.23:0/474752188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62f4101f70 0x7f62f41994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 >> 172.21.15.23:0/474752188 conn(0x7f62f40fc2f0 msgr2=0x7f62f4107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 shutdown_connections 2024-03-30T04:01:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.236+0000 7f62f9200700 1 -- 172.21.15.23:0/474752188 wait complete. 2024-03-30T04:01:38.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 -- 172.21.15.23:0/2975158785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658100c80 msgr2=0x7fb6581010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 --2- 172.21.15.23:0/2975158785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658100c80 0x7fb6581010a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00b0d0 tx=0x7fb64c00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 -- 172.21.15.23:0/2975158785 shutdown_connections 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 --2- 172.21.15.23:0/2975158785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658102810 0x7fb6581090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 --2- 172.21.15.23:0/2975158785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658100c80 0x7fb6581010a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 --2- 172.21.15.23:0/2975158785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658101e70 0x7fb6581022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 -- 172.21.15.23:0/2975158785 >> 172.21.15.23:0/2975158785 conn(0x7fb6580fc1f0 msgr2=0x7fb6580fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 -- 172.21.15.23:0/2975158785 shutdown_connections 2024-03-30T04:01:38.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.457+0000 7fb660600700 1 -- 172.21.15.23:0/2975158785 wait complete. 2024-03-30T04:01:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.458+0000 7fb660600700 1 Processor -- start 2024-03-30T04:01:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.458+0000 7fb660600700 1 -- start start 2024-03-30T04:01:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658100c80 0x7fb658199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658102810 0x7fb6581a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581a0af0 con 0x7fb658101e70 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581a0c30 con 0x7fb658102810 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb660600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581a0dc0 con 0x7fb658100c80 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39714/0 (socket says 172.21.15.23:39714) 2024-03-30T04:01:38.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 -- 172.21.15.23:0/1322693105 learned_addr learned my addr 172.21.15.23:0/1322693105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65e600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658102810 0x7fb6581a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65dc00700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658100c80 0x7fb658199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 -- 172.21.15.23:0/1322693105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658100c80 msgr2=0x7fb658199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658100c80 0x7fb658199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 -- 172.21.15.23:0/1322693105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658102810 msgr2=0x7fb6581a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658102810 0x7fb6581a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.459+0000 7fb65d200700 1 -- 172.21.15.23:0/1322693105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c009d00 con 0x7fb658101e70 2024-03-30T04:01:38.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb65d200700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb64400e9a0 tx=0x7fb64400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644009790 con 0x7fb658101e70 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb644010430 con 0x7fb658101e70 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644018680 con 0x7fb658101e70 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581a10a0 con 0x7fb658101e70 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.460+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6581a1560 con 0x7fb658101e70 2024-03-30T04:01:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.461+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb658067a10 con 0x7fb658101e70 2024-03-30T04:01:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.463+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb6440105a0 con 0x7fb658101e70 2024-03-30T04:01:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.464+0000 7fb656000700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 0x7fb6400a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.464+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fb644014070 con 0x7fb658101e70 2024-03-30T04:01:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.464+0000 7fb65dc00700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 0x7fb6400a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.465+0000 7fb65dc00700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 0x7fb6400a3670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009fd0 tx=0x7fb64c00b040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.468+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb6440529f0 con 0x7fb658101e70 2024-03-30T04:01:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:38 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/567379020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:38 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/474752188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:38 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/567379020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:38 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/474752188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.725+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb65810c4e0 con 0x7fb658101e70 2024-03-30T04:01:38.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.728+0000 7fb656000700 1 -- 172.21.15.23:0/1322693105 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7fb644096250 con 0x7fb658101e70 2024-03-30T04:01:38.727 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:38.727 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 msgr2=0x7fb6400a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 0x7fb6400a3670 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009fd0 tx=0x7fb64c00b040 comp rx=0 tx=0).stop 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 msgr2=0x7fb658199550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb64400e9a0 tx=0x7fb64400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 shutdown_connections 2024-03-30T04:01:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb658102810 0x7fb6581a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb6400a11b0 0x7fb6400a3670 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb658100c80 0x7fb658199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 --2- 172.21.15.23:0/1322693105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb658101e70 0x7fb658199550 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.729+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 >> 172.21.15.23:0/1322693105 conn(0x7fb6580fc1f0 msgr2=0x7fb658072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.730+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 shutdown_connections 2024-03-30T04:01:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.730+0000 7fb660600700 1 -- 172.21.15.23:0/1322693105 wait complete. 2024-03-30T04:01:38.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:38 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/567379020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:38 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/474752188' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:38.748 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:38.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 -- 172.21.15.23:0/2283513271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530102910 msgr2=0x7f85301091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 --2- 172.21.15.23:0/2283513271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530102910 0x7f85301091c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f852800b0d0 tx=0x7f852800b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 -- 172.21.15.23:0/2283513271 shutdown_connections 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 --2- 172.21.15.23:0/2283513271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530102910 0x7f85301091c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 --2- 172.21.15.23:0/2283513271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530100d80 0x7f85301011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 --2- 172.21.15.23:0/2283513271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 0x7f85301023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 -- 172.21.15.23:0/2283513271 >> 172.21.15.23:0/2283513271 conn(0x7f85300fc2f0 msgr2=0x7f85300fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 -- 172.21.15.23:0/2283513271 shutdown_connections 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.950+0000 7f8536c00700 1 -- 172.21.15.23:0/2283513271 wait complete. 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 Processor -- start 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 -- start start 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 0x7f8530199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530102910 0x7f85301a0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301a0b60 con 0x7f8530100d80 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301a0ca0 con 0x7f8530101f70 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.951+0000 7f8536c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85301a0e30 con 0x7f8530102910 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39720/0 (socket says 172.21.15.23:39720) 2024-03-30T04:01:38.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 -- 172.21.15.23:0/675460937 learned_addr learned my addr 172.21.15.23:0/675460937 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852f400700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 0x7f8530199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f8534c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530102910 0x7f85301a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 -- 172.21.15.23:0/675460937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530102910 msgr2=0x7f85301a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530102910 0x7f85301a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 -- 172.21.15.23:0/675460937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 msgr2=0x7f8530199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 0x7f8530199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 -- 172.21.15.23:0/675460937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8528009d00 con 0x7f8530100d80 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852fe00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f852000e970 tx=0x7f852000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520009790 con 0x7f8530100d80 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f852000c7e0 con 0x7f8530100d80 2024-03-30T04:01:38.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.952+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85301a1110 con 0x7f8530100d80 2024-03-30T04:01:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.953+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8520003c70 con 0x7f8530100d80 2024-03-30T04:01:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.953+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85301a1670 con 0x7f8530100d80 2024-03-30T04:01:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.954+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8530067af0 con 0x7f8530100d80 2024-03-30T04:01:38.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.955+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f852000cc50 con 0x7f8530100d80 2024-03-30T04:01:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.957+0000 7f852cc00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 0x7f85140a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.957+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f8520014070 con 0x7f8530100d80 2024-03-30T04:01:38.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.957+0000 7f852f400700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 0x7f85140a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:38.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.958+0000 7f852f400700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 0x7f85140a36f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f851c009c60 tx=0x7f851c009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:38.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:38.961+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f85200d1070 con 0x7f8530100d80 2024-03-30T04:01:39.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.178+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f85301a19f0 con 0x7f8530100d80 2024-03-30T04:01:39.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.179+0000 7f852cc00700 1 -- 172.21.15.23:0/675460937 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85200961c0 con 0x7f8530100d80 2024-03-30T04:01:39.178 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 msgr2=0x7f85140a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 0x7f85140a36f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f851c009c60 tx=0x7f851c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 msgr2=0x7f8530199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f852000e970 tx=0x7f852000ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 shutdown_connections 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f85140a1230 0x7f85140a36f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8530102910 0x7f85301a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8530100d80 0x7f8530199030 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 --2- 172.21.15.23:0/675460937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8530101f70 0x7f8530199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 >> 172.21.15.23:0/675460937 conn(0x7f85300fc2f0 msgr2=0x7f8530072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:39.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 shutdown_connections 2024-03-30T04:01:39.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.181+0000 7f8536c00700 1 -- 172.21.15.23:0/675460937 wait complete. 2024-03-30T04:01:39.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 -- 172.21.15.23:0/4080552753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0101f70 msgr2=0x7f16d01023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 --2- 172.21.15.23:0/4080552753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0101f70 0x7f16d01023d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0099c0 tx=0x7f16bc009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 -- 172.21.15.23:0/4080552753 shutdown_connections 2024-03-30T04:01:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 --2- 172.21.15.23:0/4080552753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0102910 0x7f16d01091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 --2- 172.21.15.23:0/4080552753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 0x7f16d01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 --2- 172.21.15.23:0/4080552753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0101f70 0x7f16d01023d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 -- 172.21.15.23:0/4080552753 >> 172.21.15.23:0/4080552753 conn(0x7f16d00fc2f0 msgr2=0x7f16d00fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 -- 172.21.15.23:0/4080552753 shutdown_connections 2024-03-30T04:01:39.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.402+0000 7f16d7400700 1 -- 172.21.15.23:0/4080552753 wait complete. 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.403+0000 7f16d7400700 1 Processor -- start 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 -- start start 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 0x7f16d0198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0101f70 0x7f16d0199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01a0a30 con 0x7f16d0102910 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01a0b70 con 0x7f16d0100d80 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d7400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16d01a0d00 con 0x7f16d0101f70 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39742/0 (socket says 172.21.15.23:39742) 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d5400700 1 -- 172.21.15.23:0/1282329008 learned_addr learned my addr 172.21.15.23:0/1282329008 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16d4a00700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 0x7f16d0198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.404+0000 7f16cfe00700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0101f70 0x7f16d0199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 -- 172.21.15.23:0/1282329008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0101f70 msgr2=0x7f16d0199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0101f70 0x7f16d0199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 -- 172.21.15.23:0/1282329008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 msgr2=0x7f16d0198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 0x7f16d0198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 -- 172.21.15.23:0/1282329008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16bc0095f0 con 0x7f16d0102910 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d5400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ef30 tx=0x7f16c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c8018070 con 0x7f16d0102910 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16c800f040 con 0x7f16d0102910 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16c800cd00 con 0x7f16d0102910 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16d01a0f80 con 0x7f16d0102910 2024-03-30T04:01:39.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.405+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16d01a13b0 con 0x7f16d0102910 2024-03-30T04:01:39.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.406+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16d0067af0 con 0x7f16d0102910 2024-03-30T04:01:39.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.408+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f16c8003a00 con 0x7f16d0102910 2024-03-30T04:01:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.410+0000 7f16cd600700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 0x7f16b80a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.410+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f16c80cd230 con 0x7f16d0102910 2024-03-30T04:01:39.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.410+0000 7f16d4a00700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 0x7f16b80a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.411+0000 7f16d4a00700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 0x7f16b80a36f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f16d0101dd0 tx=0x7f16c4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:39.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.412+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f16c8096510 con 0x7f16d0102910 2024-03-30T04:01:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:39 smithi023 ceph-mon[38254]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-30T04:01:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1322693105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:39 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/675460937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:39.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.628+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f16d010c850 con 0x7f16d0102910 2024-03-30T04:01:39.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.628+0000 7f16cd600700 1 -- 172.21.15.23:0/1282329008 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f16c8099b30 con 0x7f16d0102910 2024-03-30T04:01:39.628 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 msgr2=0x7f16b80a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 0x7f16b80a36f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f16d0101dd0 tx=0x7f16c4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 msgr2=0x7f16d01a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f16c800ef30 tx=0x7f16c800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 shutdown_connections 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16d0102910 0x7f16d01a0460 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d0100d80 0x7f16d0198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f16b80a1230 0x7f16b80a36f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 --2- 172.21.15.23:0/1282329008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16d0101f70 0x7f16d0199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 >> 172.21.15.23:0/1282329008 conn(0x7f16d00fc2f0 msgr2=0x7f16d0072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 shutdown_connections 2024-03-30T04:01:39.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.631+0000 7f16d7400700 1 -- 172.21.15.23:0/1282329008 wait complete. 2024-03-30T04:01:39.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:39 smithi154 ceph-mon[79161]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-30T04:01:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:39 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1322693105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:39 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/675460937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:39.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:39 smithi193 ceph-mon[79581]: pgmap v299: 129 pgs: 129 active+clean; 646 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-30T04:01:39.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:39 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1322693105' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:39.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:39 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/675460937' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 -- 172.21.15.23:0/866607153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc106a70 msgr2=0x7fb9bc108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 --2- 172.21.15.23:0/866607153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc106a70 0x7fb9bc108ee0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb9b400b0a0 tx=0x7fb9b400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 -- 172.21.15.23:0/866607153 shutdown_connections 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 --2- 172.21.15.23:0/866607153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc106a70 0x7fb9bc108ee0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 --2- 172.21.15.23:0/866607153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc101810 0x7fb9bc103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 --2- 172.21.15.23:0/866607153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc106530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 -- 172.21.15.23:0/866607153 >> 172.21.15.23:0/866607153 conn(0x7fb9bc0fb200 msgr2=0x7fb9bc0fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 -- 172.21.15.23:0/866607153 shutdown_connections 2024-03-30T04:01:39.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.852+0000 7fb9c3600700 1 -- 172.21.15.23:0/866607153 wait complete. 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.853+0000 7fb9c3600700 1 Processor -- start 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.853+0000 7fb9c3600700 1 -- start start 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc101810 0x7fb9bc101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc0ff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1004a0 con 0x7fb9bc106a70 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc100610 con 0x7fb9bc104140 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c3600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9bc1a0e20 con 0x7fb9bc101810 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc0ff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39750/0 (socket says 172.21.15.23:39750) 2024-03-30T04:01:39.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 -- 172.21.15.23:0/2751052936 learned_addr learned my addr 172.21.15.23:0/2751052936 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c0c00700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc101810 0x7fb9bc101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 -- 172.21.15.23:0/2751052936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc101810 msgr2=0x7fb9bc101130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc101810 0x7fb9bc101130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 -- 172.21.15.23:0/2751052936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 msgr2=0x7fb9bc0ff7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.854+0000 7fb9c1600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc0ff7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9c1600700 1 -- 172.21.15.23:0/2751052936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4009d00 con 0x7fb9bc106a70 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9bbe00700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9c1600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4007a60 tx=0x7fb9b4007930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b400e040 con 0x7fb9bc106a70 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9bc1a1050 con 0x7fb9bc106a70 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9b4004c50 con 0x7fb9bc106a70 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b40052a0 con 0x7fb9bc106a70 2024-03-30T04:01:39.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.855+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9bc1a1420 con 0x7fb9bc106a70 2024-03-30T04:01:39.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.857+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9bc0fcd80 con 0x7fb9bc106a70 2024-03-30T04:01:39.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.859+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fb9b401d470 con 0x7fb9bc106a70 2024-03-30T04:01:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.860+0000 7fb9b9600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 0x7fb9a40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:39.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.860+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fb9b40ce490 con 0x7fb9bc106a70 2024-03-30T04:01:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.860+0000 7fb9c0c00700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 0x7fb9a40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:39.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.861+0000 7fb9c0c00700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 0x7fb9a40a3670 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009c60 tx=0x7fb9b0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:39.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:39.864+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb9b4097770 con 0x7fb9bc106a70 2024-03-30T04:01:40.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.081+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb9bc10c6d0 con 0x7fb9bc106a70 2024-03-30T04:01:40.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.081+0000 7fb9b9600700 1 -- 172.21.15.23:0/2751052936 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb9b409ad90 con 0x7fb9bc106a70 2024-03-30T04:01:40.081 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 msgr2=0x7fb9a40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 0x7fb9a40a3670 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009c60 tx=0x7fb9b0009260 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 msgr2=0x7fb9bc0ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4007a60 tx=0x7fb9b4007930 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 shutdown_connections 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9bc106a70 0x7fb9bc0ffd00 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb9a40a11b0 0x7fb9a40a3670 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9bc101810 0x7fb9bc101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 --2- 172.21.15.23:0/2751052936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb9bc104140 0x7fb9bc0ff7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 >> 172.21.15.23:0/2751052936 conn(0x7fb9bc0fb200 msgr2=0x7fb9bc102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:40.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 shutdown_connections 2024-03-30T04:01:40.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.084+0000 7fb9c3600700 1 -- 172.21.15.23:0/2751052936 wait complete. 2024-03-30T04:01:40.103 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:40.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 -- 172.21.15.23:0/4155189685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64100d80 msgr2=0x7fca641011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 --2- 172.21.15.23:0/4155189685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64100d80 0x7fca641011a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fca580099c0 tx=0x7fca58009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 -- 172.21.15.23:0/4155189685 shutdown_connections 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 --2- 172.21.15.23:0/4155189685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64102910 0x7fca641091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 --2- 172.21.15.23:0/4155189685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64100d80 0x7fca641011a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 --2- 172.21.15.23:0/4155189685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64101f70 0x7fca641023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 -- 172.21.15.23:0/4155189685 >> 172.21.15.23:0/4155189685 conn(0x7fca640fc2f0 msgr2=0x7fca640fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.306+0000 7fca6b600700 1 -- 172.21.15.23:0/4155189685 shutdown_connections 2024-03-30T04:01:40.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.307+0000 7fca6b600700 1 -- 172.21.15.23:0/4155189685 wait complete. 2024-03-30T04:01:40.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 Processor -- start 2024-03-30T04:01:40.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 -- start start 2024-03-30T04:01:40.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64100d80 0x7fca64198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64102910 0x7fca641a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a0a60 con 0x7fca64101f70 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a0ba0 con 0x7fca64102910 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca6b600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca641a0d30 con 0x7fca64100d80 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca63e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca68c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64100d80 0x7fca64198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca63e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39766/0 (socket says 172.21.15.23:39766) 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca63e00700 1 -- 172.21.15.23:0/1392943275 learned_addr learned my addr 172.21.15.23:0/1392943275 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.308+0000 7fca69600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64102910 0x7fca641a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 -- 172.21.15.23:0/1392943275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64100d80 msgr2=0x7fca64198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64100d80 0x7fca64198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 -- 172.21.15.23:0/1392943275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64102910 msgr2=0x7fca641a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64102910 0x7fca641a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 -- 172.21.15.23:0/1392943275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca580095f0 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca63e00700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fca5000e9a0 tx=0x7fca5000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50009790 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca50010430 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca641a0fb0 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.309+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca50018680 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.310+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca641a14e0 con 0x7fca64101f70 2024-03-30T04:01:40.310 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.310+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca64067af0 con 0x7fca64101f70 2024-03-30T04:01:40.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.312+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fca500105a0 con 0x7fca64101f70 2024-03-30T04:01:40.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.314+0000 7fca61600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 0x7fca4c0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.314+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fca50014070 con 0x7fca64101f70 2024-03-30T04:01:40.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.314+0000 7fca68c00700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 0x7fca4c0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.315+0000 7fca68c00700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 0x7fca4c0a36f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca58009990 tx=0x7fca58000bc0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:40.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.318+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fca500529f0 con 0x7fca64101f70 2024-03-30T04:01:40.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.540+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fca641a1780 con 0x7fca64101f70 2024-03-30T04:01:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.540+0000 7fca61600700 1 -- 172.21.15.23:0/1392943275 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fca500961d0 con 0x7fca64101f70 2024-03-30T04:01:40.539 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 msgr2=0x7fca4c0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 0x7fca4c0a36f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca58009990 tx=0x7fca58000bc0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 msgr2=0x7fca64199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fca5000e9a0 tx=0x7fca5000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 shutdown_connections 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca64102910 0x7fca641a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fca4c0a1230 0x7fca4c0a36f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64100d80 0x7fca64198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 --2- 172.21.15.23:0/1392943275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fca64101f70 0x7fca64199470 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 >> 172.21.15.23:0/1392943275 conn(0x7fca640fc2f0 msgr2=0x7fca64072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 shutdown_connections 2024-03-30T04:01:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.543+0000 7fca6b600700 1 -- 172.21.15.23:0/1392943275 wait complete. 2024-03-30T04:01:40.561 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-30T04:01:40.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:40 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1282329008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:40 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2751052936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:40 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1282329008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:40 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2751052936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 -- 172.21.15.23:0/255641668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8100c80 msgr2=0x7f77d81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 --2- 172.21.15.23:0/255641668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8100c80 0x7f77d81010a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009990 tx=0x7f77c8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 -- 172.21.15.23:0/255641668 shutdown_connections 2024-03-30T04:01:40.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 --2- 172.21.15.23:0/255641668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8102810 0x7f77d81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 --2- 172.21.15.23:0/255641668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8100c80 0x7f77d81010a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 --2- 172.21.15.23:0/255641668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.761+0000 7f77dd000700 1 -- 172.21.15.23:0/255641668 >> 172.21.15.23:0/255641668 conn(0x7f77d80fc1f0 msgr2=0x7f77d80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.762+0000 7f77dd000700 1 -- 172.21.15.23:0/255641668 shutdown_connections 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.762+0000 7f77dd000700 1 -- 172.21.15.23:0/255641668 wait complete. 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 Processor -- start 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 -- start start 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d8199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8102810 0x7f77d81a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81a09d0 con 0x7f77d8100c80 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81a0b10 con 0x7f77d8101e70 2024-03-30T04:01:40.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77dd000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d81a0ca0 con 0x7f77d8102810 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77d5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77d5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39782/0 (socket says 172.21.15.23:39782) 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77d5e00700 1 -- 172.21.15.23:0/2394804651 learned_addr learned my addr 172.21.15.23:0/2394804651 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77d6800700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8102810 0x7f77d81a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.763+0000 7f77d5400700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d8199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 -- 172.21.15.23:0/2394804651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8102810 msgr2=0x7f77d81a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8102810 0x7f77d81a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 -- 172.21.15.23:0/2394804651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 msgr2=0x7f77d8199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d8199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 -- 172.21.15.23:0/2394804651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4009520 con 0x7f77d8100c80 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5400700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77d5e00700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f77c8000c00 tx=0x7f77c8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:40.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c801d070 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c80095f0 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77c8021410 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8017650 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.764+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d81a11c0 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.766+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d8067a10 con 0x7f77d8100c80 2024-03-30T04:01:40.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.767+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f77c8021580 con 0x7f77d8100c80 2024-03-30T04:01:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.770+0000 7f77cea00700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 0x7f77bc0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.770+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f77c80ce120 con 0x7f77d8100c80 2024-03-30T04:01:40.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.770+0000 7f77d5400700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 0x7f77bc0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:40.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.770+0000 7f77d5400700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 0x7f77bc0a36f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009e40 tx=0x7f77c4009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:40.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:40 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1282329008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:40 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2751052936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:40.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:40.774+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f77c8097400 con 0x7f77d8100c80 2024-03-30T04:01:41.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.040+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77d810c4e0 con 0x7f77d8100c80 2024-03-30T04:01:41.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.041+0000 7f77cea00700 1 -- 172.21.15.23:0/2394804651 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f77c809aa20 con 0x7f77d8100c80 2024-03-30T04:01:41.040 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:41.040 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:01:35.365141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 msgr2=0x7f77bc0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 0x7f77bc0a36f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009e40 tx=0x7f77c4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 msgr2=0x7f77d8198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f77c8000c00 tx=0x7f77c8009580 comp rx=0 tx=0).stop 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 shutdown_connections 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f77bc0a1230 0x7f77bc0a36f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d8102810 0x7f77d81a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f77d8100c80 0x7f77d8198ef0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 --2- 172.21.15.23:0/2394804651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77d8101e70 0x7f77d8199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.043+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 >> 172.21.15.23:0/2394804651 conn(0x7f77d80fc1f0 msgr2=0x7f77d8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.044+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 shutdown_connections 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.044+0000 7f77dd000700 1 -- 172.21.15.23:0/2394804651 wait complete. 2024-03-30T04:01:41.043 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-30T04:01:41.062 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-30T04:01:41.062 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-30T04:01:41.062 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T04:01:41.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 -- 172.21.15.23:0/2267704300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368101e70 msgr2=0x7f83681022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 --2- 172.21.15.23:0/2267704300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368101e70 0x7f83681022d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f83540099c0 tx=0x7f8354009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 -- 172.21.15.23:0/2267704300 shutdown_connections 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 --2- 172.21.15.23:0/2267704300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 0x7f83681090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 --2- 172.21.15.23:0/2267704300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368100c80 0x7f83681010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 --2- 172.21.15.23:0/2267704300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368101e70 0x7f83681022d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 -- 172.21.15.23:0/2267704300 >> 172.21.15.23:0/2267704300 conn(0x7f83680fc1f0 msgr2=0x7f83680fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 -- 172.21.15.23:0/2267704300 shutdown_connections 2024-03-30T04:01:41.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.263+0000 7f836fa00700 1 -- 172.21.15.23:0/2267704300 wait complete. 2024-03-30T04:01:41.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 Processor -- start 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 -- start start 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368101e70 0x7f8368199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 0x7f83681a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681a0900 con 0x7f8368100c80 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681a0a70 con 0x7f8368101e70 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836fa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681a0c30 con 0x7f8368102810 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.264+0000 7f836d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39798/0 (socket says 172.21.15.23:39798) 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 -- 172.21.15.23:0/1430412618 learned_addr learned my addr 172.21.15.23:0/1430412618 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836da00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 0x7f83681a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f8367e00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368101e70 0x7f8368199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 -- 172.21.15.23:0/1430412618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 msgr2=0x7f83681a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 0x7f83681a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 -- 172.21.15.23:0/1430412618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368101e70 msgr2=0x7f8368199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368101e70 0x7f8368199350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 -- 172.21.15.23:0/1430412618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83540095f0 con 0x7f8368100c80 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f836d000700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e970 tx=0x7f835c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c009790 con 0x7f8368100c80 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835c00c7e0 con 0x7f8368100c80 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.265+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c003c70 con 0x7f8368100c80 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.266+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83681a0f10 con 0x7f8368100c80 2024-03-30T04:01:41.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.266+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83681a12e0 con 0x7f8368100c80 2024-03-30T04:01:41.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.267+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8368067a10 con 0x7f8368100c80 2024-03-30T04:01:41.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.269+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f835c00cc50 con 0x7f8368100c80 2024-03-30T04:01:41.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.270+0000 7f8365600700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 0x7f83500a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.270+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f835c014070 con 0x7f8368100c80 2024-03-30T04:01:41.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.270+0000 7f8367e00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 0x7f83500a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.271+0000 7f8367e00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 0x7f83500a36f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8354006010 tx=0x7f835400b540 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:41.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.275+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f835c0961f0 con 0x7f8368100c80 2024-03-30T04:01:41.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.488+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8368074700 con 0x7f83500a1230 2024-03-30T04:01:41.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.502+0000 7f8365600700 1 -- 172.21.15.23:0/1430412618 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427389 (secure 0 0 0) 0x7f8368074700 con 0x7f83500a1230 2024-03-30T04:01:41.507 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 msgr2=0x7f83500a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 0x7f83500a36f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8354006010 tx=0x7f835400b540 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 msgr2=0x7f8368198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e970 tx=0x7f835c00ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 shutdown_connections 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f83500a1230 0x7f83500a36f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8368102810 0x7f83681a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8368100c80 0x7f8368198e10 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 --2- 172.21.15.23:0/1430412618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8368101e70 0x7f8368199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.507+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 >> 172.21.15.23:0/1430412618 conn(0x7f83680fc1f0 msgr2=0x7f8368072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.508+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 shutdown_connections 2024-03-30T04:01:41.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.508+0000 7f836fa00700 1 -- 172.21.15.23:0/1430412618 wait complete. 2024-03-30T04:01:41.513 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:01:41.525 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-03-30T04:01:40.304131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669284,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1042,"num_read_kb":662,"num_write":438,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":864,"ondisk_log_size":864,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383236,"kb_used_data":13960,"kb_used_omap":18,"kb_used_meta":369133,"kb_avail":1124558588,"statfs":{"total":1151940427776,"available":1151547994112,"internally_reserved":0,"allocated":14295040,"data_stored":7888459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":377992517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8625,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":136,"num_read_kb":82,"num_write":11,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991830+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T04:01:17.991830+0000","last_peered":"2024-03-30T04:01:17.991830+0000","last_clean":"2024-03-30T04:01:17.991830+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T04:01:17.991830+0000","last_undegraded":"2024-03-30T04:01:17.991830+0000","last_fullsized":"2024-03-30T04:01:17.991830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986818+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T04:01:17.986818+0000","last_peered":"2024-03-30T04:01:17.986818+0000","last_clean":"2024-03-30T04:01:17.986818+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T04:01:17.986818+0000","last_undegraded":"2024-03-30T04:01:17.986818+0000","last_fullsized":"2024-03-30T04:01:17.986818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986844+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T04:01:17.986844+0000","last_peered":"2024-03-30T04:01:17.986844+0000","last_clean":"2024-03-30T04:01:17.986844+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T04:01:17.986844+0000","last_undegraded":"2024-03-30T04:01:17.986844+0000","last_fullsized":"2024-03-30T04:01:17.986844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990923+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T04:01:17.990923+0000","last_peered":"2024-03-30T04:01:17.990923+0000","last_clean":"2024-03-30T04:01:17.990923+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T04:01:17.990923+0000","last_undegraded":"2024-03-30T04:01:17.990923+0000","last_fullsized":"2024-03-30T04:01:17.990923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986025+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T04:01:17.986025+0000","last_peered":"2024-03-30T04:01:17.986025+0000","last_clean":"2024-03-30T04:01:17.986025+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T04:01:17.986025+0000","last_undegraded":"2024-03-30T04:01:17.986025+0000","last_fullsized":"2024-03-30T04:01:17.986025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986574+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T04:01:17.986574+0000","last_peered":"2024-03-30T04:01:17.986574+0000","last_clean":"2024-03-30T04:01:17.986574+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T04:01:17.986574+0000","last_undegraded":"2024-03-30T04:01:17.986574+0000","last_fullsized":"2024-03-30T04:01:17.986574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991399+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T04:01:17.991399+0000","last_peered":"2024-03-30T04:01:17.991399+0000","last_clean":"2024-03-30T04:01:17.991399+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T04:01:17.991399+0000","last_undegraded":"2024-03-30T04:01:17.991399+0000","last_fullsized":"2024-03-30T04:01:17.991399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989988+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T04:01:17.989988+0000","last_peered":"2024-03-30T04:01:17.989988+0000","last_clean":"2024-03-30T04:01:17.989988+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T04:01:17.989988+0000","last_undegraded":"2024-03-30T04:01:17.989988+0000","last_fullsized":"2024-03-30T04:01:17.989988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039169+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T04:01:18.039169+0000","last_peered":"2024-03-30T04:01:18.039169+0000","last_clean":"2024-03-30T04:01:18.039169+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T04:01:18.039169+0000","last_undegraded":"2024-03-30T04:01:18.039169+0000","last_fullsized":"2024-03-30T04:01:18.039169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986792+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T04:01:17.986792+0000","last_peered":"2024-03-30T04:01:17.986792+0000","last_clean":"2024-03-30T04:01:17.986792+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T04:01:17.986792+0000","last_undegraded":"2024-03-30T04:01:17.986792+0000","last_fullsized":"2024-03-30T04:01:17.986792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990814+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T04:01:17.990814+0000","last_peered":"2024-03-30T04:01:17.990814+0000","last_clean":"2024-03-30T04:01:17.990814+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T04:01:17.990814+0000","last_undegraded":"2024-03-30T04:01:17.990814+0000","last_fullsized":"2024-03-30T04:01:17.990814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990767+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T04:01:17.990767+0000","last_peered":"2024-03-30T04:01:17.990767+0000","last_clean":"2024-03-30T04:01:17.990767+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T04:01:17.990767+0000","last_undegraded":"2024-03-30T04:01:17.990767+0000","last_fullsized":"2024-03-30T04:01:17.990767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986474+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T04:01:17.986474+0000","last_peered":"2024-03-30T04:01:17.986474+0000","last_clean":"2024-03-30T04:01:17.986474+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T04:01:17.986474+0000","last_undegraded":"2024-03-30T04:01:17.986474+0000","last_fullsized":"2024-03-30T04:01:17.986474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986493+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T04:01:17.986493+0000","last_peered":"2024-03-30T04:01:17.986493+0000","last_clean":"2024-03-30T04:01:17.986493+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T04:01:17.986493+0000","last_undegraded":"2024-03-30T04:01:17.986493+0000","last_fullsized":"2024-03-30T04:01:17.986493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989473+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T04:01:17.989473+0000","last_peered":"2024-03-30T04:01:17.989473+0000","last_clean":"2024-03-30T04:01:17.989473+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T04:01:17.989473+0000","last_undegraded":"2024-03-30T04:01:17.989473+0000","last_fullsized":"2024-03-30T04:01:17.989473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991668+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T04:01:17.991668+0000","last_peered":"2024-03-30T04:01:17.991668+0000","last_clean":"2024-03-30T04:01:17.991668+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T04:01:17.991668+0000","last_undegraded":"2024-03-30T04:01:17.991668+0000","last_fullsized":"2024-03-30T04:01:17.991668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992709+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T04:01:17.992709+0000","last_peered":"2024-03-30T04:01:17.992709+0000","last_clean":"2024-03-30T04:01:17.992709+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T04:01:17.992709+0000","last_undegraded":"2024-03-30T04:01:17.992709+0000","last_fullsized":"2024-03-30T04:01:17.992709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989001+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T04:01:17.989001+0000","last_peered":"2024-03-30T04:01:17.989001+0000","last_clean":"2024-03-30T04:01:17.989001+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T04:01:17.989001+0000","last_undegraded":"2024-03-30T04:01:17.989001+0000","last_fullsized":"2024-03-30T04:01:17.989001+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989435+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T04:01:17.989435+0000","last_peered":"2024-03-30T04:01:17.989435+0000","last_clean":"2024-03-30T04:01:17.989435+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T04:01:17.989435+0000","last_undegraded":"2024-03-30T04:01:17.989435+0000","last_fullsized":"2024-03-30T04:01:17.989435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'15","reported_seq":119,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:35.087556+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:01:35.087556+0000","last_peered":"2024-03-30T04:01:35.087556+0000","last_clean":"2024-03-30T04:01:35.087556+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:01:35.087556+0000","last_undegraded":"2024-03-30T04:01:35.087556+0000","last_fullsized":"2024-03-30T04:01:35.087556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993165+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T04:01:17.993165+0000","last_peered":"2024-03-30T04:01:17.993165+0000","last_clean":"2024-03-30T04:01:17.993165+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T04:01:17.993165+0000","last_undegraded":"2024-03-30T04:01:17.993165+0000","last_fullsized":"2024-03-30T04:01:17.993165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989691+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T04:01:17.989691+0000","last_peered":"2024-03-30T04:01:17.989691+0000","last_clean":"2024-03-30T04:01:17.989691+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T04:01:17.989691+0000","last_undegraded":"2024-03-30T04:01:17.989691+0000","last_fullsized":"2024-03-30T04:01:17.989691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993262+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T04:01:17.993262+0000","last_peered":"2024-03-30T04:01:17.993262+0000","last_clean":"2024-03-30T04:01:17.993262+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T04:01:17.993262+0000","last_undegraded":"2024-03-30T04:01:17.993262+0000","last_fullsized":"2024-03-30T04:01:17.993262+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986607+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T04:01:17.986607+0000","last_peered":"2024-03-30T04:01:17.986607+0000","last_clean":"2024-03-30T04:01:17.986607+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T04:01:17.986607+0000","last_undegraded":"2024-03-30T04:01:17.986607+0000","last_fullsized":"2024-03-30T04:01:17.986607+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990211+0000","last_change":"2024-03-30T04:00:18.019833+0000","last_active":"2024-03-30T04:01:17.990211+0000","last_peered":"2024-03-30T04:01:17.990211+0000","last_clean":"2024-03-30T04:01:17.990211+0000","last_became_active":"2024-03-30T04:00:18.019445+0000","last_became_peered":"2024-03-30T04:00:18.019445+0000","last_unstale":"2024-03-30T04:01:17.990211+0000","last_undegraded":"2024-03-30T04:01:17.990211+0000","last_fullsized":"2024-03-30T04:01:17.990211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:41:55.882417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'13","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.998604+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:01:18.998604+0000","last_peered":"2024-03-30T04:01:18.998604+0000","last_clean":"2024-03-30T04:01:18.998604+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:01:18.998604+0000","last_undegraded":"2024-03-30T04:01:18.998604+0000","last_fullsized":"2024-03-30T04:01:18.998604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990547+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T04:01:17.990547+0000","last_peered":"2024-03-30T04:01:17.990547+0000","last_clean":"2024-03-30T04:01:17.990547+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T04:01:17.990547+0000","last_undegraded":"2024-03-30T04:01:17.990547+0000","last_fullsized":"2024-03-30T04:01:17.990547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990466+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T04:01:17.990466+0000","last_peered":"2024-03-30T04:01:17.990466+0000","last_clean":"2024-03-30T04:01:17.990466+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T04:01:17.990466+0000","last_undegraded":"2024-03-30T04:01:17.990466+0000","last_fullsized":"2024-03-30T04:01:17.990466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986226+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:01:17.986226+0000","last_peered":"2024-03-30T04:01:17.986226+0000","last_clean":"2024-03-30T04:01:17.986226+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:01:17.986226+0000","last_undegraded":"2024-03-30T04:01:17.986226+0000","last_fullsized":"2024-03-30T04:01:17.986226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990105+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T04:01:17.990105+0000","last_peered":"2024-03-30T04:01:17.990105+0000","last_clean":"2024-03-30T04:01:17.990105+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T04:01:17.990105+0000","last_undegraded":"2024-03-30T04:01:17.990105+0000","last_fullsized":"2024-03-30T04:01:17.990105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988764+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T04:01:17.988764+0000","last_peered":"2024-03-30T04:01:17.988764+0000","last_clean":"2024-03-30T04:01:17.988764+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T04:01:17.988764+0000","last_undegraded":"2024-03-30T04:01:17.988764+0000","last_fullsized":"2024-03-30T04:01:17.988764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.068405+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:01:37.068405+0000","last_peered":"2024-03-30T04:01:37.068405+0000","last_clean":"2024-03-30T04:01:37.068405+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:01:37.068405+0000","last_undegraded":"2024-03-30T04:01:37.068405+0000","last_fullsized":"2024-03-30T04:01:37.068405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986076+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T04:01:17.986076+0000","last_peered":"2024-03-30T04:01:17.986076+0000","last_clean":"2024-03-30T04:01:17.986076+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T04:01:17.986076+0000","last_undegraded":"2024-03-30T04:01:17.986076+0000","last_fullsized":"2024-03-30T04:01:17.986076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988916+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T04:01:17.988916+0000","last_peered":"2024-03-30T04:01:17.988916+0000","last_clean":"2024-03-30T04:01:17.988916+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T04:01:17.988916+0000","last_undegraded":"2024-03-30T04:01:17.988916+0000","last_fullsized":"2024-03-30T04:01:17.988916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990838+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T04:01:17.990838+0000","last_peered":"2024-03-30T04:01:17.990838+0000","last_clean":"2024-03-30T04:01:17.990838+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T04:01:17.990838+0000","last_undegraded":"2024-03-30T04:01:17.990838+0000","last_fullsized":"2024-03-30T04:01:17.990838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991514+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T04:01:17.991514+0000","last_peered":"2024-03-30T04:01:17.991514+0000","last_clean":"2024-03-30T04:01:17.991514+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T04:01:17.991514+0000","last_undegraded":"2024-03-30T04:01:17.991514+0000","last_fullsized":"2024-03-30T04:01:17.991514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990637+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:01:17.990637+0000","last_peered":"2024-03-30T04:01:17.990637+0000","last_clean":"2024-03-30T04:01:17.990637+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:01:17.990637+0000","last_undegraded":"2024-03-30T04:01:17.990637+0000","last_fullsized":"2024-03-30T04:01:17.990637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991305+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T04:01:17.991305+0000","last_peered":"2024-03-30T04:01:17.991305+0000","last_clean":"2024-03-30T04:01:17.991305+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T04:01:17.991305+0000","last_undegraded":"2024-03-30T04:01:17.991305+0000","last_fullsized":"2024-03-30T04:01:17.991305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986726+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T04:01:17.986726+0000","last_peered":"2024-03-30T04:01:17.986726+0000","last_clean":"2024-03-30T04:01:17.986726+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T04:01:17.986726+0000","last_undegraded":"2024-03-30T04:01:17.986726+0000","last_fullsized":"2024-03-30T04:01:17.986726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990017+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:01:17.990017+0000","last_peered":"2024-03-30T04:01:17.990017+0000","last_clean":"2024-03-30T04:01:17.990017+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:01:17.990017+0000","last_undegraded":"2024-03-30T04:01:17.990017+0000","last_fullsized":"2024-03-30T04:01:17.990017+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992437+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T04:01:17.992437+0000","last_peered":"2024-03-30T04:01:17.992437+0000","last_clean":"2024-03-30T04:01:17.992437+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T04:01:17.992437+0000","last_undegraded":"2024-03-30T04:01:17.992437+0000","last_fullsized":"2024-03-30T04:01:17.992437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993426+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T04:01:17.993426+0000","last_peered":"2024-03-30T04:01:17.993426+0000","last_clean":"2024-03-30T04:01:17.993426+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T04:01:17.993426+0000","last_undegraded":"2024-03-30T04:01:17.993426+0000","last_fullsized":"2024-03-30T04:01:17.993426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":200,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.070807+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:01:39.070807+0000","last_peered":"2024-03-30T04:01:39.070807+0000","last_clean":"2024-03-30T04:01:39.070807+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:01:39.070807+0000","last_undegraded":"2024-03-30T04:01:39.070807+0000","last_fullsized":"2024-03-30T04:01:39.070807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990158+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T04:01:17.990158+0000","last_peered":"2024-03-30T04:01:17.990158+0000","last_clean":"2024-03-30T04:01:17.990158+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T04:01:17.990158+0000","last_undegraded":"2024-03-30T04:01:17.990158+0000","last_fullsized":"2024-03-30T04:01:17.990158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990202+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T04:01:17.990202+0000","last_peered":"2024-03-30T04:01:17.990202+0000","last_clean":"2024-03-30T04:01:17.990202+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T04:01:17.990202+0000","last_undegraded":"2024-03-30T04:01:17.990202+0000","last_fullsized":"2024-03-30T04:01:17.990202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986623+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:01:17.986623+0000","last_peered":"2024-03-30T04:01:17.986623+0000","last_clean":"2024-03-30T04:01:17.986623+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:01:17.986623+0000","last_undegraded":"2024-03-30T04:01:17.986623+0000","last_fullsized":"2024-03-30T04:01:17.986623+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.000506+0000","last_change":"2024-03-30T04:01:19.000506+0000","last_active":"2024-03-30T04:01:19.000506+0000","last_peered":"2024-03-30T04:01:19.000506+0000","last_clean":"2024-03-30T04:01:19.000506+0000","last_became_active":"2024-03-30T04:01:18.999323+0000","last_became_peered":"2024-03-30T04:01:18.999323+0000","last_unstale":"2024-03-30T04:01:19.000506+0000","last_undegraded":"2024-03-30T04:01:19.000506+0000","last_fullsized":"2024-03-30T04:01:19.000506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:05:01.500866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990438+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T04:01:17.990438+0000","last_peered":"2024-03-30T04:01:17.990438+0000","last_clean":"2024-03-30T04:01:17.990438+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T04:01:17.990438+0000","last_undegraded":"2024-03-30T04:01:17.990438+0000","last_fullsized":"2024-03-30T04:01:17.990438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989996+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:01:17.989996+0000","last_peered":"2024-03-30T04:01:17.989996+0000","last_clean":"2024-03-30T04:01:17.989996+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:01:17.989996+0000","last_undegraded":"2024-03-30T04:01:17.989996+0000","last_fullsized":"2024-03-30T04:01:17.989996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990732+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T04:01:17.990732+0000","last_peered":"2024-03-30T04:01:17.990732+0000","last_clean":"2024-03-30T04:01:17.990732+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T04:01:17.990732+0000","last_undegraded":"2024-03-30T04:01:17.990732+0000","last_fullsized":"2024-03-30T04:01:17.990732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990391+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T04:01:17.990391+0000","last_peered":"2024-03-30T04:01:17.990391+0000","last_clean":"2024-03-30T04:01:17.990391+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T04:01:17.990391+0000","last_undegraded":"2024-03-30T04:01:17.990391+0000","last_fullsized":"2024-03-30T04:01:17.990391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'41","reported_seq":251,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.071776+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:01:39.071776+0000","last_peered":"2024-03-30T04:01:39.071776+0000","last_clean":"2024-03-30T04:01:39.071776+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:01:39.071776+0000","last_undegraded":"2024-03-30T04:01:39.071776+0000","last_fullsized":"2024-03-30T04:01:39.071776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70292,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":190,"num_read_kb":165,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986450+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T04:01:17.986450+0000","last_peered":"2024-03-30T04:01:17.986450+0000","last_clean":"2024-03-30T04:01:17.986450+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T04:01:17.986450+0000","last_undegraded":"2024-03-30T04:01:17.986450+0000","last_fullsized":"2024-03-30T04:01:17.986450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990463+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T04:01:17.990463+0000","last_peered":"2024-03-30T04:01:17.990463+0000","last_clean":"2024-03-30T04:01:17.990463+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T04:01:17.990463+0000","last_undegraded":"2024-03-30T04:01:17.990463+0000","last_fullsized":"2024-03-30T04:01:17.990463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988801+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:01:17.988801+0000","last_peered":"2024-03-30T04:01:17.988801+0000","last_clean":"2024-03-30T04:01:17.988801+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:01:17.988801+0000","last_undegraded":"2024-03-30T04:01:17.988801+0000","last_fullsized":"2024-03-30T04:01:17.988801+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991071+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T04:01:17.991071+0000","last_peered":"2024-03-30T04:01:17.991071+0000","last_clean":"2024-03-30T04:01:17.991071+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T04:01:17.991071+0000","last_undegraded":"2024-03-30T04:01:17.991071+0000","last_fullsized":"2024-03-30T04:01:17.991071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990443+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T04:01:17.990443+0000","last_peered":"2024-03-30T04:01:17.990443+0000","last_clean":"2024-03-30T04:01:17.990443+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T04:01:17.990443+0000","last_undegraded":"2024-03-30T04:01:17.990443+0000","last_fullsized":"2024-03-30T04:01:17.990443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990549+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:01:17.990549+0000","last_peered":"2024-03-30T04:01:17.990549+0000","last_clean":"2024-03-30T04:01:17.990549+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:01:17.990549+0000","last_undegraded":"2024-03-30T04:01:17.990549+0000","last_fullsized":"2024-03-30T04:01:17.990549+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986863+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T04:01:17.986863+0000","last_peered":"2024-03-30T04:01:17.986863+0000","last_clean":"2024-03-30T04:01:17.986863+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T04:01:17.986863+0000","last_undegraded":"2024-03-30T04:01:17.986863+0000","last_fullsized":"2024-03-30T04:01:17.986863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039141+0000","last_change":"2024-03-30T04:00:18.020869+0000","last_active":"2024-03-30T04:01:18.039141+0000","last_peered":"2024-03-30T04:01:18.039141+0000","last_clean":"2024-03-30T04:01:18.039141+0000","last_became_active":"2024-03-30T04:00:18.020410+0000","last_became_peered":"2024-03-30T04:00:18.020410+0000","last_unstale":"2024-03-30T04:01:18.039141+0000","last_undegraded":"2024-03-30T04:01:18.039141+0000","last_fullsized":"2024-03-30T04:01:18.039141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:53.675022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"86'7","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:34.091171+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:01:34.091171+0000","last_peered":"2024-03-30T04:01:34.091171+0000","last_clean":"2024-03-30T04:01:34.091171+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:01:34.091171+0000","last_undegraded":"2024-03-30T04:01:34.091171+0000","last_fullsized":"2024-03-30T04:01:34.091171+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986379+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T04:01:17.986379+0000","last_peered":"2024-03-30T04:01:17.986379+0000","last_clean":"2024-03-30T04:01:17.986379+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T04:01:17.986379+0000","last_undegraded":"2024-03-30T04:01:17.986379+0000","last_fullsized":"2024-03-30T04:01:17.986379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992090+0000","last_change":"2024-03-30T04:00:18.288237+0000","last_active":"2024-03-30T04:01:17.992090+0000","last_peered":"2024-03-30T04:01:17.992090+0000","last_clean":"2024-03-30T04:01:17.992090+0000","last_became_active":"2024-03-30T04:00:18.287748+0000","last_became_peered":"2024-03-30T04:00:18.287748+0000","last_unstale":"2024-03-30T04:01:17.992090+0000","last_undegraded":"2024-03-30T04:01:17.992090+0000","last_fullsized":"2024-03-30T04:01:17.992090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:50.893522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985742+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:01:17.985742+0000","last_peered":"2024-03-30T04:01:17.985742+0000","last_clean":"2024-03-30T04:01:17.985742+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:01:17.985742+0000","last_undegraded":"2024-03-30T04:01:17.985742+0000","last_fullsized":"2024-03-30T04:01:17.985742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991765+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T04:01:17.991765+0000","last_peered":"2024-03-30T04:01:17.991765+0000","last_clean":"2024-03-30T04:01:17.991765+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T04:01:17.991765+0000","last_undegraded":"2024-03-30T04:01:17.991765+0000","last_fullsized":"2024-03-30T04:01:17.991765+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991661+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T04:01:17.991661+0000","last_peered":"2024-03-30T04:01:17.991661+0000","last_clean":"2024-03-30T04:01:17.991661+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T04:01:17.991661+0000","last_undegraded":"2024-03-30T04:01:17.991661+0000","last_fullsized":"2024-03-30T04:01:17.991661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986615+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:01:17.986615+0000","last_peered":"2024-03-30T04:01:17.986615+0000","last_clean":"2024-03-30T04:01:17.986615+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:01:17.986615+0000","last_undegraded":"2024-03-30T04:01:17.986615+0000","last_fullsized":"2024-03-30T04:01:17.986615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990240+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T04:01:17.990240+0000","last_peered":"2024-03-30T04:01:17.990240+0000","last_clean":"2024-03-30T04:01:17.990240+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T04:01:17.990240+0000","last_undegraded":"2024-03-30T04:01:17.990240+0000","last_fullsized":"2024-03-30T04:01:17.990240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986221+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T04:01:17.986221+0000","last_peered":"2024-03-30T04:01:17.986221+0000","last_clean":"2024-03-30T04:01:17.986221+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T04:01:17.986221+0000","last_undegraded":"2024-03-30T04:01:17.986221+0000","last_fullsized":"2024-03-30T04:01:17.986221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992002+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T04:01:17.992002+0000","last_peered":"2024-03-30T04:01:17.992002+0000","last_clean":"2024-03-30T04:01:17.992002+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T04:01:17.992002+0000","last_undegraded":"2024-03-30T04:01:17.992002+0000","last_fullsized":"2024-03-30T04:01:17.992002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038965+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T04:01:18.038965+0000","last_peered":"2024-03-30T04:01:18.038965+0000","last_clean":"2024-03-30T04:01:18.038965+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T04:01:18.038965+0000","last_undegraded":"2024-03-30T04:01:18.038965+0000","last_fullsized":"2024-03-30T04:01:18.038965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991065+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T04:01:17.991065+0000","last_peered":"2024-03-30T04:01:17.991065+0000","last_clean":"2024-03-30T04:01:17.991065+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T04:01:17.991065+0000","last_undegraded":"2024-03-30T04:01:17.991065+0000","last_fullsized":"2024-03-30T04:01:17.991065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'9","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:36.090477+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T04:01:36.090477+0000","last_peered":"2024-03-30T04:01:36.090477+0000","last_clean":"2024-03-30T04:01:36.090477+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T04:01:36.090477+0000","last_undegraded":"2024-03-30T04:01:36.090477+0000","last_fullsized":"2024-03-30T04:01:36.090477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1773,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038870+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T04:01:18.038870+0000","last_peered":"2024-03-30T04:01:18.038870+0000","last_clean":"2024-03-30T04:01:18.038870+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T04:01:18.038870+0000","last_undegraded":"2024-03-30T04:01:18.038870+0000","last_fullsized":"2024-03-30T04:01:18.038870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992237+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T04:01:17.992237+0000","last_peered":"2024-03-30T04:01:17.992237+0000","last_clean":"2024-03-30T04:01:17.992237+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T04:01:17.992237+0000","last_undegraded":"2024-03-30T04:01:17.992237+0000","last_fullsized":"2024-03-30T04:01:17.992237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990251+0000","last_change":"2024-03-30T04:00:18.018515+0000","last_active":"2024-03-30T04:01:17.990251+0000","last_peered":"2024-03-30T04:01:17.990251+0000","last_clean":"2024-03-30T04:01:17.990251+0000","last_became_active":"2024-03-30T04:00:18.017823+0000","last_became_peered":"2024-03-30T04:00:18.017823+0000","last_unstale":"2024-03-30T04:01:17.990251+0000","last_undegraded":"2024-03-30T04:01:17.990251+0000","last_fullsized":"2024-03-30T04:01:17.990251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:15:54.957524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.094872+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:01:37.094872+0000","last_peered":"2024-03-30T04:01:37.094872+0000","last_clean":"2024-03-30T04:01:37.094872+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:01:37.094872+0000","last_undegraded":"2024-03-30T04:01:37.094872+0000","last_fullsized":"2024-03-30T04:01:37.094872+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986577+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T04:01:17.986577+0000","last_peered":"2024-03-30T04:01:17.986577+0000","last_clean":"2024-03-30T04:01:17.986577+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T04:01:17.986577+0000","last_undegraded":"2024-03-30T04:01:17.986577+0000","last_fullsized":"2024-03-30T04:01:17.986577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992703+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T04:01:17.992703+0000","last_peered":"2024-03-30T04:01:17.992703+0000","last_clean":"2024-03-30T04:01:17.992703+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T04:01:17.992703+0000","last_undegraded":"2024-03-30T04:01:17.992703+0000","last_fullsized":"2024-03-30T04:01:17.992703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990294+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:01:17.990294+0000","last_peered":"2024-03-30T04:01:17.990294+0000","last_clean":"2024-03-30T04:01:17.990294+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:01:17.990294+0000","last_undegraded":"2024-03-30T04:01:17.990294+0000","last_fullsized":"2024-03-30T04:01:17.990294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986260+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T04:01:17.986260+0000","last_peered":"2024-03-30T04:01:17.986260+0000","last_clean":"2024-03-30T04:01:17.986260+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T04:01:17.986260+0000","last_undegraded":"2024-03-30T04:01:17.986260+0000","last_fullsized":"2024-03-30T04:01:17.986260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986543+0000","last_change":"2024-03-30T04:00:18.017573+0000","last_active":"2024-03-30T04:01:17.986543+0000","last_peered":"2024-03-30T04:01:17.986543+0000","last_clean":"2024-03-30T04:01:17.986543+0000","last_became_active":"2024-03-30T04:00:18.016937+0000","last_became_peered":"2024-03-30T04:00:18.016937+0000","last_unstale":"2024-03-30T04:01:17.986543+0000","last_undegraded":"2024-03-30T04:01:17.986543+0000","last_fullsized":"2024-03-30T04:01:17.986543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:26:20.863180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990431+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T04:01:17.990431+0000","last_peered":"2024-03-30T04:01:17.990431+0000","last_clean":"2024-03-30T04:01:17.990431+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T04:01:17.990431+0000","last_undegraded":"2024-03-30T04:01:17.990431+0000","last_fullsized":"2024-03-30T04:01:17.990431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986318+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:01:17.986318+0000","last_peered":"2024-03-30T04:01:17.986318+0000","last_clean":"2024-03-30T04:01:17.986318+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:01:17.986318+0000","last_undegraded":"2024-03-30T04:01:17.986318+0000","last_fullsized":"2024-03-30T04:01:17.986318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991339+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T04:01:17.991339+0000","last_peered":"2024-03-30T04:01:17.991339+0000","last_clean":"2024-03-30T04:01:17.991339+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T04:01:17.991339+0000","last_undegraded":"2024-03-30T04:01:17.991339+0000","last_fullsized":"2024-03-30T04:01:17.991339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986517+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:01:17.986517+0000","last_peered":"2024-03-30T04:01:17.986517+0000","last_clean":"2024-03-30T04:01:17.986517+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:01:17.986517+0000","last_undegraded":"2024-03-30T04:01:17.986517+0000","last_fullsized":"2024-03-30T04:01:17.986517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991400+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T04:01:17.991400+0000","last_peered":"2024-03-30T04:01:17.991400+0000","last_clean":"2024-03-30T04:01:17.991400+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T04:01:17.991400+0000","last_undegraded":"2024-03-30T04:01:17.991400+0000","last_fullsized":"2024-03-30T04:01:17.991400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991296+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T04:01:17.991296+0000","last_peered":"2024-03-30T04:01:17.991296+0000","last_clean":"2024-03-30T04:01:17.991296+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T04:01:17.991296+0000","last_undegraded":"2024-03-30T04:01:17.991296+0000","last_fullsized":"2024-03-30T04:01:17.991296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.322725+0000","last_change":"2024-03-30T04:00:18.297312+0000","last_active":"2024-03-30T04:01:18.322725+0000","last_peered":"2024-03-30T04:01:18.322725+0000","last_clean":"2024-03-30T04:01:18.322725+0000","last_became_active":"2024-03-30T04:00:18.288640+0000","last_became_peered":"2024-03-30T04:00:18.288640+0000","last_unstale":"2024-03-30T04:01:18.322725+0000","last_undegraded":"2024-03-30T04:01:18.322725+0000","last_fullsized":"2024-03-30T04:01:18.322725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:21.664643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992944+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T04:01:17.992944+0000","last_peered":"2024-03-30T04:01:17.992944+0000","last_clean":"2024-03-30T04:01:17.992944+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T04:01:17.992944+0000","last_undegraded":"2024-03-30T04:01:17.992944+0000","last_fullsized":"2024-03-30T04:01:17.992944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989266+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T04:01:17.989266+0000","last_peered":"2024-03-30T04:01:17.989266+0000","last_clean":"2024-03-30T04:01:17.989266+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T04:01:17.989266+0000","last_undegraded":"2024-03-30T04:01:17.989266+0000","last_fullsized":"2024-03-30T04:01:17.989266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990084+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:01:17.990084+0000","last_peered":"2024-03-30T04:01:17.990084+0000","last_clean":"2024-03-30T04:01:17.990084+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:01:17.990084+0000","last_undegraded":"2024-03-30T04:01:17.990084+0000","last_fullsized":"2024-03-30T04:01:17.990084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.005003+0000","last_change":"2024-03-30T04:01:19.005003+0000","last_active":"2024-03-30T04:01:19.005003+0000","last_peered":"2024-03-30T04:01:19.005003+0000","last_clean":"2024-03-30T04:01:19.005003+0000","last_became_active":"2024-03-30T04:01:19.004334+0000","last_became_peered":"2024-03-30T04:01:19.004334+0000","last_unstale":"2024-03-30T04:01:19.005003+0000","last_undegraded":"2024-03-30T04:01:19.005003+0000","last_fullsized":"2024-03-30T04:01:19.005003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:41:54.609191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990019+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T04:01:17.990019+0000","last_peered":"2024-03-30T04:01:17.990019+0000","last_clean":"2024-03-30T04:01:17.990019+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T04:01:17.990019+0000","last_undegraded":"2024-03-30T04:01:17.990019+0000","last_fullsized":"2024-03-30T04:01:17.990019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989613+0000","last_change":"2024-03-30T04:00:18.291229+0000","last_active":"2024-03-30T04:01:17.989613+0000","last_peered":"2024-03-30T04:01:17.989613+0000","last_clean":"2024-03-30T04:01:17.989613+0000","last_became_active":"2024-03-30T04:00:18.287954+0000","last_became_peered":"2024-03-30T04:00:18.287954+0000","last_unstale":"2024-03-30T04:01:17.989613+0000","last_undegraded":"2024-03-30T04:01:17.989613+0000","last_fullsized":"2024-03-30T04:01:17.989613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:54.678907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985778+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T04:01:17.985778+0000","last_peered":"2024-03-30T04:01:17.985778+0000","last_clean":"2024-03-30T04:01:17.985778+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T04:01:17.985778+0000","last_undegraded":"2024-03-30T04:01:17.985778+0000","last_fullsized":"2024-03-30T04:01:17.985778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993013+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T04:01:17.993013+0000","last_peered":"2024-03-30T04:01:17.993013+0000","last_clean":"2024-03-30T04:01:17.993013+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T04:01:17.993013+0000","last_undegraded":"2024-03-30T04:01:17.993013+0000","last_fullsized":"2024-03-30T04:01:17.993013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991372+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:01:17.991372+0000","last_peered":"2024-03-30T04:01:17.991372+0000","last_clean":"2024-03-30T04:01:17.991372+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:01:17.991372+0000","last_undegraded":"2024-03-30T04:01:17.991372+0000","last_fullsized":"2024-03-30T04:01:17.991372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990785+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T04:01:17.990785+0000","last_peered":"2024-03-30T04:01:17.990785+0000","last_clean":"2024-03-30T04:01:17.990785+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T04:01:17.990785+0000","last_undegraded":"2024-03-30T04:01:17.990785+0000","last_fullsized":"2024-03-30T04:01:17.990785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990718+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T04:01:17.990718+0000","last_peered":"2024-03-30T04:01:17.990718+0000","last_clean":"2024-03-30T04:01:17.990718+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T04:01:17.990718+0000","last_undegraded":"2024-03-30T04:01:17.990718+0000","last_fullsized":"2024-03-30T04:01:17.990718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988611+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T04:01:17.988611+0000","last_peered":"2024-03-30T04:01:17.988611+0000","last_clean":"2024-03-30T04:01:17.988611+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T04:01:17.988611+0000","last_undegraded":"2024-03-30T04:01:17.988611+0000","last_fullsized":"2024-03-30T04:01:17.988611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993400+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T04:01:17.993400+0000","last_peered":"2024-03-30T04:01:17.993400+0000","last_clean":"2024-03-30T04:01:17.993400+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T04:01:17.993400+0000","last_undegraded":"2024-03-30T04:01:17.993400+0000","last_fullsized":"2024-03-30T04:01:17.993400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989641+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T04:01:17.989641+0000","last_peered":"2024-03-30T04:01:17.989641+0000","last_clean":"2024-03-30T04:01:17.989641+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T04:01:17.989641+0000","last_undegraded":"2024-03-30T04:01:17.989641+0000","last_fullsized":"2024-03-30T04:01:17.989641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985994+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:01:17.985994+0000","last_peered":"2024-03-30T04:01:17.985994+0000","last_clean":"2024-03-30T04:01:17.985994+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:01:17.985994+0000","last_undegraded":"2024-03-30T04:01:17.985994+0000","last_fullsized":"2024-03-30T04:01:17.985994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986683+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T04:01:17.986683+0000","last_peered":"2024-03-30T04:01:17.986683+0000","last_clean":"2024-03-30T04:01:17.986683+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T04:01:17.986683+0000","last_undegraded":"2024-03-30T04:01:17.986683+0000","last_fullsized":"2024-03-30T04:01:17.986683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990340+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T04:01:17.990340+0000","last_peered":"2024-03-30T04:01:17.990340+0000","last_clean":"2024-03-30T04:01:17.990340+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T04:01:17.990340+0000","last_undegraded":"2024-03-30T04:01:17.990340+0000","last_fullsized":"2024-03-30T04:01:17.990340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":111,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:36.093875+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:01:36.093875+0000","last_peered":"2024-03-30T04:01:36.093875+0000","last_clean":"2024-03-30T04:01:36.093875+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:01:36.093875+0000","last_undegraded":"2024-03-30T04:01:36.093875+0000","last_fullsized":"2024-03-30T04:01:36.093875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.999932+0000","last_change":"2024-03-30T04:01:18.999932+0000","last_active":"2024-03-30T04:01:18.999932+0000","last_peered":"2024-03-30T04:01:18.999932+0000","last_clean":"2024-03-30T04:01:18.999932+0000","last_became_active":"2024-03-30T04:01:18.999645+0000","last_became_peered":"2024-03-30T04:01:18.999645+0000","last_unstale":"2024-03-30T04:01:18.999932+0000","last_undegraded":"2024-03-30T04:01:18.999932+0000","last_fullsized":"2024-03-30T04:01:18.999932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:22:33.394341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986480+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T04:01:17.986480+0000","last_peered":"2024-03-30T04:01:17.986480+0000","last_clean":"2024-03-30T04:01:17.986480+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T04:01:17.986480+0000","last_undegraded":"2024-03-30T04:01:17.986480+0000","last_fullsized":"2024-03-30T04:01:17.986480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990215+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:01:17.990215+0000","last_peered":"2024-03-30T04:01:17.990215+0000","last_clean":"2024-03-30T04:01:17.990215+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:01:17.990215+0000","last_undegraded":"2024-03-30T04:01:17.990215+0000","last_fullsized":"2024-03-30T04:01:17.990215+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986563+0000","last_change":"2024-03-30T04:00:18.016967+0000","last_active":"2024-03-30T04:01:17.986563+0000","last_peered":"2024-03-30T04:01:17.986563+0000","last_clean":"2024-03-30T04:01:17.986563+0000","last_became_active":"2024-03-30T04:00:18.016555+0000","last_became_peered":"2024-03-30T04:00:18.016555+0000","last_unstale":"2024-03-30T04:01:17.986563+0000","last_undegraded":"2024-03-30T04:01:17.986563+0000","last_fullsized":"2024-03-30T04:01:17.986563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:34:50.538682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989682+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T04:01:17.989682+0000","last_peered":"2024-03-30T04:01:17.989682+0000","last_clean":"2024-03-30T04:01:17.989682+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T04:01:17.989682+0000","last_undegraded":"2024-03-30T04:01:17.989682+0000","last_fullsized":"2024-03-30T04:01:17.989682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992001+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:01:17.992001+0000","last_peered":"2024-03-30T04:01:17.992001+0000","last_clean":"2024-03-30T04:01:17.992001+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:01:17.992001+0000","last_undegraded":"2024-03-30T04:01:17.992001+0000","last_fullsized":"2024-03-30T04:01:17.992001+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.987339+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T04:01:17.987339+0000","last_peered":"2024-03-30T04:01:17.987339+0000","last_clean":"2024-03-30T04:01:17.987339+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T04:01:17.987339+0000","last_undegraded":"2024-03-30T04:01:17.987339+0000","last_fullsized":"2024-03-30T04:01:17.987339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990824+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T04:01:17.990824+0000","last_peered":"2024-03-30T04:01:17.990824+0000","last_clean":"2024-03-30T04:01:17.990824+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T04:01:17.990824+0000","last_undegraded":"2024-03-30T04:01:17.990824+0000","last_fullsized":"2024-03-30T04:01:17.990824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.001518+0000","last_change":"2024-03-30T04:01:19.001518+0000","last_active":"2024-03-30T04:01:19.001518+0000","last_peered":"2024-03-30T04:01:19.001518+0000","last_clean":"2024-03-30T04:01:19.001518+0000","last_became_active":"2024-03-30T04:01:19.001118+0000","last_became_peered":"2024-03-30T04:01:19.001118+0000","last_unstale":"2024-03-30T04:01:19.001518+0000","last_undegraded":"2024-03-30T04:01:19.001518+0000","last_fullsized":"2024-03-30T04:01:19.001518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:50:08.382669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988506+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T04:01:17.988506+0000","last_peered":"2024-03-30T04:01:17.988506+0000","last_clean":"2024-03-30T04:01:17.988506+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T04:01:17.988506+0000","last_undegraded":"2024-03-30T04:01:17.988506+0000","last_fullsized":"2024-03-30T04:01:17.988506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.004456+0000","last_change":"2024-03-30T04:01:19.004456+0000","last_active":"2024-03-30T04:01:19.004456+0000","last_peered":"2024-03-30T04:01:19.004456+0000","last_clean":"2024-03-30T04:01:19.004456+0000","last_became_active":"2024-03-30T04:01:19.004121+0000","last_became_peered":"2024-03-30T04:01:19.004121+0000","last_unstale":"2024-03-30T04:01:19.004456+0000","last_undegraded":"2024-03-30T04:01:19.004456+0000","last_fullsized":"2024-03-30T04:01:19.004456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:58:37.404002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991230+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T04:01:17.991230+0000","last_peered":"2024-03-30T04:01:17.991230+0000","last_clean":"2024-03-30T04:01:17.991230+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T04:01:17.991230+0000","last_undegraded":"2024-03-30T04:01:17.991230+0000","last_fullsized":"2024-03-30T04:01:17.991230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990493+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T04:01:17.990493+0000","last_peered":"2024-03-30T04:01:17.990493+0000","last_clean":"2024-03-30T04:01:17.990493+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T04:01:17.990493+0000","last_undegraded":"2024-03-30T04:01:17.990493+0000","last_fullsized":"2024-03-30T04:01:17.990493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988935+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T04:01:17.988935+0000","last_peered":"2024-03-30T04:01:17.988935+0000","last_clean":"2024-03-30T04:01:17.988935+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T04:01:17.988935+0000","last_undegraded":"2024-03-30T04:01:17.988935+0000","last_fullsized":"2024-03-30T04:01:17.988935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993122+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T04:01:17.993122+0000","last_peered":"2024-03-30T04:01:17.993122+0000","last_clean":"2024-03-30T04:01:17.993122+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T04:01:17.993122+0000","last_undegraded":"2024-03-30T04:01:17.993122+0000","last_fullsized":"2024-03-30T04:01:17.993122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990134+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T04:01:17.990134+0000","last_peered":"2024-03-30T04:01:17.990134+0000","last_clean":"2024-03-30T04:01:17.990134+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T04:01:17.990134+0000","last_undegraded":"2024-03-30T04:01:17.990134+0000","last_fullsized":"2024-03-30T04:01:17.990134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992792+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T04:01:17.992792+0000","last_peered":"2024-03-30T04:01:17.992792+0000","last_clean":"2024-03-30T04:01:17.992792+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T04:01:17.992792+0000","last_undegraded":"2024-03-30T04:01:17.992792+0000","last_fullsized":"2024-03-30T04:01:17.992792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990666+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T04:01:17.990666+0000","last_peered":"2024-03-30T04:01:17.990666+0000","last_clean":"2024-03-30T04:01:17.990666+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T04:01:17.990666+0000","last_undegraded":"2024-03-30T04:01:17.990666+0000","last_fullsized":"2024-03-30T04:01:17.990666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990825+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T04:01:17.990825+0000","last_peered":"2024-03-30T04:01:17.990825+0000","last_clean":"2024-03-30T04:01:17.990825+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T04:01:17.990825+0000","last_undegraded":"2024-03-30T04:01:17.990825+0000","last_fullsized":"2024-03-30T04:01:17.990825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990548+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T04:01:17.990548+0000","last_peered":"2024-03-30T04:01:17.990548+0000","last_clean":"2024-03-30T04:01:17.990548+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T04:01:17.990548+0000","last_undegraded":"2024-03-30T04:01:17.990548+0000","last_fullsized":"2024-03-30T04:01:17.990548+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78682,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":906,"num_read_kb":543,"num_write":255,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":233526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547235,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":1007616,"data_stored":492856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.752},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78500000000000003,"5min":0.996,"15min":0.996},"last":0.73399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47899999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.81499999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.876,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.89200000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73399999999999999,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.76700000000000002}]},{"osd":8,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85899999999999999}]},{"osd":9,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.60499999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.052,"5min":1.097,"15min":1.097},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79200000000000004,"15min":0.78200000000000003},"min":{"1min":0.58299999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.91500000000000004}]},{"osd":10,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.625,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.93899999999999995,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.93799999999999994,"5min":1.536,"15min":1.536},"last":0.92300000000000004}]}]},{"osd":10,"up_from":68,"seq":292057776166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1708032,"data_stored":1150498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36043201},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.623,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68500000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48999999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.51100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.77500000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.01,"15min":1.01},"last":0.626},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95499999999999996,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.66800000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97599999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.83599999999999997}]},{"osd":9,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.71599999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.018,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65200000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.95699999999999996,"5min":1.147,"15min":1.147},"last":0.751}]},{"osd":11,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75600000000000001,"15min":0.746},"min":{"1min":0.63400000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79600000000000004,"15min":0.78900000000000003},"min":{"1min":0.68600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.98599999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85699999999999998}]}]},{"osd":9,"up_from":63,"seq":270582939690,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":1036288,"data_stored":496930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.2589999999999999,"5min":3.706,"15min":3.706},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60199999999999998,"15min":0.63800000000000001},"min":{"1min":0.374,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":1.018,"15min":1.018},"last":0.372}]},{"osd":1,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.69599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.74099999999999999,"15min":0.76300000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77500000000000002,"15min":0.80700000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.501,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67000000000000004,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86599999999999999,"5min":1.155,"15min":1.155},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6819999999999999,"5min":1.72,"15min":1.72},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.88100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.097,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.89200000000000002,"15min":0.94999999999999996},"min":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0489999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.93600000000000005,"15min":0.998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.107,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.752},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.95799999999999996,"15min":1.03},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.887,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.57999999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94099999999999995,"15min":0.98799999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.997,"15min":1.0629999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1910000000000001,"5min":2.379,"15min":2.379},"last":0.72599999999999998}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.92700000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2430000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.78000000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.64000000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168623,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":1028096,"data_stored":493212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.76200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57199999999999995,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.40999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.755,"5min":0.876,"15min":0.876},"last":0.54100000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.36799999999999999},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.434},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.48099999999999998,"15min":0.47099999999999997},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.55600000000000005,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.78700000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.48199999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.621},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.56799999999999995,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97199999999999998,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93899999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.64800000000000002},"min":{"1min":0.52500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.81799999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.60199999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.93899999999999995,"5min":1.006,"15min":1.006},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.76600000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.80300000000000005}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.55200000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.84999999999999998}]},{"osd":11,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.629},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70799999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397552,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1091334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.58999999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.437,"5min":0.372,"15min":0.372},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.753,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.498}]},{"osd":4,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60799999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.434}]},{"osd":5,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.505,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.67000000000000004,"15min":0.68700000000000006},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72099999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.82899999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63400000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79500000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.53100000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88900000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59699999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.76400000000000001},"min":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.85599999999999998,"5min":1.258,"15min":1.258},"last":0.69699999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96599999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.754}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561077,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":489523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.626},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.56999999999999995,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.65100000000000002,"15min":0.63200000000000001},"min":{"1min":0.51200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.67100000000000004},"min":{"1min":0.55200000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.57299999999999995,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.442,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.58299999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76700000000000002,"15min":0.749},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.61399999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.337,"5min":1.462,"15min":1.462},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.68300000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.72399999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.82399999999999995,"15min":0.79800000000000004},"min":{"1min":0.52000000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.64200000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.073}]},{"osd":8,"last update":"Sat Mar 30 04:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.71899999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.88800000000000001,"15min":0.86699999999999999},"min":{"1min":0.76500000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.109}]},{"osd":9,"last update":"Sat Mar 30 04:01:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.92300000000000004,"15min":0.91200000000000003},"min":{"1min":0.68000000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.512,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.877,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.109,"5min":1.488,"15min":1.488},"last":0.751}]},{"osd":10,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95199999999999996,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.075,"5min":0.97499999999999998,"15min":0.95899999999999996},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.82699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":1081344,"data_stored":560845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.503},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.57099999999999995,"15min":0.52100000000000002},"min":{"1min":0.40100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55100000000000005,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.51100000000000001},"min":{"1min":0.46400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.498},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.53800000000000003},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.871,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.442,"15min":0.40200000000000002},"min":{"1min":0.36199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.41899999999999998},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69199999999999995,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60799999999999998,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.622,"15min":0.57599999999999996},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92300000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91900000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.61499999999999999},"min":{"1min":0.503,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82699999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.59799999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.82199999999999995,"15min":0.80000000000000004},"min":{"1min":0.61699999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89500000000000002,"5min":3.2080000000000002,"15min":3.2080000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.755},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.79800000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.746,"15min":0.745},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.93600000000000005,"5min":0.999,"15min":0.999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78300000000000003,"15min":0.78200000000000003},"min":{"1min":0.58099999999999996,"5min":0.501,"15min":0.501},"max":{"1min":0.94099999999999995,"5min":1.073,"15min":1.073},"last":0.65300000000000002}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.755},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.76900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":489746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.59499999999999997,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60299999999999998,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.67800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.378,"15min":0.378},"max":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.9859999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.96,"5min":6.96,"15min":6.96},"last":0.92900000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.61799999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71999999999999997,"15min":0.67500000000000004},"min":{"1min":0.51000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73999999999999999,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72299999999999998,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.113,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Sat Mar 30 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67900000000000005,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.76200000000000001,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.78700000000000003}]},{"osd":7,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.76100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88200000000000001,"15min":0.88800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.93300000000000005,"15min":0.88800000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":9.7439999999999998,"5min":9.7439999999999998,"15min":9.7439999999999998},"last":0.95699999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86899999999999999,"15min":0.878},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.2290000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.93200000000000005,"15min":0.95399999999999996},"min":{"1min":0.53000000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88700000000000001,"15min":0.879},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.873},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92100000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.91400000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:01:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.2809999999999999,"5min":1.298,"15min":1.298},"last":0.96399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":557657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78900000000000003,"5min":1.133,"15min":1.133},"last":0.57999999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.56499999999999995,"15min":0.51900000000000002},"min":{"1min":0.47499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78600000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.51700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.71199999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.871}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.47699999999999998,"15min":0.44900000000000001},"min":{"1min":0.41599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72599999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.623,"15min":0.61099999999999999},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68500000000000005}]},{"osd":7,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.84899999999999998}]},{"osd":8,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55700000000000005},"min":{"1min":0.503,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91000000000000003,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.56200000000000006,"15min":0.53000000000000003},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66200000000000003}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69999999999999996,"15min":0.66300000000000003},"min":{"1min":0.65200000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.754,"15min":0.72899999999999998},"min":{"1min":0.64700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":0.995,"15min":0.995},"last":0.755}]},{"osd":10,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71299999999999997,"15min":0.69999999999999996},"min":{"1min":0.58799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":0.93700000000000006,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.82999999999999996}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.81000000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":488046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.42299999999999999},"min":{"1min":0.34599999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.443},"min":{"1min":0.36699999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64500000000000002,"15min":0.61399999999999999},"min":{"1min":0.44800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.633,"5min":1.762,"15min":1.762},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.71399999999999997,"15min":0.628},"min":{"1min":0.56000000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.601,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.626},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66000000000000003,"15min":0.63},"min":{"1min":0.495,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.67700000000000005,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.748,"15min":0.71399999999999997},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72799999999999998,"15min":0.67300000000000004},"min":{"1min":0.58099999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79600000000000004,"15min":0.73899999999999999},"min":{"1min":0.69899999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79600000000000004,"15min":0.76200000000000001},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.72899999999999998},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89500000000000002,"15min":0.88300000000000001},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.58899999999999997}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.99199999999999999,"15min":1.018},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.61899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.61899999999999999}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.02,"5min":0.99299999999999999,"15min":0.98899999999999999},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.71399999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.73799999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953533,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":1015808,"data_stored":492898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.372},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53800000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.376},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.32800000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55700000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.41799999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.53400000000000003},"min":{"1min":0.47899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81200000000000006,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60599999999999998,"15min":0.58099999999999996},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90900000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59099999999999997,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.57199999999999995},"min":{"1min":0.52500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85299999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.70899999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.441,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.54300000000000004},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.871,"15min":0.871},"last":0.55900000000000005}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.56999999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89300000000000002,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68100000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.78200000000000003}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.76600000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.79100000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91100000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.76800000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.749,"15min":0.75700000000000001},"min":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.746,"15min":0.754},"min":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.621}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.747}]}]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1630208,"data_stored":1084914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.373}]},{"osd":1,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57299999999999995,"15min":0.55300000000000005},"min":{"1min":0.39700000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.754,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40500000000000003,"15min":0.39200000000000002},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.626,"5min":0.747,"15min":0.747},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.438,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84899999999999998,"5min":0.873,"15min":0.873},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.60299999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.504}]},{"osd":4,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.753,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73399999999999999,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77000000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.622},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.016,"15min":1.016},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.53900000000000003}]},{"osd":8,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68799999999999994,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.59699999999999998,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.57199999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73599999999999999,"15min":0.71999999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.68999999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.84799999999999998,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.84299999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90800000000000003,"5min":1.222,"15min":1.222},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":72514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:01:41.533 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T04:01:41.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:41 smithi154 ceph-mon[79161]: pgmap v300: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-30T04:01:41.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:41 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1392943275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:41.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:41 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2394804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:41 smithi193 ceph-mon[79581]: pgmap v300: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-30T04:01:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:41 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1392943275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:41.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:41 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2394804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 -- 172.21.15.23:0/4290794269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 msgr2=0x7fc9341090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 --2- 172.21.15.23:0/4290794269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341090c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00b0d0 tx=0x7fc92c00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 -- 172.21.15.23:0/4290794269 shutdown_connections 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 --2- 172.21.15.23:0/4290794269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341090c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 --2- 172.21.15.23:0/4290794269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934100c80 0x7fc9341010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 --2- 172.21.15.23:0/4290794269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934101e70 0x7fc9341022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 -- 172.21.15.23:0/4290794269 >> 172.21.15.23:0/4290794269 conn(0x7fc9340fc1f0 msgr2=0x7fc9340fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 -- 172.21.15.23:0/4290794269 shutdown_connections 2024-03-30T04:01:41.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.765+0000 7fc939800700 1 -- 172.21.15.23:0/4290794269 wait complete. 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.766+0000 7fc939800700 1 Processor -- start 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 -- start start 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934100c80 0x7fc934198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934101e70 0x7fc934199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9341a0a10 con 0x7fc934102810 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9341a0b50 con 0x7fc934101e70 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc939800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9341a0ce0 con 0x7fc934100c80 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc933400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc933400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39820/0 (socket says 172.21.15.23:39820) 2024-03-30T04:01:41.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc933400700 1 -- 172.21.15.23:0/3889328724 learned_addr learned my addr 172.21.15.23:0/3889328724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc932000700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934101e70 0x7fc934199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.767+0000 7fc932a00700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934100c80 0x7fc934198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 -- 172.21.15.23:0/3889328724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934100c80 msgr2=0x7fc934198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934100c80 0x7fc934198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 -- 172.21.15.23:0/3889328724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934101e70 msgr2=0x7fc934199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934101e70 0x7fc934199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 -- 172.21.15.23:0/3889328724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92c009d00 con 0x7fc934102810 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc933400700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc92c007b60 tx=0x7fc92c008e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c00e040 con 0x7fc934102810 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc92c01d930 con 0x7fc934102810 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9341a0f60 con 0x7fc934102810 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c01b9f0 con 0x7fc934102810 2024-03-30T04:01:41.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.768+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9341a1380 con 0x7fc934102810 2024-03-30T04:01:41.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.769+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934067a10 con 0x7fc934102810 2024-03-30T04:01:41.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.772+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fc92c01daa0 con 0x7fc934102810 2024-03-30T04:01:41.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:41 smithi023 ceph-mon[38254]: pgmap v300: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-30T04:01:41.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:41 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1392943275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:41.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:41 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2394804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:41.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.774+0000 7fc92b400700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 0x7fc9180a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:41.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.774+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fc92c0ce8e0 con 0x7fc934102810 2024-03-30T04:01:41.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.774+0000 7fc932a00700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 0x7fc9180a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:41.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.775+0000 7fc932a00700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 0x7fc9180a36f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc934101cd0 tx=0x7fc924009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:41.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.776+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc92c097d40 con 0x7fc934102810 2024-03-30T04:01:41.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:41.990+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc934074700 con 0x7fc9180a1230 2024-03-30T04:01:42.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.003+0000 7fc92b400700 1 -- 172.21.15.23:0/3889328724 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427389 (secure 0 0 0) 0x7fc934074700 con 0x7fc9180a1230 2024-03-30T04:01:42.008 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:42.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.008+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 msgr2=0x7fc9180a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.008+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 0x7fc9180a36f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc934101cd0 tx=0x7fc924009260 comp rx=0 tx=0).stop 2024-03-30T04:01:42.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.008+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 msgr2=0x7fc9341a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.008+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc92c007b60 tx=0x7fc92c008e40 comp rx=0 tx=0).stop 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 shutdown_connections 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc934102810 0x7fc9341a0440 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fc9180a1230 0x7fc9180a36f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc934100c80 0x7fc934198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 --2- 172.21.15.23:0/3889328724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc934101e70 0x7fc934199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 >> 172.21.15.23:0/3889328724 conn(0x7fc9340fc1f0 msgr2=0x7fc934072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 shutdown_connections 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.009+0000 7fc939800700 1 -- 172.21.15.23:0/3889328724 wait complete. 2024-03-30T04:01:42.013 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:01:42.026 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-03-30T04:01:40.304131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669284,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1042,"num_read_kb":662,"num_write":438,"num_write_kb":3101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":864,"ondisk_log_size":864,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383236,"kb_used_data":13960,"kb_used_omap":18,"kb_used_meta":369133,"kb_avail":1124558588,"statfs":{"total":1151940427776,"available":1151547994112,"internally_reserved":0,"allocated":14295040,"data_stored":7888459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":377992517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8625,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":136,"num_read_kb":82,"num_write":11,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991830+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T04:01:17.991830+0000","last_peered":"2024-03-30T04:01:17.991830+0000","last_clean":"2024-03-30T04:01:17.991830+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T04:01:17.991830+0000","last_undegraded":"2024-03-30T04:01:17.991830+0000","last_fullsized":"2024-03-30T04:01:17.991830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986818+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T04:01:17.986818+0000","last_peered":"2024-03-30T04:01:17.986818+0000","last_clean":"2024-03-30T04:01:17.986818+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T04:01:17.986818+0000","last_undegraded":"2024-03-30T04:01:17.986818+0000","last_fullsized":"2024-03-30T04:01:17.986818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986844+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T04:01:17.986844+0000","last_peered":"2024-03-30T04:01:17.986844+0000","last_clean":"2024-03-30T04:01:17.986844+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T04:01:17.986844+0000","last_undegraded":"2024-03-30T04:01:17.986844+0000","last_fullsized":"2024-03-30T04:01:17.986844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990923+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T04:01:17.990923+0000","last_peered":"2024-03-30T04:01:17.990923+0000","last_clean":"2024-03-30T04:01:17.990923+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T04:01:17.990923+0000","last_undegraded":"2024-03-30T04:01:17.990923+0000","last_fullsized":"2024-03-30T04:01:17.990923+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986025+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T04:01:17.986025+0000","last_peered":"2024-03-30T04:01:17.986025+0000","last_clean":"2024-03-30T04:01:17.986025+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T04:01:17.986025+0000","last_undegraded":"2024-03-30T04:01:17.986025+0000","last_fullsized":"2024-03-30T04:01:17.986025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986574+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T04:01:17.986574+0000","last_peered":"2024-03-30T04:01:17.986574+0000","last_clean":"2024-03-30T04:01:17.986574+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T04:01:17.986574+0000","last_undegraded":"2024-03-30T04:01:17.986574+0000","last_fullsized":"2024-03-30T04:01:17.986574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991399+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T04:01:17.991399+0000","last_peered":"2024-03-30T04:01:17.991399+0000","last_clean":"2024-03-30T04:01:17.991399+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T04:01:17.991399+0000","last_undegraded":"2024-03-30T04:01:17.991399+0000","last_fullsized":"2024-03-30T04:01:17.991399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989988+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T04:01:17.989988+0000","last_peered":"2024-03-30T04:01:17.989988+0000","last_clean":"2024-03-30T04:01:17.989988+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T04:01:17.989988+0000","last_undegraded":"2024-03-30T04:01:17.989988+0000","last_fullsized":"2024-03-30T04:01:17.989988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039169+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T04:01:18.039169+0000","last_peered":"2024-03-30T04:01:18.039169+0000","last_clean":"2024-03-30T04:01:18.039169+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T04:01:18.039169+0000","last_undegraded":"2024-03-30T04:01:18.039169+0000","last_fullsized":"2024-03-30T04:01:18.039169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986792+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T04:01:17.986792+0000","last_peered":"2024-03-30T04:01:17.986792+0000","last_clean":"2024-03-30T04:01:17.986792+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T04:01:17.986792+0000","last_undegraded":"2024-03-30T04:01:17.986792+0000","last_fullsized":"2024-03-30T04:01:17.986792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990814+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T04:01:17.990814+0000","last_peered":"2024-03-30T04:01:17.990814+0000","last_clean":"2024-03-30T04:01:17.990814+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T04:01:17.990814+0000","last_undegraded":"2024-03-30T04:01:17.990814+0000","last_fullsized":"2024-03-30T04:01:17.990814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990767+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T04:01:17.990767+0000","last_peered":"2024-03-30T04:01:17.990767+0000","last_clean":"2024-03-30T04:01:17.990767+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T04:01:17.990767+0000","last_undegraded":"2024-03-30T04:01:17.990767+0000","last_fullsized":"2024-03-30T04:01:17.990767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986474+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T04:01:17.986474+0000","last_peered":"2024-03-30T04:01:17.986474+0000","last_clean":"2024-03-30T04:01:17.986474+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T04:01:17.986474+0000","last_undegraded":"2024-03-30T04:01:17.986474+0000","last_fullsized":"2024-03-30T04:01:17.986474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986493+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T04:01:17.986493+0000","last_peered":"2024-03-30T04:01:17.986493+0000","last_clean":"2024-03-30T04:01:17.986493+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T04:01:17.986493+0000","last_undegraded":"2024-03-30T04:01:17.986493+0000","last_fullsized":"2024-03-30T04:01:17.986493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989473+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T04:01:17.989473+0000","last_peered":"2024-03-30T04:01:17.989473+0000","last_clean":"2024-03-30T04:01:17.989473+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T04:01:17.989473+0000","last_undegraded":"2024-03-30T04:01:17.989473+0000","last_fullsized":"2024-03-30T04:01:17.989473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991668+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T04:01:17.991668+0000","last_peered":"2024-03-30T04:01:17.991668+0000","last_clean":"2024-03-30T04:01:17.991668+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T04:01:17.991668+0000","last_undegraded":"2024-03-30T04:01:17.991668+0000","last_fullsized":"2024-03-30T04:01:17.991668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992709+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T04:01:17.992709+0000","last_peered":"2024-03-30T04:01:17.992709+0000","last_clean":"2024-03-30T04:01:17.992709+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T04:01:17.992709+0000","last_undegraded":"2024-03-30T04:01:17.992709+0000","last_fullsized":"2024-03-30T04:01:17.992709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989001+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T04:01:17.989001+0000","last_peered":"2024-03-30T04:01:17.989001+0000","last_clean":"2024-03-30T04:01:17.989001+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T04:01:17.989001+0000","last_undegraded":"2024-03-30T04:01:17.989001+0000","last_fullsized":"2024-03-30T04:01:17.989001+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989435+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T04:01:17.989435+0000","last_peered":"2024-03-30T04:01:17.989435+0000","last_clean":"2024-03-30T04:01:17.989435+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T04:01:17.989435+0000","last_undegraded":"2024-03-30T04:01:17.989435+0000","last_fullsized":"2024-03-30T04:01:17.989435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'15","reported_seq":119,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:35.087556+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:01:35.087556+0000","last_peered":"2024-03-30T04:01:35.087556+0000","last_clean":"2024-03-30T04:01:35.087556+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:01:35.087556+0000","last_undegraded":"2024-03-30T04:01:35.087556+0000","last_fullsized":"2024-03-30T04:01:35.087556+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993165+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T04:01:17.993165+0000","last_peered":"2024-03-30T04:01:17.993165+0000","last_clean":"2024-03-30T04:01:17.993165+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T04:01:17.993165+0000","last_undegraded":"2024-03-30T04:01:17.993165+0000","last_fullsized":"2024-03-30T04:01:17.993165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989691+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T04:01:17.989691+0000","last_peered":"2024-03-30T04:01:17.989691+0000","last_clean":"2024-03-30T04:01:17.989691+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T04:01:17.989691+0000","last_undegraded":"2024-03-30T04:01:17.989691+0000","last_fullsized":"2024-03-30T04:01:17.989691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993262+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T04:01:17.993262+0000","last_peered":"2024-03-30T04:01:17.993262+0000","last_clean":"2024-03-30T04:01:17.993262+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T04:01:17.993262+0000","last_undegraded":"2024-03-30T04:01:17.993262+0000","last_fullsized":"2024-03-30T04:01:17.993262+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986607+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T04:01:17.986607+0000","last_peered":"2024-03-30T04:01:17.986607+0000","last_clean":"2024-03-30T04:01:17.986607+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T04:01:17.986607+0000","last_undegraded":"2024-03-30T04:01:17.986607+0000","last_fullsized":"2024-03-30T04:01:17.986607+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990211+0000","last_change":"2024-03-30T04:00:18.019833+0000","last_active":"2024-03-30T04:01:17.990211+0000","last_peered":"2024-03-30T04:01:17.990211+0000","last_clean":"2024-03-30T04:01:17.990211+0000","last_became_active":"2024-03-30T04:00:18.019445+0000","last_became_peered":"2024-03-30T04:00:18.019445+0000","last_unstale":"2024-03-30T04:01:17.990211+0000","last_undegraded":"2024-03-30T04:01:17.990211+0000","last_fullsized":"2024-03-30T04:01:17.990211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:41:55.882417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'13","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.998604+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:01:18.998604+0000","last_peered":"2024-03-30T04:01:18.998604+0000","last_clean":"2024-03-30T04:01:18.998604+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:01:18.998604+0000","last_undegraded":"2024-03-30T04:01:18.998604+0000","last_fullsized":"2024-03-30T04:01:18.998604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990547+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T04:01:17.990547+0000","last_peered":"2024-03-30T04:01:17.990547+0000","last_clean":"2024-03-30T04:01:17.990547+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T04:01:17.990547+0000","last_undegraded":"2024-03-30T04:01:17.990547+0000","last_fullsized":"2024-03-30T04:01:17.990547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990466+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T04:01:17.990466+0000","last_peered":"2024-03-30T04:01:17.990466+0000","last_clean":"2024-03-30T04:01:17.990466+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T04:01:17.990466+0000","last_undegraded":"2024-03-30T04:01:17.990466+0000","last_fullsized":"2024-03-30T04:01:17.990466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986226+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:01:17.986226+0000","last_peered":"2024-03-30T04:01:17.986226+0000","last_clean":"2024-03-30T04:01:17.986226+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:01:17.986226+0000","last_undegraded":"2024-03-30T04:01:17.986226+0000","last_fullsized":"2024-03-30T04:01:17.986226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990105+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T04:01:17.990105+0000","last_peered":"2024-03-30T04:01:17.990105+0000","last_clean":"2024-03-30T04:01:17.990105+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T04:01:17.990105+0000","last_undegraded":"2024-03-30T04:01:17.990105+0000","last_fullsized":"2024-03-30T04:01:17.990105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988764+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T04:01:17.988764+0000","last_peered":"2024-03-30T04:01:17.988764+0000","last_clean":"2024-03-30T04:01:17.988764+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T04:01:17.988764+0000","last_undegraded":"2024-03-30T04:01:17.988764+0000","last_fullsized":"2024-03-30T04:01:17.988764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.068405+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:01:37.068405+0000","last_peered":"2024-03-30T04:01:37.068405+0000","last_clean":"2024-03-30T04:01:37.068405+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:01:37.068405+0000","last_undegraded":"2024-03-30T04:01:37.068405+0000","last_fullsized":"2024-03-30T04:01:37.068405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986076+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T04:01:17.986076+0000","last_peered":"2024-03-30T04:01:17.986076+0000","last_clean":"2024-03-30T04:01:17.986076+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T04:01:17.986076+0000","last_undegraded":"2024-03-30T04:01:17.986076+0000","last_fullsized":"2024-03-30T04:01:17.986076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988916+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T04:01:17.988916+0000","last_peered":"2024-03-30T04:01:17.988916+0000","last_clean":"2024-03-30T04:01:17.988916+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T04:01:17.988916+0000","last_undegraded":"2024-03-30T04:01:17.988916+0000","last_fullsized":"2024-03-30T04:01:17.988916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990838+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T04:01:17.990838+0000","last_peered":"2024-03-30T04:01:17.990838+0000","last_clean":"2024-03-30T04:01:17.990838+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T04:01:17.990838+0000","last_undegraded":"2024-03-30T04:01:17.990838+0000","last_fullsized":"2024-03-30T04:01:17.990838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991514+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T04:01:17.991514+0000","last_peered":"2024-03-30T04:01:17.991514+0000","last_clean":"2024-03-30T04:01:17.991514+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T04:01:17.991514+0000","last_undegraded":"2024-03-30T04:01:17.991514+0000","last_fullsized":"2024-03-30T04:01:17.991514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990637+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:01:17.990637+0000","last_peered":"2024-03-30T04:01:17.990637+0000","last_clean":"2024-03-30T04:01:17.990637+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:01:17.990637+0000","last_undegraded":"2024-03-30T04:01:17.990637+0000","last_fullsized":"2024-03-30T04:01:17.990637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991305+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T04:01:17.991305+0000","last_peered":"2024-03-30T04:01:17.991305+0000","last_clean":"2024-03-30T04:01:17.991305+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T04:01:17.991305+0000","last_undegraded":"2024-03-30T04:01:17.991305+0000","last_fullsized":"2024-03-30T04:01:17.991305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986726+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T04:01:17.986726+0000","last_peered":"2024-03-30T04:01:17.986726+0000","last_clean":"2024-03-30T04:01:17.986726+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T04:01:17.986726+0000","last_undegraded":"2024-03-30T04:01:17.986726+0000","last_fullsized":"2024-03-30T04:01:17.986726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990017+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:01:17.990017+0000","last_peered":"2024-03-30T04:01:17.990017+0000","last_clean":"2024-03-30T04:01:17.990017+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:01:17.990017+0000","last_undegraded":"2024-03-30T04:01:17.990017+0000","last_fullsized":"2024-03-30T04:01:17.990017+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992437+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T04:01:17.992437+0000","last_peered":"2024-03-30T04:01:17.992437+0000","last_clean":"2024-03-30T04:01:17.992437+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T04:01:17.992437+0000","last_undegraded":"2024-03-30T04:01:17.992437+0000","last_fullsized":"2024-03-30T04:01:17.992437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993426+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T04:01:17.993426+0000","last_peered":"2024-03-30T04:01:17.993426+0000","last_clean":"2024-03-30T04:01:17.993426+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T04:01:17.993426+0000","last_undegraded":"2024-03-30T04:01:17.993426+0000","last_fullsized":"2024-03-30T04:01:17.993426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":200,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.070807+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:01:39.070807+0000","last_peered":"2024-03-30T04:01:39.070807+0000","last_clean":"2024-03-30T04:01:39.070807+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:01:39.070807+0000","last_undegraded":"2024-03-30T04:01:39.070807+0000","last_fullsized":"2024-03-30T04:01:39.070807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990158+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T04:01:17.990158+0000","last_peered":"2024-03-30T04:01:17.990158+0000","last_clean":"2024-03-30T04:01:17.990158+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T04:01:17.990158+0000","last_undegraded":"2024-03-30T04:01:17.990158+0000","last_fullsized":"2024-03-30T04:01:17.990158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990202+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T04:01:17.990202+0000","last_peered":"2024-03-30T04:01:17.990202+0000","last_clean":"2024-03-30T04:01:17.990202+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T04:01:17.990202+0000","last_undegraded":"2024-03-30T04:01:17.990202+0000","last_fullsized":"2024-03-30T04:01:17.990202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986623+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:01:17.986623+0000","last_peered":"2024-03-30T04:01:17.986623+0000","last_clean":"2024-03-30T04:01:17.986623+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:01:17.986623+0000","last_undegraded":"2024-03-30T04:01:17.986623+0000","last_fullsized":"2024-03-30T04:01:17.986623+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.000506+0000","last_change":"2024-03-30T04:01:19.000506+0000","last_active":"2024-03-30T04:01:19.000506+0000","last_peered":"2024-03-30T04:01:19.000506+0000","last_clean":"2024-03-30T04:01:19.000506+0000","last_became_active":"2024-03-30T04:01:18.999323+0000","last_became_peered":"2024-03-30T04:01:18.999323+0000","last_unstale":"2024-03-30T04:01:19.000506+0000","last_undegraded":"2024-03-30T04:01:19.000506+0000","last_fullsized":"2024-03-30T04:01:19.000506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:05:01.500866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990438+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T04:01:17.990438+0000","last_peered":"2024-03-30T04:01:17.990438+0000","last_clean":"2024-03-30T04:01:17.990438+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T04:01:17.990438+0000","last_undegraded":"2024-03-30T04:01:17.990438+0000","last_fullsized":"2024-03-30T04:01:17.990438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989996+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:01:17.989996+0000","last_peered":"2024-03-30T04:01:17.989996+0000","last_clean":"2024-03-30T04:01:17.989996+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:01:17.989996+0000","last_undegraded":"2024-03-30T04:01:17.989996+0000","last_fullsized":"2024-03-30T04:01:17.989996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990732+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T04:01:17.990732+0000","last_peered":"2024-03-30T04:01:17.990732+0000","last_clean":"2024-03-30T04:01:17.990732+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T04:01:17.990732+0000","last_undegraded":"2024-03-30T04:01:17.990732+0000","last_fullsized":"2024-03-30T04:01:17.990732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990391+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T04:01:17.990391+0000","last_peered":"2024-03-30T04:01:17.990391+0000","last_clean":"2024-03-30T04:01:17.990391+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T04:01:17.990391+0000","last_undegraded":"2024-03-30T04:01:17.990391+0000","last_fullsized":"2024-03-30T04:01:17.990391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'41","reported_seq":251,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.071776+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:01:39.071776+0000","last_peered":"2024-03-30T04:01:39.071776+0000","last_clean":"2024-03-30T04:01:39.071776+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:01:39.071776+0000","last_undegraded":"2024-03-30T04:01:39.071776+0000","last_fullsized":"2024-03-30T04:01:39.071776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70292,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":190,"num_read_kb":165,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986450+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T04:01:17.986450+0000","last_peered":"2024-03-30T04:01:17.986450+0000","last_clean":"2024-03-30T04:01:17.986450+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T04:01:17.986450+0000","last_undegraded":"2024-03-30T04:01:17.986450+0000","last_fullsized":"2024-03-30T04:01:17.986450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990463+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T04:01:17.990463+0000","last_peered":"2024-03-30T04:01:17.990463+0000","last_clean":"2024-03-30T04:01:17.990463+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T04:01:17.990463+0000","last_undegraded":"2024-03-30T04:01:17.990463+0000","last_fullsized":"2024-03-30T04:01:17.990463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988801+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:01:17.988801+0000","last_peered":"2024-03-30T04:01:17.988801+0000","last_clean":"2024-03-30T04:01:17.988801+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:01:17.988801+0000","last_undegraded":"2024-03-30T04:01:17.988801+0000","last_fullsized":"2024-03-30T04:01:17.988801+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991071+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T04:01:17.991071+0000","last_peered":"2024-03-30T04:01:17.991071+0000","last_clean":"2024-03-30T04:01:17.991071+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T04:01:17.991071+0000","last_undegraded":"2024-03-30T04:01:17.991071+0000","last_fullsized":"2024-03-30T04:01:17.991071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990443+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T04:01:17.990443+0000","last_peered":"2024-03-30T04:01:17.990443+0000","last_clean":"2024-03-30T04:01:17.990443+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T04:01:17.990443+0000","last_undegraded":"2024-03-30T04:01:17.990443+0000","last_fullsized":"2024-03-30T04:01:17.990443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990549+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:01:17.990549+0000","last_peered":"2024-03-30T04:01:17.990549+0000","last_clean":"2024-03-30T04:01:17.990549+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:01:17.990549+0000","last_undegraded":"2024-03-30T04:01:17.990549+0000","last_fullsized":"2024-03-30T04:01:17.990549+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986863+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T04:01:17.986863+0000","last_peered":"2024-03-30T04:01:17.986863+0000","last_clean":"2024-03-30T04:01:17.986863+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T04:01:17.986863+0000","last_undegraded":"2024-03-30T04:01:17.986863+0000","last_fullsized":"2024-03-30T04:01:17.986863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039141+0000","last_change":"2024-03-30T04:00:18.020869+0000","last_active":"2024-03-30T04:01:18.039141+0000","last_peered":"2024-03-30T04:01:18.039141+0000","last_clean":"2024-03-30T04:01:18.039141+0000","last_became_active":"2024-03-30T04:00:18.020410+0000","last_became_peered":"2024-03-30T04:00:18.020410+0000","last_unstale":"2024-03-30T04:01:18.039141+0000","last_undegraded":"2024-03-30T04:01:18.039141+0000","last_fullsized":"2024-03-30T04:01:18.039141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:53.675022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"86'7","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:34.091171+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:01:34.091171+0000","last_peered":"2024-03-30T04:01:34.091171+0000","last_clean":"2024-03-30T04:01:34.091171+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:01:34.091171+0000","last_undegraded":"2024-03-30T04:01:34.091171+0000","last_fullsized":"2024-03-30T04:01:34.091171+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986379+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T04:01:17.986379+0000","last_peered":"2024-03-30T04:01:17.986379+0000","last_clean":"2024-03-30T04:01:17.986379+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T04:01:17.986379+0000","last_undegraded":"2024-03-30T04:01:17.986379+0000","last_fullsized":"2024-03-30T04:01:17.986379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992090+0000","last_change":"2024-03-30T04:00:18.288237+0000","last_active":"2024-03-30T04:01:17.992090+0000","last_peered":"2024-03-30T04:01:17.992090+0000","last_clean":"2024-03-30T04:01:17.992090+0000","last_became_active":"2024-03-30T04:00:18.287748+0000","last_became_peered":"2024-03-30T04:00:18.287748+0000","last_unstale":"2024-03-30T04:01:17.992090+0000","last_undegraded":"2024-03-30T04:01:17.992090+0000","last_fullsized":"2024-03-30T04:01:17.992090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:50.893522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985742+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:01:17.985742+0000","last_peered":"2024-03-30T04:01:17.985742+0000","last_clean":"2024-03-30T04:01:17.985742+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:01:17.985742+0000","last_undegraded":"2024-03-30T04:01:17.985742+0000","last_fullsized":"2024-03-30T04:01:17.985742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991765+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T04:01:17.991765+0000","last_peered":"2024-03-30T04:01:17.991765+0000","last_clean":"2024-03-30T04:01:17.991765+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T04:01:17.991765+0000","last_undegraded":"2024-03-30T04:01:17.991765+0000","last_fullsized":"2024-03-30T04:01:17.991765+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991661+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T04:01:17.991661+0000","last_peered":"2024-03-30T04:01:17.991661+0000","last_clean":"2024-03-30T04:01:17.991661+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T04:01:17.991661+0000","last_undegraded":"2024-03-30T04:01:17.991661+0000","last_fullsized":"2024-03-30T04:01:17.991661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986615+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:01:17.986615+0000","last_peered":"2024-03-30T04:01:17.986615+0000","last_clean":"2024-03-30T04:01:17.986615+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:01:17.986615+0000","last_undegraded":"2024-03-30T04:01:17.986615+0000","last_fullsized":"2024-03-30T04:01:17.986615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990240+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T04:01:17.990240+0000","last_peered":"2024-03-30T04:01:17.990240+0000","last_clean":"2024-03-30T04:01:17.990240+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T04:01:17.990240+0000","last_undegraded":"2024-03-30T04:01:17.990240+0000","last_fullsized":"2024-03-30T04:01:17.990240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986221+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T04:01:17.986221+0000","last_peered":"2024-03-30T04:01:17.986221+0000","last_clean":"2024-03-30T04:01:17.986221+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T04:01:17.986221+0000","last_undegraded":"2024-03-30T04:01:17.986221+0000","last_fullsized":"2024-03-30T04:01:17.986221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992002+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T04:01:17.992002+0000","last_peered":"2024-03-30T04:01:17.992002+0000","last_clean":"2024-03-30T04:01:17.992002+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T04:01:17.992002+0000","last_undegraded":"2024-03-30T04:01:17.992002+0000","last_fullsized":"2024-03-30T04:01:17.992002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038965+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T04:01:18.038965+0000","last_peered":"2024-03-30T04:01:18.038965+0000","last_clean":"2024-03-30T04:01:18.038965+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T04:01:18.038965+0000","last_undegraded":"2024-03-30T04:01:18.038965+0000","last_fullsized":"2024-03-30T04:01:18.038965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991065+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T04:01:17.991065+0000","last_peered":"2024-03-30T04:01:17.991065+0000","last_clean":"2024-03-30T04:01:17.991065+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T04:01:17.991065+0000","last_undegraded":"2024-03-30T04:01:17.991065+0000","last_fullsized":"2024-03-30T04:01:17.991065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'9","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:36.090477+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T04:01:36.090477+0000","last_peered":"2024-03-30T04:01:36.090477+0000","last_clean":"2024-03-30T04:01:36.090477+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T04:01:36.090477+0000","last_undegraded":"2024-03-30T04:01:36.090477+0000","last_fullsized":"2024-03-30T04:01:36.090477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1773,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038870+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T04:01:18.038870+0000","last_peered":"2024-03-30T04:01:18.038870+0000","last_clean":"2024-03-30T04:01:18.038870+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T04:01:18.038870+0000","last_undegraded":"2024-03-30T04:01:18.038870+0000","last_fullsized":"2024-03-30T04:01:18.038870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992237+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T04:01:17.992237+0000","last_peered":"2024-03-30T04:01:17.992237+0000","last_clean":"2024-03-30T04:01:17.992237+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T04:01:17.992237+0000","last_undegraded":"2024-03-30T04:01:17.992237+0000","last_fullsized":"2024-03-30T04:01:17.992237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990251+0000","last_change":"2024-03-30T04:00:18.018515+0000","last_active":"2024-03-30T04:01:17.990251+0000","last_peered":"2024-03-30T04:01:17.990251+0000","last_clean":"2024-03-30T04:01:17.990251+0000","last_became_active":"2024-03-30T04:00:18.017823+0000","last_became_peered":"2024-03-30T04:00:18.017823+0000","last_unstale":"2024-03-30T04:01:17.990251+0000","last_undegraded":"2024-03-30T04:01:17.990251+0000","last_fullsized":"2024-03-30T04:01:17.990251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:15:54.957524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.094872+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:01:37.094872+0000","last_peered":"2024-03-30T04:01:37.094872+0000","last_clean":"2024-03-30T04:01:37.094872+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:01:37.094872+0000","last_undegraded":"2024-03-30T04:01:37.094872+0000","last_fullsized":"2024-03-30T04:01:37.094872+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986577+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T04:01:17.986577+0000","last_peered":"2024-03-30T04:01:17.986577+0000","last_clean":"2024-03-30T04:01:17.986577+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T04:01:17.986577+0000","last_undegraded":"2024-03-30T04:01:17.986577+0000","last_fullsized":"2024-03-30T04:01:17.986577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992703+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T04:01:17.992703+0000","last_peered":"2024-03-30T04:01:17.992703+0000","last_clean":"2024-03-30T04:01:17.992703+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T04:01:17.992703+0000","last_undegraded":"2024-03-30T04:01:17.992703+0000","last_fullsized":"2024-03-30T04:01:17.992703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990294+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:01:17.990294+0000","last_peered":"2024-03-30T04:01:17.990294+0000","last_clean":"2024-03-30T04:01:17.990294+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:01:17.990294+0000","last_undegraded":"2024-03-30T04:01:17.990294+0000","last_fullsized":"2024-03-30T04:01:17.990294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986260+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T04:01:17.986260+0000","last_peered":"2024-03-30T04:01:17.986260+0000","last_clean":"2024-03-30T04:01:17.986260+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T04:01:17.986260+0000","last_undegraded":"2024-03-30T04:01:17.986260+0000","last_fullsized":"2024-03-30T04:01:17.986260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986543+0000","last_change":"2024-03-30T04:00:18.017573+0000","last_active":"2024-03-30T04:01:17.986543+0000","last_peered":"2024-03-30T04:01:17.986543+0000","last_clean":"2024-03-30T04:01:17.986543+0000","last_became_active":"2024-03-30T04:00:18.016937+0000","last_became_peered":"2024-03-30T04:00:18.016937+0000","last_unstale":"2024-03-30T04:01:17.986543+0000","last_undegraded":"2024-03-30T04:01:17.986543+0000","last_fullsized":"2024-03-30T04:01:17.986543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:26:20.863180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990431+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T04:01:17.990431+0000","last_peered":"2024-03-30T04:01:17.990431+0000","last_clean":"2024-03-30T04:01:17.990431+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T04:01:17.990431+0000","last_undegraded":"2024-03-30T04:01:17.990431+0000","last_fullsized":"2024-03-30T04:01:17.990431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986318+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:01:17.986318+0000","last_peered":"2024-03-30T04:01:17.986318+0000","last_clean":"2024-03-30T04:01:17.986318+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:01:17.986318+0000","last_undegraded":"2024-03-30T04:01:17.986318+0000","last_fullsized":"2024-03-30T04:01:17.986318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991339+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T04:01:17.991339+0000","last_peered":"2024-03-30T04:01:17.991339+0000","last_clean":"2024-03-30T04:01:17.991339+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T04:01:17.991339+0000","last_undegraded":"2024-03-30T04:01:17.991339+0000","last_fullsized":"2024-03-30T04:01:17.991339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986517+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:01:17.986517+0000","last_peered":"2024-03-30T04:01:17.986517+0000","last_clean":"2024-03-30T04:01:17.986517+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:01:17.986517+0000","last_undegraded":"2024-03-30T04:01:17.986517+0000","last_fullsized":"2024-03-30T04:01:17.986517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991400+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T04:01:17.991400+0000","last_peered":"2024-03-30T04:01:17.991400+0000","last_clean":"2024-03-30T04:01:17.991400+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T04:01:17.991400+0000","last_undegraded":"2024-03-30T04:01:17.991400+0000","last_fullsized":"2024-03-30T04:01:17.991400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991296+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T04:01:17.991296+0000","last_peered":"2024-03-30T04:01:17.991296+0000","last_clean":"2024-03-30T04:01:17.991296+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T04:01:17.991296+0000","last_undegraded":"2024-03-30T04:01:17.991296+0000","last_fullsized":"2024-03-30T04:01:17.991296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'10","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.322725+0000","last_change":"2024-03-30T04:00:18.297312+0000","last_active":"2024-03-30T04:01:18.322725+0000","last_peered":"2024-03-30T04:01:18.322725+0000","last_clean":"2024-03-30T04:01:18.322725+0000","last_became_active":"2024-03-30T04:00:18.288640+0000","last_became_peered":"2024-03-30T04:00:18.288640+0000","last_unstale":"2024-03-30T04:01:18.322725+0000","last_undegraded":"2024-03-30T04:01:18.322725+0000","last_fullsized":"2024-03-30T04:01:18.322725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:21.664643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992944+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T04:01:17.992944+0000","last_peered":"2024-03-30T04:01:17.992944+0000","last_clean":"2024-03-30T04:01:17.992944+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T04:01:17.992944+0000","last_undegraded":"2024-03-30T04:01:17.992944+0000","last_fullsized":"2024-03-30T04:01:17.992944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989266+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T04:01:17.989266+0000","last_peered":"2024-03-30T04:01:17.989266+0000","last_clean":"2024-03-30T04:01:17.989266+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T04:01:17.989266+0000","last_undegraded":"2024-03-30T04:01:17.989266+0000","last_fullsized":"2024-03-30T04:01:17.989266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990084+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:01:17.990084+0000","last_peered":"2024-03-30T04:01:17.990084+0000","last_clean":"2024-03-30T04:01:17.990084+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:01:17.990084+0000","last_undegraded":"2024-03-30T04:01:17.990084+0000","last_fullsized":"2024-03-30T04:01:17.990084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.005003+0000","last_change":"2024-03-30T04:01:19.005003+0000","last_active":"2024-03-30T04:01:19.005003+0000","last_peered":"2024-03-30T04:01:19.005003+0000","last_clean":"2024-03-30T04:01:19.005003+0000","last_became_active":"2024-03-30T04:01:19.004334+0000","last_became_peered":"2024-03-30T04:01:19.004334+0000","last_unstale":"2024-03-30T04:01:19.005003+0000","last_undegraded":"2024-03-30T04:01:19.005003+0000","last_fullsized":"2024-03-30T04:01:19.005003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:41:54.609191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990019+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T04:01:17.990019+0000","last_peered":"2024-03-30T04:01:17.990019+0000","last_clean":"2024-03-30T04:01:17.990019+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T04:01:17.990019+0000","last_undegraded":"2024-03-30T04:01:17.990019+0000","last_fullsized":"2024-03-30T04:01:17.990019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989613+0000","last_change":"2024-03-30T04:00:18.291229+0000","last_active":"2024-03-30T04:01:17.989613+0000","last_peered":"2024-03-30T04:01:17.989613+0000","last_clean":"2024-03-30T04:01:17.989613+0000","last_became_active":"2024-03-30T04:00:18.287954+0000","last_became_peered":"2024-03-30T04:00:18.287954+0000","last_unstale":"2024-03-30T04:01:17.989613+0000","last_undegraded":"2024-03-30T04:01:17.989613+0000","last_fullsized":"2024-03-30T04:01:17.989613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:54.678907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985778+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T04:01:17.985778+0000","last_peered":"2024-03-30T04:01:17.985778+0000","last_clean":"2024-03-30T04:01:17.985778+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T04:01:17.985778+0000","last_undegraded":"2024-03-30T04:01:17.985778+0000","last_fullsized":"2024-03-30T04:01:17.985778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993013+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T04:01:17.993013+0000","last_peered":"2024-03-30T04:01:17.993013+0000","last_clean":"2024-03-30T04:01:17.993013+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T04:01:17.993013+0000","last_undegraded":"2024-03-30T04:01:17.993013+0000","last_fullsized":"2024-03-30T04:01:17.993013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991372+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:01:17.991372+0000","last_peered":"2024-03-30T04:01:17.991372+0000","last_clean":"2024-03-30T04:01:17.991372+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:01:17.991372+0000","last_undegraded":"2024-03-30T04:01:17.991372+0000","last_fullsized":"2024-03-30T04:01:17.991372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990785+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T04:01:17.990785+0000","last_peered":"2024-03-30T04:01:17.990785+0000","last_clean":"2024-03-30T04:01:17.990785+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T04:01:17.990785+0000","last_undegraded":"2024-03-30T04:01:17.990785+0000","last_fullsized":"2024-03-30T04:01:17.990785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990718+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T04:01:17.990718+0000","last_peered":"2024-03-30T04:01:17.990718+0000","last_clean":"2024-03-30T04:01:17.990718+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T04:01:17.990718+0000","last_undegraded":"2024-03-30T04:01:17.990718+0000","last_fullsized":"2024-03-30T04:01:17.990718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988611+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T04:01:17.988611+0000","last_peered":"2024-03-30T04:01:17.988611+0000","last_clean":"2024-03-30T04:01:17.988611+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T04:01:17.988611+0000","last_undegraded":"2024-03-30T04:01:17.988611+0000","last_fullsized":"2024-03-30T04:01:17.988611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993400+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T04:01:17.993400+0000","last_peered":"2024-03-30T04:01:17.993400+0000","last_clean":"2024-03-30T04:01:17.993400+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T04:01:17.993400+0000","last_undegraded":"2024-03-30T04:01:17.993400+0000","last_fullsized":"2024-03-30T04:01:17.993400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989641+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T04:01:17.989641+0000","last_peered":"2024-03-30T04:01:17.989641+0000","last_clean":"2024-03-30T04:01:17.989641+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T04:01:17.989641+0000","last_undegraded":"2024-03-30T04:01:17.989641+0000","last_fullsized":"2024-03-30T04:01:17.989641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985994+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:01:17.985994+0000","last_peered":"2024-03-30T04:01:17.985994+0000","last_clean":"2024-03-30T04:01:17.985994+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:01:17.985994+0000","last_undegraded":"2024-03-30T04:01:17.985994+0000","last_fullsized":"2024-03-30T04:01:17.985994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986683+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T04:01:17.986683+0000","last_peered":"2024-03-30T04:01:17.986683+0000","last_clean":"2024-03-30T04:01:17.986683+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T04:01:17.986683+0000","last_undegraded":"2024-03-30T04:01:17.986683+0000","last_fullsized":"2024-03-30T04:01:17.986683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990340+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T04:01:17.990340+0000","last_peered":"2024-03-30T04:01:17.990340+0000","last_clean":"2024-03-30T04:01:17.990340+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T04:01:17.990340+0000","last_undegraded":"2024-03-30T04:01:17.990340+0000","last_fullsized":"2024-03-30T04:01:17.990340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":111,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:36.093875+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:01:36.093875+0000","last_peered":"2024-03-30T04:01:36.093875+0000","last_clean":"2024-03-30T04:01:36.093875+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:01:36.093875+0000","last_undegraded":"2024-03-30T04:01:36.093875+0000","last_fullsized":"2024-03-30T04:01:36.093875+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.999932+0000","last_change":"2024-03-30T04:01:18.999932+0000","last_active":"2024-03-30T04:01:18.999932+0000","last_peered":"2024-03-30T04:01:18.999932+0000","last_clean":"2024-03-30T04:01:18.999932+0000","last_became_active":"2024-03-30T04:01:18.999645+0000","last_became_peered":"2024-03-30T04:01:18.999645+0000","last_unstale":"2024-03-30T04:01:18.999932+0000","last_undegraded":"2024-03-30T04:01:18.999932+0000","last_fullsized":"2024-03-30T04:01:18.999932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:22:33.394341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986480+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T04:01:17.986480+0000","last_peered":"2024-03-30T04:01:17.986480+0000","last_clean":"2024-03-30T04:01:17.986480+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T04:01:17.986480+0000","last_undegraded":"2024-03-30T04:01:17.986480+0000","last_fullsized":"2024-03-30T04:01:17.986480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990215+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:01:17.990215+0000","last_peered":"2024-03-30T04:01:17.990215+0000","last_clean":"2024-03-30T04:01:17.990215+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:01:17.990215+0000","last_undegraded":"2024-03-30T04:01:17.990215+0000","last_fullsized":"2024-03-30T04:01:17.990215+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986563+0000","last_change":"2024-03-30T04:00:18.016967+0000","last_active":"2024-03-30T04:01:17.986563+0000","last_peered":"2024-03-30T04:01:17.986563+0000","last_clean":"2024-03-30T04:01:17.986563+0000","last_became_active":"2024-03-30T04:00:18.016555+0000","last_became_peered":"2024-03-30T04:00:18.016555+0000","last_unstale":"2024-03-30T04:01:17.986563+0000","last_undegraded":"2024-03-30T04:01:17.986563+0000","last_fullsized":"2024-03-30T04:01:17.986563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:34:50.538682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989682+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T04:01:17.989682+0000","last_peered":"2024-03-30T04:01:17.989682+0000","last_clean":"2024-03-30T04:01:17.989682+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T04:01:17.989682+0000","last_undegraded":"2024-03-30T04:01:17.989682+0000","last_fullsized":"2024-03-30T04:01:17.989682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992001+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:01:17.992001+0000","last_peered":"2024-03-30T04:01:17.992001+0000","last_clean":"2024-03-30T04:01:17.992001+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:01:17.992001+0000","last_undegraded":"2024-03-30T04:01:17.992001+0000","last_fullsized":"2024-03-30T04:01:17.992001+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.987339+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T04:01:17.987339+0000","last_peered":"2024-03-30T04:01:17.987339+0000","last_clean":"2024-03-30T04:01:17.987339+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T04:01:17.987339+0000","last_undegraded":"2024-03-30T04:01:17.987339+0000","last_fullsized":"2024-03-30T04:01:17.987339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990824+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T04:01:17.990824+0000","last_peered":"2024-03-30T04:01:17.990824+0000","last_clean":"2024-03-30T04:01:17.990824+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T04:01:17.990824+0000","last_undegraded":"2024-03-30T04:01:17.990824+0000","last_fullsized":"2024-03-30T04:01:17.990824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.001518+0000","last_change":"2024-03-30T04:01:19.001518+0000","last_active":"2024-03-30T04:01:19.001518+0000","last_peered":"2024-03-30T04:01:19.001518+0000","last_clean":"2024-03-30T04:01:19.001518+0000","last_became_active":"2024-03-30T04:01:19.001118+0000","last_became_peered":"2024-03-30T04:01:19.001118+0000","last_unstale":"2024-03-30T04:01:19.001518+0000","last_undegraded":"2024-03-30T04:01:19.001518+0000","last_fullsized":"2024-03-30T04:01:19.001518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:50:08.382669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988506+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T04:01:17.988506+0000","last_peered":"2024-03-30T04:01:17.988506+0000","last_clean":"2024-03-30T04:01:17.988506+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T04:01:17.988506+0000","last_undegraded":"2024-03-30T04:01:17.988506+0000","last_fullsized":"2024-03-30T04:01:17.988506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.004456+0000","last_change":"2024-03-30T04:01:19.004456+0000","last_active":"2024-03-30T04:01:19.004456+0000","last_peered":"2024-03-30T04:01:19.004456+0000","last_clean":"2024-03-30T04:01:19.004456+0000","last_became_active":"2024-03-30T04:01:19.004121+0000","last_became_peered":"2024-03-30T04:01:19.004121+0000","last_unstale":"2024-03-30T04:01:19.004456+0000","last_undegraded":"2024-03-30T04:01:19.004456+0000","last_fullsized":"2024-03-30T04:01:19.004456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:58:37.404002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991230+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T04:01:17.991230+0000","last_peered":"2024-03-30T04:01:17.991230+0000","last_clean":"2024-03-30T04:01:17.991230+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T04:01:17.991230+0000","last_undegraded":"2024-03-30T04:01:17.991230+0000","last_fullsized":"2024-03-30T04:01:17.991230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990493+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T04:01:17.990493+0000","last_peered":"2024-03-30T04:01:17.990493+0000","last_clean":"2024-03-30T04:01:17.990493+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T04:01:17.990493+0000","last_undegraded":"2024-03-30T04:01:17.990493+0000","last_fullsized":"2024-03-30T04:01:17.990493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988935+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T04:01:17.988935+0000","last_peered":"2024-03-30T04:01:17.988935+0000","last_clean":"2024-03-30T04:01:17.988935+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T04:01:17.988935+0000","last_undegraded":"2024-03-30T04:01:17.988935+0000","last_fullsized":"2024-03-30T04:01:17.988935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993122+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T04:01:17.993122+0000","last_peered":"2024-03-30T04:01:17.993122+0000","last_clean":"2024-03-30T04:01:17.993122+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T04:01:17.993122+0000","last_undegraded":"2024-03-30T04:01:17.993122+0000","last_fullsized":"2024-03-30T04:01:17.993122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990134+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T04:01:17.990134+0000","last_peered":"2024-03-30T04:01:17.990134+0000","last_clean":"2024-03-30T04:01:17.990134+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T04:01:17.990134+0000","last_undegraded":"2024-03-30T04:01:17.990134+0000","last_fullsized":"2024-03-30T04:01:17.990134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992792+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T04:01:17.992792+0000","last_peered":"2024-03-30T04:01:17.992792+0000","last_clean":"2024-03-30T04:01:17.992792+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T04:01:17.992792+0000","last_undegraded":"2024-03-30T04:01:17.992792+0000","last_fullsized":"2024-03-30T04:01:17.992792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990666+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T04:01:17.990666+0000","last_peered":"2024-03-30T04:01:17.990666+0000","last_clean":"2024-03-30T04:01:17.990666+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T04:01:17.990666+0000","last_undegraded":"2024-03-30T04:01:17.990666+0000","last_fullsized":"2024-03-30T04:01:17.990666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990825+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T04:01:17.990825+0000","last_peered":"2024-03-30T04:01:17.990825+0000","last_clean":"2024-03-30T04:01:17.990825+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T04:01:17.990825+0000","last_undegraded":"2024-03-30T04:01:17.990825+0000","last_fullsized":"2024-03-30T04:01:17.990825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990548+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T04:01:17.990548+0000","last_peered":"2024-03-30T04:01:17.990548+0000","last_clean":"2024-03-30T04:01:17.990548+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T04:01:17.990548+0000","last_undegraded":"2024-03-30T04:01:17.990548+0000","last_fullsized":"2024-03-30T04:01:17.990548+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78682,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":906,"num_read_kb":543,"num_write":255,"num_write_kb":161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":233526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547235,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":1007616,"data_stored":492856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.752},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78500000000000003,"5min":0.996,"15min":0.996},"last":0.73399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47899999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.81499999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.876,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.89200000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73399999999999999,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.76700000000000002}]},{"osd":8,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85899999999999999}]},{"osd":9,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.60499999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.052,"5min":1.097,"15min":1.097},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79200000000000004,"15min":0.78200000000000003},"min":{"1min":0.58299999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.91500000000000004}]},{"osd":10,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.625,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.93899999999999995,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.93799999999999994,"5min":1.536,"15min":1.536},"last":0.92300000000000004}]}]},{"osd":10,"up_from":68,"seq":292057776166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1708032,"data_stored":1150498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36043201},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.623,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68500000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48999999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.51100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.77500000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.01,"15min":1.01},"last":0.626},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95499999999999996,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.66800000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97599999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.83599999999999997}]},{"osd":9,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.71599999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.018,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65200000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.95699999999999996,"5min":1.147,"15min":1.147},"last":0.751}]},{"osd":11,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75600000000000001,"15min":0.746},"min":{"1min":0.63400000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79600000000000004,"15min":0.78900000000000003},"min":{"1min":0.68600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.98599999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85699999999999998}]}]},{"osd":9,"up_from":63,"seq":270582939690,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":1036288,"data_stored":496930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.2589999999999999,"5min":3.706,"15min":3.706},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60199999999999998,"15min":0.63800000000000001},"min":{"1min":0.374,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":1.018,"15min":1.018},"last":0.372}]},{"osd":1,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.69599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.74099999999999999,"15min":0.76300000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77500000000000002,"15min":0.80700000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.501,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.47699999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67000000000000004,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86599999999999999,"5min":1.155,"15min":1.155},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.46899999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6819999999999999,"5min":1.72,"15min":1.72},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.88100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.097,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.89200000000000002,"15min":0.94999999999999996},"min":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0489999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.51000000000000001}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.93600000000000005,"15min":0.998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.107,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.752},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.95799999999999996,"15min":1.03},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.887,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.57999999999999996}]},{"osd":8,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94099999999999995,"15min":0.98799999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.997,"15min":1.0629999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1910000000000001,"5min":2.379,"15min":2.379},"last":0.72599999999999998}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.92700000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2430000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.78000000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.64000000000000001}]}]},{"osd":8,"up_from":56,"seq":240518168623,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":1028096,"data_stored":493212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.76200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57199999999999995,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.40999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.755,"5min":0.876,"15min":0.876},"last":0.54100000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.36799999999999999},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.434},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.48099999999999998,"15min":0.47099999999999997},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.55600000000000005,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.78700000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.48199999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.621},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.56799999999999995,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97199999999999998,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93899999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.64800000000000002},"min":{"1min":0.52500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.81799999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.60199999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.93899999999999995,"5min":1.006,"15min":1.006},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.76600000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.80300000000000005}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.55200000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.84999999999999998}]},{"osd":11,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.629},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70799999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397552,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1091334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.58999999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.437,"5min":0.372,"15min":0.372},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.753,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.498}]},{"osd":4,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60799999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.434}]},{"osd":5,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.505,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.67000000000000004,"15min":0.68700000000000006},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72099999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.82899999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63400000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79500000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.53100000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88900000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59699999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.76400000000000001},"min":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.85599999999999998,"5min":1.258,"15min":1.258},"last":0.69699999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96599999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.754}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561077,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":489523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.626},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.56999999999999995,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.65100000000000002,"15min":0.63200000000000001},"min":{"1min":0.51200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.67100000000000004},"min":{"1min":0.55200000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.57299999999999995,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.442,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.58299999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76700000000000002,"15min":0.749},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.61399999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.337,"5min":1.462,"15min":1.462},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.68300000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.72399999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.82399999999999995,"15min":0.79800000000000004},"min":{"1min":0.52000000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.64200000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.073}]},{"osd":8,"last update":"Sat Mar 30 04:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.71899999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.88800000000000001,"15min":0.86699999999999999},"min":{"1min":0.76500000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.109}]},{"osd":9,"last update":"Sat Mar 30 04:01:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.92300000000000004,"15min":0.91200000000000003},"min":{"1min":0.68000000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.512,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.877,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.109,"5min":1.488,"15min":1.488},"last":0.751}]},{"osd":10,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95199999999999996,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.075,"5min":0.97499999999999998,"15min":0.95899999999999996},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.82699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":1081344,"data_stored":560845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.503},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.57099999999999995,"15min":0.52100000000000002},"min":{"1min":0.40100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55100000000000005,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.51100000000000001},"min":{"1min":0.46400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.498},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.53800000000000003},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.871,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.442,"15min":0.40200000000000002},"min":{"1min":0.36199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.41899999999999998},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69199999999999995,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60799999999999998,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.622,"15min":0.57599999999999996},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92300000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91900000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.61499999999999999},"min":{"1min":0.503,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82699999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.59799999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.82199999999999995,"15min":0.80000000000000004},"min":{"1min":0.61699999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89500000000000002,"5min":3.2080000000000002,"15min":3.2080000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.755},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.79800000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.746,"15min":0.745},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.93600000000000005,"5min":0.999,"15min":0.999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78300000000000003,"15min":0.78200000000000003},"min":{"1min":0.58099999999999996,"5min":0.501,"15min":0.501},"max":{"1min":0.94099999999999995,"5min":1.073,"15min":1.073},"last":0.65300000000000002}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.755},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.76900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":489746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.59499999999999997,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60299999999999998,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.67800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.378,"15min":0.378},"max":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.9859999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.96,"5min":6.96,"15min":6.96},"last":0.92900000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.61799999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71999999999999997,"15min":0.67500000000000004},"min":{"1min":0.51000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73999999999999999,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72299999999999998,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.113,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Sat Mar 30 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67900000000000005,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.76200000000000001,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.78700000000000003}]},{"osd":7,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.76100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88200000000000001,"15min":0.88800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.93300000000000005,"15min":0.88800000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":9.7439999999999998,"5min":9.7439999999999998,"15min":9.7439999999999998},"last":0.95699999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86899999999999999,"15min":0.878},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.2290000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.93200000000000005,"15min":0.95399999999999996},"min":{"1min":0.53000000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88700000000000001,"15min":0.879},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.873},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92100000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.91400000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:01:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.2809999999999999,"5min":1.298,"15min":1.298},"last":0.96399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":557657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78900000000000003,"5min":1.133,"15min":1.133},"last":0.57999999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.56499999999999995,"15min":0.51900000000000002},"min":{"1min":0.47499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78600000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.51700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.71199999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.47699999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.56999999999999995,"15min":0.56399999999999995},"min":{"1min":0.45700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.871}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.47699999999999998,"15min":0.44900000000000001},"min":{"1min":0.41599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72599999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.55300000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.623,"15min":0.61099999999999999},"min":{"1min":0.46600000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.874,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.54500000000000004,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.94199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.68500000000000005}]},{"osd":7,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.84899999999999998}]},{"osd":8,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55700000000000005},"min":{"1min":0.503,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91000000000000003,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.56200000000000006,"15min":0.53000000000000003},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66200000000000003}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69999999999999996,"15min":0.66300000000000003},"min":{"1min":0.65200000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.754,"15min":0.72899999999999998},"min":{"1min":0.64700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":0.995,"15min":0.995},"last":0.755}]},{"osd":10,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71299999999999997,"15min":0.69999999999999996},"min":{"1min":0.58799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":0.93700000000000006,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.82999999999999996}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.81000000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":488046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.42299999999999999},"min":{"1min":0.34599999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.443},"min":{"1min":0.36699999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64500000000000002,"15min":0.61399999999999999},"min":{"1min":0.44800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.633,"5min":1.762,"15min":1.762},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.71399999999999997,"15min":0.628},"min":{"1min":0.56000000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.601,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.626},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66000000000000003,"15min":0.63},"min":{"1min":0.495,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.67700000000000005,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.748,"15min":0.71399999999999997},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72799999999999998,"15min":0.67300000000000004},"min":{"1min":0.58099999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79600000000000004,"15min":0.73899999999999999},"min":{"1min":0.69899999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79600000000000004,"15min":0.76200000000000001},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.72899999999999998},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.45600000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89500000000000002,"15min":0.88300000000000001},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.58899999999999997}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.99199999999999999,"15min":1.018},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.61899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.61899999999999999}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.02,"5min":0.99299999999999999,"15min":0.98899999999999999},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.71399999999999997}]},{"osd":11,"last update":"Sat Mar 30 04:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.73799999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953533,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":1015808,"data_stored":492898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.372},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53800000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.376},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.32800000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55700000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40400000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.41799999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.53400000000000003},"min":{"1min":0.47899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81200000000000006,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.80700000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60599999999999998,"15min":0.58099999999999996},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90900000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59099999999999997,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.57199999999999995},"min":{"1min":0.52500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85299999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.70899999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.441,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.64400000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.54300000000000004},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.871,"15min":0.871},"last":0.55900000000000005}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.56999999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89300000000000002,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68100000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.78200000000000003}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.76600000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.79100000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91100000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.76800000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.749,"15min":0.75700000000000001},"min":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.746,"15min":0.754},"min":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.621}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.747}]}]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1630208,"data_stored":1084914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.373}]},{"osd":1,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57299999999999995,"15min":0.55300000000000005},"min":{"1min":0.39700000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.754,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40500000000000003,"15min":0.39200000000000002},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.626,"5min":0.747,"15min":0.747},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.438,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84899999999999998,"5min":0.873,"15min":0.873},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.60299999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.504}]},{"osd":4,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.753,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73399999999999999,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77000000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.622},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.016,"15min":1.016},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.53900000000000003}]},{"osd":8,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68799999999999994,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.59699999999999998,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.57199999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73599999999999999,"15min":0.71999999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.68999999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.84799999999999998,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.84299999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90800000000000003,"5min":1.222,"15min":1.222},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":70041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":72514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:01:42.069 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-30T04:01:42.069 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 -- 172.21.15.23:0/1442029466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e00737b0 msgr2=0x7f98e0073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 --2- 172.21.15.23:0/1442029466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e00737b0 0x7f98e0073c30 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009990 tx=0x7f98cc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 -- 172.21.15.23:0/1442029466 shutdown_connections 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 --2- 172.21.15.23:0/1442029466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e0108b60 0x7f98e010afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 --2- 172.21.15.23:0/1442029466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e00731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 --2- 172.21.15.23:0/1442029466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e00737b0 0x7f98e0073c30 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 -- 172.21.15.23:0/1442029466 >> 172.21.15.23:0/1442029466 conn(0x7f98e00fc200 msgr2=0x7f98e00fe640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:42.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 -- 172.21.15.23:0/1442029466 shutdown_connections 2024-03-30T04:01:42.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.271+0000 7f98e5e00700 1 -- 172.21.15.23:0/1442029466 wait complete. 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 Processor -- start 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 -- start start 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 0x7f98e0198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 0x7f98e01a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e01a08f0 con 0x7f98e0108b60 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e01a0a60 con 0x7f98e0074d90 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.272+0000 7f98e5e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e01a0c20 con 0x7f98e00737b0 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98df400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 0x7f98e01a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38172/0 (socket says 172.21.15.23:38172) 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 -- 172.21.15.23:0/4257224107 learned_addr learned my addr 172.21.15.23:0/4257224107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98dea00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 0x7f98e0198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 -- 172.21.15.23:0/4257224107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 msgr2=0x7f98e0198e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 0x7f98e0198e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 -- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 msgr2=0x7f98e01a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 0x7f98e01a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 -- 172.21.15.23:0/4257224107 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc0095f0 con 0x7f98e0074d90 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98dea00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 0x7f98e0198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98df400700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 0x7f98e01a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.273+0000 7f98de000700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:42.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.274+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc01d070 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.274+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e01a0ea0 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.274+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f98cc021410 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.274+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc017650 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.274+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e01a1260 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.275+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e0107970 con 0x7f98e0074d90 2024-03-30T04:01:42.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.277+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f98cc021810 con 0x7f98e0074d90 2024-03-30T04:01:42.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.278+0000 7f98e4a00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 0x7f98c40a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.278+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f98cc0cea00 con 0x7f98e0074d90 2024-03-30T04:01:42.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.278+0000 7f98dea00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 0x7f98c40a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.279+0000 7f98dea00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 0x7f98c40a3670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f98e019a280 tx=0x7f98d0009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:42.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.280+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f98cc097ce0 con 0x7f98e0074d90 2024-03-30T04:01:42.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.494+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98e00611f0 con 0x7f98c40a11b0 2024-03-30T04:01:42.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.507+0000 7f98e4a00700 1 -- 172.21.15.23:0/4257224107 <== mgr.14154 v2:172.21.15.23:6800/2266065890 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427331 (secure 0 0 0) 0x7f98e00611f0 con 0x7f98c40a11b0 2024-03-30T04:01:42.511 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 msgr2=0x7f98c40a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 0x7f98c40a3670 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f98e019a280 tx=0x7f98d0009260 comp rx=0 tx=0).stop 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 msgr2=0x7f98e0199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc005740 comp rx=0 tx=0).stop 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 shutdown_connections 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f98e0108b60 0x7f98e01a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f98c40a11b0 0x7f98c40a3670 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e00737b0 0x7f98e0198e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 --2- 172.21.15.23:0/4257224107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e0074d90 0x7f98e0199340 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 >> 172.21.15.23:0/4257224107 conn(0x7f98e00fc200 msgr2=0x7f98e0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 shutdown_connections 2024-03-30T04:01:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.512+0000 7f98e5e00700 1 -- 172.21.15.23:0/4257224107 wait complete. 2024-03-30T04:01:42.516 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-30T04:01:42.529 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-03-30T04:01:42.304674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669284,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1056,"num_read_kb":667,"num_write":440,"num_write_kb":3102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":865,"ondisk_log_size":865,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383236,"kb_used_data":13960,"kb_used_omap":18,"kb_used_meta":369133,"kb_avail":1124558588,"statfs":{"total":1151940427776,"available":1151547994112,"internally_reserved":0,"allocated":14295040,"data_stored":7890979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19131,"internal_metadata":377992517},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8625,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":67,"num_write":13,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003412"},"pg_stats":[{"pgid":"3.3f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991830+0000","last_change":"2024-03-30T03:59:40.463593+0000","last_active":"2024-03-30T04:01:17.991830+0000","last_peered":"2024-03-30T04:01:17.991830+0000","last_clean":"2024-03-30T04:01:17.991830+0000","last_became_active":"2024-03-30T03:59:40.463242+0000","last_became_peered":"2024-03-30T03:59:40.463242+0000","last_unstale":"2024-03-30T04:01:17.991830+0000","last_undegraded":"2024-03-30T04:01:17.991830+0000","last_fullsized":"2024-03-30T04:01:17.991830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:42:35.569753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986818+0000","last_change":"2024-03-30T03:59:40.464578+0000","last_active":"2024-03-30T04:01:17.986818+0000","last_peered":"2024-03-30T04:01:17.986818+0000","last_clean":"2024-03-30T04:01:17.986818+0000","last_became_active":"2024-03-30T03:59:40.461640+0000","last_became_peered":"2024-03-30T03:59:40.461640+0000","last_unstale":"2024-03-30T04:01:17.986818+0000","last_undegraded":"2024-03-30T04:01:17.986818+0000","last_fullsized":"2024-03-30T04:01:17.986818+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:54.300359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986844+0000","last_change":"2024-03-30T03:59:40.457190+0000","last_active":"2024-03-30T04:01:17.986844+0000","last_peered":"2024-03-30T04:01:17.986844+0000","last_clean":"2024-03-30T04:01:17.986844+0000","last_became_active":"2024-03-30T03:59:40.455614+0000","last_became_peered":"2024-03-30T03:59:40.455614+0000","last_unstale":"2024-03-30T04:01:17.986844+0000","last_undegraded":"2024-03-30T04:01:17.986844+0000","last_fullsized":"2024-03-30T04:01:17.986844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:18:22.075225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"86'8","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.997523+0000","last_change":"2024-03-30T03:59:40.462530+0000","last_active":"2024-03-30T04:01:18.997523+0000","last_peered":"2024-03-30T04:01:18.997523+0000","last_clean":"2024-03-30T04:01:18.997523+0000","last_became_active":"2024-03-30T03:59:40.462217+0000","last_became_peered":"2024-03-30T03:59:40.462217+0000","last_unstale":"2024-03-30T04:01:18.997523+0000","last_undegraded":"2024-03-30T04:01:18.997523+0000","last_fullsized":"2024-03-30T04:01:18.997523+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:06:34.077038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986025+0000","last_change":"2024-03-30T03:59:40.468124+0000","last_active":"2024-03-30T04:01:17.986025+0000","last_peered":"2024-03-30T04:01:17.986025+0000","last_clean":"2024-03-30T04:01:17.986025+0000","last_became_active":"2024-03-30T03:59:40.467764+0000","last_became_peered":"2024-03-30T03:59:40.467764+0000","last_unstale":"2024-03-30T04:01:17.986025+0000","last_undegraded":"2024-03-30T04:01:17.986025+0000","last_fullsized":"2024-03-30T04:01:17.986025+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:15:57.825270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:40.454292+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:40.453948+0000","last_became_peered":"2024-03-30T03:59:40.453948+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:09:50.909879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986574+0000","last_change":"2024-03-30T03:59:40.459726+0000","last_active":"2024-03-30T04:01:17.986574+0000","last_peered":"2024-03-30T04:01:17.986574+0000","last_clean":"2024-03-30T04:01:17.986574+0000","last_became_active":"2024-03-30T03:59:40.459094+0000","last_became_peered":"2024-03-30T03:59:40.459094+0000","last_unstale":"2024-03-30T04:01:17.986574+0000","last_undegraded":"2024-03-30T04:01:17.986574+0000","last_fullsized":"2024-03-30T04:01:17.986574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:51:38.979328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991399+0000","last_change":"2024-03-30T03:59:40.466688+0000","last_active":"2024-03-30T04:01:17.991399+0000","last_peered":"2024-03-30T04:01:17.991399+0000","last_clean":"2024-03-30T04:01:17.991399+0000","last_became_active":"2024-03-30T03:59:40.466333+0000","last_became_peered":"2024-03-30T03:59:40.466333+0000","last_unstale":"2024-03-30T04:01:17.991399+0000","last_undegraded":"2024-03-30T04:01:17.991399+0000","last_fullsized":"2024-03-30T04:01:17.991399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:10:52.548643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989988+0000","last_change":"2024-03-30T03:59:40.454777+0000","last_active":"2024-03-30T04:01:17.989988+0000","last_peered":"2024-03-30T04:01:17.989988+0000","last_clean":"2024-03-30T04:01:17.989988+0000","last_became_active":"2024-03-30T03:59:40.454405+0000","last_became_peered":"2024-03-30T03:59:40.454405+0000","last_unstale":"2024-03-30T04:01:17.989988+0000","last_undegraded":"2024-03-30T04:01:17.989988+0000","last_fullsized":"2024-03-30T04:01:17.989988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:01:02.474274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"86'16","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039169+0000","last_change":"2024-03-30T03:59:40.457350+0000","last_active":"2024-03-30T04:01:18.039169+0000","last_peered":"2024-03-30T04:01:18.039169+0000","last_clean":"2024-03-30T04:01:18.039169+0000","last_became_active":"2024-03-30T03:59:40.457015+0000","last_became_peered":"2024-03-30T03:59:40.457015+0000","last_unstale":"2024-03-30T04:01:18.039169+0000","last_undegraded":"2024-03-30T04:01:18.039169+0000","last_fullsized":"2024-03-30T04:01:18.039169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:35:55.020784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986792+0000","last_change":"2024-03-30T03:59:40.462680+0000","last_active":"2024-03-30T04:01:17.986792+0000","last_peered":"2024-03-30T04:01:17.986792+0000","last_clean":"2024-03-30T04:01:17.986792+0000","last_became_active":"2024-03-30T03:59:40.461493+0000","last_became_peered":"2024-03-30T03:59:40.461493+0000","last_unstale":"2024-03-30T04:01:17.986792+0000","last_undegraded":"2024-03-30T04:01:17.986792+0000","last_fullsized":"2024-03-30T04:01:17.986792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:09:05.115628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990814+0000","last_change":"2024-03-30T03:59:40.462007+0000","last_active":"2024-03-30T04:01:17.990814+0000","last_peered":"2024-03-30T04:01:17.990814+0000","last_clean":"2024-03-30T04:01:17.990814+0000","last_became_active":"2024-03-30T03:59:40.461688+0000","last_became_peered":"2024-03-30T03:59:40.461688+0000","last_unstale":"2024-03-30T04:01:17.990814+0000","last_undegraded":"2024-03-30T04:01:17.990814+0000","last_fullsized":"2024-03-30T04:01:17.990814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:12:13.202974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990767+0000","last_change":"2024-03-30T03:59:40.461286+0000","last_active":"2024-03-30T04:01:17.990767+0000","last_peered":"2024-03-30T04:01:17.990767+0000","last_clean":"2024-03-30T04:01:17.990767+0000","last_became_active":"2024-03-30T03:59:40.460947+0000","last_became_peered":"2024-03-30T03:59:40.460947+0000","last_unstale":"2024-03-30T04:01:17.990767+0000","last_undegraded":"2024-03-30T04:01:17.990767+0000","last_fullsized":"2024-03-30T04:01:17.990767+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:47:27.870022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986474+0000","last_change":"2024-03-30T03:59:40.460513+0000","last_active":"2024-03-30T04:01:17.986474+0000","last_peered":"2024-03-30T04:01:17.986474+0000","last_clean":"2024-03-30T04:01:17.986474+0000","last_became_active":"2024-03-30T03:59:40.457088+0000","last_became_peered":"2024-03-30T03:59:40.457088+0000","last_unstale":"2024-03-30T04:01:17.986474+0000","last_undegraded":"2024-03-30T04:01:17.986474+0000","last_fullsized":"2024-03-30T04:01:17.986474+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:35.410107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986493+0000","last_change":"2024-03-30T03:59:40.465609+0000","last_active":"2024-03-30T04:01:17.986493+0000","last_peered":"2024-03-30T04:01:17.986493+0000","last_clean":"2024-03-30T04:01:17.986493+0000","last_became_active":"2024-03-30T03:59:40.460877+0000","last_became_peered":"2024-03-30T03:59:40.460877+0000","last_unstale":"2024-03-30T04:01:17.986493+0000","last_undegraded":"2024-03-30T04:01:17.986493+0000","last_fullsized":"2024-03-30T04:01:17.986493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:02.372891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989473+0000","last_change":"2024-03-30T03:59:40.455102+0000","last_active":"2024-03-30T04:01:17.989473+0000","last_peered":"2024-03-30T04:01:17.989473+0000","last_clean":"2024-03-30T04:01:17.989473+0000","last_became_active":"2024-03-30T03:59:40.454712+0000","last_became_peered":"2024-03-30T03:59:40.454712+0000","last_unstale":"2024-03-30T04:01:17.989473+0000","last_undegraded":"2024-03-30T04:01:17.989473+0000","last_fullsized":"2024-03-30T04:01:17.989473+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:37:11.717411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991668+0000","last_change":"2024-03-30T03:59:40.464365+0000","last_active":"2024-03-30T04:01:17.991668+0000","last_peered":"2024-03-30T04:01:17.991668+0000","last_clean":"2024-03-30T04:01:17.991668+0000","last_became_active":"2024-03-30T03:59:40.463302+0000","last_became_peered":"2024-03-30T03:59:40.463302+0000","last_unstale":"2024-03-30T04:01:17.991668+0000","last_undegraded":"2024-03-30T04:01:17.991668+0000","last_fullsized":"2024-03-30T04:01:17.991668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:59:24.073044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992709+0000","last_change":"2024-03-30T03:59:40.459027+0000","last_active":"2024-03-30T04:01:17.992709+0000","last_peered":"2024-03-30T04:01:17.992709+0000","last_clean":"2024-03-30T04:01:17.992709+0000","last_became_active":"2024-03-30T03:59:40.458635+0000","last_became_peered":"2024-03-30T03:59:40.458635+0000","last_unstale":"2024-03-30T04:01:17.992709+0000","last_undegraded":"2024-03-30T04:01:17.992709+0000","last_fullsized":"2024-03-30T04:01:17.992709+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:19:08.922608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989001+0000","last_change":"2024-03-30T03:59:40.467111+0000","last_active":"2024-03-30T04:01:17.989001+0000","last_peered":"2024-03-30T04:01:17.989001+0000","last_clean":"2024-03-30T04:01:17.989001+0000","last_became_active":"2024-03-30T03:59:40.466495+0000","last_became_peered":"2024-03-30T03:59:40.466495+0000","last_unstale":"2024-03-30T04:01:17.989001+0000","last_undegraded":"2024-03-30T04:01:17.989001+0000","last_fullsized":"2024-03-30T04:01:17.989001+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:07.740286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989435+0000","last_change":"2024-03-30T03:59:44.472830+0000","last_active":"2024-03-30T04:01:17.989435+0000","last_peered":"2024-03-30T04:01:17.989435+0000","last_clean":"2024-03-30T04:01:17.989435+0000","last_became_active":"2024-03-30T03:59:44.472452+0000","last_became_peered":"2024-03-30T03:59:44.472452+0000","last_unstale":"2024-03-30T04:01:17.989435+0000","last_undegraded":"2024-03-30T04:01:17.989435+0000","last_fullsized":"2024-03-30T04:01:17.989435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:51:33.364523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'15","reported_seq":124,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:40.095591+0000","last_change":"2024-03-30T03:59:39.929857+0000","last_active":"2024-03-30T04:01:40.095591+0000","last_peered":"2024-03-30T04:01:40.095591+0000","last_clean":"2024-03-30T04:01:40.095591+0000","last_became_active":"2024-03-30T03:59:39.928570+0000","last_became_peered":"2024-03-30T03:59:39.928570+0000","last_unstale":"2024-03-30T04:01:40.095591+0000","last_undegraded":"2024-03-30T04:01:40.095591+0000","last_fullsized":"2024-03-30T04:01:40.095591+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:13:23.178730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.998931+0000","last_change":"2024-03-30T03:59:40.475417+0000","last_active":"2024-03-30T04:01:18.998931+0000","last_peered":"2024-03-30T04:01:18.998931+0000","last_clean":"2024-03-30T04:01:18.998931+0000","last_became_active":"2024-03-30T03:59:40.473624+0000","last_became_peered":"2024-03-30T03:59:40.473624+0000","last_unstale":"2024-03-30T04:01:18.998931+0000","last_undegraded":"2024-03-30T04:01:18.998931+0000","last_fullsized":"2024-03-30T04:01:18.998931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:11:23.495433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989691+0000","last_change":"2024-03-30T03:59:44.472693+0000","last_active":"2024-03-30T04:01:17.989691+0000","last_peered":"2024-03-30T04:01:17.989691+0000","last_clean":"2024-03-30T04:01:17.989691+0000","last_became_active":"2024-03-30T03:59:44.472316+0000","last_became_peered":"2024-03-30T03:59:44.472316+0000","last_unstale":"2024-03-30T04:01:17.989691+0000","last_undegraded":"2024-03-30T04:01:17.989691+0000","last_fullsized":"2024-03-30T04:01:17.989691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:06:07.888193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'2","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.999025+0000","last_change":"2024-03-30T03:59:39.930019+0000","last_active":"2024-03-30T04:01:18.999025+0000","last_peered":"2024-03-30T04:01:18.999025+0000","last_clean":"2024-03-30T04:01:18.999025+0000","last_became_active":"2024-03-30T03:59:39.928186+0000","last_became_peered":"2024-03-30T03:59:39.928186+0000","last_unstale":"2024-03-30T04:01:18.999025+0000","last_undegraded":"2024-03-30T04:01:18.999025+0000","last_fullsized":"2024-03-30T04:01:18.999025+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:51:41.001693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986607+0000","last_change":"2024-03-30T03:59:40.455659+0000","last_active":"2024-03-30T04:01:17.986607+0000","last_peered":"2024-03-30T04:01:17.986607+0000","last_clean":"2024-03-30T04:01:17.986607+0000","last_became_active":"2024-03-30T03:59:40.455360+0000","last_became_peered":"2024-03-30T03:59:40.455360+0000","last_unstale":"2024-03-30T04:01:17.986607+0000","last_undegraded":"2024-03-30T04:01:17.986607+0000","last_fullsized":"2024-03-30T04:01:17.986607+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:12:54.938257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990211+0000","last_change":"2024-03-30T04:00:18.019833+0000","last_active":"2024-03-30T04:01:17.990211+0000","last_peered":"2024-03-30T04:01:17.990211+0000","last_clean":"2024-03-30T04:01:17.990211+0000","last_became_active":"2024-03-30T04:00:18.019445+0000","last_became_peered":"2024-03-30T04:00:18.019445+0000","last_unstale":"2024-03-30T04:01:17.990211+0000","last_undegraded":"2024-03-30T04:01:17.990211+0000","last_fullsized":"2024-03-30T04:01:17.990211+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:41:55.882417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,8,5],"acting":[4,3,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"88'13","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.998604+0000","last_change":"2024-03-30T03:59:39.927529+0000","last_active":"2024-03-30T04:01:18.998604+0000","last_peered":"2024-03-30T04:01:18.998604+0000","last_clean":"2024-03-30T04:01:18.998604+0000","last_became_active":"2024-03-30T03:59:39.927202+0000","last_became_peered":"2024-03-30T03:59:39.927202+0000","last_unstale":"2024-03-30T04:01:18.998604+0000","last_undegraded":"2024-03-30T04:01:18.998604+0000","last_fullsized":"2024-03-30T04:01:18.998604+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:24:50.627198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990547+0000","last_change":"2024-03-30T03:59:40.457092+0000","last_active":"2024-03-30T04:01:17.990547+0000","last_peered":"2024-03-30T04:01:17.990547+0000","last_clean":"2024-03-30T04:01:17.990547+0000","last_became_active":"2024-03-30T03:59:40.455997+0000","last_became_peered":"2024-03-30T03:59:40.455997+0000","last_unstale":"2024-03-30T04:01:17.990547+0000","last_undegraded":"2024-03-30T04:01:17.990547+0000","last_fullsized":"2024-03-30T04:01:17.990547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:45:28.898901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990466+0000","last_change":"2024-03-30T03:59:44.470409+0000","last_active":"2024-03-30T04:01:17.990466+0000","last_peered":"2024-03-30T04:01:17.990466+0000","last_clean":"2024-03-30T04:01:17.990466+0000","last_became_active":"2024-03-30T03:59:44.470045+0000","last_became_peered":"2024-03-30T03:59:44.470045+0000","last_unstale":"2024-03-30T04:01:17.990466+0000","last_undegraded":"2024-03-30T04:01:17.990466+0000","last_fullsized":"2024-03-30T04:01:17.990466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:03:05.890338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986226+0000","last_change":"2024-03-30T03:59:39.920885+0000","last_active":"2024-03-30T04:01:17.986226+0000","last_peered":"2024-03-30T04:01:17.986226+0000","last_clean":"2024-03-30T04:01:17.986226+0000","last_became_active":"2024-03-30T03:59:39.920501+0000","last_became_peered":"2024-03-30T03:59:39.920501+0000","last_unstale":"2024-03-30T04:01:17.986226+0000","last_undegraded":"2024-03-30T04:01:17.986226+0000","last_fullsized":"2024-03-30T04:01:17.986226+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:20:06.671602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990105+0000","last_change":"2024-03-30T03:59:40.460686+0000","last_active":"2024-03-30T04:01:17.990105+0000","last_peered":"2024-03-30T04:01:17.990105+0000","last_clean":"2024-03-30T04:01:17.990105+0000","last_became_active":"2024-03-30T03:59:40.460402+0000","last_became_peered":"2024-03-30T03:59:40.460402+0000","last_unstale":"2024-03-30T04:01:17.990105+0000","last_undegraded":"2024-03-30T04:01:17.990105+0000","last_fullsized":"2024-03-30T04:01:17.990105+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:41.578692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988764+0000","last_change":"2024-03-30T03:59:44.463863+0000","last_active":"2024-03-30T04:01:17.988764+0000","last_peered":"2024-03-30T04:01:17.988764+0000","last_clean":"2024-03-30T04:01:17.988764+0000","last_became_active":"2024-03-30T03:59:44.463367+0000","last_became_peered":"2024-03-30T03:59:44.463367+0000","last_unstale":"2024-03-30T04:01:17.988764+0000","last_undegraded":"2024-03-30T04:01:17.988764+0000","last_fullsized":"2024-03-30T04:01:17.988764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:23.066381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"88'6","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.068405+0000","last_change":"2024-03-30T03:59:39.927638+0000","last_active":"2024-03-30T04:01:37.068405+0000","last_peered":"2024-03-30T04:01:37.068405+0000","last_clean":"2024-03-30T04:01:37.068405+0000","last_became_active":"2024-03-30T03:59:39.927297+0000","last_became_peered":"2024-03-30T03:59:39.927297+0000","last_unstale":"2024-03-30T04:01:37.068405+0000","last_undegraded":"2024-03-30T04:01:37.068405+0000","last_fullsized":"2024-03-30T04:01:37.068405+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:59.826146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":967,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986076+0000","last_change":"2024-03-30T03:59:40.455590+0000","last_active":"2024-03-30T04:01:17.986076+0000","last_peered":"2024-03-30T04:01:17.986076+0000","last_clean":"2024-03-30T04:01:17.986076+0000","last_became_active":"2024-03-30T03:59:40.455285+0000","last_became_peered":"2024-03-30T03:59:40.455285+0000","last_unstale":"2024-03-30T04:01:17.986076+0000","last_undegraded":"2024-03-30T04:01:17.986076+0000","last_fullsized":"2024-03-30T04:01:17.986076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:59:36.721566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988916+0000","last_change":"2024-03-30T03:59:44.467684+0000","last_active":"2024-03-30T04:01:17.988916+0000","last_peered":"2024-03-30T04:01:17.988916+0000","last_clean":"2024-03-30T04:01:17.988916+0000","last_became_active":"2024-03-30T03:59:44.467279+0000","last_became_peered":"2024-03-30T03:59:44.467279+0000","last_unstale":"2024-03-30T04:01:17.988916+0000","last_undegraded":"2024-03-30T04:01:17.988916+0000","last_fullsized":"2024-03-30T04:01:17.988916+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:52:29.319627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038734+0000","last_change":"2024-03-30T03:59:39.924613+0000","last_active":"2024-03-30T04:01:18.038734+0000","last_peered":"2024-03-30T04:01:18.038734+0000","last_clean":"2024-03-30T04:01:18.038734+0000","last_became_active":"2024-03-30T03:59:39.924268+0000","last_became_peered":"2024-03-30T03:59:39.924268+0000","last_unstale":"2024-03-30T04:01:18.038734+0000","last_undegraded":"2024-03-30T04:01:18.038734+0000","last_fullsized":"2024-03-30T04:01:18.038734+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:19:28.543969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990838+0000","last_change":"2024-03-30T03:59:40.466843+0000","last_active":"2024-03-30T04:01:17.990838+0000","last_peered":"2024-03-30T04:01:17.990838+0000","last_clean":"2024-03-30T04:01:17.990838+0000","last_became_active":"2024-03-30T03:59:40.466475+0000","last_became_peered":"2024-03-30T03:59:40.466475+0000","last_unstale":"2024-03-30T04:01:17.990838+0000","last_undegraded":"2024-03-30T04:01:17.990838+0000","last_fullsized":"2024-03-30T04:01:17.990838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:34:11.411336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991514+0000","last_change":"2024-03-30T03:59:44.468923+0000","last_active":"2024-03-30T04:01:17.991514+0000","last_peered":"2024-03-30T04:01:17.991514+0000","last_clean":"2024-03-30T04:01:17.991514+0000","last_became_active":"2024-03-30T03:59:44.468523+0000","last_became_peered":"2024-03-30T03:59:44.468523+0000","last_unstale":"2024-03-30T04:01:17.991514+0000","last_undegraded":"2024-03-30T04:01:17.991514+0000","last_fullsized":"2024-03-30T04:01:17.991514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:12:21.604534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990637+0000","last_change":"2024-03-30T03:59:39.922517+0000","last_active":"2024-03-30T04:01:17.990637+0000","last_peered":"2024-03-30T04:01:17.990637+0000","last_clean":"2024-03-30T04:01:17.990637+0000","last_became_active":"2024-03-30T03:59:39.922153+0000","last_became_peered":"2024-03-30T03:59:39.922153+0000","last_unstale":"2024-03-30T04:01:17.990637+0000","last_undegraded":"2024-03-30T04:01:17.990637+0000","last_fullsized":"2024-03-30T04:01:17.990637+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:48:28.260222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991305+0000","last_change":"2024-03-30T03:59:40.472845+0000","last_active":"2024-03-30T04:01:17.991305+0000","last_peered":"2024-03-30T04:01:17.991305+0000","last_clean":"2024-03-30T04:01:17.991305+0000","last_became_active":"2024-03-30T03:59:40.469121+0000","last_became_peered":"2024-03-30T03:59:40.469121+0000","last_unstale":"2024-03-30T04:01:17.991305+0000","last_undegraded":"2024-03-30T04:01:17.991305+0000","last_fullsized":"2024-03-30T04:01:17.991305+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:28:04.886074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986726+0000","last_change":"2024-03-30T03:59:44.474402+0000","last_active":"2024-03-30T04:01:17.986726+0000","last_peered":"2024-03-30T04:01:17.986726+0000","last_clean":"2024-03-30T04:01:17.986726+0000","last_became_active":"2024-03-30T03:59:44.473910+0000","last_became_peered":"2024-03-30T03:59:44.473910+0000","last_unstale":"2024-03-30T04:01:17.986726+0000","last_undegraded":"2024-03-30T04:01:17.986726+0000","last_fullsized":"2024-03-30T04:01:17.986726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:33:45.721430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990017+0000","last_change":"2024-03-30T03:59:39.927780+0000","last_active":"2024-03-30T04:01:17.990017+0000","last_peered":"2024-03-30T04:01:17.990017+0000","last_clean":"2024-03-30T04:01:17.990017+0000","last_became_active":"2024-03-30T03:59:39.927087+0000","last_became_peered":"2024-03-30T03:59:39.927087+0000","last_unstale":"2024-03-30T04:01:17.990017+0000","last_undegraded":"2024-03-30T04:01:17.990017+0000","last_fullsized":"2024-03-30T04:01:17.990017+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:34:34.972098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992437+0000","last_change":"2024-03-30T03:59:40.461391+0000","last_active":"2024-03-30T04:01:17.992437+0000","last_peered":"2024-03-30T04:01:17.992437+0000","last_clean":"2024-03-30T04:01:17.992437+0000","last_became_active":"2024-03-30T03:59:40.460930+0000","last_became_peered":"2024-03-30T03:59:40.460930+0000","last_unstale":"2024-03-30T04:01:17.992437+0000","last_undegraded":"2024-03-30T04:01:17.992437+0000","last_fullsized":"2024-03-30T04:01:17.992437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:42:36.379009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993426+0000","last_change":"2024-03-30T03:59:44.478274+0000","last_active":"2024-03-30T04:01:17.993426+0000","last_peered":"2024-03-30T04:01:17.993426+0000","last_clean":"2024-03-30T04:01:17.993426+0000","last_became_active":"2024-03-30T03:59:44.472875+0000","last_became_peered":"2024-03-30T03:59:44.472875+0000","last_unstale":"2024-03-30T04:01:17.993426+0000","last_undegraded":"2024-03-30T04:01:17.993426+0000","last_fullsized":"2024-03-30T04:01:17.993426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:19:52.246789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":200,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.070807+0000","last_change":"2024-03-30T03:59:39.933390+0000","last_active":"2024-03-30T04:01:39.070807+0000","last_peered":"2024-03-30T04:01:39.070807+0000","last_clean":"2024-03-30T04:01:39.070807+0000","last_became_active":"2024-03-30T03:59:39.932865+0000","last_became_peered":"2024-03-30T03:59:39.932865+0000","last_unstale":"2024-03-30T04:01:39.070807+0000","last_undegraded":"2024-03-30T04:01:39.070807+0000","last_fullsized":"2024-03-30T04:01:39.070807+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:41:02.501465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":175,"num_read_kb":175,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990158+0000","last_change":"2024-03-30T03:59:40.458917+0000","last_active":"2024-03-30T04:01:17.990158+0000","last_peered":"2024-03-30T04:01:17.990158+0000","last_clean":"2024-03-30T04:01:17.990158+0000","last_became_active":"2024-03-30T03:59:40.458617+0000","last_became_peered":"2024-03-30T03:59:40.458617+0000","last_unstale":"2024-03-30T04:01:17.990158+0000","last_undegraded":"2024-03-30T04:01:17.990158+0000","last_fullsized":"2024-03-30T04:01:17.990158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:25:44.811909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990202+0000","last_change":"2024-03-30T03:59:44.466314+0000","last_active":"2024-03-30T04:01:17.990202+0000","last_peered":"2024-03-30T04:01:17.990202+0000","last_clean":"2024-03-30T04:01:17.990202+0000","last_became_active":"2024-03-30T03:59:44.464171+0000","last_became_peered":"2024-03-30T03:59:44.464171+0000","last_unstale":"2024-03-30T04:01:17.990202+0000","last_undegraded":"2024-03-30T04:01:17.990202+0000","last_fullsized":"2024-03-30T04:01:17.990202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:05:49.519621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986623+0000","last_change":"2024-03-30T03:59:39.926947+0000","last_active":"2024-03-30T04:01:17.986623+0000","last_peered":"2024-03-30T04:01:17.986623+0000","last_clean":"2024-03-30T04:01:17.986623+0000","last_became_active":"2024-03-30T03:59:39.925872+0000","last_became_peered":"2024-03-30T03:59:39.925872+0000","last_unstale":"2024-03-30T04:01:17.986623+0000","last_undegraded":"2024-03-30T04:01:17.986623+0000","last_fullsized":"2024-03-30T04:01:17.986623+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:13:38.659046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"86'9","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.000506+0000","last_change":"2024-03-30T04:01:19.000506+0000","last_active":"2024-03-30T04:01:19.000506+0000","last_peered":"2024-03-30T04:01:19.000506+0000","last_clean":"2024-03-30T04:01:19.000506+0000","last_became_active":"2024-03-30T04:01:18.999323+0000","last_became_peered":"2024-03-30T04:01:18.999323+0000","last_unstale":"2024-03-30T04:01:19.000506+0000","last_undegraded":"2024-03-30T04:01:19.000506+0000","last_fullsized":"2024-03-30T04:01:19.000506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:05:01.500866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,8],"acting":[0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990438+0000","last_change":"2024-03-30T03:59:44.461992+0000","last_active":"2024-03-30T04:01:17.990438+0000","last_peered":"2024-03-30T04:01:17.990438+0000","last_clean":"2024-03-30T04:01:17.990438+0000","last_became_active":"2024-03-30T03:59:44.461525+0000","last_became_peered":"2024-03-30T03:59:44.461525+0000","last_unstale":"2024-03-30T04:01:17.990438+0000","last_undegraded":"2024-03-30T04:01:17.990438+0000","last_fullsized":"2024-03-30T04:01:17.990438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:06:55.259192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989996+0000","last_change":"2024-03-30T03:59:39.930942+0000","last_active":"2024-03-30T04:01:17.989996+0000","last_peered":"2024-03-30T04:01:17.989996+0000","last_clean":"2024-03-30T04:01:17.989996+0000","last_became_active":"2024-03-30T03:59:39.930627+0000","last_became_peered":"2024-03-30T03:59:39.930627+0000","last_unstale":"2024-03-30T04:01:17.989996+0000","last_undegraded":"2024-03-30T04:01:17.989996+0000","last_fullsized":"2024-03-30T04:01:17.989996+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:32:41.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990732+0000","last_change":"2024-03-30T03:59:40.456344+0000","last_active":"2024-03-30T04:01:17.990732+0000","last_peered":"2024-03-30T04:01:17.990732+0000","last_clean":"2024-03-30T04:01:17.990732+0000","last_became_active":"2024-03-30T03:59:40.455492+0000","last_became_peered":"2024-03-30T03:59:40.455492+0000","last_unstale":"2024-03-30T04:01:17.990732+0000","last_undegraded":"2024-03-30T04:01:17.990732+0000","last_fullsized":"2024-03-30T04:01:17.990732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:44:37.971131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990391+0000","last_change":"2024-03-30T03:59:44.465686+0000","last_active":"2024-03-30T04:01:17.990391+0000","last_peered":"2024-03-30T04:01:17.990391+0000","last_clean":"2024-03-30T04:01:17.990391+0000","last_became_active":"2024-03-30T03:59:44.464764+0000","last_became_peered":"2024-03-30T03:59:44.464764+0000","last_unstale":"2024-03-30T04:01:17.990391+0000","last_undegraded":"2024-03-30T04:01:17.990391+0000","last_fullsized":"2024-03-30T04:01:17.990391+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:43:52.580139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"88'41","reported_seq":251,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.071776+0000","last_change":"2024-03-30T03:59:39.927202+0000","last_active":"2024-03-30T04:01:39.071776+0000","last_peered":"2024-03-30T04:01:39.071776+0000","last_clean":"2024-03-30T04:01:39.071776+0000","last_became_active":"2024-03-30T03:59:39.926140+0000","last_became_peered":"2024-03-30T03:59:39.926140+0000","last_unstale":"2024-03-30T04:01:39.071776+0000","last_undegraded":"2024-03-30T04:01:39.071776+0000","last_fullsized":"2024-03-30T04:01:39.071776+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:57:53.388712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70292,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":190,"num_read_kb":165,"num_write":43,"num_write_kb":89,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986450+0000","last_change":"2024-03-30T03:59:40.454140+0000","last_active":"2024-03-30T04:01:17.986450+0000","last_peered":"2024-03-30T04:01:17.986450+0000","last_clean":"2024-03-30T04:01:17.986450+0000","last_became_active":"2024-03-30T03:59:40.452762+0000","last_became_peered":"2024-03-30T03:59:40.452762+0000","last_unstale":"2024-03-30T04:01:17.986450+0000","last_undegraded":"2024-03-30T04:01:17.986450+0000","last_fullsized":"2024-03-30T04:01:17.986450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:08:54.502636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990463+0000","last_change":"2024-03-30T03:59:44.477961+0000","last_active":"2024-03-30T04:01:17.990463+0000","last_peered":"2024-03-30T04:01:17.990463+0000","last_clean":"2024-03-30T04:01:17.990463+0000","last_became_active":"2024-03-30T03:59:44.477531+0000","last_became_peered":"2024-03-30T03:59:44.477531+0000","last_unstale":"2024-03-30T04:01:17.990463+0000","last_undegraded":"2024-03-30T04:01:17.990463+0000","last_fullsized":"2024-03-30T04:01:17.990463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:37.908415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'9","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988801+0000","last_change":"2024-03-30T03:59:39.929562+0000","last_active":"2024-03-30T04:01:17.988801+0000","last_peered":"2024-03-30T04:01:17.988801+0000","last_clean":"2024-03-30T04:01:17.988801+0000","last_became_active":"2024-03-30T03:59:39.929137+0000","last_became_peered":"2024-03-30T03:59:39.929137+0000","last_unstale":"2024-03-30T04:01:17.988801+0000","last_undegraded":"2024-03-30T04:01:17.988801+0000","last_fullsized":"2024-03-30T04:01:17.988801+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:28:09.380073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991071+0000","last_change":"2024-03-30T03:59:40.476865+0000","last_active":"2024-03-30T04:01:17.991071+0000","last_peered":"2024-03-30T04:01:17.991071+0000","last_clean":"2024-03-30T04:01:17.991071+0000","last_became_active":"2024-03-30T03:59:40.476405+0000","last_became_peered":"2024-03-30T03:59:40.476405+0000","last_unstale":"2024-03-30T04:01:17.991071+0000","last_undegraded":"2024-03-30T04:01:17.991071+0000","last_fullsized":"2024-03-30T04:01:17.991071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:43:04.923097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990443+0000","last_change":"2024-03-30T03:59:44.471226+0000","last_active":"2024-03-30T04:01:17.990443+0000","last_peered":"2024-03-30T04:01:17.990443+0000","last_clean":"2024-03-30T04:01:17.990443+0000","last_became_active":"2024-03-30T03:59:44.470798+0000","last_became_peered":"2024-03-30T03:59:44.470798+0000","last_unstale":"2024-03-30T04:01:17.990443+0000","last_undegraded":"2024-03-30T04:01:17.990443+0000","last_fullsized":"2024-03-30T04:01:17.990443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:43:24.591676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"84'5","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990549+0000","last_change":"2024-03-30T03:59:39.927380+0000","last_active":"2024-03-30T04:01:17.990549+0000","last_peered":"2024-03-30T04:01:17.990549+0000","last_clean":"2024-03-30T04:01:17.990549+0000","last_became_active":"2024-03-30T03:59:39.927037+0000","last_became_peered":"2024-03-30T03:59:39.927037+0000","last_unstale":"2024-03-30T04:01:17.990549+0000","last_undegraded":"2024-03-30T04:01:17.990549+0000","last_fullsized":"2024-03-30T04:01:17.990549+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:54:31.359738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986863+0000","last_change":"2024-03-30T03:59:40.458728+0000","last_active":"2024-03-30T04:01:17.986863+0000","last_peered":"2024-03-30T04:01:17.986863+0000","last_clean":"2024-03-30T04:01:17.986863+0000","last_became_active":"2024-03-30T03:59:40.458314+0000","last_became_peered":"2024-03-30T03:59:40.458314+0000","last_unstale":"2024-03-30T04:01:17.986863+0000","last_undegraded":"2024-03-30T04:01:17.986863+0000","last_fullsized":"2024-03-30T04:01:17.986863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:13:28.386806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.039141+0000","last_change":"2024-03-30T04:00:18.020869+0000","last_active":"2024-03-30T04:01:18.039141+0000","last_peered":"2024-03-30T04:01:18.039141+0000","last_clean":"2024-03-30T04:01:18.039141+0000","last_became_active":"2024-03-30T04:00:18.020410+0000","last_became_peered":"2024-03-30T04:00:18.020410+0000","last_unstale":"2024-03-30T04:01:18.039141+0000","last_undegraded":"2024-03-30T04:01:18.039141+0000","last_fullsized":"2024-03-30T04:01:18.039141+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:19:53.675022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0,3],"acting":[10,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.11","version":"86'7","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:39.098316+0000","last_change":"2024-03-30T03:59:39.924505+0000","last_active":"2024-03-30T04:01:39.098316+0000","last_peered":"2024-03-30T04:01:39.098316+0000","last_clean":"2024-03-30T04:01:39.098316+0000","last_became_active":"2024-03-30T03:59:39.924140+0000","last_became_peered":"2024-03-30T03:59:39.924140+0000","last_unstale":"2024-03-30T04:01:39.098316+0000","last_undegraded":"2024-03-30T04:01:39.098316+0000","last_fullsized":"2024-03-30T04:01:39.098316+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:04:32.372206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1074,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986379+0000","last_change":"2024-03-30T03:59:40.459914+0000","last_active":"2024-03-30T04:01:17.986379+0000","last_peered":"2024-03-30T04:01:17.986379+0000","last_clean":"2024-03-30T04:01:17.986379+0000","last_became_active":"2024-03-30T03:59:40.458905+0000","last_became_peered":"2024-03-30T03:59:40.458905+0000","last_unstale":"2024-03-30T04:01:17.986379+0000","last_undegraded":"2024-03-30T04:01:17.986379+0000","last_fullsized":"2024-03-30T04:01:17.986379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:48:28.653039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992090+0000","last_change":"2024-03-30T04:00:18.288237+0000","last_active":"2024-03-30T04:01:17.992090+0000","last_peered":"2024-03-30T04:01:17.992090+0000","last_clean":"2024-03-30T04:01:17.992090+0000","last_became_active":"2024-03-30T04:00:18.287748+0000","last_became_peered":"2024-03-30T04:00:18.287748+0000","last_unstale":"2024-03-30T04:01:17.992090+0000","last_undegraded":"2024-03-30T04:01:17.992090+0000","last_fullsized":"2024-03-30T04:01:17.992090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:22:50.893522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'17","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985742+0000","last_change":"2024-03-30T03:59:39.931742+0000","last_active":"2024-03-30T04:01:17.985742+0000","last_peered":"2024-03-30T04:01:17.985742+0000","last_clean":"2024-03-30T04:01:17.985742+0000","last_became_active":"2024-03-30T03:59:39.931406+0000","last_became_peered":"2024-03-30T03:59:39.931406+0000","last_unstale":"2024-03-30T04:01:17.985742+0000","last_undegraded":"2024-03-30T04:01:17.985742+0000","last_fullsized":"2024-03-30T04:01:17.985742+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:45:07.042930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991765+0000","last_change":"2024-03-30T03:59:40.460235+0000","last_active":"2024-03-30T04:01:17.991765+0000","last_peered":"2024-03-30T04:01:17.991765+0000","last_clean":"2024-03-30T04:01:17.991765+0000","last_became_active":"2024-03-30T03:59:40.459824+0000","last_became_peered":"2024-03-30T03:59:40.459824+0000","last_unstale":"2024-03-30T04:01:17.991765+0000","last_undegraded":"2024-03-30T04:01:17.991765+0000","last_fullsized":"2024-03-30T04:01:17.991765+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:15:28.957449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991661+0000","last_change":"2024-03-30T03:59:44.472126+0000","last_active":"2024-03-30T04:01:17.991661+0000","last_peered":"2024-03-30T04:01:17.991661+0000","last_clean":"2024-03-30T04:01:17.991661+0000","last_became_active":"2024-03-30T03:59:44.471725+0000","last_became_peered":"2024-03-30T03:59:44.471725+0000","last_unstale":"2024-03-30T04:01:17.991661+0000","last_undegraded":"2024-03-30T04:01:17.991661+0000","last_fullsized":"2024-03-30T04:01:17.991661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:56.174696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'2","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986615+0000","last_change":"2024-03-30T03:59:39.926888+0000","last_active":"2024-03-30T04:01:17.986615+0000","last_peered":"2024-03-30T04:01:17.986615+0000","last_clean":"2024-03-30T04:01:17.986615+0000","last_became_active":"2024-03-30T03:59:39.926561+0000","last_became_peered":"2024-03-30T03:59:39.926561+0000","last_unstale":"2024-03-30T04:01:17.986615+0000","last_undegraded":"2024-03-30T04:01:17.986615+0000","last_fullsized":"2024-03-30T04:01:17.986615+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:23:03.859478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990240+0000","last_change":"2024-03-30T03:59:40.464886+0000","last_active":"2024-03-30T04:01:17.990240+0000","last_peered":"2024-03-30T04:01:17.990240+0000","last_clean":"2024-03-30T04:01:17.990240+0000","last_became_active":"2024-03-30T03:59:40.464419+0000","last_became_peered":"2024-03-30T03:59:40.464419+0000","last_unstale":"2024-03-30T04:01:17.990240+0000","last_undegraded":"2024-03-30T04:01:17.990240+0000","last_fullsized":"2024-03-30T04:01:17.990240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:30:59.319576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986221+0000","last_change":"2024-03-30T03:59:44.598809+0000","last_active":"2024-03-30T04:01:17.986221+0000","last_peered":"2024-03-30T04:01:17.986221+0000","last_clean":"2024-03-30T04:01:17.986221+0000","last_became_active":"2024-03-30T03:59:44.598331+0000","last_became_peered":"2024-03-30T03:59:44.598331+0000","last_unstale":"2024-03-30T04:01:17.986221+0000","last_undegraded":"2024-03-30T04:01:17.986221+0000","last_fullsized":"2024-03-30T04:01:17.986221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:20:16.015577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"84'1","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992002+0000","last_change":"2024-03-30T03:59:39.938703+0000","last_active":"2024-03-30T04:01:17.992002+0000","last_peered":"2024-03-30T04:01:17.992002+0000","last_clean":"2024-03-30T04:01:17.992002+0000","last_became_active":"2024-03-30T03:59:39.938233+0000","last_became_peered":"2024-03-30T03:59:39.938233+0000","last_unstale":"2024-03-30T04:01:17.992002+0000","last_undegraded":"2024-03-30T04:01:17.992002+0000","last_fullsized":"2024-03-30T04:01:17.992002+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:40.546637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038965+0000","last_change":"2024-03-30T03:59:40.467313+0000","last_active":"2024-03-30T04:01:18.038965+0000","last_peered":"2024-03-30T04:01:18.038965+0000","last_clean":"2024-03-30T04:01:18.038965+0000","last_became_active":"2024-03-30T03:59:40.466958+0000","last_became_peered":"2024-03-30T03:59:40.466958+0000","last_unstale":"2024-03-30T04:01:18.038965+0000","last_undegraded":"2024-03-30T04:01:18.038965+0000","last_fullsized":"2024-03-30T04:01:18.038965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:13:50.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991065+0000","last_change":"2024-03-30T03:59:44.474356+0000","last_active":"2024-03-30T04:01:17.991065+0000","last_peered":"2024-03-30T04:01:17.991065+0000","last_clean":"2024-03-30T04:01:17.991065+0000","last_became_active":"2024-03-30T03:59:44.473582+0000","last_became_peered":"2024-03-30T03:59:44.473582+0000","last_unstale":"2024-03-30T04:01:17.991065+0000","last_undegraded":"2024-03-30T04:01:17.991065+0000","last_fullsized":"2024-03-30T04:01:17.991065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:09.979460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"88'9","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:36.090477+0000","last_change":"2024-03-30T03:59:39.933082+0000","last_active":"2024-03-30T04:01:36.090477+0000","last_peered":"2024-03-30T04:01:36.090477+0000","last_clean":"2024-03-30T04:01:36.090477+0000","last_became_active":"2024-03-30T03:59:39.932769+0000","last_became_peered":"2024-03-30T03:59:39.932769+0000","last_unstale":"2024-03-30T04:01:36.090477+0000","last_undegraded":"2024-03-30T04:01:36.090477+0000","last_fullsized":"2024-03-30T04:01:36.090477+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:58:24.626248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1773,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"86'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.038870+0000","last_change":"2024-03-30T03:59:40.460771+0000","last_active":"2024-03-30T04:01:18.038870+0000","last_peered":"2024-03-30T04:01:18.038870+0000","last_clean":"2024-03-30T04:01:18.038870+0000","last_became_active":"2024-03-30T03:59:40.460437+0000","last_became_peered":"2024-03-30T03:59:40.460437+0000","last_unstale":"2024-03-30T04:01:18.038870+0000","last_undegraded":"2024-03-30T04:01:18.038870+0000","last_fullsized":"2024-03-30T04:01:18.038870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:56:15.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992237+0000","last_change":"2024-03-30T03:59:44.598259+0000","last_active":"2024-03-30T04:01:17.992237+0000","last_peered":"2024-03-30T04:01:17.992237+0000","last_clean":"2024-03-30T04:01:17.992237+0000","last_became_active":"2024-03-30T03:59:44.597676+0000","last_became_peered":"2024-03-30T03:59:44.597676+0000","last_unstale":"2024-03-30T04:01:17.992237+0000","last_undegraded":"2024-03-30T04:01:17.992237+0000","last_fullsized":"2024-03-30T04:01:17.992237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:48:07.516134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"86'8","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990251+0000","last_change":"2024-03-30T04:00:18.018515+0000","last_active":"2024-03-30T04:01:17.990251+0000","last_peered":"2024-03-30T04:01:17.990251+0000","last_clean":"2024-03-30T04:01:17.990251+0000","last_became_active":"2024-03-30T04:00:18.017823+0000","last_became_peered":"2024-03-30T04:00:18.017823+0000","last_unstale":"2024-03-30T04:01:17.990251+0000","last_undegraded":"2024-03-30T04:01:17.990251+0000","last_fullsized":"2024-03-30T04:01:17.990251+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:15:54.957524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'6","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:37.094872+0000","last_change":"2024-03-30T03:59:39.932918+0000","last_active":"2024-03-30T04:01:37.094872+0000","last_peered":"2024-03-30T04:01:37.094872+0000","last_clean":"2024-03-30T04:01:37.094872+0000","last_became_active":"2024-03-30T03:59:39.930409+0000","last_became_peered":"2024-03-30T03:59:39.930409+0000","last_unstale":"2024-03-30T04:01:37.094872+0000","last_undegraded":"2024-03-30T04:01:37.094872+0000","last_fullsized":"2024-03-30T04:01:37.094872+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:14:21.201015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986577+0000","last_change":"2024-03-30T03:59:44.467289+0000","last_active":"2024-03-30T04:01:17.986577+0000","last_peered":"2024-03-30T04:01:17.986577+0000","last_clean":"2024-03-30T04:01:17.986577+0000","last_became_active":"2024-03-30T03:59:44.466859+0000","last_became_peered":"2024-03-30T03:59:44.466859+0000","last_unstale":"2024-03-30T04:01:17.986577+0000","last_undegraded":"2024-03-30T04:01:17.986577+0000","last_fullsized":"2024-03-30T04:01:17.986577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:37:49.871403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992703+0000","last_change":"2024-03-30T03:59:40.475928+0000","last_active":"2024-03-30T04:01:17.992703+0000","last_peered":"2024-03-30T04:01:17.992703+0000","last_clean":"2024-03-30T04:01:17.992703+0000","last_became_active":"2024-03-30T03:59:40.473691+0000","last_became_peered":"2024-03-30T03:59:40.473691+0000","last_unstale":"2024-03-30T04:01:17.992703+0000","last_undegraded":"2024-03-30T04:01:17.992703+0000","last_fullsized":"2024-03-30T04:01:17.992703+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:08:19.149511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"84'6","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990294+0000","last_change":"2024-03-30T03:59:39.924003+0000","last_active":"2024-03-30T04:01:17.990294+0000","last_peered":"2024-03-30T04:01:17.990294+0000","last_clean":"2024-03-30T04:01:17.990294+0000","last_became_active":"2024-03-30T03:59:39.923637+0000","last_became_peered":"2024-03-30T03:59:39.923637+0000","last_unstale":"2024-03-30T04:01:17.990294+0000","last_undegraded":"2024-03-30T04:01:17.990294+0000","last_fullsized":"2024-03-30T04:01:17.990294+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T12:47:03.127780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":339,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986260+0000","last_change":"2024-03-30T03:58:46.925072+0000","last_active":"2024-03-30T04:01:17.986260+0000","last_peered":"2024-03-30T04:01:17.986260+0000","last_clean":"2024-03-30T04:01:17.986260+0000","last_became_active":"2024-03-30T03:58:46.924609+0000","last_became_peered":"2024-03-30T03:58:46.924609+0000","last_unstale":"2024-03-30T04:01:17.986260+0000","last_undegraded":"2024-03-30T04:01:17.986260+0000","last_fullsized":"2024-03-30T04:01:17.986260+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:56:13.091287+0000","last_clean_scrub_stamp":"2024-03-30T03:56:13.091287+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:23:00.880049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986543+0000","last_change":"2024-03-30T04:00:18.017573+0000","last_active":"2024-03-30T04:01:17.986543+0000","last_peered":"2024-03-30T04:01:17.986543+0000","last_clean":"2024-03-30T04:01:17.986543+0000","last_became_active":"2024-03-30T04:00:18.016937+0000","last_became_peered":"2024-03-30T04:00:18.016937+0000","last_unstale":"2024-03-30T04:01:17.986543+0000","last_undegraded":"2024-03-30T04:01:17.986543+0000","last_fullsized":"2024-03-30T04:01:17.986543+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:26:20.863180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990431+0000","last_change":"2024-03-30T03:59:40.457572+0000","last_active":"2024-03-30T04:01:17.990431+0000","last_peered":"2024-03-30T04:01:17.990431+0000","last_clean":"2024-03-30T04:01:17.990431+0000","last_became_active":"2024-03-30T03:59:40.457291+0000","last_became_peered":"2024-03-30T03:59:40.457291+0000","last_unstale":"2024-03-30T04:01:17.990431+0000","last_undegraded":"2024-03-30T04:01:17.990431+0000","last_fullsized":"2024-03-30T04:01:17.990431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:24:59.808566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"84'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986318+0000","last_change":"2024-03-30T03:59:39.931881+0000","last_active":"2024-03-30T04:01:17.986318+0000","last_peered":"2024-03-30T04:01:17.986318+0000","last_clean":"2024-03-30T04:01:17.986318+0000","last_became_active":"2024-03-30T03:59:39.931598+0000","last_became_peered":"2024-03-30T03:59:39.931598+0000","last_unstale":"2024-03-30T04:01:17.986318+0000","last_undegraded":"2024-03-30T04:01:17.986318+0000","last_fullsized":"2024-03-30T04:01:17.986318+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:46:49.109483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991339+0000","last_change":"2024-03-30T03:59:44.470919+0000","last_active":"2024-03-30T04:01:17.991339+0000","last_peered":"2024-03-30T04:01:17.991339+0000","last_clean":"2024-03-30T04:01:17.991339+0000","last_became_active":"2024-03-30T03:59:44.470444+0000","last_became_peered":"2024-03-30T03:59:44.470444+0000","last_unstale":"2024-03-30T04:01:17.991339+0000","last_undegraded":"2024-03-30T04:01:17.991339+0000","last_fullsized":"2024-03-30T04:01:17.991339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:39:09.890034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986517+0000","last_change":"2024-03-30T03:59:39.937156+0000","last_active":"2024-03-30T04:01:17.986517+0000","last_peered":"2024-03-30T04:01:17.986517+0000","last_clean":"2024-03-30T04:01:17.986517+0000","last_became_active":"2024-03-30T03:59:39.936855+0000","last_became_peered":"2024-03-30T03:59:39.936855+0000","last_unstale":"2024-03-30T04:01:17.986517+0000","last_undegraded":"2024-03-30T04:01:17.986517+0000","last_fullsized":"2024-03-30T04:01:17.986517+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:02:37.548426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991400+0000","last_change":"2024-03-30T03:59:40.459031+0000","last_active":"2024-03-30T04:01:17.991400+0000","last_peered":"2024-03-30T04:01:17.991400+0000","last_clean":"2024-03-30T04:01:17.991400+0000","last_became_active":"2024-03-30T03:59:40.458615+0000","last_became_peered":"2024-03-30T03:59:40.458615+0000","last_unstale":"2024-03-30T04:01:17.991400+0000","last_undegraded":"2024-03-30T04:01:17.991400+0000","last_fullsized":"2024-03-30T04:01:17.991400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:19:40.987546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991296+0000","last_change":"2024-03-30T03:59:44.599870+0000","last_active":"2024-03-30T04:01:17.991296+0000","last_peered":"2024-03-30T04:01:17.991296+0000","last_clean":"2024-03-30T04:01:17.991296+0000","last_became_active":"2024-03-30T03:59:44.598400+0000","last_became_peered":"2024-03-30T03:59:44.598400+0000","last_unstale":"2024-03-30T04:01:17.991296+0000","last_undegraded":"2024-03-30T04:01:17.991296+0000","last_fullsized":"2024-03-30T04:01:17.991296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:55:18.137494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'11","reported_seq":50,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.992945+0000","last_change":"2024-03-30T04:00:18.297312+0000","last_active":"2024-03-30T04:01:18.992945+0000","last_peered":"2024-03-30T04:01:18.992945+0000","last_clean":"2024-03-30T04:01:18.992945+0000","last_became_active":"2024-03-30T04:00:18.288640+0000","last_became_peered":"2024-03-30T04:00:18.288640+0000","last_unstale":"2024-03-30T04:01:18.992945+0000","last_undegraded":"2024-03-30T04:01:18.992945+0000","last_fullsized":"2024-03-30T04:01:18.992945+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:44:21.664643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992944+0000","last_change":"2024-03-30T03:59:40.455289+0000","last_active":"2024-03-30T04:01:17.992944+0000","last_peered":"2024-03-30T04:01:17.992944+0000","last_clean":"2024-03-30T04:01:17.992944+0000","last_became_active":"2024-03-30T03:59:40.454051+0000","last_became_peered":"2024-03-30T03:59:40.454051+0000","last_unstale":"2024-03-30T04:01:17.992944+0000","last_undegraded":"2024-03-30T04:01:17.992944+0000","last_fullsized":"2024-03-30T04:01:17.992944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:17:53.318310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989266+0000","last_change":"2024-03-30T03:59:44.458211+0000","last_active":"2024-03-30T04:01:17.989266+0000","last_peered":"2024-03-30T04:01:17.989266+0000","last_clean":"2024-03-30T04:01:17.989266+0000","last_became_active":"2024-03-30T03:59:44.457773+0000","last_became_peered":"2024-03-30T03:59:44.457773+0000","last_unstale":"2024-03-30T04:01:17.989266+0000","last_undegraded":"2024-03-30T04:01:17.989266+0000","last_fullsized":"2024-03-30T04:01:17.989266+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T15:36:53.544796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'7","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990084+0000","last_change":"2024-03-30T03:59:39.926628+0000","last_active":"2024-03-30T04:01:17.990084+0000","last_peered":"2024-03-30T04:01:17.990084+0000","last_clean":"2024-03-30T04:01:17.990084+0000","last_became_active":"2024-03-30T03:59:39.926326+0000","last_became_peered":"2024-03-30T03:59:39.926326+0000","last_unstale":"2024-03-30T04:01:17.990084+0000","last_undegraded":"2024-03-30T04:01:17.990084+0000","last_fullsized":"2024-03-30T04:01:17.990084+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:59:20.790110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.005003+0000","last_change":"2024-03-30T04:01:19.005003+0000","last_active":"2024-03-30T04:01:19.005003+0000","last_peered":"2024-03-30T04:01:19.005003+0000","last_clean":"2024-03-30T04:01:19.005003+0000","last_became_active":"2024-03-30T04:01:19.004334+0000","last_became_peered":"2024-03-30T04:01:19.004334+0000","last_unstale":"2024-03-30T04:01:19.005003+0000","last_undegraded":"2024-03-30T04:01:19.005003+0000","last_fullsized":"2024-03-30T04:01:19.005003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:41:54.609191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990019+0000","last_change":"2024-03-30T03:59:44.599264+0000","last_active":"2024-03-30T04:01:17.990019+0000","last_peered":"2024-03-30T04:01:17.990019+0000","last_clean":"2024-03-30T04:01:17.990019+0000","last_became_active":"2024-03-30T03:59:44.598854+0000","last_became_peered":"2024-03-30T03:59:44.598854+0000","last_unstale":"2024-03-30T04:01:17.990019+0000","last_undegraded":"2024-03-30T04:01:17.990019+0000","last_fullsized":"2024-03-30T04:01:17.990019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:27:30.059481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"84'2","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989613+0000","last_change":"2024-03-30T04:00:18.291229+0000","last_active":"2024-03-30T04:01:17.989613+0000","last_peered":"2024-03-30T04:01:17.989613+0000","last_clean":"2024-03-30T04:01:17.989613+0000","last_became_active":"2024-03-30T04:00:18.287954+0000","last_became_peered":"2024-03-30T04:00:18.287954+0000","last_unstale":"2024-03-30T04:01:17.989613+0000","last_undegraded":"2024-03-30T04:01:17.989613+0000","last_fullsized":"2024-03-30T04:01:17.989613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:54.678907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985778+0000","last_change":"2024-03-30T03:59:40.457615+0000","last_active":"2024-03-30T04:01:17.985778+0000","last_peered":"2024-03-30T04:01:17.985778+0000","last_clean":"2024-03-30T04:01:17.985778+0000","last_became_active":"2024-03-30T03:59:40.457358+0000","last_became_peered":"2024-03-30T03:59:40.457358+0000","last_unstale":"2024-03-30T04:01:17.985778+0000","last_undegraded":"2024-03-30T04:01:17.985778+0000","last_fullsized":"2024-03-30T04:01:17.985778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T06:20:35.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993013+0000","last_change":"2024-03-30T03:59:44.479292+0000","last_active":"2024-03-30T04:01:17.993013+0000","last_peered":"2024-03-30T04:01:17.993013+0000","last_clean":"2024-03-30T04:01:17.993013+0000","last_became_active":"2024-03-30T03:59:44.477969+0000","last_became_peered":"2024-03-30T03:59:44.477969+0000","last_unstale":"2024-03-30T04:01:17.993013+0000","last_undegraded":"2024-03-30T04:01:17.993013+0000","last_fullsized":"2024-03-30T04:01:17.993013+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:52:05.242528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"84'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991372+0000","last_change":"2024-03-30T03:59:39.926434+0000","last_active":"2024-03-30T04:01:17.991372+0000","last_peered":"2024-03-30T04:01:17.991372+0000","last_clean":"2024-03-30T04:01:17.991372+0000","last_became_active":"2024-03-30T03:59:39.926014+0000","last_became_peered":"2024-03-30T03:59:39.926014+0000","last_unstale":"2024-03-30T04:01:17.991372+0000","last_undegraded":"2024-03-30T04:01:17.991372+0000","last_fullsized":"2024-03-30T04:01:17.991372+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:21:51.623307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990785+0000","last_change":"2024-03-30T03:59:40.463807+0000","last_active":"2024-03-30T04:01:17.990785+0000","last_peered":"2024-03-30T04:01:17.990785+0000","last_clean":"2024-03-30T04:01:17.990785+0000","last_became_active":"2024-03-30T03:59:40.457938+0000","last_became_peered":"2024-03-30T03:59:40.457938+0000","last_unstale":"2024-03-30T04:01:17.990785+0000","last_undegraded":"2024-03-30T04:01:17.990785+0000","last_fullsized":"2024-03-30T04:01:17.990785+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:42:14.225273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"84'1","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990718+0000","last_change":"2024-03-30T03:59:44.460408+0000","last_active":"2024-03-30T04:01:17.990718+0000","last_peered":"2024-03-30T04:01:17.990718+0000","last_clean":"2024-03-30T04:01:17.990718+0000","last_became_active":"2024-03-30T03:59:44.459976+0000","last_became_peered":"2024-03-30T03:59:44.459976+0000","last_unstale":"2024-03-30T04:01:17.990718+0000","last_undegraded":"2024-03-30T04:01:17.990718+0000","last_fullsized":"2024-03-30T04:01:17.990718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:59:13.717501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"84'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988611+0000","last_change":"2024-03-30T03:59:39.930775+0000","last_active":"2024-03-30T04:01:17.988611+0000","last_peered":"2024-03-30T04:01:17.988611+0000","last_clean":"2024-03-30T04:01:17.988611+0000","last_became_active":"2024-03-30T03:59:39.930468+0000","last_became_peered":"2024-03-30T03:59:39.930468+0000","last_unstale":"2024-03-30T04:01:17.988611+0000","last_undegraded":"2024-03-30T04:01:17.988611+0000","last_fullsized":"2024-03-30T04:01:17.988611+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:29:57.792055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993400+0000","last_change":"2024-03-30T03:59:40.471308+0000","last_active":"2024-03-30T04:01:17.993400+0000","last_peered":"2024-03-30T04:01:17.993400+0000","last_clean":"2024-03-30T04:01:17.993400+0000","last_became_active":"2024-03-30T03:59:40.468364+0000","last_became_peered":"2024-03-30T03:59:40.468364+0000","last_unstale":"2024-03-30T04:01:17.993400+0000","last_undegraded":"2024-03-30T04:01:17.993400+0000","last_fullsized":"2024-03-30T04:01:17.993400+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:50:39.931375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989641+0000","last_change":"2024-03-30T03:59:44.598146+0000","last_active":"2024-03-30T04:01:17.989641+0000","last_peered":"2024-03-30T04:01:17.989641+0000","last_clean":"2024-03-30T04:01:17.989641+0000","last_became_active":"2024-03-30T03:59:44.597781+0000","last_became_peered":"2024-03-30T03:59:44.597781+0000","last_unstale":"2024-03-30T04:01:17.989641+0000","last_undegraded":"2024-03-30T04:01:17.989641+0000","last_fullsized":"2024-03-30T04:01:17.989641+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:10:14.436512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'4","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.985994+0000","last_change":"2024-03-30T03:59:39.926128+0000","last_active":"2024-03-30T04:01:17.985994+0000","last_peered":"2024-03-30T04:01:17.985994+0000","last_clean":"2024-03-30T04:01:17.985994+0000","last_became_active":"2024-03-30T03:59:39.925806+0000","last_became_peered":"2024-03-30T03:59:39.925806+0000","last_unstale":"2024-03-30T04:01:17.985994+0000","last_undegraded":"2024-03-30T04:01:17.985994+0000","last_fullsized":"2024-03-30T04:01:17.985994+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:58:52.725343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"86'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986683+0000","last_change":"2024-03-30T03:59:40.460434+0000","last_active":"2024-03-30T04:01:17.986683+0000","last_peered":"2024-03-30T04:01:17.986683+0000","last_clean":"2024-03-30T04:01:17.986683+0000","last_became_active":"2024-03-30T03:59:40.460126+0000","last_became_peered":"2024-03-30T03:59:40.460126+0000","last_unstale":"2024-03-30T04:01:17.986683+0000","last_undegraded":"2024-03-30T04:01:17.986683+0000","last_fullsized":"2024-03-30T04:01:17.986683+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T04:37:14.419687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990340+0000","last_change":"2024-03-30T03:59:44.462935+0000","last_active":"2024-03-30T04:01:17.990340+0000","last_peered":"2024-03-30T04:01:17.990340+0000","last_clean":"2024-03-30T04:01:17.990340+0000","last_became_active":"2024-03-30T03:59:44.462460+0000","last_became_peered":"2024-03-30T03:59:44.462460+0000","last_unstale":"2024-03-30T04:01:17.990340+0000","last_undegraded":"2024-03-30T04:01:17.990340+0000","last_fullsized":"2024-03-30T04:01:17.990340+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T11:19:11.411879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"84'5","reported_seq":115,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:40.099535+0000","last_change":"2024-03-30T03:59:39.938481+0000","last_active":"2024-03-30T04:01:40.099535+0000","last_peered":"2024-03-30T04:01:40.099535+0000","last_clean":"2024-03-30T04:01:40.099535+0000","last_became_active":"2024-03-30T03:59:39.938116+0000","last_became_peered":"2024-03-30T03:59:39.938116+0000","last_unstale":"2024-03-30T04:01:40.099535+0000","last_undegraded":"2024-03-30T04:01:40.099535+0000","last_fullsized":"2024-03-30T04:01:40.099535+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:17:58.652974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":90,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:18.999932+0000","last_change":"2024-03-30T04:01:18.999932+0000","last_active":"2024-03-30T04:01:18.999932+0000","last_peered":"2024-03-30T04:01:18.999932+0000","last_clean":"2024-03-30T04:01:18.999932+0000","last_became_active":"2024-03-30T04:01:18.999645+0000","last_became_peered":"2024-03-30T04:01:18.999645+0000","last_unstale":"2024-03-30T04:01:18.999932+0000","last_undegraded":"2024-03-30T04:01:18.999932+0000","last_fullsized":"2024-03-30T04:01:18.999932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:22:33.394341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986480+0000","last_change":"2024-03-30T03:59:44.468855+0000","last_active":"2024-03-30T04:01:17.986480+0000","last_peered":"2024-03-30T04:01:17.986480+0000","last_clean":"2024-03-30T04:01:17.986480+0000","last_became_active":"2024-03-30T03:59:44.468216+0000","last_became_peered":"2024-03-30T03:59:44.468216+0000","last_unstale":"2024-03-30T04:01:17.986480+0000","last_undegraded":"2024-03-30T04:01:17.986480+0000","last_fullsized":"2024-03-30T04:01:17.986480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:21:40.081491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'3","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990215+0000","last_change":"2024-03-30T03:59:39.932509+0000","last_active":"2024-03-30T04:01:17.990215+0000","last_peered":"2024-03-30T04:01:17.990215+0000","last_clean":"2024-03-30T04:01:17.990215+0000","last_became_active":"2024-03-30T03:59:39.932163+0000","last_became_peered":"2024-03-30T03:59:39.932163+0000","last_unstale":"2024-03-30T04:01:17.990215+0000","last_undegraded":"2024-03-30T04:01:17.990215+0000","last_fullsized":"2024-03-30T04:01:17.990215+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:21:16.622164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'6","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.986563+0000","last_change":"2024-03-30T04:00:18.016967+0000","last_active":"2024-03-30T04:01:17.986563+0000","last_peered":"2024-03-30T04:01:17.986563+0000","last_clean":"2024-03-30T04:01:17.986563+0000","last_became_active":"2024-03-30T04:00:18.016555+0000","last_became_peered":"2024-03-30T04:00:18.016555+0000","last_unstale":"2024-03-30T04:01:17.986563+0000","last_undegraded":"2024-03-30T04:01:17.986563+0000","last_fullsized":"2024-03-30T04:01:17.986563+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:34:50.538682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.989682+0000","last_change":"2024-03-30T03:59:44.462502+0000","last_active":"2024-03-30T04:01:17.989682+0000","last_peered":"2024-03-30T04:01:17.989682+0000","last_clean":"2024-03-30T04:01:17.989682+0000","last_became_active":"2024-03-30T03:59:44.462105+0000","last_became_peered":"2024-03-30T03:59:44.462105+0000","last_unstale":"2024-03-30T04:01:17.989682+0000","last_undegraded":"2024-03-30T04:01:17.989682+0000","last_fullsized":"2024-03-30T04:01:17.989682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:43.439981+0000","last_clean_scrub_stamp":"2024-03-30T03:59:43.439981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:17:57.942648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992001+0000","last_change":"2024-03-30T03:59:39.929406+0000","last_active":"2024-03-30T04:01:17.992001+0000","last_peered":"2024-03-30T04:01:17.992001+0000","last_clean":"2024-03-30T04:01:17.992001+0000","last_became_active":"2024-03-30T03:59:39.929070+0000","last_became_peered":"2024-03-30T03:59:39.929070+0000","last_unstale":"2024-03-30T04:01:17.992001+0000","last_undegraded":"2024-03-30T04:01:17.992001+0000","last_fullsized":"2024-03-30T04:01:17.992001+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":77,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:38.901658+0000","last_clean_scrub_stamp":"2024-03-30T03:59:38.901658+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:44:36.843587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.987339+0000","last_change":"2024-03-30T03:59:40.459241+0000","last_active":"2024-03-30T04:01:17.987339+0000","last_peered":"2024-03-30T04:01:17.987339+0000","last_clean":"2024-03-30T04:01:17.987339+0000","last_became_active":"2024-03-30T03:59:40.458903+0000","last_became_peered":"2024-03-30T03:59:40.458903+0000","last_unstale":"2024-03-30T04:01:17.987339+0000","last_undegraded":"2024-03-30T04:01:17.987339+0000","last_fullsized":"2024-03-30T04:01:17.987339+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:29:37.473212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"86'9","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990824+0000","last_change":"2024-03-30T03:59:40.464028+0000","last_active":"2024-03-30T04:01:17.990824+0000","last_peered":"2024-03-30T04:01:17.990824+0000","last_clean":"2024-03-30T04:01:17.990824+0000","last_became_active":"2024-03-30T03:59:40.458374+0000","last_became_peered":"2024-03-30T03:59:40.458374+0000","last_unstale":"2024-03-30T04:01:17.990824+0000","last_undegraded":"2024-03-30T04:01:17.990824+0000","last_fullsized":"2024-03-30T04:01:17.990824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:51:50.672645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"86'12","reported_seq":49,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.001518+0000","last_change":"2024-03-30T04:01:19.001518+0000","last_active":"2024-03-30T04:01:19.001518+0000","last_peered":"2024-03-30T04:01:19.001518+0000","last_clean":"2024-03-30T04:01:19.001518+0000","last_became_active":"2024-03-30T04:01:19.001118+0000","last_became_peered":"2024-03-30T04:01:19.001118+0000","last_unstale":"2024-03-30T04:01:19.001518+0000","last_undegraded":"2024-03-30T04:01:19.001518+0000","last_fullsized":"2024-03-30T04:01:19.001518+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:50:08.382669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988506+0000","last_change":"2024-03-30T03:59:40.476588+0000","last_active":"2024-03-30T04:01:17.988506+0000","last_peered":"2024-03-30T04:01:17.988506+0000","last_clean":"2024-03-30T04:01:17.988506+0000","last_became_active":"2024-03-30T03:59:40.476337+0000","last_became_peered":"2024-03-30T03:59:40.476337+0000","last_unstale":"2024-03-30T04:01:17.988506+0000","last_undegraded":"2024-03-30T04:01:17.988506+0000","last_fullsized":"2024-03-30T04:01:17.988506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:08:30.849016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"86'7","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-30T04:01:19.004456+0000","last_change":"2024-03-30T04:01:19.004456+0000","last_active":"2024-03-30T04:01:19.004456+0000","last_peered":"2024-03-30T04:01:19.004456+0000","last_clean":"2024-03-30T04:01:19.004456+0000","last_became_active":"2024-03-30T04:01:19.004121+0000","last_became_peered":"2024-03-30T04:01:19.004121+0000","last_unstale":"2024-03-30T04:01:19.004456+0000","last_undegraded":"2024-03-30T04:01:19.004456+0000","last_fullsized":"2024-03-30T04:01:19.004456+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:58:37.404002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"86'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.991230+0000","last_change":"2024-03-30T03:59:40.455527+0000","last_active":"2024-03-30T04:01:17.991230+0000","last_peered":"2024-03-30T04:01:17.991230+0000","last_clean":"2024-03-30T04:01:17.991230+0000","last_became_active":"2024-03-30T03:59:40.454690+0000","last_became_peered":"2024-03-30T03:59:40.454690+0000","last_unstale":"2024-03-30T04:01:17.991230+0000","last_undegraded":"2024-03-30T04:01:17.991230+0000","last_fullsized":"2024-03-30T04:01:17.991230+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T14:02:18.829548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990493+0000","last_change":"2024-03-30T03:59:40.451025+0000","last_active":"2024-03-30T04:01:17.990493+0000","last_peered":"2024-03-30T04:01:17.990493+0000","last_clean":"2024-03-30T04:01:17.990493+0000","last_became_active":"2024-03-30T03:59:40.450651+0000","last_became_peered":"2024-03-30T03:59:40.450651+0000","last_unstale":"2024-03-30T04:01:17.990493+0000","last_undegraded":"2024-03-30T04:01:17.990493+0000","last_fullsized":"2024-03-30T04:01:17.990493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:56:30.868223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"86'15","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.988935+0000","last_change":"2024-03-30T03:59:40.460099+0000","last_active":"2024-03-30T04:01:17.988935+0000","last_peered":"2024-03-30T04:01:17.988935+0000","last_clean":"2024-03-30T04:01:17.988935+0000","last_became_active":"2024-03-30T03:59:40.459485+0000","last_became_peered":"2024-03-30T03:59:40.459485+0000","last_unstale":"2024-03-30T04:01:17.988935+0000","last_undegraded":"2024-03-30T04:01:17.988935+0000","last_fullsized":"2024-03-30T04:01:17.988935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:15:00.564776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"86'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.993122+0000","last_change":"2024-03-30T03:59:40.467821+0000","last_active":"2024-03-30T04:01:17.993122+0000","last_peered":"2024-03-30T04:01:17.993122+0000","last_clean":"2024-03-30T04:01:17.993122+0000","last_became_active":"2024-03-30T03:59:40.467414+0000","last_became_peered":"2024-03-30T03:59:40.467414+0000","last_unstale":"2024-03-30T04:01:17.993122+0000","last_undegraded":"2024-03-30T04:01:17.993122+0000","last_fullsized":"2024-03-30T04:01:17.993122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T08:26:27.394725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"86'6","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990134+0000","last_change":"2024-03-30T03:59:40.466770+0000","last_active":"2024-03-30T04:01:17.990134+0000","last_peered":"2024-03-30T04:01:17.990134+0000","last_clean":"2024-03-30T04:01:17.990134+0000","last_became_active":"2024-03-30T03:59:40.466397+0000","last_became_peered":"2024-03-30T03:59:40.466397+0000","last_unstale":"2024-03-30T04:01:17.990134+0000","last_undegraded":"2024-03-30T04:01:17.990134+0000","last_fullsized":"2024-03-30T04:01:17.990134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T05:08:46.425560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.992792+0000","last_change":"2024-03-30T03:59:40.464805+0000","last_active":"2024-03-30T04:01:17.992792+0000","last_peered":"2024-03-30T04:01:17.992792+0000","last_clean":"2024-03-30T04:01:17.992792+0000","last_became_active":"2024-03-30T03:59:40.464238+0000","last_became_peered":"2024-03-30T03:59:40.464238+0000","last_unstale":"2024-03-30T04:01:17.992792+0000","last_undegraded":"2024-03-30T04:01:17.992792+0000","last_fullsized":"2024-03-30T04:01:17.992792+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T13:46:33.344955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"86'10","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990666+0000","last_change":"2024-03-30T03:59:40.476695+0000","last_active":"2024-03-30T04:01:17.990666+0000","last_peered":"2024-03-30T04:01:17.990666+0000","last_clean":"2024-03-30T04:01:17.990666+0000","last_became_active":"2024-03-30T03:59:40.476323+0000","last_became_peered":"2024-03-30T03:59:40.476323+0000","last_unstale":"2024-03-30T04:01:17.990666+0000","last_undegraded":"2024-03-30T04:01:17.990666+0000","last_fullsized":"2024-03-30T04:01:17.990666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T07:30:55.184453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"86'11","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990825+0000","last_change":"2024-03-30T03:59:40.456737+0000","last_active":"2024-03-30T04:01:17.990825+0000","last_peered":"2024-03-30T04:01:17.990825+0000","last_clean":"2024-03-30T04:01:17.990825+0000","last_became_active":"2024-03-30T03:59:40.456045+0000","last_became_peered":"2024-03-30T03:59:40.456045+0000","last_unstale":"2024-03-30T04:01:17.990825+0000","last_undegraded":"2024-03-30T04:01:17.990825+0000","last_fullsized":"2024-03-30T04:01:17.990825+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T09:46:05.976734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"86'8","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-30T04:01:17.990548+0000","last_change":"2024-03-30T03:59:40.457380+0000","last_active":"2024-03-30T04:01:17.990548+0000","last_peered":"2024-03-30T04:01:17.990548+0000","last_clean":"2024-03-30T04:01:17.990548+0000","last_became_active":"2024-03-30T03:59:40.456948+0000","last_became_peered":"2024-03-30T03:59:40.456948+0000","last_unstale":"2024-03-30T04:01:17.990548+0000","last_undegraded":"2024-03-30T04:01:17.990548+0000","last_fullsized":"2024-03-30T04:01:17.990548+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T03:59:39.906327+0000","last_clean_scrub_stamp":"2024-03-30T03:59:39.906327+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-31T10:46:25.417091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":78682,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":920,"num_read_kb":548,"num_write":257,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":236046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":75,"seq":322122547235,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":1007616,"data_stored":492856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31848901},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.0950000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.59399999999999997,"15min":0.58199999999999996},"min":{"1min":0.45400000000000001,"5min":0.439,"15min":0.439},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.1869999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.621,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.752},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.40200000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78500000000000003,"5min":0.996,"15min":0.996},"last":0.73399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48899999999999999,"15min":0.48799999999999999},"min":{"1min":0.34799999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34100000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":4,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.66800000000000004,"15min":0.67000000000000004},"min":{"1min":0.47899999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.80600000000000005,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.70499999999999996}]},{"osd":5,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79700000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57099999999999995,"15min":0.56899999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71499999999999997}]},{"osd":6,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.80100000000000005,"15min":0.81499999999999995},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.876,"5min":2.9340000000000002,"15min":2.9340000000000002},"last":0.89200000000000002}]},{"osd":7,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85399999999999998,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.73399999999999999,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91600000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.76700000000000002}]},{"osd":8,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.85899999999999999}]},{"osd":9,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.77600000000000002,"15min":0.77100000000000002},"min":{"1min":0.60499999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.052,"5min":1.097,"15min":1.097},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79200000000000004,"15min":0.78200000000000003},"min":{"1min":0.58299999999999996,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.0089999999999999},"last":0.91500000000000004}]},{"osd":10,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.625,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":0.93899999999999995,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.93799999999999994,"5min":1.536,"15min":1.536},"last":0.92300000000000004}]}]},{"osd":10,"up_from":68,"seq":292057776166,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36876,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708276,"statfs":{"total":95995035648,"available":95957274624,"internally_reserved":0,"allocated":1708032,"data_stored":1150498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36043201},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.48999999999999999},"min":{"1min":0.38,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.623,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.504,"15min":0.502},"min":{"1min":0.41899999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68500000000000005,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.61499999999999999,"15min":0.61099999999999999},"min":{"1min":0.48999999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.622,"15min":0.61599999999999999},"min":{"1min":0.47699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.81399999999999995,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.63600000000000001,"15min":0.64000000000000001},"min":{"1min":0.496,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.67400000000000004,"15min":0.67000000000000004},"min":{"1min":0.58199999999999996,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60399999999999998,"15min":0.59999999999999998},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.67600000000000005,"15min":0.67500000000000004},"min":{"1min":0.53500000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81899999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.67800000000000005,"15min":0.67400000000000004},"min":{"1min":0.51100000000000001,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.77500000000000002}]},{"osd":6,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.99299999999999999,"5min":1.01,"15min":1.01},"last":0.626},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.60399999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95499999999999996,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:01:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.66800000000000004}]},{"osd":8,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.55100000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97599999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.57299999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91100000000000003,"5min":1,"15min":1},"last":0.83599999999999997}]},{"osd":9,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.84899999999999998,"15min":0.85099999999999998},"min":{"1min":0.71599999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.018,"5min":1.3700000000000001,"15min":1.3700000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.65200000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.95699999999999996,"5min":1.147,"15min":1.147},"last":0.751}]},{"osd":11,"last update":"Sat Mar 30 04:01:04 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75600000000000001,"15min":0.746},"min":{"1min":0.63400000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.96999999999999997,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.79600000000000004,"15min":0.78900000000000003},"min":{"1min":0.68600000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":0.98599999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.85699999999999998}]}]},{"osd":9,"up_from":63,"seq":270582939691,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":1036288,"data_stored":496930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31848897},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.32700000000000001,"5min":0.311,"15min":0.311},"max":{"1min":1.2589999999999999,"5min":3.706,"15min":3.706},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.60199999999999998,"15min":0.63800000000000001},"min":{"1min":0.374,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74199999999999999,"5min":1.018,"15min":1.018},"last":0.57099999999999995}]},{"osd":1,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.69599999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":1.8700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.74099999999999999,"15min":0.76300000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.77500000000000002,"15min":0.80700000000000005},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.78900000000000003,"15min":0.81899999999999995},"min":{"1min":0.501,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.85599999999999998}]},{"osd":3,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.67000000000000004,"15min":0.69899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86599999999999999,"5min":1.155,"15min":1.155},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.74299999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.5900000000000001,"5min":1.5900000000000001,"15min":1.5900000000000001},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.81599999999999995,"15min":0.85299999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6819999999999999,"5min":1.72,"15min":1.72},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.82999999999999996,"15min":0.88100000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.097,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.89200000000000002,"15min":0.94999999999999996},"min":{"1min":0.59099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0489999999999999,"5min":2.4860000000000002,"15min":2.4860000000000002},"last":1.028},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.87,"15min":0.90400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.9219999999999999,"5min":1.9219999999999999,"15min":1.9219999999999999},"last":0.87}]},{"osd":6,"last update":"Sat Mar 30 03:59:26 2024","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.617,"5min":1.617,"15min":1.617},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.581,"5min":1.581,"15min":1.581},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.93600000000000005,"15min":0.998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.107,"5min":2.2240000000000002,"15min":2.2240000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.95799999999999996,"15min":1.03},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.887,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":0.62}]},{"osd":8,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.94099999999999995,"15min":0.98799999999999999},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.997,"15min":1.0629999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.1910000000000001,"5min":2.379,"15min":2.379},"last":0.99199999999999999}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91300000000000003,"15min":0.90700000000000003},"min":{"1min":0.60399999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":1.7050000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.92700000000000005},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2430000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.83099999999999996}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.997,"5min":0.997,"15min":0.997},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.905,"5min":1.905,"15min":1.905},"last":0.92700000000000005}]}]},{"osd":8,"up_from":56,"seq":240518168623,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36212,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708940,"statfs":{"total":95995035648,"available":95957954560,"internally_reserved":0,"allocated":1028096,"data_stored":493212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.60499999999999998,"15min":0.625},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78900000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.53400000000000003},"min":{"1min":0.44,"5min":0.371,"15min":0.371},"max":{"1min":0.76200000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":1,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.57199999999999995,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53500000000000003,"15min":0.52000000000000002},"min":{"1min":0.40999999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.755,"5min":0.876,"15min":0.876},"last":0.54100000000000004}]},{"osd":2,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.36799999999999999},"min":{"1min":0.30299999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64200000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.434},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.48099999999999998,"15min":0.47099999999999997},"min":{"1min":0.32200000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.64700000000000002,"15min":0.64500000000000002},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51400000000000001}]},{"osd":4,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.45500000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76500000000000001,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.55600000000000005,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.379,"15min":0.379},"max":{"1min":0.78700000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64800000000000002}]},{"osd":5,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.41999999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41699999999999998}]},{"osd":6,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.63800000000000001,"15min":0.63100000000000001},"min":{"1min":0.48199999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.621},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.71699999999999997,"15min":0.72099999999999997},"min":{"1min":0.56799999999999995,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.97199999999999998,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73099999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68700000000000006,"15min":0.67300000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93899999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.67000000000000004,"15min":0.64800000000000002},"min":{"1min":0.52500000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.81799999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:38 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.71399999999999997,"15min":0.68600000000000005},"min":{"1min":0.60199999999999998,"5min":0.495,"15min":0.495},"max":{"1min":0.93899999999999995,"5min":1.006,"15min":1.006},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.76600000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":3.7160000000000002,"5min":3.7160000000000002,"15min":3.7160000000000002},"last":0.80300000000000005}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.69699999999999995,"15min":0.68300000000000005},"min":{"1min":0.55200000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.70199999999999996,"15min":0.68999999999999995},"min":{"1min":0.56000000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.84999999999999998}]},{"osd":11,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.629},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70799999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397552,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28604,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716548,"statfs":{"total":95995035648,"available":95965745152,"internally_reserved":0,"allocated":1626112,"data_stored":1091334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.41099999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.40400000000000003,"15min":0.39200000000000002},"min":{"1min":0.31,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.626,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.32800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.58999999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.437,"5min":0.372,"15min":0.372},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.753,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64400000000000002}]},{"osd":3,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57499999999999996},"min":{"1min":0.46000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81899999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.82799999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.498}]},{"osd":4,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.499,"15min":0.501},"min":{"1min":0.33600000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60799999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49399999999999999,"15min":0.496},"min":{"1min":0.34799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.72899999999999998,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.434}]},{"osd":5,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.93100000000000005,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.505,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81100000000000005,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.58199999999999996}]},{"osd":6,"last update":"Sat Mar 30 04:00:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.67000000000000004,"15min":0.68700000000000006},"min":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.72099999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.82899999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63400000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66800000000000004,"15min":0.66500000000000004},"min":{"1min":0.47999999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79500000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.70099999999999996},"min":{"1min":0.53100000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88900000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59699999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.77800000000000002,"15min":0.78600000000000003},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.75800000000000001,"15min":0.76400000000000001},"min":{"1min":0.60399999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":0.85599999999999998,"5min":1.258,"15min":1.258},"last":0.69699999999999995}]},{"osd":10,"last update":"Sat Mar 30 04:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":0.74299999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.96599999999999997,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71599999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.754}]},{"osd":11,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77500000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561077,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":489523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.52200000000000002,"15min":0.51000000000000001},"min":{"1min":0.38400000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.626},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.56999999999999995,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.45700000000000002}]},{"osd":1,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.65100000000000002,"15min":0.63200000000000001},"min":{"1min":0.51200000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.69499999999999995,"15min":0.67100000000000004},"min":{"1min":0.55200000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.71099999999999997,"15min":0.69299999999999995},"min":{"1min":0.60199999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.165,"5min":1.165,"15min":1.165},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.70099999999999996,"15min":0.67900000000000005},"min":{"1min":0.57299999999999995,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.63500000000000001,"15min":0.61699999999999999},"min":{"1min":0.48399999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.70099999999999996,"15min":0.69699999999999995},"min":{"1min":0.442,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.75,"15min":0.73499999999999999},"min":{"1min":0.58299999999999996,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.83199999999999996,"5min":0.76700000000000002,"15min":0.749},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.65800000000000003}]},{"osd":5,"last update":"Sat Mar 30 04:00:39 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.84899999999999998,"15min":0.84799999999999998},"min":{"1min":0.61399999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.337,"5min":1.462,"15min":1.462},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.68300000000000005,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.72399999999999998}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.82399999999999995,"15min":0.79800000000000004},"min":{"1min":0.52000000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.83199999999999996,"15min":0.82199999999999995},"min":{"1min":0.64200000000000002,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.073}]},{"osd":8,"last update":"Sat Mar 30 04:01:12 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.88600000000000001,"15min":0.85999999999999999},"min":{"1min":0.71899999999999997,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.88800000000000001,"15min":0.86699999999999999},"min":{"1min":0.76500000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.109}]},{"osd":9,"last update":"Sat Mar 30 04:01:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.92300000000000004,"15min":0.91200000000000003},"min":{"1min":0.68000000000000005,"5min":0.63,"15min":0.63},"max":{"1min":1.512,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.877,"15min":0.86499999999999999},"min":{"1min":0.65900000000000003,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.109,"5min":1.488,"15min":1.488},"last":0.751}]},{"osd":10,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.95199999999999996,"15min":0.95399999999999996},"min":{"1min":0.73299999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.278,"5min":1.278,"15min":1.278},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.075,"5min":0.97499999999999998,"15min":0.95899999999999996},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76300000000000001}]},{"osd":11,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.875,"5min":0.875,"15min":0.875},"max":{"1min":1.8680000000000001,"5min":1.8680000000000001,"15min":1.8680000000000001},"last":1.149},{"interface":"front","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":0.82699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":1081344,"data_stored":560845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.56200000000000006,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.84499999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.503},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.57099999999999995,"15min":0.52100000000000002},"min":{"1min":0.40100000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.72899999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:01:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55100000000000005,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.58499999999999996,"15min":0.51100000000000001},"min":{"1min":0.46400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:00:38 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.53700000000000003,"15min":0.498},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.59199999999999997,"15min":0.53800000000000003},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.871,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55600000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.442,"15min":0.40200000000000002},"min":{"1min":0.36199999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.71999999999999997,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.41899999999999998},"min":{"1min":0.34300000000000003,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69199999999999995,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47599999999999998}]},{"osd":5,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60799999999999998,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.81299999999999994,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.622,"15min":0.57599999999999996},"min":{"1min":0.39800000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Sat Mar 30 04:00:35 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.92300000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.65000000000000002,"15min":0.64000000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.40100000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91900000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.51500000000000001},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78500000000000003,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.624,"15min":0.61499999999999999},"min":{"1min":0.503,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82699999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70799999999999996,"15min":0.69799999999999995},"min":{"1min":0.54700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.90600000000000003,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.59799999999999998}]},{"osd":9,"last update":"Sat Mar 30 04:01:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.82199999999999995,"15min":0.80000000000000004},"min":{"1min":0.61699999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89500000000000002,"5min":3.2080000000000002,"15min":3.2080000000000002},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.745,"15min":0.755},"min":{"1min":0.55700000000000005,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":0.79800000000000004,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.63}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.746,"15min":0.745},"min":{"1min":0.51200000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.93600000000000005,"5min":0.999,"15min":0.999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.78300000000000003,"15min":0.78200000000000003},"min":{"1min":0.58099999999999996,"5min":0.501,"15min":0.501},"max":{"1min":0.94099999999999995,"5min":1.073,"15min":1.073},"last":0.65300000000000002}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.755},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.76900000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738444,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":489746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.59499999999999997,"15min":0.54800000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.60299999999999998,"15min":0.56299999999999994},"min":{"1min":0.372,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.738,"5min":1.738,"15min":1.738},"last":0.55600000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.67800000000000005,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.378,"15min":0.378},"max":{"1min":6.9859999999999998,"5min":6.9859999999999998,"15min":6.9859999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0580000000000001,"5min":0.71599999999999997,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":6.96,"5min":6.96,"15min":6.96},"last":0.92900000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.55300000000000005}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63700000000000001,"15min":0.61799999999999999},"min":{"1min":0.44800000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.71999999999999997,"15min":0.67500000000000004},"min":{"1min":0.51000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.73699999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:01:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.73999999999999999,"15min":0.68200000000000005},"min":{"1min":0.48099999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72299999999999998,"15min":0.67600000000000005},"min":{"1min":0.51900000000000002,"5min":0.497,"15min":0.497},"max":{"1min":1.113,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.94199999999999995}]},{"osd":6,"last update":"Sat Mar 30 04:00:37 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.67900000000000005,"15min":0.64100000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.462,"5min":1.462,"15min":1.462},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.76200000000000001,"15min":0.72499999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.78700000000000003}]},{"osd":7,"last update":"Sat Mar 30 04:00:46 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.24,"5min":1.24,"15min":1.24},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":0.76100000000000001}]},{"osd":8,"last update":"Sat Mar 30 04:01:08 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.88200000000000001,"15min":0.88800000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":0.93300000000000005,"15min":0.88800000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":9.7439999999999998,"5min":9.7439999999999998,"15min":9.7439999999999998},"last":0.95699999999999996}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.86899999999999999,"15min":0.878},"min":{"1min":0.50700000000000001,"5min":0.443,"15min":0.443},"max":{"1min":1.2290000000000001,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.93200000000000005,"15min":0.95399999999999996},"min":{"1min":0.53000000000000003,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.89600000000000002}]},{"osd":10,"last update":"Sat Mar 30 04:00:52 2024","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.88700000000000001,"15min":0.879},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.25,"5min":2.25,"15min":2.25},"last":0.873},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.92300000000000004,"15min":0.92100000000000004},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.91400000000000003}]},{"osd":11,"last update":"Sat Mar 30 04:01:03 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93899999999999995,"15min":0.93500000000000005},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.94399999999999995,"15min":0.94599999999999995},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.2809999999999999,"5min":1.298,"15min":1.298},"last":0.96399999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":560177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.56999999999999995},"min":{"1min":0.45100000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56100000000000005,"15min":0.57899999999999996},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78900000000000003,"5min":1.133,"15min":1.133},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.56499999999999995,"15min":0.51900000000000002},"min":{"1min":0.47499999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.78600000000000003,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.56599999999999995,"15min":0.52600000000000002},"min":{"1min":0.51700000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Mar 30 04:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60199999999999998,"15min":0.57999999999999996},"min":{"1min":0.50600000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.58999999999999997,"15min":0.57099999999999995},"min":{"1min":0.48199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.624},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.55700000000000005}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.47699999999999998,"15min":0.44900000000000001},"min":{"1min":0.41599999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.439},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.46200000000000002,"15min":0.44600000000000001},"min":{"1min":0.317,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.72599999999999998,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38800000000000001}]},{"osd":6,"last update":"Sat Mar 30 04:01:39 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.64300000000000002,"15min":0.61799999999999999},"min":{"1min":0.53200000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.67000000000000004,"15min":0.64000000000000001},"min":{"1min":0.52800000000000002,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91200000000000003,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67000000000000004}]},{"osd":7,"last update":"Sat Mar 30 04:00:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.69399999999999995}]},{"osd":8,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.58299999999999996,"15min":0.55700000000000005},"min":{"1min":0.503,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91000000000000003,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.56200000000000006,"15min":0.53000000000000003},"min":{"1min":0.42499999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51100000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69999999999999996,"15min":0.66300000000000003},"min":{"1min":0.65200000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.754,"15min":0.72899999999999998},"min":{"1min":0.64700000000000002,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.94799999999999995,"5min":0.995,"15min":0.995},"last":0.70499999999999996}]},{"osd":10,"last update":"Sat Mar 30 04:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.52800000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.97299999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.71299999999999997,"15min":0.69999999999999996},"min":{"1min":0.58799999999999997,"5min":0.442,"15min":0.442},"max":{"1min":0.93700000000000006,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.68300000000000005}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.64800000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182464,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":488046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.48699999999999999,"15min":0.42299999999999999},"min":{"1min":0.34599999999999997,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.373},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.443},"min":{"1min":0.36699999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.64500000000000002,"15min":0.61399999999999999},"min":{"1min":0.44800000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.633,"5min":1.762,"15min":1.762},"last":0.624},{"interface":"front","average":{"1min":0.754,"5min":0.71399999999999997,"15min":0.628},"min":{"1min":0.56000000000000005,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.601,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.68400000000000005,"15min":0.626},"min":{"1min":0.54200000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66000000000000003,"15min":0.63},"min":{"1min":0.495,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Sat Mar 30 04:01:00 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.67700000000000005,"15min":0.64800000000000002},"min":{"1min":0.46300000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.748,"15min":0.71399999999999997},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.61499999999999999}]},{"osd":5,"last update":"Sat Mar 30 04:01:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.72799999999999998,"15min":0.67300000000000004},"min":{"1min":0.58099999999999996,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.397,"5min":1.397,"15min":1.397},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.79600000000000004,"15min":0.73899999999999999},"min":{"1min":0.69899999999999995,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.482,"5min":1.482,"15min":1.482},"last":0.55900000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.79600000000000004,"15min":0.76200000000000001},"min":{"1min":0.47299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76100000000000001,"15min":0.72899999999999998},"min":{"1min":0.45600000000000002,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Mar 30 04:00:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.80700000000000005,"15min":0.80200000000000005},"min":{"1min":0.59799999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.89500000000000002,"15min":0.88300000000000001},"min":{"1min":0.65200000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.89600000000000002}]},{"osd":8,"last update":"Sat Mar 30 03:59:08 2024","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.63300000000000001}]},{"osd":9,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.99199999999999999,"15min":1.018},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.755},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.61899999999999999,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.474,"5min":1.474,"15min":1.474},"last":0.70799999999999996}]},{"osd":10,"last update":"Sat Mar 30 04:00:48 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.67000000000000004,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.02,"5min":0.99299999999999999,"15min":0.98899999999999999},"min":{"1min":0.749,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.716,"5min":1.716,"15min":1.716},"last":0.85399999999999998}]},{"osd":11,"last update":"Sat Mar 30 04:00:43 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.872}]}]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36208,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":93708944,"statfs":{"total":95995035648,"available":95957958656,"internally_reserved":0,"allocated":1015808,"data_stored":492898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 03:58:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38900000000000001,"15min":0.372},"min":{"1min":0.29999999999999999,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53800000000000003,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.32800000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55700000000000005,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40999999999999998}]},{"osd":2,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.41799999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.53400000000000003},"min":{"1min":0.47899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.81200000000000006,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.80400000000000005}]},{"osd":3,"last update":"Sat Mar 30 04:00:59 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60599999999999998,"15min":0.58099999999999996},"min":{"1min":0.45500000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57099999999999995},"min":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.90900000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Sat Mar 30 04:01:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.59099999999999997,"15min":0.55700000000000005},"min":{"1min":0.45200000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.61799999999999999,"15min":0.57199999999999995},"min":{"1min":0.52500000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.85299999999999998,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.78400000000000003}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.65100000000000002,"15min":0.64500000000000002},"min":{"1min":0.41099999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77900000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65700000000000003,"15min":0.64100000000000001},"min":{"1min":0.441,"5min":0.375,"15min":0.375},"max":{"1min":0.79000000000000004,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.495}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55200000000000005,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.74299999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56200000000000006,"15min":0.54300000000000004},"min":{"1min":0.45300000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.871,"15min":0.871},"last":0.48099999999999998}]},{"osd":8,"last update":"Sat Mar 30 04:01:11 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.69499999999999995,"15min":0.67500000000000004},"min":{"1min":0.56999999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89300000000000002,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68100000000000005},"min":{"1min":0.59699999999999998,"5min":0.442,"15min":0.442},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.81799999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:32 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.76700000000000002,"15min":0.76600000000000001},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.77900000000000003,"15min":0.79100000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.91100000000000003,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.68000000000000005}]},{"osd":10,"last update":"Sat Mar 30 04:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.749,"15min":0.75700000000000001},"min":{"1min":0.53900000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.91800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.746,"15min":0.754},"min":{"1min":0.58399999999999996,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.85399999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.60999999999999999}]},{"osd":11,"last update":"Sat Mar 30 04:00:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.72799999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32704,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712448,"statfs":{"total":95995035648,"available":95961546752,"internally_reserved":0,"allocated":1630208,"data_stored":1084914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.42499999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55000000000000004,"15min":0.53600000000000003},"min":{"1min":0.35899999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.373}]},{"osd":1,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.57299999999999995,"15min":0.55300000000000005},"min":{"1min":0.39700000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73399999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57299999999999995,"15min":0.56200000000000006},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.754,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40500000000000003,"15min":0.39200000000000002},"min":{"1min":0.308,"5min":0.248,"15min":0.248},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.41499999999999998,"15min":0.40100000000000002},"min":{"1min":0.32300000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.626,"5min":0.747,"15min":0.747},"last":0.40899999999999997}]},{"osd":3,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.438,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84899999999999998,"5min":0.873,"15min":0.873},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.60299999999999998,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.504}]},{"osd":4,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60199999999999998,"15min":0.58999999999999997},"min":{"1min":0.46899999999999997,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.753,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57599999999999996,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73399999999999999,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.55700000000000005}]},{"osd":6,"last update":"Sat Mar 30 04:01:35 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.63600000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77000000000000002,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.622},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.64100000000000001,"15min":0.63100000000000001},"min":{"1min":0.501,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80400000000000005,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Mar 30 04:00:54 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64600000000000002,"15min":0.65400000000000003},"min":{"1min":0.497,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.77800000000000002,"5min":1.016,"15min":1.016},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67500000000000004,"15min":0.68100000000000005},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.53900000000000003}]},{"osd":8,"last update":"Sat Mar 30 04:01:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.374,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68799999999999994,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.59699999999999998,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85399999999999998,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.57199999999999995}]},{"osd":9,"last update":"Sat Mar 30 04:01:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73599999999999999,"15min":0.71999999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.71099999999999997,"15min":0.68999999999999995},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.84799999999999998,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.66400000000000003}]},{"osd":10,"last update":"Sat Mar 30 04:00:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.73199999999999998,"15min":0.73399999999999999},"min":{"1min":0.59599999999999997,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.84299999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73999999999999999,"15min":0.74199999999999999},"min":{"1min":0.55200000000000005,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.90800000000000003,"5min":1.222,"15min":1.222},"last":0.68799999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":73229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":72561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":72514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T04:01:42.537 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:42.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 -- 172.21.15.23:0/1812901216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124101e70 msgr2=0x7f51241022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 --2- 172.21.15.23:0/1812901216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124101e70 0x7f51241022d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f51100099c0 tx=0x7f5110009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 -- 172.21.15.23:0/1812901216 shutdown_connections 2024-03-30T04:01:42.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 --2- 172.21.15.23:0/1812901216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124102810 0x7f51241090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 --2- 172.21.15.23:0/1812901216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124100c80 0x7f51241010a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 --2- 172.21.15.23:0/1812901216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124101e70 0x7f51241022d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 -- 172.21.15.23:0/1812901216 >> 172.21.15.23:0/1812901216 conn(0x7f51240fc1f0 msgr2=0x7f51240fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:42.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 -- 172.21.15.23:0/1812901216 shutdown_connections 2024-03-30T04:01:42.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.769+0000 7f512a200700 1 -- 172.21.15.23:0/1812901216 wait complete. 2024-03-30T04:01:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.770+0000 7f512a200700 1 Processor -- start 2024-03-30T04:01:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 -- start start 2024-03-30T04:01:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124100c80 0x7f5124198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124101e70 0x7f5124199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241a08e0 con 0x7f5124102810 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241a0a50 con 0x7f5124101e70 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f512a200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51241a0c10 con 0x7f5124100c80 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f5123e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f5123e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39860/0 (socket says 172.21.15.23:39860) 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f5123e00700 1 -- 172.21.15.23:0/3186753637 learned_addr learned my addr 172.21.15.23:0/3186753637 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f5123400700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124100c80 0x7f5124198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.771+0000 7f5122a00700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124101e70 0x7f5124199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 -- 172.21.15.23:0/3186753637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124100c80 msgr2=0x7f5124198df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124100c80 0x7f5124198df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 -- 172.21.15.23:0/3186753637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124101e70 msgr2=0x7f5124199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124101e70 0x7f5124199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:42.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 -- 172.21.15.23:0/3186753637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51100095f0 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f5123e00700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f511c00ef30 tx=0x7f511c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c018070 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f511c00f040 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511c00cd00 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51241a0e90 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.772+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51241a1260 con 0x7f5124102810 2024-03-30T04:01:42.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.773+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5124067a10 con 0x7f5124102810 2024-03-30T04:01:42.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.775+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f511c003a00 con 0x7f5124102810 2024-03-30T04:01:42.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.777+0000 7f511be00700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 0x7f51080a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:42.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.778+0000 7f5123400700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 0x7f51080a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:42.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.778+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f511c0cc970 con 0x7f5124102810 2024-03-30T04:01:42.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.778+0000 7f5123400700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 0x7f51080a3670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f512419a270 tx=0x7f5114009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:42.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:42.782+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f511c095c50 con 0x7f5124102810 2024-03-30T04:01:43.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.041+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f512410c6e0 con 0x7f5124102810 2024-03-30T04:01:43.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.042+0000 7f511be00700 1 -- 172.21.15.23:0/3186753637 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f511c099270 con 0x7f5124102810 2024-03-30T04:01:43.041 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:43.041 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 msgr2=0x7f51080a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 0x7f51080a3670 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f512419a270 tx=0x7f5114009260 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 msgr2=0x7f51241a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f511c00ef30 tx=0x7f511c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 shutdown_connections 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5124102810 0x7f51241a0310 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f51080a11b0 0x7f51080a3670 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124100c80 0x7f5124198df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 --2- 172.21.15.23:0/3186753637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5124101e70 0x7f5124199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 >> 172.21.15.23:0/3186753637 conn(0x7f51240fc1f0 msgr2=0x7f5124072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 shutdown_connections 2024-03-30T04:01:43.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.045+0000 7f512a200700 1 -- 172.21.15.23:0/3186753637 wait complete. 2024-03-30T04:01:43.063 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:01:43.063 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:01:43.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 -- 172.21.15.23:0/2649047309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88100d80 msgr2=0x7fef881011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 --2- 172.21.15.23:0/2649047309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88100d80 0x7fef881011a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0099c0 tx=0x7fef7c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 -- 172.21.15.23:0/2649047309 shutdown_connections 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 --2- 172.21.15.23:0/2649047309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88102910 0x7fef881091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 --2- 172.21.15.23:0/2649047309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88100d80 0x7fef881011a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 --2- 172.21.15.23:0/2649047309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88101f70 0x7fef881023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 -- 172.21.15.23:0/2649047309 >> 172.21.15.23:0/2649047309 conn(0x7fef880fc2f0 msgr2=0x7fef880fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 -- 172.21.15.23:0/2649047309 shutdown_connections 2024-03-30T04:01:43.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.267+0000 7fef8f600700 1 -- 172.21.15.23:0/2649047309 wait complete. 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 Processor -- start 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 -- start start 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 0x7fef88196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88101f70 0x7fef881972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8819e830 con 0x7fef88102910 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8819e9a0 con 0x7fef88101f70 2024-03-30T04:01:43.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8f600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8819eb60 con 0x7fef88100d80 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.268+0000 7fef8d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39874/0 (socket says 172.21.15.23:39874) 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 -- 172.21.15.23:0/4113762844 learned_addr learned my addr 172.21.15.23:0/4113762844 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef87e00700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88101f70 0x7fef881972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8cc00700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 0x7fef88196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 -- 172.21.15.23:0/4113762844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 msgr2=0x7fef88196d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 0x7fef88196d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 -- 172.21.15.23:0/4113762844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88101f70 msgr2=0x7fef881972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88101f70 0x7fef881972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 -- 172.21.15.23:0/4113762844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0095f0 con 0x7fef88102910 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8cc00700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 0x7fef88196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8d600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fef8000ef30 tx=0x7fef8000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:43.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80018070 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef8000f040 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.269+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8819ede0 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.270+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000cdb0 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.270+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8819f260 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.271+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef88108250 con 0x7fef88102910 2024-03-30T04:01:43.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.273+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fef80003a00 con 0x7fef88102910 2024-03-30T04:01:43.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.275+0000 7fef85600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 0x7fef700a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.275+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fef800cca00 con 0x7fef88102910 2024-03-30T04:01:43.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.275+0000 7fef8cc00700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 0x7fef700a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.276+0000 7fef8cc00700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 0x7fef700a3670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009990 tx=0x7fef7c01a040 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:43.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.279+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fef80095ce0 con 0x7fef88102910 2024-03-30T04:01:43.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.496+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef88107c30 con 0x7fef88102910 2024-03-30T04:01:43.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.497+0000 7fef85600700 1 -- 172.21.15.23:0/4113762844 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fef80099300 con 0x7fef88102910 2024-03-30T04:01:43.496 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:01:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 msgr2=0x7fef700a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 0x7fef700a3670 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009990 tx=0x7fef7c01a040 comp rx=0 tx=0).stop 2024-03-30T04:01:43.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 msgr2=0x7fef8819e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fef8000ef30 tx=0x7fef8000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 shutdown_connections 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef88102910 0x7fef8819e260 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fef700a11b0 0x7fef700a3670 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fef88100d80 0x7fef88196d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 --2- 172.21.15.23:0/4113762844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef88101f70 0x7fef881972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.499+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 >> 172.21.15.23:0/4113762844 conn(0x7fef880fc2f0 msgr2=0x7fef88105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.500+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 shutdown_connections 2024-03-30T04:01:43.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.500+0000 7fef8f600700 1 -- 172.21.15.23:0/4113762844 wait complete. 2024-03-30T04:01:43.517 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:01:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:43 smithi154 ceph-mon[79161]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:43 smithi154 ceph-mon[79161]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:43 smithi154 ceph-mon[79161]: pgmap v301: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-30T04:01:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:43 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3186753637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 -- 172.21.15.23:0/1098936751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c102910 msgr2=0x7f5e3c1091c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 --2- 172.21.15.23:0/1098936751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c102910 0x7f5e3c1091c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e34009fc0 tx=0x7f5e3400b330 comp rx=0 tx=0).stop 2024-03-30T04:01:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 -- 172.21.15.23:0/1098936751 shutdown_connections 2024-03-30T04:01:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 --2- 172.21.15.23:0/1098936751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c102910 0x7f5e3c1091c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 --2- 172.21.15.23:0/1098936751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c100d80 0x7f5e3c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 --2- 172.21.15.23:0/1098936751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c101f70 0x7f5e3c1023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 -- 172.21.15.23:0/1098936751 >> 172.21.15.23:0/1098936751 conn(0x7f5e3c0fc2f0 msgr2=0x7f5e3c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:43.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 -- 172.21.15.23:0/1098936751 shutdown_connections 2024-03-30T04:01:43.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.719+0000 7f5e41600700 1 -- 172.21.15.23:0/1098936751 wait complete. 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.720+0000 7f5e41600700 1 Processor -- start 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 -- start start 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 0x7f5e3c199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c102910 0x7f5e3c1a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1a0b80 con 0x7f5e3c101f70 2024-03-30T04:01:43.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1a0cc0 con 0x7f5e3c100d80 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e41600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c1a0e50 con 0x7f5e3c102910 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e3a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e3a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39890/0 (socket says 172.21.15.23:39890) 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e3a000700 1 -- 172.21.15.23:0/2695660596 learned_addr learned my addr 172.21.15.23:0/2695660596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e3b400700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c102910 0x7f5e3c1a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.721+0000 7f5e3aa00700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 0x7f5e3c199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 -- 172.21.15.23:0/2695660596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c102910 msgr2=0x7f5e3c1a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c102910 0x7f5e3c1a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 -- 172.21.15.23:0/2695660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 msgr2=0x7f5e3c199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 0x7f5e3c199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 -- 172.21.15.23:0/2695660596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c0095f0 con 0x7f5e3c101f70 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3aa00700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 0x7f5e3c199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e3a000700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9a0 tx=0x7f5e2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28009790 con 0x7f5e3c101f70 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e2800cbd0 con 0x7f5e3c101f70 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28018680 con 0x7f5e3c101f70 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e34009c40 con 0x7f5e3c101f70 2024-03-30T04:01:43.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.722+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1a1460 con 0x7f5e3c101f70 2024-03-30T04:01:43.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.723+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c067af0 con 0x7f5e3c101f70 2024-03-30T04:01:43.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.725+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f5e2800cd40 con 0x7f5e3c101f70 2024-03-30T04:01:43.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.726+0000 7f5e33400700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 0x7f5e200a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:43.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.727+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f5e28014070 con 0x7f5e3c101f70 2024-03-30T04:01:43.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.727+0000 7f5e3aa00700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 0x7f5e200a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:43.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.727+0000 7f5e3aa00700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 0x7f5e200a3670 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c101dd0 tx=0x7f5e2c009310 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:43.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.730+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5e280529f0 con 0x7f5e3c101f70 2024-03-30T04:01:43.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:43 smithi193 ceph-mon[79581]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:43 smithi193 ceph-mon[79581]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:43 smithi193 ceph-mon[79581]: pgmap v301: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-30T04:01:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:43 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3186753637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:43.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:43 smithi023 ceph-mon[38254]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:43 smithi023 ceph-mon[38254]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:43 smithi023 ceph-mon[38254]: pgmap v301: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-30T04:01:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3186753637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.951+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5e3c10c840 con 0x7f5e3c101f70 2024-03-30T04:01:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.952+0000 7f5e33400700 1 -- 172.21.15.23:0/2695660596 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5e28096300 con 0x7f5e3c101f70 2024-03-30T04:01:43.951 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 msgr2=0x7f5e200a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 0x7f5e200a3670 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c101dd0 tx=0x7f5e2c009310 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 msgr2=0x7f5e3c199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800e9a0 tx=0x7f5e2800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 shutdown_connections 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f5e200a11b0 0x7f5e200a3670 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c102910 0x7f5e3c1a05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100d80 0x7f5e3c199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 --2- 172.21.15.23:0/2695660596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c101f70 0x7f5e3c199590 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 >> 172.21.15.23:0/2695660596 conn(0x7f5e3c0fc2f0 msgr2=0x7f5e3c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 shutdown_connections 2024-03-30T04:01:43.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:43.954+0000 7f5e41600700 1 -- 172.21.15.23:0/2695660596 wait complete. 2024-03-30T04:01:43.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:01:44.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 -- 172.21.15.23:0/3243913648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 msgr2=0x7fa844102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 --2- 172.21.15.23:0/3243913648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844102210 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa8300099c0 tx=0x7fa830009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 -- 172.21.15.23:0/3243913648 shutdown_connections 2024-03-30T04:01:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 --2- 172.21.15.23:0/3243913648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844102750 0x7fa844109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 --2- 172.21.15.23:0/3243913648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844100bc0 0x7fa844100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 --2- 172.21.15.23:0/3243913648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844102210 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.174+0000 7fa849200700 1 -- 172.21.15.23:0/3243913648 >> 172.21.15.23:0/3243913648 conn(0x7fa8440fc110 msgr2=0x7fa8440fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:44.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.175+0000 7fa849200700 1 -- 172.21.15.23:0/3243913648 shutdown_connections 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.175+0000 7fa849200700 1 -- 172.21.15.23:0/3243913648 wait complete. 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 Processor -- start 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 -- start start 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 0x7fa844198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844102750 0x7fa8441a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441a0850 con 0x7fa844101db0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441a0990 con 0x7fa844102750 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa849200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8441a0b50 con 0x7fa844100bc0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa842a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.176+0000 7fa843400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 0x7fa844198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa843400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 0x7fa844198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48122/0 (socket says 172.21.15.23:48122) 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa843400700 1 -- 172.21.15.23:0/1794922184 learned_addr learned my addr 172.21.15.23:0/1794922184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa843e00700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844102750 0x7fa8441a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 -- 172.21.15.23:0/1794922184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 msgr2=0x7fa844198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 0x7fa844198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 -- 172.21.15.23:0/1794922184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844102750 msgr2=0x7fa8441a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844102750 0x7fa8441a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 -- 172.21.15.23:0/1794922184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300095f0 con 0x7fa844101db0 2024-03-30T04:01:44.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa842a00700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844199300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa8300098e0 tx=0x7fa83000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83001d070 con 0x7fa844101db0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa830005030 con 0x7fa844101db0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830004a50 con 0x7fa844101db0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.177+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8441a0dd0 con 0x7fa844101db0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.178+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8441a11a0 con 0x7fa844101db0 2024-03-30T04:01:44.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.179+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8440679a0 con 0x7fa844101db0 2024-03-30T04:01:44.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.181+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fa8300054a0 con 0x7fa844101db0 2024-03-30T04:01:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.182+0000 7fa83be00700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 0x7fa8280a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.182+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fa8300cda20 con 0x7fa844101db0 2024-03-30T04:01:44.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.182+0000 7fa843400700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 0x7fa8280a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.183+0000 7fa843400700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 0x7fa8280a3670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa84419a1e0 tx=0x7fa834009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:44.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.186+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa830096d00 con 0x7fa844101db0 2024-03-30T04:01:44.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.405+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa84410c630 con 0x7fa844101db0 2024-03-30T04:01:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.408+0000 7fa83be00700 1 -- 172.21.15.23:0/1794922184 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa83009a320 con 0x7fa844101db0 2024-03-30T04:01:44.407 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:01:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 msgr2=0x7fa8280a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 0x7fa8280a3670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa84419a1e0 tx=0x7fa834009260 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 msgr2=0x7fa844199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844199300 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa8300098e0 tx=0x7fa83000f690 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 shutdown_connections 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa844102750 0x7fa8441a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa8280a11b0 0x7fa8280a3670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa844100bc0 0x7fa844198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 --2- 172.21.15.23:0/1794922184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa844101db0 0x7fa844199300 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 >> 172.21.15.23:0/1794922184 conn(0x7fa8440fc110 msgr2=0x7fa844072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 shutdown_connections 2024-03-30T04:01:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.411+0000 7fa849200700 1 -- 172.21.15.23:0/1794922184 wait complete. 2024-03-30T04:01:44.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.635+0000 7f641bc00700 1 -- 172.21.15.23:0/2817109261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414074df0 msgr2=0x7f6414073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.635+0000 7f641bc00700 1 --2- 172.21.15.23:0/2817109261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414074df0 0x7f6414073240 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6408009990 tx=0x7f6408009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 -- 172.21.15.23:0/2817109261 shutdown_connections 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 --2- 172.21.15.23:0/2817109261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414108c40 0x7f641410b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 --2- 172.21.15.23:0/2817109261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414074df0 0x7f6414073240 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 --2- 172.21.15.23:0/2817109261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414073810 0x7f6414073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 -- 172.21.15.23:0/2817109261 >> 172.21.15.23:0/2817109261 conn(0x7f64140fc2f0 msgr2=0x7f64140fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 -- 172.21.15.23:0/2817109261 shutdown_connections 2024-03-30T04:01:44.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.636+0000 7f641bc00700 1 -- 172.21.15.23:0/2817109261 wait complete. 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.637+0000 7f641bc00700 1 Processor -- start 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.637+0000 7f641bc00700 1 -- start start 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414074df0 0x7f64141994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 0x7f64141a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141a0a50 con 0x7f6414073810 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141a0b90 con 0x7f6414108c40 2024-03-30T04:01:44.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f641bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64141a0d20 con 0x7f6414074df0 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39928/0 (socket says 172.21.15.23:39928) 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 -- 172.21.15.23:0/3183509479 learned_addr learned my addr 172.21.15.23:0/3183509479 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6413e00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414074df0 0x7f64141994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419c00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 0x7f64141a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 -- 172.21.15.23:0/3183509479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414074df0 msgr2=0x7f64141994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414074df0 0x7f64141994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 -- 172.21.15.23:0/3183509479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 msgr2=0x7f64141a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 0x7f64141a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419200700 1 -- 172.21.15.23:0/3183509479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64080095f0 con 0x7f6414073810 2024-03-30T04:01:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.638+0000 7f6419c00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 0x7f64141a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:44.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f6419200700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f64080098e0 tx=0x7f640800f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640801d070 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6408005050 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408004bd0 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64141a0fa0 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.639+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64141a1310 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.640+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6414107a00 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.642+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f6408004d30 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.644+0000 7f6411600700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 0x7f63fc0a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.644+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f64080cdd70 con 0x7f6414073810 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.644+0000 7f6413e00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 0x7f63fc0a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:44.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.645+0000 7f6413e00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 0x7f63fc0a36f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6414074b10 tx=0x7f6400007400 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:44.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.648+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6408097050 con 0x7f6414073810 2024-03-30T04:01:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:44 smithi154 ceph-mon[79161]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4113762844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:44 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2695660596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:44 smithi154 ceph-mon[79161]: pgmap v302: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-30T04:01:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:44 smithi193 ceph-mon[79581]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4113762844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:44 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2695660596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:44 smithi193 ceph-mon[79581]: pgmap v302: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-30T04:01:44.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:44 smithi023 ceph-mon[38254]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-30T04:01:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4113762844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:44 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2695660596' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:01:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:44 smithi023 ceph-mon[38254]: pgmap v302: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-30T04:01:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.863+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f641410e7c0 con 0x7f6414073810 2024-03-30T04:01:44.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.864+0000 7f6411600700 1 -- 172.21.15.23:0/3183509479 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f640809a670 con 0x7f6414073810 2024-03-30T04:01:44.863 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 msgr2=0x7f63fc0a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 0x7f63fc0a36f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6414074b10 tx=0x7f6400007400 comp rx=0 tx=0).stop 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 msgr2=0x7f6414198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f64080098e0 tx=0x7f640800f670 comp rx=0 tx=0).stop 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 shutdown_connections 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6414108c40 0x7f64141a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6414073810 0x7f6414198f70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f63fc0a1230 0x7f63fc0a36f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 --2- 172.21.15.23:0/3183509479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6414074df0 0x7f64141994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 >> 172.21.15.23:0/3183509479 conn(0x7f64140fc2f0 msgr2=0x7f6414104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 shutdown_connections 2024-03-30T04:01:44.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:44.866+0000 7f641bc00700 1 -- 172.21.15.23:0/3183509479 wait complete. 2024-03-30T04:01:44.884 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 -- 172.21.15.23:0/1779366975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c100c80 msgr2=0x7fd24c1010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 --2- 172.21.15.23:0/1779366975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c100c80 0x7fd24c1010a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009990 tx=0x7fd23c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 -- 172.21.15.23:0/1779366975 shutdown_connections 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 --2- 172.21.15.23:0/1779366975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c102810 0x7fd24c1090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 --2- 172.21.15.23:0/1779366975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c100c80 0x7fd24c1010a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 --2- 172.21.15.23:0/1779366975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c101e70 0x7fd24c1022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 -- 172.21.15.23:0/1779366975 >> 172.21.15.23:0/1779366975 conn(0x7fd24c0fc1f0 msgr2=0x7fd24c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:45.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 -- 172.21.15.23:0/1779366975 shutdown_connections 2024-03-30T04:01:45.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.086+0000 7fd251400700 1 -- 172.21.15.23:0/1779366975 wait complete. 2024-03-30T04:01:45.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 Processor -- start 2024-03-30T04:01:45.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 -- start start 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c100c80 0x7fd24c074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c101e70 0x7fd24c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c073de0 con 0x7fd24c102810 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c073f50 con 0x7fd24c100c80 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.087+0000 7fd251400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24c1a0cd0 con 0x7fd24c101e70 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39936/0 (socket says 172.21.15.23:39936) 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 -- 172.21.15.23:0/2011904405 learned_addr learned my addr 172.21.15.23:0/2011904405 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd249e00700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c101e70 0x7fd24c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24a800700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c100c80 0x7fd24c074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 -- 172.21.15.23:0/2011904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c101e70 msgr2=0x7fd24c0731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c101e70 0x7fd24c0731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 -- 172.21.15.23:0/2011904405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c100c80 msgr2=0x7fd24c074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c100c80 0x7fd24c074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 -- 172.21.15.23:0/2011904405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd238009520 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.088+0000 7fd24b200700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c370 tx=0x7fd24400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.089+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd244017070 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.089+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd244009c40 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.089+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd24400f040 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.089+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd24400e050 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.089+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c0095f0 con 0x7fd24c102810 2024-03-30T04:01:45.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.090+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24c108170 con 0x7fd24c102810 2024-03-30T04:01:45.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.092+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fd24400cb60 con 0x7fd24c102810 2024-03-30T04:01:45.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.093+0000 7fd243400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 0x7fd2300a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.093+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fd2440cc710 con 0x7fd24c102810 2024-03-30T04:01:45.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.093+0000 7fd24a800700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 0x7fd2300a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.094+0000 7fd24a800700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 0x7fd2300a36f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00b680 tx=0x7fd23c00b750 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:45.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.097+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd244095960 con 0x7fd24c102810 2024-03-30T04:01:45.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.363+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd24c067a10 con 0x7fd24c102810 2024-03-30T04:01:45.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.364+0000 7fd243400700 1 -- 172.21.15.23:0/2011904405 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fd244098f80 con 0x7fd24c102810 2024-03-30T04:01:45.363 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:45.363 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:01:35.365141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:01:45.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.366+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 msgr2=0x7fd2300a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.366+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 0x7fd2300a36f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00b680 tx=0x7fd23c00b750 comp rx=0 tx=0).stop 2024-03-30T04:01:45.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.366+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 msgr2=0x7fd24c0736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.366+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd24400c370 tx=0x7fd24400c680 comp rx=0 tx=0).stop 2024-03-30T04:01:45.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 shutdown_connections 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd24c102810 0x7fd24c0736e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd24c100c80 0x7fd24c074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fd2300a1230 0x7fd2300a36f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 --2- 172.21.15.23:0/2011904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd24c101e70 0x7fd24c0731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 >> 172.21.15.23:0/2011904405 conn(0x7fd24c0fc1f0 msgr2=0x7fd24c1052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 shutdown_connections 2024-03-30T04:01:45.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.367+0000 7fd251400700 1 -- 172.21.15.23:0/2011904405 wait complete. 2024-03-30T04:01:45.367 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-30T04:01:45.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 -- 172.21.15.23:0/3340933043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 msgr2=0x7f17c81010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 --2- 172.21.15.23:0/3340933043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c81010a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f17b80099c0 tx=0x7f17b8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 -- 172.21.15.23:0/3340933043 shutdown_connections 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 --2- 172.21.15.23:0/3340933043 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 --2- 172.21.15.23:0/3340933043 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c81010a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 --2- 172.21.15.23:0/3340933043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 0x7f17c81022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 -- 172.21.15.23:0/3340933043 >> 172.21.15.23:0/3340933043 conn(0x7f17c80fc1f0 msgr2=0x7f17c80fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 -- 172.21.15.23:0/3340933043 shutdown_connections 2024-03-30T04:01:45.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.584+0000 7f17ce800700 1 -- 172.21.15.23:0/3340933043 wait complete. 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.585+0000 7f17ce800700 1 Processor -- start 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.585+0000 7f17ce800700 1 -- start start 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 0x7f17c81993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a0990 con 0x7f17c8100c80 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a0ad0 con 0x7f17c8102810 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17ce800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a0c60 con 0x7f17c8101e70 2024-03-30T04:01:45.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39952/0 (socket says 172.21.15.23:39952) 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 -- 172.21.15.23:0/3982327754 learned_addr learned my addr 172.21.15.23:0/3982327754 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7e00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c6a00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 0x7f17c81993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 -- 172.21.15.23:0/3982327754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 msgr2=0x7f17c81993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 0x7f17c81993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 -- 172.21.15.23:0/3982327754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 msgr2=0x7f17c81a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7400700 1 -- 172.21.15.23:0/3982327754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b80095f0 con 0x7f17c8100c80 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.586+0000 7f17c7e00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17c7400700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f17b8006010 tx=0x7f17b8005af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b801d070 con 0x7f17c8100c80 2024-03-30T04:01:45.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17b8004150 con 0x7f17c8100c80 2024-03-30T04:01:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c81a0ee0 con 0x7f17c8100c80 2024-03-30T04:01:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b8021410 con 0x7f17c8100c80 2024-03-30T04:01:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.587+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81a1250 con 0x7f17c8100c80 2024-03-30T04:01:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.588+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8067a10 con 0x7f17c8100c80 2024-03-30T04:01:45.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.590+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f17b800f6f0 con 0x7f17c8100c80 2024-03-30T04:01:45.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.591+0000 7f17bfe00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 0x7f17ac0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.591+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f17b80cdd70 con 0x7f17c8100c80 2024-03-30T04:01:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.591+0000 7f17c6a00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 0x7f17ac0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:45.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.592+0000 7f17c6a00700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 0x7f17ac0a3670 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005950 tx=0x7f17b40058e0 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:45.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.595+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f17b8097050 con 0x7f17c8100c80 2024-03-30T04:01:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1794922184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3183509479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:45 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2011904405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:45.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1794922184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3183509479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:45 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2011904405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:45.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1794922184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3183509479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:01:45.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:45 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2011904405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:01:45.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.854+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17c810c4e0 con 0x7f17c8100c80 2024-03-30T04:01:45.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.855+0000 7f17bfe00700 1 -- 172.21.15.23:0/3982327754 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24603 (secure 0 0 0) 0x7f17b809a670 con 0x7f17c8100c80 2024-03-30T04:01:45.854 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:01:45.854 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":88,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:01:18.983496+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:01:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 msgr2=0x7f17ac0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 0x7f17ac0a3670 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f17b4005950 tx=0x7f17b40058e0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 msgr2=0x7f17c8198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f17b8006010 tx=0x7f17b8005af0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 shutdown_connections 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c8102810 0x7f17c81a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17c8100c80 0x7f17c8198eb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f17ac0a11b0 0x7f17ac0a3670 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 --2- 172.21.15.23:0/3982327754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17c8101e70 0x7f17c81993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 >> 172.21.15.23:0/3982327754 conn(0x7f17c80fc1f0 msgr2=0x7f17c8072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 shutdown_connections 2024-03-30T04:01:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:45.858+0000 7f17ce800700 1 -- 172.21.15.23:0/3982327754 wait complete. 2024-03-30T04:01:45.876 DEBUG:tasks.fwd_scrub:Yielding 2024-03-30T04:01:45.877 INFO:teuthology.run_tasks:Running task exec... 2024-03-30T04:01:45.886 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T04:01:45.886 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:01:45.886 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-30T04:01:45.887 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-30T04:01:45.887 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:01:45.887 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:01:46.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.088+0000 7f2670c00700 1 -- 172.21.15.23:0/2655738783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c074df0 msgr2=0x7f266c073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.088+0000 7f2670c00700 1 --2- 172.21.15.23:0/2655738783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c074df0 0x7f266c073240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f265c009990 tx=0x7f265c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 -- 172.21.15.23:0/2655738783 shutdown_connections 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 --2- 172.21.15.23:0/2655738783 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 0x7f266c10f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 --2- 172.21.15.23:0/2655738783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c074df0 0x7f266c073240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 --2- 172.21.15.23:0/2655738783 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c073810 0x7f266c073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 -- 172.21.15.23:0/2655738783 >> 172.21.15.23:0/2655738783 conn(0x7f266c0fc280 msgr2=0x7f266c0fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 -- 172.21.15.23:0/2655738783 shutdown_connections 2024-03-30T04:01:46.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.089+0000 7f2670c00700 1 -- 172.21.15.23:0/2655738783 wait complete. 2024-03-30T04:01:46.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 Processor -- start 2024-03-30T04:01:46.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 -- start start 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c074df0 0x7f266c19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 0x7f266c1a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1a4da0 con 0x7f266c10d2e0 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1a4ee0 con 0x7f266c074df0 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2670c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f266c1a5070 con 0x7f266c073810 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f266a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 0x7f266c1a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.090+0000 7f2669e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48176/0 (socket says 172.21.15.23:48176) 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 -- 172.21.15.23:0/3280806265 learned_addr learned my addr 172.21.15.23:0/3280806265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669400700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c074df0 0x7f266c19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 -- 172.21.15.23:0/3280806265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c074df0 msgr2=0x7f266c19d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c074df0 0x7f266c19d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 -- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 msgr2=0x7f266c1a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 0x7f266c1a47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 -- 172.21.15.23:0/3280806265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0095f0 con 0x7f266c073810 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.091+0000 7f2669e00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f265c00b680 tx=0x7f265c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.092+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c01d070 con 0x7f266c073810 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.092+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f265c022470 con 0x7f266c073810 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.092+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c017650 con 0x7f266c073810 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.092+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266c1a52f0 con 0x7f266c073810 2024-03-30T04:01:46.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.092+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266c1a5710 con 0x7f266c073810 2024-03-30T04:01:46.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.093+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266c103540 con 0x7f266c073810 2024-03-30T04:01:46.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.096+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f265c0225e0 con 0x7f266c073810 2024-03-30T04:01:46.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7f2662a00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 0x7f26500a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 -- 172.21.15.23:0/4263667702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c101f70 msgr2=0x7fe61c1023d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 --2- 172.21.15.23:0/4263667702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c101f70 0x7fe61c1023d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe6080099c0 tx=0x7fe608009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f265c0ce1e0 con 0x7f266c073810 2024-03-30T04:01:46.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 -- 172.21.15.23:0/4263667702 shutdown_connections 2024-03-30T04:01:46.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 --2- 172.21.15.23:0/4263667702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c102910 0x7fe61c1091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 --2- 172.21.15.23:0/4263667702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 --2- 172.21.15.23:0/4263667702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c101f70 0x7fe61c1023d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 -- 172.21.15.23:0/4263667702 >> 172.21.15.23:0/4263667702 conn(0x7fe61c0fc2f0 msgr2=0x7fe61c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7f2669400700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 0x7f26500a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.097+0000 7fe624600700 1 -- 172.21.15.23:0/4263667702 shutdown_connections 2024-03-30T04:01:46.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.098+0000 7fe624600700 1 -- 172.21.15.23:0/4263667702 wait complete. 2024-03-30T04:01:46.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.098+0000 7f2669400700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 0x7f26500a36f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f266c074b10 tx=0x7f2658009260 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f265c0974c0 con 0x7f266c073810 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 Processor -- start 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 -- start start 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c102910 0x7fe61c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c073e80 con 0x7fe61c102910 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c073ff0 con 0x7fe61c100d80 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe624600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1a0d60 con 0x7fe61c101f70 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.099+0000 7fe621200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48198/0 (socket says 172.21.15.23:48198) 2024-03-30T04:01:46.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 -- 172.21.15.23:0/305274696 learned_addr learned my addr 172.21.15.23:0/305274696 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe622600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c102910 0x7fe61c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621c00700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 -- 172.21.15.23:0/305274696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 msgr2=0x7fe61c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 -- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c102910 msgr2=0x7fe61c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c102910 0x7fe61c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 -- 172.21.15.23:0/305274696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080095f0 con 0x7fe61c101f70 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621c00700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe621200700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c010 tx=0x7fe60800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.100+0000 7fe61a000700 1 -- 172.21.15.23:0/305274696 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60801d070 con 0x7fe61c101f70 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.101+0000 7fe61a000700 1 -- 172.21.15.23:0/305274696 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe608005510 con 0x7fe61c101f70 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.101+0000 7fe61a000700 1 -- 172.21.15.23:0/305274696 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608004b30 con 0x7fe61c101f70 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.101+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c1a0f90 con 0x7fe61c101f70 2024-03-30T04:01:46.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.101+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c1a1300 con 0x7fe61c101f70 2024-03-30T04:01:46.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.105+0000 7fe61a000700 1 -- 172.21.15.23:0/305274696 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7fe608004c90 con 0x7fe61c101f70 2024-03-30T04:01:46.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.106+0000 7fe638134b80 1 Processor -- start 2024-03-30T04:01:46.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 -- start start 2024-03-30T04:01:46.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe61a000700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 0x7fe6040a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe61a000700 1 -- 172.21.15.23:0/305274696 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fe6080cdbc0 con 0x7fe61c101f70 2024-03-30T04:01:46.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9efa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8e9f04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e946a10 con 0x559b8e9ee850 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e9443d0 con 0x559b8e9ef1b0 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe638134b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e93fcc0 con 0x559b8e9f0050 2024-03-30T04:01:46.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe622600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 0x7fe6040a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.107+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39994/0 (socket says 172.21.15.23:39994) 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 -- 172.21.15.23:0/3604451166 learned_addr learned my addr 172.21.15.23:0/3604451166 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe622600700 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8e9f04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621200700 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 -- 172.21.15.23:0/3604451166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 msgr2=0x559b8e9f04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8e9f04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 -- 172.21.15.23:0/3604451166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 msgr2=0x559b8e9efa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9efa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 -- 172.21.15.23:0/3604451166 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b8e7c1430 con 0x559b8e9ee850 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe621c00700 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe610009d60 tx=0x7fe6100116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cb6535d799b0d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.108+0000 7fe622600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 0x7fe6040a36f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x559b8e9ef830 tx=0x7fe614009920 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe60ea00700 1 -- 172.21.15.23:0/3604451166 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610011ed0 con 0x559b8e9ee850 2024-03-30T04:01:46.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe60ea00700 1 -- 172.21.15.23:0/3604451166 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe610012910 con 0x559b8e9ee850 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe60ea00700 1 -- 172.21.15.23:0/3604451166 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000f830 con 0x559b8e9ee850 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 -- 172.21.15.23:0/3604451166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 msgr2=0x559b8e9eec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe610009d60 tx=0x7fe6100116c0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 -- 172.21.15.23:0/3604451166 shutdown_connections 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8e9f04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9eec70 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 --2- 172.21.15.23:0/3604451166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9efa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.109+0000 7fe638134b80 1 -- 172.21.15.23:0/3604451166 >> 172.21.15.23:0/3604451166 conn(0x559b8e94d720 msgr2=0x559b8e9e9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.110+0000 7fe638134b80 1 -- 172.21.15.23:0/3604451166 shutdown_connections 2024-03-30T04:01:46.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.110+0000 7fe638134b80 1 -- 172.21.15.23:0/3604451166 wait complete. 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.110+0000 7fe638134b80 1 -- start start 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.110+0000 7fe638134b80 1 Processor -- start 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 0x559b8e9f8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e8dbe00 con 0x559b8e9ee850 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e8dbaa0 con 0x559b8e9f0050 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e7c3720 con 0x559b8e9ef1b0 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe638134b80 10 client.0 _ll_register_callbacks cb 0x559b8e8e54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40002/0 (socket says 172.21.15.23:40002) 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 0x559b8e9f8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 -- 172.21.15.23:0/3338111575 learned_addr learned my addr 172.21.15.23:0/3338111575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe622600700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 -- 172.21.15.23:0/3338111575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 msgr2=0x559b8e9f8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 0x559b8e9f8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.111+0000 7fe621c00700 1 -- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 msgr2=0x559b8e9ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe621c00700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe621c00700 1 -- 172.21.15.23:0/3338111575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe610009990 con 0x559b8e9ee850 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe621200700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 0x559b8e9f8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:46.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe621c00700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe610009890 tx=0x7fe610012df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1a5b8e687abbe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe603400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6100126e0 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b8e900400 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fe610004040 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe610004200 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe638134b80 10 client.15588 fetch_fsmap learned FSMap version 28 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.112+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559b8e76dbe0 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559b8e900400 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe610022070 con 0x559b8e9ee850 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 10 client.15588 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 20 client.15588 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea04300 0x559b8ea06720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 4 client.15588 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b8e938780 con 0x559b8ea04300 2024-03-30T04:01:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.113+0000 7fe621200700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea04300 0x559b8ea06720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.114+0000 7fe621200700 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea04300 0x559b8ea06720 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7ff7c72ccde61 server_cookie=619b0e4f0f263c33 in_seq=0 out_seq=0 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.114+0000 7fe603400700 10 client.15588 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe603400700 1 -- 172.21.15.23:0/3338111575 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559b8e938780 con 0x559b8ea04300 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe603400700 10 client.15588 handle_command_reply: tid=0 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 client.15588 shutdown 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 msgr2=0x559b8e9f8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe610009890 tx=0x7fe610012df0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 shutdown_connections 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f8690 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea04300 0x559b8ea06720 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 --2- 172.21.15.23:0/3338111575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ef1b0 0x559b8e9f8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 >> 172.21.15.23:0/3338111575 conn(0x559b8e94d720 msgr2=0x559b8e9ed1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe603400700 0 client.15588 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 shutdown_connections 2024-03-30T04:01:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 1 -- 172.21.15.23:0/3338111575 wait complete. 2024-03-30T04:01:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.115+0000 7fe638134b80 20 client.15588 trim_cache size 0 max 16384 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.133+0000 7fe638134b80 1 Processor -- start 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.133+0000 7fe638134b80 1 -- start start 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9ffbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8ea06310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8ea00dd0 con 0x559b8e9ee850 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8ea03e90 con 0x559b8e9ef1b0 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe638134b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e9ef8e0 con 0x559b8e9f0050 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40014/0 (socket says 172.21.15.23:40014) 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 -- 172.21.15.23:0/1802138814 learned_addr learned my addr 172.21.15.23:0/1802138814 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621200700 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9ffbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe622600700 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8ea06310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 -- 172.21.15.23:0/1802138814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 msgr2=0x559b8ea06310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8ea06310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 -- 172.21.15.23:0/1802138814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 msgr2=0x559b8e9ffbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9ffbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.134+0000 7fe621c00700 1 -- 172.21.15.23:0/1802138814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b8e905f20 con 0x559b8e9ee850 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.135+0000 7fe621c00700 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe6100092a0 tx=0x7fe610009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf94a726729e402 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.135+0000 7fe60f400700 1 -- 172.21.15.23:0/1802138814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61000f440 con 0x559b8e9ee850 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.135+0000 7fe60f400700 1 -- 172.21.15.23:0/1802138814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe61000f920 con 0x559b8e9ee850 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.135+0000 7fe60f400700 1 -- 172.21.15.23:0/1802138814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe610005570 con 0x559b8e9ee850 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- 172.21.15.23:0/1802138814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 msgr2=0x559b8e9f7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe6100092a0 tx=0x7fe610009900 comp rx=0 tx=0).stop 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- 172.21.15.23:0/1802138814 shutdown_connections 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9f0050 0x559b8ea06310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ee850 0x559b8e9f7ba0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 --2- 172.21.15.23:0/1802138814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9ef1b0 0x559b8e9ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- 172.21.15.23:0/1802138814 >> 172.21.15.23:0/1802138814 conn(0x559b8e94d720 msgr2=0x559b8e9f6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- 172.21.15.23:0/1802138814 shutdown_connections 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- 172.21.15.23:0/1802138814 wait complete. 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.136+0000 7fe638134b80 1 -- start start 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 Processor -- start 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 0x559b8e9fe2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e9feff0 con 0x559b8e9ef1b0 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8e9ff160 con 0x559b8e9f0050 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b8ea58110 con 0x559b8e9ee850 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe638134b80 10 client.0 _ll_register_callbacks cb 0x559b8e8e54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe621c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe621c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48234/0 (socket says 172.21.15.23:48234) 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe621c00700 1 -- 172.21.15.23:0/1988202833 learned_addr learned my addr 172.21.15.23:0/1988202833 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.137+0000 7fe621200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 0x559b8e9fe2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe622600700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 -- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 msgr2=0x559b8e9fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 -- 172.21.15.23:0/1988202833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 msgr2=0x559b8e9fe2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 0x559b8e9fe2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 -- 172.21.15.23:0/1988202833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b8e76dbe0 con 0x559b8e9ee850 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621200700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 0x559b8e9fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe621c00700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe61000f7a0 tx=0x7fe61000fd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81c850c5853eba42 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe603400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61003d070 con 0x559b8e9ee850 2024-03-30T04:01:46.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fe610000e40 con 0x559b8e9ee850 2024-03-30T04:01:46.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.138+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe610012910 con 0x559b8e9ee850 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe610005620 con 0x559b8e9ee850 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 10 client.25597 fetch_fsmap learned FSMap version 28 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559b8e7c5d50 con 0x559b8e9ee850 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe610012910 con 0x559b8e9ee850 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe6100117e0 con 0x559b8e9ee850 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 10 client.25597 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 10 client.25597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 20 client.25597 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea59650 0x559b8ea5ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 4 client.25597 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.139+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559b8e78ed70 con 0x559b8ea59650 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.140+0000 7fe621200700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea59650 0x559b8ea5ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.140+0000 7fe621200700 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea59650 0x559b8ea5ba40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6b11ab0100086b server_cookie=8010bf4b16bcfb96 in_seq=0 out_seq=0 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.140+0000 7fe603400700 10 client.25597 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe603400700 1 -- 172.21.15.23:0/1988202833 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559b8e78ed70 con 0x559b8ea59650 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe603400700 10 client.25597 handle_command_reply: tid=0 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 client.25597 shutdown 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 msgr2=0x559b8e9ffb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe61000f7a0 tx=0x7fe61000fd90 comp rx=0 tx=0).stop 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 shutdown_connections 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559b8e9f0050 0x559b8e9fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559b8ea59650 0x559b8ea5ba40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559b8e9ee850 0x559b8e9ffb00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 --2- 172.21.15.23:0/1988202833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b8e9ef1b0 0x559b8e9fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe603400700 0 client.25597 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 >> 172.21.15.23:0/1988202833 conn(0x559b8e94d720 msgr2=0x559b8ea06310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 shutdown_connections 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.142+0000 7fe638134b80 1 -- 172.21.15.23:0/1988202833 wait complete. 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe638134b80 20 client.25597 trim_cache size 0 max 16384 2024-03-30T04:01:46.144 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:46.145 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:01:46.145 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "49a6ee14-0015-4150-a6b1-8ed65fc73c12", 2024-03-30T04:01:46.145 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:01:46.145 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 msgr2=0x7fe6040a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 0x7fe6040a36f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x559b8e9ef830 tx=0x7fe614009920 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 msgr2=0x7fe61c0731a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe60800c010 tx=0x7fe60800f690 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 shutdown_connections 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe61c102910 0x7fe61c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe61c100d80 0x7fe61c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fe6040a1230 0x7fe6040a36f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 --2- 172.21.15.23:0/305274696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c101f70 0x7fe61c0731a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 >> 172.21.15.23:0/305274696 conn(0x7fe61c0fc2f0 msgr2=0x7fe61c105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 shutdown_connections 2024-03-30T04:01:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.143+0000 7fe624600700 1 -- 172.21.15.23:0/305274696 wait complete. 2024-03-30T04:01:46.160 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:01:46.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 -- 172.21.15.23:0/982525095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 msgr2=0x7f9798073c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 --2- 172.21.15.23:0/982525095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f9798073c90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97840098f0 tx=0x7f9784009c00 comp rx=0 tx=0).stop 2024-03-30T04:01:46.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 -- 172.21.15.23:0/982525095 shutdown_connections 2024-03-30T04:01:46.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 --2- 172.21.15.23:0/982525095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f979810d350 0x7f979810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 --2- 172.21.15.23:0/982525095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9798074df0 0x7f9798073240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 --2- 172.21.15.23:0/982525095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f9798073c90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 -- 172.21.15.23:0/982525095 >> 172.21.15.23:0/982525095 conn(0x7f97980fc2f0 msgr2=0x7f97980fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 -- 172.21.15.23:0/982525095 shutdown_connections 2024-03-30T04:01:46.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.363+0000 7f979d200700 1 -- 172.21.15.23:0/982525095 wait complete. 2024-03-30T04:01:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.364+0000 7f979d200700 1 Processor -- start 2024-03-30T04:01:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.364+0000 7f979d200700 1 -- start start 2024-03-30T04:01:46.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9798074df0 0x7f979819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 0x7f97981a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981a4ee0 con 0x7f9798073810 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981a5020 con 0x7f979810d350 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f979d200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981a51b0 con 0x7f9798074df0 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f9795e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f9795e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40034/0 (socket says 172.21.15.23:40034) 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f9795e00700 1 -- 172.21.15.23:0/368793235 learned_addr learned my addr 172.21.15.23:0/368793235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f9796800700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 0x7f97981a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.365+0000 7f9795400700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9798074df0 0x7f979819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 -- 172.21.15.23:0/368793235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9798074df0 msgr2=0x7f979819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9798074df0 0x7f979819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 -- 172.21.15.23:0/368793235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 msgr2=0x7f97981a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 0x7f97981a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 -- 172.21.15.23:0/368793235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97880095f0 con 0x7f9798073810 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9796800700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 0x7f97981a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f9795e00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f978800ec10 tx=0x7f978800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f978ea00700 1 -- 172.21.15.23:0/368793235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978800cd40 con 0x7f9798073810 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9784009520 con 0x7f9798073810 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f978ea00700 1 -- 172.21.15.23:0/368793235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97880044e0 con 0x7f9798073810 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f978ea00700 1 -- 172.21.15.23:0/368793235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9788010690 con 0x7f9798073810 2024-03-30T04:01:46.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.366+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97981a5860 con 0x7f9798073810 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.370+0000 7f97b0dcdb80 1 Processor -- start 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 -- start start 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d2e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907c84990 con 0x560907d2d130 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907c82350 con 0x560907d2dfd0 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907c7dc40 con 0x560907d2c7d0 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f9796800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f978ea00700 1 -- 172.21.15.23:0/368793235 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 135916+0+0 (secure 0 0 0) 0x7f9788003710 con 0x7f9798073810 2024-03-30T04:01:46.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f9796800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48266/0 (socket says 172.21.15.23:48266) 2024-03-30T04:01:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f9796800700 1 -- 172.21.15.23:0/742481162 learned_addr learned my addr 172.21.15.23:0/742481162 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.371+0000 7f9795400700 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d2e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 -- 172.21.15.23:0/742481162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 msgr2=0x560907d2e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d2e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 -- 172.21.15.23:0/742481162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 msgr2=0x560907d2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 -- 172.21.15.23:0/742481162 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560907aff3b0 con 0x560907d2c7d0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9796800700 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9790009440 tx=0x7f9790009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83748636a05f01a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9783400700 1 -- 172.21.15.23:0/742481162 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979000e070 con 0x560907d2c7d0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9783400700 1 -- 172.21.15.23:0/742481162 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f979000f040 con 0x560907d2c7d0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.372+0000 7f9783400700 1 -- 172.21.15.23:0/742481162 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9790015c90 con 0x560907d2c7d0 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f978ea00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 0x7f977c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/742481162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 msgr2=0x560907d2cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9790009440 tx=0x7f9790009920 comp rx=0 tx=0).stop 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f978ea00700 1 -- 172.21.15.23:0/368793235 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f9788014070 con 0x7f9798073810 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/742481162 shutdown_connections 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d2e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2cbf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/742481162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/742481162 >> 172.21.15.23:0/742481162 conn(0x560907c8b6a0 msgr2=0x560907d279a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.373+0000 7f9795e00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 0x7f977c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.374+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/742481162 shutdown_connections 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.374+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/742481162 wait complete. 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.374+0000 7f97b0dcdb80 1 -- start start 2024-03-30T04:01:46.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.374+0000 7f9795e00700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 0x7f977c0a3670 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9788096d60 tx=0x7f978800ef90 comp rx=0 tx=0).ready entity=mgr.14154 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 Processor -- start 2024-03-30T04:01:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2c7d0 0x560907d366a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907c19d80 con 0x560907d2dfd0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907c19a20 con 0x560907d2c7d0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907b016a0 con 0x560907d2d130 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f97b0dcdb80 10 client.0 _ll_register_callbacks cb 0x560907c23470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f9796800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2c7d0 0x560907d366a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f9795400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f9795400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48280/0 (socket says 172.21.15.23:48280) 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f9796800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.375+0000 7f9795400700 1 -- 172.21.15.23:0/953016317 learned_addr learned my addr 172.21.15.23:0/953016317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 -- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2c7d0 msgr2=0x560907d366a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2c7d0 0x560907d366a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 -- 172.21.15.23:0/953016317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 msgr2=0x560907d3dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 -- 172.21.15.23:0/953016317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97900090f0 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9795400700 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9798074b10 tx=0x7f978400b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6f8ab9283b14ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.376+0000 7f9777e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784005040 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560907c3e380 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f97840051a0 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9784005360 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f97b0dcdb80 10 client.25609 fetch_fsmap learned FSMap version 28 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560907aabb60 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.377+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560907c3e380 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9784015020 con 0x560907d2d130 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 10 client.25609 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 10 client.25609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 20 client.25609 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d424c0 0x560907d448e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 4 client.25609 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:01:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560907c76700 con 0x560907d424c0 2024-03-30T04:01:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.378+0000 7f9796800700 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d424c0 0x560907d448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f9796800700 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d424c0 0x560907d448e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7dd1176a60d77d server_cookie=7d8366ff05b20bb3 in_seq=0 out_seq=0 2024-03-30T04:01:46.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f9777e00700 10 client.25609 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f9777e00700 1 -- 172.21.15.23:0/953016317 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560907c76700 con 0x560907d424c0 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f9777e00700 10 client.25609 handle_command_reply: tid=0 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f97b0dcdb80 1 client.25609 shutdown 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 msgr2=0x560907d36be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.379+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9798074b10 tx=0x7f978400b790 comp rx=0 tx=0).stop 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 shutdown_connections 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2c7d0 0x560907d366a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d424c0 0x560907d448e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/953016317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2d130 0x560907d36be0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f9777e00700 0 client.25609 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 >> 172.21.15.23:0/953016317 conn(0x560907c8b6a0 msgr2=0x560907d2b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 shutdown_connections 2024-03-30T04:01:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/953016317 wait complete. 2024-03-30T04:01:46.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.380+0000 7f97b0dcdb80 20 client.25609 trim_cache size 0 max 16384 2024-03-30T04:01:46.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 Processor -- start 2024-03-30T04:01:46.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 -- start start 2024-03-30T04:01:46.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d34540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d3eda0 con 0x560907d2d130 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d36210 con 0x560907d2dfd0 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d28660 con 0x560907d2c7d0 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f9795400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f9795400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40062/0 (socket says 172.21.15.23:40062) 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.398+0000 7f9795400700 1 -- 172.21.15.23:0/3483014435 learned_addr learned my addr 172.21.15.23:0/3483014435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9796800700 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9796800700 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d34540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f266c112b50 con 0x7f266c073810 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 -- 172.21.15.23:0/3483014435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 msgr2=0x560907d2e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 -- 172.21.15.23:0/3483014435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 msgr2=0x560907d34540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d34540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 -- 172.21.15.23:0/3483014435 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560907c43ea0 con 0x560907d2d130 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.399+0000 7f9795400700 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f97840224e0 tx=0x7f9784009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8842b9598884a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.400+0000 7f9783e00700 1 -- 172.21.15.23:0/3483014435 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784005b40 con 0x560907d2d130 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.400+0000 7f9783e00700 1 -- 172.21.15.23:0/3483014435 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f978401d030 con 0x560907d2d130 2024-03-30T04:01:46.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.400+0000 7f9783e00700 1 -- 172.21.15.23:0/3483014435 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784022840 con 0x560907d2d130 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.400+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/3483014435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 msgr2=0x560907d3dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.400+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f97840224e0 tx=0x7f9784009c00 comp rx=0 tx=0).stop 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/3483014435 shutdown_connections 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2dfd0 0x560907d34540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d2e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/3483014435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2d130 0x560907d3dbe0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/3483014435 >> 172.21.15.23:0/3483014435 conn(0x560907c8b6a0 msgr2=0x560907d205a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/3483014435 shutdown_connections 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/3483014435 wait complete. 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 -- start start 2024-03-30T04:01:46.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.401+0000 7f97b0dcdb80 1 Processor -- start 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d3d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 0x560907d3bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d3c920 con 0x560907d2dfd0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d3ca90 con 0x560907d2d130 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560907d3cc50 con 0x560907d2c7d0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f97b0dcdb80 10 client.0 _ll_register_callbacks cb 0x560907c23470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f9796800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d3d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f9796800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f9796800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40078/0 (socket says 172.21.15.23:40078) 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f9796800700 1 -- 172.21.15.23:0/217562967 learned_addr learned my addr 172.21.15.23:0/217562967 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.402+0000 7f9795400700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 0x560907d3bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 -- 172.21.15.23:0/217562967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 msgr2=0x560907d3d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d3d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 -- 172.21.15.23:0/217562967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 msgr2=0x560907d3c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 -- 172.21.15.23:0/217562967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560907aabb60 con 0x560907d2d130 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9796800700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9795400700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 0x560907d3bea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9784005a80 tx=0x7f97840057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e6eb81f943b1120 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9777e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784022600 con 0x560907d2d130 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f979000f040 con 0x560907d2d130 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.403+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f9784000e40 con 0x560907d2d130 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9784005110 con 0x560907d2d130 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 10 client.25625 fetch_fsmap learned FSMap version 28 2024-03-30T04:01:46.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560907b03cd0 con 0x560907d2d130 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f979000f040 con 0x560907d2d130 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9784009c70 con 0x560907d2d130 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 10 client.25625 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 10 client.25625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 20 client.25625 populate_metadata read hostname 'smithi023' 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d97610 0x560907d99a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 4 client.25625 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560907acccf0 con 0x560907d97610 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.404+0000 7f9796800700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d97610 0x560907d99a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f9796800700 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d97610 0x560907d99a30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5be7a6889b2fa379 server_cookie=fe807ada500529da in_seq=0 out_seq=0 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f9777e00700 10 client.25625 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f9777e00700 1 -- 172.21.15.23:0/217562967 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560907acccf0 con 0x560907d97610 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f9777e00700 10 client.25625 handle_command_reply: tid=0 2024-03-30T04:01:46.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 client.25625 shutdown 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 msgr2=0x560907d3bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 0x560907d3bea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9784005a80 tx=0x7f97840057c0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 shutdown_connections 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560907d2dfd0 0x560907d3c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560907d97610 0x560907d99a30 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560907d2c7d0 0x560907d3d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 --2- 172.21.15.23:0/217562967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560907d2d130 0x560907d3bea0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 >> 172.21.15.23:0/217562967 conn(0x560907c8b6a0 msgr2=0x560907d344e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.405+0000 7f9777e00700 0 client.25625 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 shutdown_connections 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f97b0dcdb80 1 -- 172.21.15.23:0/217562967 wait complete. 2024-03-30T04:01:46.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f97b0dcdb80 20 client.25625 trim_cache size 0 max 16384 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "49a6ee14-0015-4150-a6b1-8ed65fc73c12": { 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "49a6ee14-0015-4150-a6b1-8ed65fc73c12", 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:01:46.409 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 msgr2=0x7f977c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 0x7f977c0a3670 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9788096d60 tx=0x7f978800ef90 comp rx=0 tx=0).stop 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 msgr2=0x7f979819d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f978800ec10 tx=0x7f978800ef20 comp rx=0 tx=0).stop 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 shutdown_connections 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f979810d350 0x7f97981a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9798073810 0x7f979819d3b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f977c0a11b0 0x7f977c0a3670 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 --2- 172.21.15.23:0/368793235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9798074df0 0x7f979819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.406+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 >> 172.21.15.23:0/368793235 conn(0x7f97980fc2f0 msgr2=0x7f97981006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.407+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 shutdown_connections 2024-03-30T04:01:46.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.407+0000 7f979d200700 1 -- 172.21.15.23:0/368793235 wait complete. 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3982327754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: pgmap v303: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-30T04:01:46.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3280806265' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:46 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.423 INFO:tasks.cephfs.filesystem:scrub status for tag:49a6ee14-0015-4150-a6b1-8ed65fc73c12 - {'path': '/', 'tag': '49a6ee14-0015-4150-a6b1-8ed65fc73c12', 'options': 'recursive,force'} 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3982327754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: pgmap v303: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-30T04:01:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3280806265' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:46 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3982327754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: pgmap v303: 129 pgs: 129 active+clean; 654 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3280806265' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:46 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-30T04:01:46.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.978+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 20) v1 ==== 135933+0+0 (secure 0 0 0) 0x7f265c097890 con 0x7f266c073810 2024-03-30T04:01:46.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.979+0000 7f2662a00700 1 -- 172.21.15.23:0/3280806265 <== mon.1 v2:172.21.15.193:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v20) v1 ==== 92+0+0 (secure 0 0 0) 0x7f265c0cf460 con 0x7f266c073810 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 msgr2=0x7f26500a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 0x7f26500a36f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f266c074b10 tx=0x7f2658009260 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 msgr2=0x7f266c19d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f265c00b680 tx=0x7f265c009580 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 shutdown_connections 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f266c10d2e0 0x7f266c1a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f26500a1230 0x7f26500a36f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f266c073810 0x7f266c19d270 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 --2- 172.21.15.23:0/3280806265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f266c074df0 0x7f266c19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 >> 172.21.15.23:0/3280806265 conn(0x7f266c0fc280 msgr2=0x7f266c100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 shutdown_connections 2024-03-30T04:01:46.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:46.983+0000 7f2670c00700 1 -- 172.21.15.23:0/3280806265 wait complete. 2024-03-30T04:01:47.002 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 -- 172.21.15.23:0/2974967850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 msgr2=0x7fa48c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 --2- 172.21.15.23:0/2974967850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c100020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0099c0 tx=0x7fa47c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 -- 172.21.15.23:0/2974967850 shutdown_connections 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 --2- 172.21.15.23:0/2974967850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 0x7fa48c108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 --2- 172.21.15.23:0/2974967850 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c100020 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 --2- 172.21.15.23:0/2974967850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 -- 172.21.15.23:0/2974967850 >> 172.21.15.23:0/2974967850 conn(0x7fa48c0fb170 msgr2=0x7fa48c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 -- 172.21.15.23:0/2974967850 shutdown_connections 2024-03-30T04:01:47.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.216+0000 7fa493200700 1 -- 172.21.15.23:0/2974967850 wait complete. 2024-03-30T04:01:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.217+0000 7fa493200700 1 Processor -- start 2024-03-30T04:01:47.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 -- start start 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 0x7fa48c1a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a0900 con 0x7fa48c0ffc00 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a0a70 con 0x7fa48c101790 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa493200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c1a0c30 con 0x7fa48c100df0 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa48b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa48be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa491200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 0x7fa48c1a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa48b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48312/0 (socket says 172.21.15.23:48312) 2024-03-30T04:01:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.218+0000 7fa48b400700 1 -- 172.21.15.23:0/613070826 learned_addr learned my addr 172.21.15.23:0/613070826 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 msgr2=0x7fa48c1a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 0x7fa48c1a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 msgr2=0x7fa48c198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 -- 172.21.15.23:0/613070826 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0095f0 con 0x7fa48c100df0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa48b400700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9a0 tx=0x7fa47800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478009790 con 0x7fa48c100df0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c1a0f10 con 0x7fa48c100df0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa478010430 con 0x7fa48c100df0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa478018680 con 0x7fa48c100df0 2024-03-30T04:01:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.219+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c1a1330 con 0x7fa48c100df0 2024-03-30T04:01:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.220+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c067c30 con 0x7fa48c100df0 2024-03-30T04:01:47.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.223+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 135933+0+0 (secure 0 0 0) 0x7fa4780105a0 con 0x7fa48c100df0 2024-03-30T04:01:47.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.224+0000 7fa488c00700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 0x7fa4700a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.224+0000 7fa48be00700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 msgr2=0x7fa4700a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:47.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.224+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fa478014070 con 0x7fa48c100df0 2024-03-30T04:01:47.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.224+0000 7fa48be00700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 0x7fa4700a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:01:47.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.230+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa478096480 con 0x7fa48c100df0 2024-03-30T04:01:47.271 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setuser ceph since I am not root 2024-03-30T04:01:47.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: ignoring --setgroup ceph since I am not root 2024-03-30T04:01:47.415 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: ignoring --setuser ceph since I am not root 2024-03-30T04:01:47.415 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: ignoring --setgroup ceph since I am not root 2024-03-30T04:01:47.415 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-mgr[80658]: -- 172.21.15.154:0/2840558655 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dd972be1a0 con 0x55dd97294800 2024-03-30T04:01:47.417 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: ignoring --setuser ceph since I am not root 2024-03-30T04:01:47.417 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: ignoring --setgroup ceph since I am not root 2024-03-30T04:01:47.417 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-mgr[81468]: -- 172.21.15.193:0/2246559851 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2c5e981a0 con 0x55f2c5e6ec00 2024-03-30T04:01:47.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.425+0000 7fa48be00700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 msgr2=0x7fa4700a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:47.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.425+0000 7fa48be00700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 0x7fa4700a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:01:47.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.466+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fa48c10b410 con 0x7fa48c100df0 2024-03-30T04:01:47.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.474+0000 7fa488c00700 1 -- 172.21.15.23:0/613070826 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v30)=0 v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa4780260d0 con 0x7fa48c100df0 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 msgr2=0x7fa4700a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 0x7fa4700a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 msgr2=0x7fa48c199350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa47800e9a0 tx=0x7fa47800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 shutdown_connections 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa48c101790 0x7fa48c1a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa48c0ffc00 0x7fa48c198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fa4700a1200 0x7fa4700a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 --2- 172.21.15.23:0/613070826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa48c100df0 0x7fa48c199350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 >> 172.21.15.23:0/613070826 conn(0x7fa48c0fb170 msgr2=0x7fa48c0fcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.476+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 shutdown_connections 2024-03-30T04:01:47.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.477+0000 7fa493200700 1 -- 172.21.15.23:0/613070826 wait complete. 2024-03-30T04:01:47.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-30T04:01:47.680 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:47.415+0000 7fc769538200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T04:01:47.680 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:47.509+0000 7fc769538200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T04:01:47.680 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:47.596+0000 7fc769538200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T04:01:47.683 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:47.417+0000 7f2f416e4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T04:01:47.683 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:47.513+0000 7f2f416e4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T04:01:47.683 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:47.600+0000 7f2f416e4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T04:01:47.683 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:47.684+0000 7f2f416e4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 -- 172.21.15.23:0/726462320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 msgr2=0x7fb27c109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 --2- 172.21.15.23:0/726462320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c109160 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb27400b0a0 tx=0x7fb27400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 -- 172.21.15.23:0/726462320 shutdown_connections 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 --2- 172.21.15.23:0/726462320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c109160 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 --2- 172.21.15.23:0/726462320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 --2- 172.21.15.23:0/726462320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 0x7fb27c102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 -- 172.21.15.23:0/726462320 >> 172.21.15.23:0/726462320 conn(0x7fb27c0fc2b0 msgr2=0x7fb27c0fe6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 -- 172.21.15.23:0/726462320 shutdown_connections 2024-03-30T04:01:47.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.705+0000 7fb283c00700 1 -- 172.21.15.23:0/726462320 wait complete. 2024-03-30T04:01:47.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.706+0000 7fb283c00700 1 Processor -- start 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.706+0000 7fb283c00700 1 -- start start 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 0x7fb27c1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0ac0 con 0x7fb27c100d20 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0c00 con 0x7fb27c101f10 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb283c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27c1a0d90 con 0x7fb27c1028b0 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48336/0 (socket says 172.21.15.23:48336) 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 -- 172.21.15.23:0/2391935629 learned_addr learned my addr 172.21.15.23:0/2391935629 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c198f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40138/0 (socket says 172.21.15.23:40138) 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb27be00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 0x7fb27c1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:47.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 msgr2=0x7fb27c1994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 0x7fb27c1994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 msgr2=0x7fb27c198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.707+0000 7fb281c00700 1 -- 172.21.15.23:0/2391935629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb274009d00 con 0x7fb27c1028b0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb281c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb274007a90 tx=0x7fb27400bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27400e040 con 0x7fb27c1028b0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27c1a1010 con 0x7fb27c1028b0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb274004e60 con 0x7fb27c1028b0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740052a0 con 0x7fb27c1028b0 2024-03-30T04:01:47.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.708+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27c1a1440 con 0x7fb27c1028b0 2024-03-30T04:01:47.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.709+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27c067c30 con 0x7fb27c1028b0 2024-03-30T04:01:47.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.711+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 135933+0+0 (secure 0 0 0) 0x7fb27401d470 con 0x7fb27c1028b0 2024-03-30T04:01:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.712+0000 7fb279600700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 0x7fb2640a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.713+0000 7fb281200700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 msgr2=0x7fb2640a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.713+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7fb2740ce5d0 con 0x7fb27c1028b0 2024-03-30T04:01:47.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.713+0000 7fb281200700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 0x7fb2640a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:01:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.720+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb2740978b0 con 0x7fb27c1028b0 2024-03-30T04:01:47.751 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:47.622+0000 7fe20825f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T04:01:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.914+0000 7fb281200700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 msgr2=0x7fb2640a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.914+0000 7fb281200700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 0x7fb2640a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:01:47.931 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:47.680+0000 7fc769538200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T04:01:47.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.981+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fb27c10c7a0 con 0x7fb27c1028b0 2024-03-30T04:01:47.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.991+0000 7fb279600700 1 -- 172.21.15.23:0/2391935629 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v31)=0 v31) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb27401ae60 con 0x7fb27c1028b0 2024-03-30T04:01:47.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 msgr2=0x7fb2640a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 0x7fb2640a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 msgr2=0x7fb27c1a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb274007a90 tx=0x7fb27400bce0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 shutdown_connections 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7fb2640a1200 0x7fb2640a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb27c1028b0 0x7fb27c1a04f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb27c100d20 0x7fb27c198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 --2- 172.21.15.23:0/2391935629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb27c101f10 0x7fb27c1994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 >> 172.21.15.23:0/2391935629 conn(0x7fb27c0fc2b0 msgr2=0x7fb27c072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 shutdown_connections 2024-03-30T04:01:47.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:47.994+0000 7fb283c00700 1 -- 172.21.15.23:0/2391935629 wait complete. 2024-03-30T04:01:48.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-30T04:01:48.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-mon[38254]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-30T04:01:48.009 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' 2024-03-30T04:01:48.010 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:47.752+0000 7fe20825f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T04:01:48.010 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:47.872+0000 7fe20825f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T04:01:48.010 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:47.990+0000 7fe20825f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T04:01:48.012 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 -- 172.21.15.23:0/4157328008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 msgr2=0x7f6bd8103350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4157328008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8103350 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009990 tx=0x7f6bc8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 -- 172.21.15.23:0/4157328008 shutdown_connections 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4157328008 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd8104a40 0x7f6bd810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4157328008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8103350 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4157328008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8104080 0x7f6bd8104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 -- 172.21.15.23:0/4157328008 >> 172.21.15.23:0/4157328008 conn(0x7f6bd80fe480 msgr2=0x7f6bd81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:48.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 -- 172.21.15.23:0/4157328008 shutdown_connections 2024-03-30T04:01:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.215+0000 7f6bdee00700 1 -- 172.21.15.23:0/4157328008 wait complete. 2024-03-30T04:01:48.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.217+0000 7f6bdee00700 1 Processor -- start 2024-03-30T04:01:48.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.217+0000 7f6bdee00700 1 -- start start 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8198e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8104080 0x7f6bd81993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd8104a40 0x7f6bd81a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81a0990 con 0x7f6bd8104080 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81a0ad0 con 0x7f6bd8104a40 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdee00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd81a0c60 con 0x7f6bd8102f30 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8198e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8104080 0x7f6bd81993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8198e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59660/0 (socket says 172.21.15.23:59660) 2024-03-30T04:01:48.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 -- 172.21.15.23:0/4006325145 learned_addr learned my addr 172.21.15.23:0/4006325145 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bdce00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd8104a40 0x7f6bd81a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd8104a40 msgr2=0x7f6bd81a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd8104a40 0x7f6bd81a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8104080 msgr2=0x7f6bd81993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bd8104080 0x7f6bd81993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.218+0000 7f6bd7e00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80095f0 con 0x7f6bd8102f30 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bd7e00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8198e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b680 tx=0x7f6bc8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc801d070 con 0x7f6bd8102f30 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6bc8022470 con 0x7f6bd8102f30 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd81a0ee0 con 0x7f6bd8102f30 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8017650 con 0x7f6bd8102f30 2024-03-30T04:01:48.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.219+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81a12a0 con 0x7f6bd8102f30 2024-03-30T04:01:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.220+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd810a3a0 con 0x7f6bd8102f30 2024-03-30T04:01:48.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.222+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 135933+0+0 (secure 0 0 0) 0x7f6bc80225e0 con 0x7f6bd8102f30 2024-03-30T04:01:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.223+0000 7f6bd4c00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.223+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9123+0+0 (secure 0 0 0) 0x7f6bc80ce120 con 0x7f6bd8102f30 2024-03-30T04:01:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.223+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:48.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.223+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-30T04:01:48.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.227+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6bc8097400 con 0x7f6bd8102f30 2024-03-30T04:01:48.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-30T04:01:48.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-mon[79581]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-30T04:01:48.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:47 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' 2024-03-30T04:01:48.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-30T04:01:48.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-mon[79161]: mgrmap e20: x(active, since 7m), standbys: z, y 2024-03-30T04:01:48.313 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:47 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' 2024-03-30T04:01:48.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.424+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:48.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.424+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-30T04:01:48.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.480+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bd802cd00 con 0x7f6bbc0a1200 2024-03-30T04:01:48.571 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:48.317+0000 7f2f416e4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:48.571 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:48.313+0000 7fc769538200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:48.572 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:48.572+0000 7fc769538200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T04:01:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.825+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:48.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:48.825+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-30T04:01:48.921 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:48.571+0000 7f2f416e4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T04:01:48.921 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:48.657+0000 7f2f416e4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T04:01:48.921 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:48.764+0000 7f2f416e4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:48.924 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:48.658+0000 7fc769538200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T04:01:48.924 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:48.766+0000 7fc769538200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:49.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' 2024-03-30T04:01:49.182 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:48.925+0000 7fc769538200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T04:01:49.182 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:49.055+0000 7fc769538200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T04:01:49.182 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:49.141+0000 7fc769538200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T04:01:49.195 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:48 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' 2024-03-30T04:01:49.195 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:48.867+0000 7fe20825f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:49.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' 2024-03-30T04:01:49.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:48.922+0000 7f2f416e4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T04:01:49.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:49.050+0000 7f2f416e4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T04:01:49.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:49.136+0000 7f2f416e4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T04:01:49.457 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.196+0000 7fe20825f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T04:01:49.457 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.312+0000 7fe20825f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T04:01:49.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:49.626+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:49.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:49.626+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-30T04:01:49.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.458+0000 7fe20825f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:49.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.660+0000 7fe20825f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T04:01:49.889 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:49.504+0000 7f2f416e4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T04:01:49.901 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:49.514+0000 7fc769538200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T04:01:50.176 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:49.889+0000 7f2f416e4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T04:01:50.176 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:50.083+0000 7f2f416e4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T04:01:50.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:49.902+0000 7fc769538200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T04:01:50.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:50.097+0000 7fc769538200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T04:01:50.271 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.835+0000 7fe20825f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T04:01:50.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:49.951+0000 7fe20825f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T04:01:50.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:50.176+0000 7f2f416e4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T04:01:50.681 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:50.191+0000 7fc769538200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T04:01:50.771 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:50.435+0000 7fe20825f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T04:01:51.205 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:50.942+0000 7fe20825f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T04:01:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:51.227+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:51.227+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-30T04:01:51.242 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:50.936+0000 7f2f416e4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T04:01:51.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:51.033+0000 7f2f416e4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T04:01:51.243 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:51.181+0000 7f2f416e4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T04:01:51.431 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:50.955+0000 7fc769538200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T04:01:51.431 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:51.054+0000 7fc769538200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T04:01:51.431 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:51.204+0000 7fc769538200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T04:01:51.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:51.206+0000 7fe20825f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-30T04:01:51.522 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:51.335+0000 7fe20825f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T04:01:52.177 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:51.803+0000 7f2f416e4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T04:01:52.177 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:51.984+0000 7f2f416e4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T04:01:52.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:51.836+0000 7fc769538200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T04:01:52.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:52.018+0000 7fc769538200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T04:01:52.492 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:52.177+0000 7f2f416e4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T04:01:52.672 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:52.336+0000 7fe20825f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T04:01:52.672 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:52.468+0000 7fe20825f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T04:01:52.681 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:52.212+0000 7fc769538200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T04:01:53.021 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:52.673+0000 7fe20825f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T04:01:53.391 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:53.108+0000 7f2f416e4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T04:01:53.431 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:53.152+0000 7fc769538200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T04:01:53.668 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:53.392+0000 7f2f416e4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:53.668 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:53.480+0000 7f2f416e4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:53.716 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:53.438+0000 7fc769538200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:53.716 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:53.527+0000 7fc769538200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:53.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:53.745+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 21) v1 ==== 135933+0+0 (secure 0 0 0) 0x7f6bc80977d0 con 0x7f6bd8102f30 2024-03-30T04:01:53.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: Standby manager daemon z restarted 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: Standby manager daemon z started 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: Standby manager daemon y restarted 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: Standby manager daemon y started 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:53.772 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:53.520+0000 7fe20825f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-30T04:01:53.993 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z[81445]: 2024-03-30T04:01:53.668+0000 7f2f416e4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T04:01:53.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: Standby manager daemon z restarted 2024-03-30T04:01:53.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: Standby manager daemon z started 2024-03-30T04:01:53.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: Standby manager daemon y restarted 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: Standby manager daemon y started 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T04:01:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:53 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:54.042 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:53.774+0000 7fe20825f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T04:01:54.181 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y[80635]: 2024-03-30T04:01:53.718+0000 7fc769538200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T04:01:54.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: Standby manager daemon z restarted 2024-03-30T04:01:54.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: Standby manager daemon z started 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.193:0/1098757666' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: Standby manager daemon y restarted 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: Standby manager daemon y started 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-30T04:01:54.182 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:53 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-30T04:01:54.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:54.430+0000 7f6bd7400700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/2266065890 2024-03-30T04:01:54.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:54.430+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-30T04:01:54.521 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:54.043+0000 7fe20825f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T04:01:55.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:54 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T04:01:55.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:54 smithi023 ceph-mon[38254]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:55.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:54 smithi023 ceph-mon[38254]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-30T04:01:55.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:54 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T04:01:55.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:54 smithi154 ceph-mon[79161]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:55.181 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:54 smithi154 ceph-mon[79161]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-30T04:01:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:54 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-30T04:01:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:54 smithi193 ceph-mon[79581]: from='mgr.? 172.21.15.154:0/839849500' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-30T04:01:55.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:54 smithi193 ceph-mon[79581]: mgrmap e21: x(active, since 7m), standbys: z, y 2024-03-30T04:01:55.521 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:55.247+0000 7fe20825f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T04:01:56.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:55.643+0000 7fe20825f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T04:01:56.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:55.769+0000 7fe20825f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T04:01:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:56.062+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 8 ==== mgrmap(e 22) v1 ==== 135550+0+0 (secure 0 0 0) 0x7f6bc80cb940 con 0x7f6bd8102f30 2024-03-30T04:01:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:56.063+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 msgr2=0x7f6bbc0a36c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-30T04:01:56.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:56.063+0000 7f6bd4c00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/2266065890,v1:172.21.15.23:6801/2266065890] conn(0x7f6bbc0a1200 0x7f6bbc0a36c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:56.371 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.043+0000 7fe20825f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.370+0000 7fe15ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.370+0000 7fe15ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.370+0000 7fe15ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.370+0000 7fe15ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.370+0000 7fe15ea00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.371+0000 7fe159000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.371+0000 7fe159000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.371+0000 7fe159000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.371+0000 7fe159000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:56.372 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:01:56.371+0000 7fe159000700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:01:57.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.066+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mon.1 v2:172.21.15.193:3300/0 9 ==== mgrmap(e 23) v1 ==== 135677+0+0 (secure 0 0 0) 0x7f6bc80228e0 con 0x7f6bd8102f30 2024-03-30T04:01:57.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.067+0000 7f6bd4c00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6bbc01b7d0 0x7f6bbc01bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.067+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bd802cd00 con 0x7f6bbc01b7d0 2024-03-30T04:01:57.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.070+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6bbc01b7d0 0x7f6bbc01bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.075+0000 7f6bd7400700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6bbc01b7d0 0x7f6bbc01bbb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009880 tx=0x7f6bc4009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:57.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.156+0000 7f6bd4c00700 1 -- 172.21.15.23:0/4006325145 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6bd802cd00 con 0x7f6bbc01b7d0 2024-03-30T04:01:57.155 INFO:teuthology.orchestra.run.smithi023.stdout:Schedule set for path / 2024-03-30T04:01:57.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6bbc01b7d0 msgr2=0x7f6bbc01bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6bbc01b7d0 0x7f6bbc01bbb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009880 tx=0x7f6bc4009260 comp rx=0 tx=0).stop 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 msgr2=0x7f6bd8198e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 --2- 172.21.15.23:0/4006325145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bd8102f30 0x7f6bd8198e60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800b680 tx=0x7f6bc8009580 comp rx=0 tx=0).stop 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdce00700 1 -- 172.21.15.23:0/4006325145 reap_dead start 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 shutdown_connections 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 >> 172.21.15.23:0/4006325145 conn(0x7f6bd80fe480 msgr2=0x7f6bd81074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 shutdown_connections 2024-03-30T04:01:57.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.160+0000 7f6bdee00700 1 -- 172.21.15.23:0/4006325145 wait complete. 2024-03-30T04:01:57.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-30T04:01:57.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: Active manager daemon x restarted 2024-03-30T04:01:57.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: Activating manager daemon x 2024-03-30T04:01:57.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: osdmap e89: 12 total, 12 up, 12 in 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: mgrmap e22: x(active, starting, since 0.0147592s), standbys: z, y 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:01:57.243 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:01:57.244 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: Manager daemon x is now available 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:01:57.245 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:56 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: Active manager daemon x restarted 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: Activating manager daemon x 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: osdmap e89: 12 total, 12 up, 12 in 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: mgrmap e22: x(active, starting, since 0.0147592s), standbys: z, y 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T04:01:57.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:01:57.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T04:01:57.248 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: Manager daemon x is now available 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:01:57.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:56 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T04:01:57.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 -- 172.21.15.23:0/3852747010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 msgr2=0x7ff3b0073240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3852747010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0073240 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0009990 tx=0x7ff3a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 -- 172.21.15.23:0/3852747010 shutdown_connections 2024-03-30T04:01:57.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3852747010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0108bd0 0x7ff3b010b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3852747010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0073240 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3852747010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0073810 0x7ff3b0073c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 -- 172.21.15.23:0/3852747010 >> 172.21.15.23:0/3852747010 conn(0x7ff3b00fc280 msgr2=0x7ff3b00fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:57.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 -- 172.21.15.23:0/3852747010 shutdown_connections 2024-03-30T04:01:57.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.391+0000 7ff3b5400700 1 -- 172.21.15.23:0/3852747010 wait complete. 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.392+0000 7ff3b5400700 1 Processor -- start 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 -- start start 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0108bd0 0x7ff3b01a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01a09e0 con 0x7ff3b0108bd0 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01a0b20 con 0x7ff3b0073810 2024-03-30T04:01:57.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3b5400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b01a0cb0 con 0x7ff3b0074df0 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3ae800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3ae800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43942/0 (socket says 172.21.15.23:43942) 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3ae800700 1 -- 172.21.15.23:0/3302322298 learned_addr learned my addr 172.21.15.23:0/3302322298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3ade00700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.393+0000 7ff3af200700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0108bd0 0x7ff3b01a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 -- 172.21.15.23:0/3302322298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 msgr2=0x7ff3b0199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 -- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0108bd0 msgr2=0x7ff3b01a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0108bd0 0x7ff3b01a0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 -- 172.21.15.23:0/3302322298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c009520 con 0x7ff3b0073810 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3ae800700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a000f660 con 0x7ff3b0073810 2024-03-30T04:01:57.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a00095f0 con 0x7ff3b0073810 2024-03-30T04:01:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff3a000fc80 con 0x7ff3b0073810 2024-03-30T04:01:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a002ac00 con 0x7ff3b0073810 2024-03-30T04:01:57.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.394+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01009b0 con 0x7ff3b0073810 2024-03-30T04:01:57.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.395+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b0107990 con 0x7ff3b0073810 2024-03-30T04:01:57.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.397+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 23) v1 ==== 135677+0+0 (secure 0 0 0) 0x7ff3a002a3f0 con 0x7ff3b0073810 2024-03-30T04:01:57.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.398+0000 7ff3a7400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 0x7ff3940a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.399+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff3a00d6dd0 con 0x7ff3b0073810 2024-03-30T04:01:57.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.399+0000 7ff3ade00700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 0x7ff3940a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.399+0000 7ff3ade00700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 0x7ff3940a33e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0074b10 tx=0x7ff39c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:57.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.402+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff3a00a0420 con 0x7ff3b0073810 2024-03-30T04:01:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: Active manager daemon x restarted 2024-03-30T04:01:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: Activating manager daemon x 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: osdmap e89: 12 total, 12 up, 12 in 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: mgrmap e22: x(active, starting, since 0.0147592s), standbys: z, y 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:01:57.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-30T04:01:57.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: Manager daemon x is now available 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-30T04:01:57.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:56 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-30T04:01:57.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.678+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3b0061290 con 0x7ff3940a0f20 2024-03-30T04:01:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.746+0000 7ff3a7400700 1 -- 172.21.15.23:0/3302322298 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff3b0061290 con 0x7ff3940a0f20 2024-03-30T04:01:57.745 INFO:teuthology.orchestra.run.smithi023.stdout:Retention added to path / 2024-03-30T04:01:57.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 msgr2=0x7ff3940a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 0x7ff3940a33e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0074b10 tx=0x7ff39c009260 comp rx=0 tx=0).stop 2024-03-30T04:01:57.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 msgr2=0x7ff3b0198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff3a0000c00 tx=0x7ff3a00057c0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 shutdown_connections 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff3b0108bd0 0x7ff3b01a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3940a0f20 0x7ff3940a33e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3b0073810 0x7ff3b0198f00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 --2- 172.21.15.23:0/3302322298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3b0074df0 0x7ff3b0199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 >> 172.21.15.23:0/3302322298 conn(0x7ff3b00fc280 msgr2=0x7ff3b0104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.749+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 shutdown_connections 2024-03-30T04:01:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.750+0000 7ff3b5400700 1 -- 172.21.15.23:0/3302322298 wait complete. 2024-03-30T04:01:57.767 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-30T04:01:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 -- 172.21.15.23:0/2991599049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 msgr2=0x7fc71c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 --2- 172.21.15.23:0/2991599049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c101250 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc7080099c0 tx=0x7fc708009cd0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 -- 172.21.15.23:0/2991599049 shutdown_connections 2024-03-30T04:01:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 --2- 172.21.15.23:0/2991599049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c101790 0x7fc71c108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 --2- 172.21.15.23:0/2991599049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c0ffc00 0x7fc71c100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 --2- 172.21.15.23:0/2991599049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c101250 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 -- 172.21.15.23:0/2991599049 >> 172.21.15.23:0/2991599049 conn(0x7fc71c0fb170 msgr2=0x7fc71c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:57.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 -- 172.21.15.23:0/2991599049 shutdown_connections 2024-03-30T04:01:57.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.977+0000 7fc722600700 1 -- 172.21.15.23:0/2991599049 wait complete. 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.978+0000 7fc722600700 1 Processor -- start 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.978+0000 7fc722600700 1 -- start start 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c101790 0x7fc71c1a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1a0a60 con 0x7fc71c100df0 2024-03-30T04:01:57.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1a0ba0 con 0x7fc71c101790 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc722600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1a0d30 con 0x7fc71c0ffc00 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc71b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc71b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59816/0 (socket says 172.21.15.23:59816) 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc71b400700 1 -- 172.21.15.23:0/2257503538 learned_addr learned my addr 172.21.15.23:0/2257503538 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc71be00700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c101790 0x7fc71c1a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.979+0000 7fc71aa00700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 -- 172.21.15.23:0/2257503538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c101790 msgr2=0x7fc71c1a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c101790 0x7fc71c1a0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 -- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 msgr2=0x7fc71c199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 -- 172.21.15.23:0/2257503538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7080095f0 con 0x7fc71c0ffc00 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc71b400700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e970 tx=0x7fc70c00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c009790 con 0x7fc71c0ffc00 2024-03-30T04:01:57.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc70c00c7e0 con 0x7fc71c0ffc00 2024-03-30T04:01:57.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c1a0fb0 con 0x7fc71c0ffc00 2024-03-30T04:01:57.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c003c70 con 0x7fc71c0ffc00 2024-03-30T04:01:57.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.980+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c1a1430 con 0x7fc71c0ffc00 2024-03-30T04:01:57.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.981+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c067c30 con 0x7fc71c0ffc00 2024-03-30T04:01:57.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.983+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 23) v1 ==== 135677+0+0 (secure 0 0 0) 0x7fc70c00cc50 con 0x7fc71c0ffc00 2024-03-30T04:01:57.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.985+0000 7fc713e00700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 0x7fc7000a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:57.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.985+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc70c014070 con 0x7fc71c0ffc00 2024-03-30T04:01:57.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.985+0000 7fc71aa00700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 0x7fc7000a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:57.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.986+0000 7fc71aa00700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 0x7fc7000a33e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b580 tx=0x7fc708005fb0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:57.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:57.988+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc70c096120 con 0x7fc71c0ffc00 2024-03-30T04:01:58.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.253+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fc71c0fd3c0 con 0x7fc7000a0f20 2024-03-30T04:01:58.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.261+0000 7fc713e00700 1 -- 172.21.15.23:0/2257503538 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7fc71c0fd3c0 con 0x7fc7000a0f20 2024-03-30T04:01:58.259 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-30T00:00:00", "created": "2024-03-30T04:01:57", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 msgr2=0x7fc7000a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 0x7fc7000a33e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b580 tx=0x7fc708005fb0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 msgr2=0x7fc71c198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e970 tx=0x7fc70c00ec80 comp rx=0 tx=0).stop 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 shutdown_connections 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7000a0f20 0x7fc7000a33e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc71c101790 0x7fc71c1a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc71c0ffc00 0x7fc71c198f30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 --2- 172.21.15.23:0/2257503538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc71c100df0 0x7fc71c199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 >> 172.21.15.23:0/2257503538 conn(0x7fc71c0fb170 msgr2=0x7fc71c0fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:58.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 shutdown_connections 2024-03-30T04:01:58.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.263+0000 7fc722600700 1 -- 172.21.15.23:0/2257503538 wait complete. 2024-03-30T04:01:58.281 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-30T04:01:58.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: [30/Mar/2024:04:01:57] ENGINE Bus STARTING 2024-03-30T04:01:58.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: mgrmap e23: x(active, since 1.02156s), standbys: z, y 2024-03-30T04:01:58.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:58.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:58.345 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.346 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:58 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: [30/Mar/2024:04:01:57] ENGINE Bus STARTING 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: mgrmap e23: x(active, since 1.02156s), standbys: z, y 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:58 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 -- 172.21.15.23:0/946783710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4108101f00 msgr2=0x7f4108102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 --2- 172.21.15.23:0/946783710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4108101f00 0x7f4108102360 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009990 tx=0x7f40f4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 -- 172.21.15.23:0/946783710 shutdown_connections 2024-03-30T04:01:58.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 --2- 172.21.15.23:0/946783710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41081028a0 0x7f4108109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 --2- 172.21.15.23:0/946783710 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108100d10 0x7f4108101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 --2- 172.21.15.23:0/946783710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4108101f00 0x7f4108102360 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 -- 172.21.15.23:0/946783710 >> 172.21.15.23:0/946783710 conn(0x7f41080fc2a0 msgr2=0x7f41080fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 -- 172.21.15.23:0/946783710 shutdown_connections 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.494+0000 7f410d000700 1 -- 172.21.15.23:0/946783710 wait complete. 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.495+0000 7f410d000700 1 Processor -- start 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.495+0000 7f410d000700 1 -- start start 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.495+0000 7f410d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.495+0000 7f410d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f410d000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41081028a0 0x7f41081a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f410d000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081a0a20 con 0x7f4108101f00 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f410d000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081a0b60 con 0x7f4108100d10 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f410d000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41081a0cf0 con 0x7f41081028a0 2024-03-30T04:01:58.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58482/0 (socket says 172.21.15.23:58482) 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48658/0 (socket says 172.21.15.23:48658) 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105e00700 1 -- 172.21.15.23:0/3147206640 learned_addr learned my addr 172.21.15.23:0/3147206640 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4106800700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41081028a0 0x7f41081a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 -- 172.21.15.23:0/3147206640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41081028a0 msgr2=0x7f41081a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41081028a0 0x7f41081a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 -- 172.21.15.23:0/3147206640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 msgr2=0x7f4108198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105400700 1 -- 172.21.15.23:0/3147206640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40095f0 con 0x7f4108101f00 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.496+0000 7f4105e00700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:01:58.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f4105400700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000c00 tx=0x7f40f4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f401d070 con 0x7f4108101f00 2024-03-30T04:01:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f4021410 con 0x7f4108101f00 2024-03-30T04:01:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4017650 con 0x7f4108101f00 2024-03-30T04:01:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41081a0f70 con 0x7f4108101f00 2024-03-30T04:01:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.497+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41081a12e0 con 0x7f4108101f00 2024-03-30T04:01:58.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: [30/Mar/2024:04:01:57] ENGINE Bus STARTING 2024-03-30T04:01:58.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: mgrmap e23: x(active, since 1.02156s), standbys: z, y 2024-03-30T04:01:58.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:58.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:58.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.500 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:58 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:58.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.499+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41080673a0 con 0x7f4108101f00 2024-03-30T04:01:58.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.500+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 135677+0+0 (secure 0 0 0) 0x7f40f4021580 con 0x7f4108101f00 2024-03-30T04:01:58.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.502+0000 7f40fea00700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 0x7f40ec0a3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:01:58.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.503+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f40f40cdc20 con 0x7f4108101f00 2024-03-30T04:01:58.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.503+0000 7f4105e00700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 0x7f40ec0a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:01:58.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.504+0000 7f4105e00700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 0x7f40ec0a3460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40f8005950 tx=0x7f40f8009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:01:58.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.507+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f40f4097270 con 0x7f4108101f00 2024-03-30T04:01:58.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.606+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 24) v1 ==== 135985+0+0 (secure 0 0 0) 0x7f40f4097640 con 0x7f4108101f00 2024-03-30T04:01:58.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.774+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4108074700 con 0x7f40ec0a0fa0 2024-03-30T04:01:58.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.781+0000 7f40fea00700 1 -- 172.21.15.23:0/3147206640 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f4108074700 con 0x7f40ec0a0fa0 2024-03-30T04:01:58.781 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 msgr2=0x7f40ec0a3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 0x7f40ec0a3460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40f8005950 tx=0x7f40f8009260 comp rx=0 tx=0).stop 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 msgr2=0x7f4108199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000c00 tx=0x7f40f4009580 comp rx=0 tx=0).stop 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 shutdown_connections 2024-03-30T04:01:58.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41081028a0 0x7f41081a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f40ec0a0fa0 0x7f40ec0a3460 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4108100d10 0x7f4108198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 --2- 172.21.15.23:0/3147206640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4108101f00 0x7f4108199430 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 >> 172.21.15.23:0/3147206640 conn(0x7f41080fc2a0 msgr2=0x7f4108072fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 shutdown_connections 2024-03-30T04:01:58.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:01:58.784+0000 7f410d000700 1 -- 172.21.15.23:0/3147206640 wait complete. 2024-03-30T04:01:58.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-30T04:01:58.873 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-30T04:01:58.884 INFO:teuthology.run_tasks:Running task workunit... 2024-03-30T04:01:58.899 INFO:tasks.workunit:Pulling workunits from ref 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:01:58.899 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-30T04:01:58.899 INFO:tasks.workunit:timeout=3h 2024-03-30T04:01:58.899 INFO:tasks.workunit:cleanup=True 2024-03-30T04:01:58.899 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:58.942 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:58.943 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-30T04:01:58.943 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 7ah/122d Inode: 1099511628283 Links: 2 2024-03-30T04:01:58.943 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T04:01:58.944 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:cephfs_t:s0 2024-03-30T04:01:58.944 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-30 04:00:15.514555254 +0000 2024-03-30T04:01:58.944 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-30 04:00:15.514555254 +0000 2024-03-30T04:01:58.944 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-30 04:01:34.010759931 +0000 2024-03-30T04:01:58.944 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-30 04:00:15.514555254 +0000 2024-03-30T04:01:58.944 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-30T04:01:58.944 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-30T04:01:59.062 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 470100a4ccbcf7769052bf09a5413881ccf1b97f 2024-03-30T04:01:59.108 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: [30/Mar/2024:04:01:57] ENGINE Serving on https://172.21.15.23:7150 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: [30/Mar/2024:04:01:57] ENGINE Serving on http://172.21.15.23:8765 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: [30/Mar/2024:04:01:57] ENGINE Bus STARTED 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:01:59 smithi154 ceph-mon[79161]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: [30/Mar/2024:04:01:57] ENGINE Serving on https://172.21.15.23:7150 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: [30/Mar/2024:04:01:57] ENGINE Serving on http://172.21.15.23:8765 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: [30/Mar/2024:04:01:57] ENGINE Bus STARTED 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:01:59 smithi193 ceph-mon[79581]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: [30/Mar/2024:04:01:57] ENGINE Serving on https://172.21.15.23:7150 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: [30/Mar/2024:04:01:57] ENGINE Serving on http://172.21.15.23:8765 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: [30/Mar/2024:04:01:57] ENGINE Bus STARTED 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:01:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:01:59 smithi023 ceph-mon[38254]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:00.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:00.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='client.25690 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:02:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:01.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:02:00.758+0000 7fe134200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:02:01.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:02:00.758+0000 7fe134200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:02:01.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:02:00.758+0000 7fe134200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:02:01.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:02:00.758+0000 7fe134200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:02:01.022 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:02:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:02:00.758+0000 7fe134200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-30T04:02:01.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:01 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:01 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:02:01.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:02:01.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Updating smithi154:/etc/ceph/ceph.conf 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:01.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:01 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:02.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:02.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:02 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:02.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:02 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.conf 2024-03-30T04:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:02:02.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:02 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:02:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:03 smithi154 ceph-mon[79161]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:03 smithi154 ceph-mon[79161]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T04:02:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:03 smithi154 ceph-mon[79161]: mgrmap e26: x(active, since 7s), standbys: z, y 2024-03-30T04:02:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:03 smithi193 ceph-mon[79581]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:03 smithi193 ceph-mon[79581]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T04:02:03.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:03 smithi193 ceph-mon[79581]: mgrmap e26: x(active, since 7s), standbys: z, y 2024-03-30T04:02:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:03 smithi023 ceph-mon[38254]: Updating smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/config/ceph.client.admin.keyring 2024-03-30T04:02:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:03 smithi023 ceph-mon[38254]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-30T04:02:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:03 smithi023 ceph-mon[38254]: mgrmap e26: x(active, since 7s), standbys: z, y 2024-03-30T04:02:05.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:05 smithi154 ceph-mon[79161]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 170 KiB/s wr, 106 op/s 2024-03-30T04:02:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:05 smithi193 ceph-mon[79581]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 170 KiB/s wr, 106 op/s 2024-03-30T04:02:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:05 smithi023 ceph-mon[38254]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 170 KiB/s wr, 106 op/s 2024-03-30T04:02:07.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:06 smithi193 ceph-mon[79581]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 82 op/s 2024-03-30T04:02:07.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:06 smithi023 ceph-mon[38254]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 82 op/s 2024-03-30T04:02:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:06 smithi154 ceph-mon[79161]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 133 KiB/s wr, 82 op/s 2024-03-30T04:02:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:08 smithi154 ceph-mon[79161]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-30T04:02:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:08 smithi193 ceph-mon[79581]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-30T04:02:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:08 smithi023 ceph-mon[38254]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 109 KiB/s wr, 72 op/s 2024-03-30T04:02:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:10 smithi154 ceph-mon[79161]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-30T04:02:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:10 smithi193 ceph-mon[79581]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-30T04:02:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:10 smithi023 ceph-mon[38254]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 66 op/s 2024-03-30T04:02:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:12 smithi154 ceph-mon[79161]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-30T04:02:12.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:12 smithi193 ceph-mon[79581]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-30T04:02:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:12 smithi023 ceph-mon[38254]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 100 KiB/s wr, 67 op/s 2024-03-30T04:02:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:14 smithi154 ceph-mon[79161]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-30T04:02:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:14 smithi193 ceph-mon[79581]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-30T04:02:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:14 smithi023 ceph-mon[38254]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 100 KiB/s wr, 70 op/s 2024-03-30T04:02:16.424 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 -- 172.21.15.23:0/3126606234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac100f60 msgr2=0x7f0aac101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3126606234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac100f60 0x7f0aac101340 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c009990 tx=0x7f0a9c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 -- 172.21.15.23:0/3126606234 shutdown_connections 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3126606234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3126606234 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac100f60 0x7f0aac101340 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3126606234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac101910 0x7f0aac105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 -- 172.21.15.23:0/3126606234 >> 172.21.15.23:0/3126606234 conn(0x7f0aac0fc7c0 msgr2=0x7f0aac0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 -- 172.21.15.23:0/3126606234 shutdown_connections 2024-03-30T04:02:16.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.644+0000 7f0ab2200700 1 -- 172.21.15.23:0/3126606234 wait complete. 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 Processor -- start 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 -- start start 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac100f60 0x7f0aac0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 0x7f0aac1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1a59f0 con 0x7f0aac073b30 2024-03-30T04:02:16.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1a5b60 con 0x7f0aac106080 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.645+0000 7f0ab2200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aac1a5d20 con 0x7f0aac100f60 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aabe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 0x7f0aac1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50938/0 (socket says 172.21.15.23:50938) 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3236066534 learned_addr learned my addr 172.21.15.23:0/3236066534 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aab400700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac100f60 0x7f0aac0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3236066534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac100f60 msgr2=0x7f0aac0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac100f60 0x7f0aac0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 msgr2=0x7f0aac1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 0x7f0aac1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3236066534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0095f0 con 0x7f0aac106080 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aabe00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 0x7f0aac1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.646+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800e970 tx=0x7f0a9800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.647+0000 7f0aa3e00700 1 -- 172.21.15.23:0/3236066534 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98009790 con 0x7f0aac106080 2024-03-30T04:02:16.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.647+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aac1a5fa0 con 0x7f0aac106080 2024-03-30T04:02:16.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.647+0000 7f0aa3e00700 1 -- 172.21.15.23:0/3236066534 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a9800c7e0 con 0x7f0aac106080 2024-03-30T04:02:16.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.647+0000 7f0aa3e00700 1 -- 172.21.15.23:0/3236066534 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980054e0 con 0x7f0aac106080 2024-03-30T04:02:16.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.647+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac1a6480 con 0x7f0aac106080 2024-03-30T04:02:16.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.650+0000 7f0aa3e00700 1 -- 172.21.15.23:0/3236066534 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f0a9800c950 con 0x7f0aac106080 2024-03-30T04:02:16.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 Processor -- start 2024-03-30T04:02:16.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 -- start start 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df7d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dec8990 con 0x55fc9df7adf0 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dec6350 con 0x55fc9df6fef0 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.651+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dec1c40 con 0x55fc9df6f5d0 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aa3e00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 0x7f0a900a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50946/0 (socket says 172.21.15.23:50946) 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 -- 172.21.15.23:0/1201668298 learned_addr learned my addr 172.21.15.23:0/1201668298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aa3e00700 1 -- 172.21.15.23:0/3236066534 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a98014070 con 0x7f0aac106080 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aabe00700 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df7d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aabe00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 0x7f0a900a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 -- 172.21.15.23:0/1201668298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 msgr2=0x55fc9df6f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 -- 172.21.15.23:0/1201668298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 msgr2=0x55fc9df7d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df7d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 -- 172.21.15.23:0/1201668298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9dd433b0 con 0x55fc9df6fef0 2024-03-30T04:02:16.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.652+0000 7f0aab400700 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c005ec0 tx=0x7f0a9c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb72fc08e6393940 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0a97e00700 1 -- 172.21.15.23:0/1201668298 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c01d070 con 0x55fc9df6fef0 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0aabe00700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 0x7f0a900a3870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009fd0 tx=0x7f0aa400b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0a97e00700 1 -- 172.21.15.23:0/1201668298 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a9c01e040 con 0x55fc9df6fef0 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0a97e00700 1 -- 172.21.15.23:0/1201668298 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c004880 con 0x55fc9df6fef0 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1201668298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 msgr2=0x55fc9df7a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.653+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c005ec0 tx=0x7f0a9c005ef0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1201668298 shutdown_connections 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df7d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1201668298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6fef0 0x55fc9df7a8b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1201668298 >> 172.21.15.23:0/1201668298 conn(0x55fc9decf6a0 msgr2=0x55fc9df6ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1201668298 shutdown_connections 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1201668298 wait complete. 2024-03-30T04:02:16.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 -- start start 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.654+0000 7f0ac5c43b80 1 Processor -- start 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 0x55fc9df77ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9de5dd80 con 0x55fc9df7adf0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9de5da20 con 0x55fc9df6f5d0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dd456a0 con 0x55fc9df6fef0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0ac5c43b80 10 client.0 _ll_register_callbacks cb 0x55fc9de67470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0aabe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0aabe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48972/0 (socket says 172.21.15.23:48972) 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0aabe00700 1 -- 172.21.15.23:0/4183760434 learned_addr learned my addr 172.21.15.23:0/4183760434 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 0x55fc9df77ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.655+0000 7f0aab400700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 -- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 msgr2=0x55fc9df7a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 -- 172.21.15.23:0/4183760434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 msgr2=0x55fc9df780f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 -- 172.21.15.23:0/4183760434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aac10e470 con 0x55fc9df6fef0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aabe00700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 0x55fc9df77ae0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a980ca7a0 tx=0x7f0a980659c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45680a80cd74186f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0a94c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980ca450 con 0x55fc9df6fef0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fc9de82380 con 0x55fc9df6fef0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a9809a790 con 0x55fc9df6fef0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a98065a30 con 0x55fc9df6fef0 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0ac5c43b80 10 client.25717 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:16.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.656+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fc9dcefb60 con 0x55fc9df6fef0 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fc9de82380 con 0x55fc9df6fef0 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a980cd160 con 0x55fc9df6fef0 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 10 client.25717 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 10 client.25717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 20 client.25717 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9df8a920 0x55fc9df8cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 4 client.25717 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:16.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.657+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc9deba700 con 0x55fc9df8a920 2024-03-30T04:02:16.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.658+0000 7f0aab400700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9df8a920 0x55fc9df8cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.658+0000 7f0aab400700 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9df8a920 0x55fc9df8cd40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e5e0748c4e98b70 server_cookie=32227dd5245d154b in_seq=0 out_seq=0 2024-03-30T04:02:16.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.658+0000 7f0a94c00700 10 client.25717 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0a94c00700 1 -- 172.21.15.23:0/4183760434 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fc9deba700 con 0x55fc9df8a920 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0a94c00700 10 client.25717 handle_command_reply: tid=0 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 client.25717 shutdown 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 msgr2=0x55fc9df77ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 0x55fc9df77ae0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0a980ca7a0 tx=0x7f0a980659c0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 shutdown_connections 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df7adf0 0x55fc9df780f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9df8a920 0x55fc9df8cd40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/4183760434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6fef0 0x55fc9df77ae0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0a94c00700 0 client.25717 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 >> 172.21.15.23:0/4183760434 conn(0x55fc9decf6a0 msgr2=0x55fc9df71530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 shutdown_connections 2024-03-30T04:02:16.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/4183760434 wait complete. 2024-03-30T04:02:16.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.659+0000 7f0ac5c43b80 20 client.25717 trim_cache size 0 max 16384 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.676+0000 7f0ac5c43b80 1 Processor -- start 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 -- start start 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df7adf0 0x55fc9df645a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9df708c0 con 0x55fc9df6fef0 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9df8d100 con 0x55fc9df6f5d0 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9df81fd0 con 0x55fc9df7adf0 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0aab400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48978/0 (socket says 172.21.15.23:48978) 2024-03-30T04:02:16.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3119404862 learned_addr learned my addr 172.21.15.23:0/3119404862 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.677+0000 7f0aabe00700 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df7adf0 0x55fc9df645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3119404862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df7adf0 msgr2=0x55fc9df645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df7adf0 0x55fc9df645a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3119404862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 msgr2=0x55fc9df6af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 -- 172.21.15.23:0/3119404862 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9de87ea0 con 0x55fc9df6fef0 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aab400700 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004800 tx=0x7f0a98005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67cd0bb7ee80742 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aa1200700 1 -- 172.21.15.23:0/3119404862 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98069160 con 0x55fc9df6fef0 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aa1200700 1 -- 172.21.15.23:0/3119404862 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a980ca450 con 0x55fc9df6fef0 2024-03-30T04:02:16.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.678+0000 7f0aa1200700 1 -- 172.21.15.23:0/3119404862 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980a2510 con 0x55fc9df6fef0 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/3119404862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 msgr2=0x55fc9df81a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004800 tx=0x7f0a98005b40 comp rx=0 tx=0).stop 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/3119404862 shutdown_connections 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df7adf0 0x55fc9df645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df6f5d0 0x55fc9df6af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/3119404862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df81a10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/3119404862 >> 172.21.15.23:0/3119404862 conn(0x55fc9decf6a0 msgr2=0x55fc9df77370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/3119404862 shutdown_connections 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/3119404862 wait complete. 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 -- start start 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.679+0000 7f0ac5c43b80 1 Processor -- start 2024-03-30T04:02:16.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df7adf0 0x55fc9dfdd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dfddd50 con 0x55fc9df6fef0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dfddec0 con 0x55fc9df7adf0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9dfde080 con 0x55fc9df6f5d0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0ac5c43b80 10 client.0 _ll_register_callbacks cb 0x55fc9de67470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0aaaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48992/0 (socket says 172.21.15.23:48992) 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0aaaa00700 1 -- 172.21.15.23:0/1639433233 learned_addr learned my addr 172.21.15.23:0/1639433233 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0aabe00700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df7adf0 0x55fc9dfdd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.680+0000 7f0aab400700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 -- 172.21.15.23:0/1639433233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 msgr2=0x55fc9df7fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 -- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df7adf0 msgr2=0x55fc9dfdd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df7adf0 0x55fc9dfdd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 -- 172.21.15.23:0/1639433233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc9dd433b0 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0aaaa00700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0a98005950 tx=0x7f0a98065b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=824b32082458df81 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0a94c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98069160 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a9809a790 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a9c01e040 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a980ca6e0 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0ac5c43b80 10 client.15726 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.681+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fc9dcefb60 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a9c01e040 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a980b82f0 con 0x55fc9df6fef0 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 10 client.15726 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:16.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 20 client.15726 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9dfdf360 0x55fc9dfe1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 4 client.15726 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fc9dd10cf0 con 0x55fc9dfdf360 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.682+0000 7f0aab400700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9dfdf360 0x55fc9dfe1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0aab400700 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9dfdf360 0x55fc9dfe1750 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6de626540a25c941 server_cookie=85a441c5af256837 in_seq=0 out_seq=0 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0a94c00700 10 client.15726 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0a94c00700 1 -- 172.21.15.23:0/1639433233 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fc9dd10cf0 con 0x55fc9dfdf360 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0a94c00700 10 client.15726 handle_command_reply: tid=0 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 client.15726 shutdown 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 msgr2=0x55fc9df80180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0a98005950 tx=0x7f0a98065b40 comp rx=0 tx=0).stop 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 shutdown_connections 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fc9df7adf0 0x55fc9dfdd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fc9dfdf360 0x55fc9dfe1750 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fc9df6f5d0 0x55fc9df7fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 --2- 172.21.15.23:0/1639433233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fc9df6fef0 0x55fc9df80180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 >> 172.21.15.23:0/1639433233 conn(0x55fc9decf6a0 msgr2=0x55fc9df76370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0a94c00700 0 client.15726 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 shutdown_connections 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.683+0000 7f0ac5c43b80 1 -- 172.21.15.23:0/1639433233 wait complete. 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ac5c43b80 20 client.15726 trim_cache size 0 max 16384 2024-03-30T04:02:16.687 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 msgr2=0x7f0a900a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 0x7f0a900a3870 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009fd0 tx=0x7f0aa400b040 comp rx=0 tx=0).stop 2024-03-30T04:02:16.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 msgr2=0x7f0aac0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800e970 tx=0x7f0a9800ec80 comp rx=0 tx=0).stop 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 shutdown_connections 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0aac073b30 0x7f0aac1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0aac100f60 0x7f0aac0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a900a13b0 0x7f0a900a3870 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 --2- 172.21.15.23:0/3236066534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aac106080 0x7f0aac0735f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 >> 172.21.15.23:0/3236066534 conn(0x7f0aac0fc7c0 msgr2=0x7f0aac0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 shutdown_connections 2024-03-30T04:02:16.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.684+0000 7f0ab2200700 1 -- 172.21.15.23:0/3236066534 wait complete. 2024-03-30T04:02:16.702 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:02:16.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:02:16.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.902+0000 7f794ea00700 1 -- 172.21.15.23:0/1229154561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948105f80 msgr2=0x7f794810db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.902+0000 7f794ea00700 1 --2- 172.21.15.23:0/1229154561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948105f80 0x7f794810db00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f794000b0a0 tx=0x7f794000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 -- 172.21.15.23:0/1229154561 shutdown_connections 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 --2- 172.21.15.23:0/1229154561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948105f80 0x7f794810db00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 --2- 172.21.15.23:0/1229154561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f7948101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 --2- 172.21.15.23:0/1229154561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948101810 0x7f7948105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 -- 172.21.15.23:0/1229154561 >> 172.21.15.23:0/1229154561 conn(0x7f79480fc760 msgr2=0x7f79480feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 -- 172.21.15.23:0/1229154561 shutdown_connections 2024-03-30T04:02:16.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.903+0000 7f794ea00700 1 -- 172.21.15.23:0/1229154561 wait complete. 2024-03-30T04:02:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 Processor -- start 2024-03-30T04:02:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 -- start start 2024-03-30T04:02:16.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948105f80 0x7f79480735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 0x7f79481a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79481a5840 con 0x7f7948100e60 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79481a59b0 con 0x7f7948073b30 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f794ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79481a5b70 con 0x7f7948105f80 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f7947e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.904+0000 7f7947e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49010/0 (socket says 172.21.15.23:49010) 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 -- 172.21.15.23:0/722186520 learned_addr learned my addr 172.21.15.23:0/722186520 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947400700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948105f80 0x7f79480735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f794ca00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 0x7f79481a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 -- 172.21.15.23:0/722186520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948105f80 msgr2=0x7f79480735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948105f80 0x7f79480735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 -- 172.21.15.23:0/722186520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 msgr2=0x7f79481a5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:16.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 0x7f79481a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 -- 172.21.15.23:0/722186520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940009d00 con 0x7f7948100e60 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f794ca00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 0x7f79481a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7947e00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f793800e9a0 tx=0x7f793800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938009790 con 0x7f7948100e60 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79481a5e50 con 0x7f7948100e60 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.905+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7938010430 con 0x7f7948100e60 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.906+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938018680 con 0x7f7948100e60 2024-03-30T04:02:16.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.906+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79481a62d0 con 0x7f7948100e60 2024-03-30T04:02:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.907+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7948109250 con 0x7f7948100e60 2024-03-30T04:02:16.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.908+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f79380105a0 con 0x7f7948100e60 2024-03-30T04:02:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.910+0000 7f7944c00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 0x7f792c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:16.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.910+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7938014070 con 0x7f7948100e60 2024-03-30T04:02:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.910+0000 7f7947400700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 0x7f792c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:16.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.911+0000 7f7947400700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 0x7f792c0a3870 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79480747e0 tx=0x7f7934009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:16.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:16.914+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f79380529f0 con 0x7f7948100e60 2024-03-30T04:02:17.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:16 smithi023 ceph-mon[38254]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.188+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7948102330 con 0x7f7948100e60 2024-03-30T04:02:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.188+0000 7f7944c00700 1 -- 172.21.15.23:0/722186520 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f7938096360 con 0x7f7948100e60 2024-03-30T04:02:17.187 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:02:17.187 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:01:35.365141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:02:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 msgr2=0x7f792c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 0x7f792c0a3870 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f79480747e0 tx=0x7f7934009260 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 msgr2=0x7f79480730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f793800e9a0 tx=0x7f793800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 shutdown_connections 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f792c0a13b0 0x7f792c0a3870 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7948073b30 0x7f79481a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7948100e60 0x7f79480730b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 --2- 172.21.15.23:0/722186520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7948105f80 0x7f79480735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 >> 172.21.15.23:0/722186520 conn(0x7f79480fc760 msgr2=0x7f79480fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 shutdown_connections 2024-03-30T04:02:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.191+0000 7f794ea00700 1 -- 172.21.15.23:0/722186520 wait complete. 2024-03-30T04:02:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-30T04:02:17.209 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:02:17.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:16 smithi193 ceph-mon[79581]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 -- 172.21.15.23:0/4094726047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 msgr2=0x7f91d0101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4094726047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0101340 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f91c0009990 tx=0x7f91c0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 -- 172.21.15.23:0/4094726047 shutdown_connections 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4094726047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 0x7f91d010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4094726047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0101340 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4094726047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d0101910 0x7f91d0105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 -- 172.21.15.23:0/4094726047 >> 172.21.15.23:0/4094726047 conn(0x7f91d00fc7c0 msgr2=0x7f91d00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 -- 172.21.15.23:0/4094726047 shutdown_connections 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.412+0000 7f91d5a00700 1 -- 172.21.15.23:0/4094726047 wait complete. 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.413+0000 7f91d5a00700 1 Processor -- start 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.413+0000 7f91d5a00700 1 -- start start 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 0x7f91d0197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d019bb10 0x7f91d019bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d019c590 con 0x7f91d0100f60 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d019c700 con 0x7f91d019bb10 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91d5a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d019c8c0 con 0x7f91d0106080 2024-03-30T04:02:17.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49036/0 (socket says 172.21.15.23:49036) 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 -- 172.21.15.23:0/4292792308 learned_addr learned my addr 172.21.15.23:0/4292792308 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91c7e00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 0x7f91d0197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cf400700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d019bb10 0x7f91d019bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 -- 172.21.15.23:0/4292792308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 msgr2=0x7f91d0197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 0x7f91d0197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 -- 172.21.15.23:0/4292792308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d019bb10 msgr2=0x7f91d019bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d019bb10 0x7f91d019bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.414+0000 7f91cea00700 1 -- 172.21.15.23:0/4292792308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c00095f0 con 0x7f91d0100f60 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91cea00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f91c0005910 tx=0x7f91c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91c7400700 1 -- 172.21.15.23:0/4292792308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c001d070 con 0x7f91d0100f60 2024-03-30T04:02:17.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91c7400700 1 -- 172.21.15.23:0/4292792308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91c0022470 con 0x7f91d0100f60 2024-03-30T04:02:17.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d01a3dc0 con 0x7f91d0100f60 2024-03-30T04:02:17.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91c7400700 1 -- 172.21.15.23:0/4292792308 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c0017650 con 0x7f91d0100f60 2024-03-30T04:02:17.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.415+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d01a41e0 con 0x7f91d0100f60 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.418+0000 7f91c7400700 1 -- 172.21.15.23:0/4292792308 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f91c00225e0 con 0x7f91d0100f60 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.420+0000 7f91e9430b80 1 Processor -- start 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.420+0000 7f91e9430b80 1 -- start start 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.420+0000 7f91c7400700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 0x7f91b40a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.420+0000 7f91c7400700 1 -- 172.21.15.23:0/4292792308 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f91c00ce6e0 con 0x7f91d0100f60 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91cf400700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 0x7f91b40a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b9199b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b924df0 0x555d8b9271e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b872990 con 0x555d8b919ef0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b870350 con 0x555d8b924df0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91e9430b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b86bc40 con 0x555d8b9195d0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49050/0 (socket says 172.21.15.23:49050) 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 -- 172.21.15.23:0/513625602 learned_addr learned my addr 172.21.15.23:0/513625602 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91cea00700 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b9199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91cf400700 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b924df0 0x555d8b9271e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 -- 172.21.15.23:0/513625602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 msgr2=0x555d8b9199b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b9199b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 -- 172.21.15.23:0/513625602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b924df0 msgr2=0x555d8b9271e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b924df0 0x555d8b9271e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91cf400700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 0x7f91b40a38f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91c80093c0 tx=0x7f91c8009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.421+0000 7f91c7e00700 1 -- 172.21.15.23:0/513625602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8b6ed3b0 con 0x555d8b919ef0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.422+0000 7f91c7e00700 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009cf0 tx=0x7f91b8011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9667d90ea1220f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.422+0000 7f91bfe00700 1 -- 172.21.15.23:0/513625602 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8011d80 con 0x555d8b919ef0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.422+0000 7f91bfe00700 1 -- 172.21.15.23:0/513625602 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91b8004510 con 0x555d8b919ef0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.422+0000 7f91bfe00700 1 -- 172.21.15.23:0/513625602 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8005240 con 0x555d8b919ef0 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- 172.21.15.23:0/513625602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 msgr2=0x555d8b9248b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f91b8009cf0 tx=0x7f91b8011440 comp rx=0 tx=0).stop 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- 172.21.15.23:0/513625602 shutdown_connections 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b924df0 0x555d8b9271e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b9199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 --2- 172.21.15.23:0/513625602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b9248b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- 172.21.15.23:0/513625602 >> 172.21.15.23:0/513625602 conn(0x555d8b8796a0 msgr2=0x555d8b915a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- 172.21.15.23:0/513625602 shutdown_connections 2024-03-30T04:02:17.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- 172.21.15.23:0/513625602 wait complete. 2024-03-30T04:02:17.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.423+0000 7f91e9430b80 1 -- start start 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 Processor -- start 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b9195d0 0x555d8b924780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b807d80 con 0x555d8b919ef0 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b807a20 con 0x555d8b9195d0 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b6ef6a0 con 0x555d8b924df0 2024-03-30T04:02:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91e9430b80 10 client.0 _ll_register_callbacks cb 0x555d8b811470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91c7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49060/0 (socket says 172.21.15.23:49060) 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91cf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91c7e00700 1 -- 172.21.15.23:0/3717772192 learned_addr learned my addr 172.21.15.23:0/3717772192 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.424+0000 7f91cea00700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b9195d0 0x555d8b924780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 -- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b9195d0 msgr2=0x555d8b924780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b9195d0 0x555d8b924780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 -- 172.21.15.23:0/3717772192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 msgr2=0x555d8b91da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 -- 172.21.15.23:0/3717772192 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b8009920 con 0x555d8b924df0 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91c7e00700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91cf400700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91e040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009fd0 tx=0x7f91c8004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f03a1bca2d082ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.425+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c800e050 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f91c8017e10 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d8b82c380 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91c8007e80 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 10 client.25741 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555d8b699b60 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d8b82c380 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91c80163d0 con 0x555d8b924df0 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 10 client.25741 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 10 client.25741 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:02:17.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 20 client.25741 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b93d2f0 0x555d8b93f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.426+0000 7f91e9430b80 4 client.25741 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.427+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d8b864700 con 0x555d8b93d2f0 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.427+0000 7f91cea00700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b93d2f0 0x555d8b93f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.427+0000 7f91cea00700 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b93d2f0 0x555d8b93f710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=567989a73093e038 server_cookie=b0f1622f2b073146 in_seq=0 out_seq=0 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.427+0000 7f91bcc00700 10 client.25741 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91bcc00700 1 -- 172.21.15.23:0/3717772192 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555d8b864700 con 0x555d8b93d2f0 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91bcc00700 10 client.25741 handle_command_reply: tid=0 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 client.25741 shutdown 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 msgr2=0x555d8b91e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91e040 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009fd0 tx=0x7f91c8004b40 comp rx=0 tx=0).stop 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 shutdown_connections 2024-03-30T04:02:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b93d2f0 0x555d8b93f710 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91e040 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b9195d0 0x555d8b924780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3717772192 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b919ef0 0x555d8b91da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91bcc00700 0 client.25741 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.428+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 >> 172.21.15.23:0/3717772192 conn(0x555d8b8796a0 msgr2=0x555d8b91b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.429+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 shutdown_connections 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.429+0000 7f91e9430b80 1 -- 172.21.15.23:0/3717772192 wait complete. 2024-03-30T04:02:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.429+0000 7f91e9430b80 20 client.25741 trim_cache size 0 max 16384 2024-03-30T04:02:17.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:16 smithi154 ceph-mon[79161]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:17.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.442+0000 7f91e9430b80 1 Processor -- start 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 -- start start 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b924440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b922c80 con 0x555d8b9195d0 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b91f060 con 0x555d8b919ef0 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91e9430b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b928970 con 0x555d8b924df0 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49074/0 (socket says 172.21.15.23:49074) 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 -- 172.21.15.23:0/3591100305 learned_addr learned my addr 172.21.15.23:0/3591100305 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91c7e00700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b924440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cf400700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 -- 172.21.15.23:0/3591100305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 msgr2=0x555d8b91d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 -- 172.21.15.23:0/3591100305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 msgr2=0x555d8b924440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b924440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cea00700 1 -- 172.21.15.23:0/3591100305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d8b831ea0 con 0x555d8b9195d0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91c7e00700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b924440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.443+0000 7f91cf400700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91cea00700 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f91c0097e00 tx=0x7f91c001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65161b068839d350 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91c4c00700 1 -- 172.21.15.23:0/3591100305 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c00b8070 con 0x555d8b9195d0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91c4c00700 1 -- 172.21.15.23:0/3591100305 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91c009ba90 con 0x555d8b9195d0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91c4c00700 1 -- 172.21.15.23:0/3591100305 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c00995a0 con 0x555d8b9195d0 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91e9430b80 1 -- 172.21.15.23:0/3591100305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 msgr2=0x555d8b90e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.444+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f91c0097e00 tx=0x7f91c001a040 comp rx=0 tx=0).stop 2024-03-30T04:02:17.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 -- 172.21.15.23:0/3591100305 shutdown_connections 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b924df0 0x555d8b91d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b9195d0 0x555d8b90e5a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 --2- 172.21.15.23:0/3591100305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b924440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 -- 172.21.15.23:0/3591100305 >> 172.21.15.23:0/3591100305 conn(0x555d8b8796a0 msgr2=0x555d8b91d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 -- 172.21.15.23:0/3591100305 shutdown_connections 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 -- 172.21.15.23:0/3591100305 wait complete. 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 -- start start 2024-03-30T04:02:17.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.445+0000 7f91e9430b80 1 Processor -- start 2024-03-30T04:02:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b939570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b939ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b93a1c0 con 0x555d8b924df0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b936890 con 0x555d8b919ef0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d8b936a50 con 0x555d8b9195d0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91e9430b80 10 client.0 _ll_register_callbacks cb 0x555d8b811470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49076/0 (socket says 172.21.15.23:49076) 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 -- 172.21.15.23:0/1513911210 learned_addr learned my addr 172.21.15.23:0/1513911210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91c7e00700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b939ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cea00700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b939570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 -- 172.21.15.23:0/1513911210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 msgr2=0x555d8b939570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b939570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 -- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 msgr2=0x555d8b939ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b939ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.446+0000 7f91cf400700 1 -- 172.21.15.23:0/1513911210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d01a41e0 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91cf400700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f91c80061e0 tx=0x7f91c8009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d5448c31b9b7ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91bcc00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8039070 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91b8004510 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f91c8016ce0 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91c800e040 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91e9430b80 10 client.15750 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555d8b6f1cd0 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.447+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91b8004510 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91c803c020 con 0x555d8b924df0 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 10 client.15750 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 10 client.15750 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:02:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 20 client.15750 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b991f20 0x555d8b994340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 4 client.15750 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555d8b6bacf0 con 0x555d8b991f20 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.448+0000 7f91cea00700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b991f20 0x555d8b994340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91cea00700 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b991f20 0x555d8b994340 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0a308bc9c9c5213 server_cookie=82be72e8bfdc825c in_seq=0 out_seq=0 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91bcc00700 10 client.15750 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91bcc00700 1 -- 172.21.15.23:0/1513911210 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555d8b6bacf0 con 0x555d8b991f20 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91bcc00700 10 client.15750 handle_command_reply: tid=0 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 client.15750 shutdown 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 msgr2=0x555d8b936350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f91c80061e0 tx=0x7f91c8009e20 comp rx=0 tx=0).stop 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 shutdown_connections 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d8b924df0 0x555d8b936350 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d8b991f20 0x555d8b994340 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d8b9195d0 0x555d8b939570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 --2- 172.21.15.23:0/1513911210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d8b919ef0 0x555d8b939ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91bcc00700 0 client.15750 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 >> 172.21.15.23:0/1513911210 conn(0x555d8b8796a0 msgr2=0x555d8b91b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 shutdown_connections 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.449+0000 7f91e9430b80 1 -- 172.21.15.23:0/1513911210 wait complete. 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91e9430b80 20 client.15750 trim_cache size 0 max 16384 2024-03-30T04:02:17.453 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 msgr2=0x7f91b40a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 0x7f91b40a38f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f91c80093c0 tx=0x7f91c8009920 comp rx=0 tx=0).stop 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 msgr2=0x7f91d0197230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f91c0005910 tx=0x7f91c0004730 comp rx=0 tx=0).stop 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 shutdown_connections 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f91b40a1430 0x7f91b40a38f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91d019bb10 0x7f91d019bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f91d0100f60 0x7f91d0197230 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 --2- 172.21.15.23:0/4292792308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d0106080 0x7f91d0197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 >> 172.21.15.23:0/4292792308 conn(0x7f91d00fc7c0 msgr2=0x7f91d00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 shutdown_connections 2024-03-30T04:02:17.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.450+0000 7f91d5a00700 1 -- 172.21.15.23:0/4292792308 wait complete. 2024-03-30T04:02:17.467 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.669+0000 7f616aa00700 1 -- 172.21.15.23:0/76184794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164101910 msgr2=0x7f6164105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.669+0000 7f616aa00700 1 --2- 172.21.15.23:0/76184794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164101910 0x7f6164105950 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f61500098f0 tx=0x7f6150009c00 comp rx=0 tx=0).stop 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 -- 172.21.15.23:0/76184794 shutdown_connections 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 --2- 172.21.15.23:0/76184794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6164106080 0x7f616410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 --2- 172.21.15.23:0/76184794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164100f60 0x7f6164101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 --2- 172.21.15.23:0/76184794 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164101910 0x7f6164105950 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 -- 172.21.15.23:0/76184794 >> 172.21.15.23:0/76184794 conn(0x7f61640fc7c0 msgr2=0x7f61640febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 -- 172.21.15.23:0/76184794 shutdown_connections 2024-03-30T04:02:17.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.670+0000 7f616aa00700 1 -- 172.21.15.23:0/76184794 wait complete. 2024-03-30T04:02:17.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 Processor -- start 2024-03-30T04:02:17.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 -- start start 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164106080 0x7f61641999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f616419dd40 0x7f616419e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616419e7c0 con 0x7f6164100f60 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616419e930 con 0x7f6164106080 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.671+0000 7f616aa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616419eaf0 con 0x7f616419dd40 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49088/0 (socket says 172.21.15.23:49088) 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 -- 172.21.15.23:0/1682617169 learned_addr learned my addr 172.21.15.23:0/1682617169 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163400700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164106080 0x7f61641999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6168a00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f616419dd40 0x7f616419e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 -- 172.21.15.23:0/1682617169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f616419dd40 msgr2=0x7f616419e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f616419dd40 0x7f616419e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 -- 172.21.15.23:0/1682617169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164106080 msgr2=0x7f61641999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164106080 0x7f61641999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 -- 172.21.15.23:0/1682617169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61540095f0 con 0x7f6164100f60 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.672+0000 7f6163e00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f615400ec10 tx=0x7f615400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.673+0000 7f6160c00700 1 -- 172.21.15.23:0/1682617169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615400cd40 con 0x7f6164100f60 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.673+0000 7f6160c00700 1 -- 172.21.15.23:0/1682617169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61540044e0 con 0x7f6164100f60 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.673+0000 7f6160c00700 1 -- 172.21.15.23:0/1682617169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6154010690 con 0x7f6164100f60 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.673+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6150009520 con 0x7f6164100f60 2024-03-30T04:02:17.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.673+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61641a6290 con 0x7f6164100f60 2024-03-30T04:02:17.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.676+0000 7f6160c00700 1 -- 172.21.15.23:0/1682617169 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f6154003710 con 0x7f6164100f60 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.677+0000 7f617e427b80 1 Processor -- start 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.677+0000 7f617e427b80 1 -- start start 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.677+0000 7f6160c00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 0x7f61480a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.677+0000 7f6160c00700 1 -- 172.21.15.23:0/1682617169 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6154014070 con 0x7f6164100f60 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9de89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9d41990 con 0x55c3c9df3df0 2024-03-30T04:02:17.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9d3f350 con 0x55c3c9de8ef0 2024-03-30T04:02:17.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6168a00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 0x7f61480a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f617e427b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9d3ac40 con 0x55c3c9de85d0 2024-03-30T04:02:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51072/0 (socket says 172.21.15.23:51072) 2024-03-30T04:02:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 -- 172.21.15.23:0/315238869 learned_addr learned my addr 172.21.15.23:0/315238869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163e00700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9de89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6168a00700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 -- 172.21.15.23:0/315238869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 msgr2=0x55c3c9de89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9de89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 -- 172.21.15.23:0/315238869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 msgr2=0x55c3c9df61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163400700 1 -- 172.21.15.23:0/315238869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3c9bbc3b0 con 0x55c3c9de8ef0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.678+0000 7f6163e00700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9de89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.679+0000 7f6163400700 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f616406a380 tx=0x7f6150005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=793627c7c0049c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.679+0000 7f6168a00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 0x7f61480a38f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f615c007b50 tx=0x7f615c009200 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.679+0000 7f6159200700 1 -- 172.21.15.23:0/315238869 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000fd70 con 0x55c3c9de8ef0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.679+0000 7f6159200700 1 -- 172.21.15.23:0/315238869 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615001d040 con 0x55c3c9de8ef0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.679+0000 7f6159200700 1 -- 172.21.15.23:0/315238869 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6150005040 con 0x55c3c9de8ef0 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 -- 172.21.15.23:0/315238869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 msgr2=0x55c3c9df38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f616406a380 tx=0x7f6150005dc0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 -- 172.21.15.23:0/315238869 shutdown_connections 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9de89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 --2- 172.21.15.23:0/315238869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df38b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 -- 172.21.15.23:0/315238869 >> 172.21.15.23:0/315238869 conn(0x55c3c9d486a0 msgr2=0x55c3c9de4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 -- 172.21.15.23:0/315238869 shutdown_connections 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.680+0000 7f617e427b80 1 -- 172.21.15.23:0/315238869 wait complete. 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.681+0000 7f617e427b80 1 -- start start 2024-03-30T04:02:17.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.681+0000 7f617e427b80 1 Processor -- start 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9cd6d80 con 0x55c3c9df3df0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9cd6a20 con 0x55c3c9de85d0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9bbe6a0 con 0x55c3c9de8ef0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f617e427b80 10 client.0 _ll_register_callbacks cb 0x55c3c9ce0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49096/0 (socket says 172.21.15.23:49096) 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 -- 172.21.15.23:0/3485558484 learned_addr learned my addr 172.21.15.23:0/3485558484 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6163400700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6163e00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 -- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 msgr2=0x55c3c9df0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 -- 172.21.15.23:0/3485558484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 msgr2=0x55c3c9df37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.682+0000 7f6168a00700 1 -- 172.21.15.23:0/3485558484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61641a6290 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f6168a00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f615c019fa0 tx=0x7f615c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fed0b2fe7644cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f614d600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c02a070 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f615c021070 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3c9cfb380 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615c006000 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f617e427b80 10 client.15762 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.683+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f616410e470 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c3c9cfb380 con 0x55c3c9df3df0 2024-03-30T04:02:17.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f615c02d020 con 0x55c3c9df3df0 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 10 client.15762 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 10 client.15762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 20 client.15762 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e03920 0x55c3c9e05d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 4 client.15762 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3c9d33700 con 0x55c3c9e03920 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.684+0000 7f6163e00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e03920 0x55c3c9e05d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.685+0000 7f6163e00700 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e03920 0x55c3c9e05d40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c1a502f86a3b1ace server_cookie=df05dc23775b4204 in_seq=0 out_seq=0 2024-03-30T04:02:17.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.685+0000 7f614d600700 10 client.15762 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:17.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.685+0000 7f614d600700 1 -- 172.21.15.23:0/3485558484 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c3c9d33700 con 0x55c3c9e03920 2024-03-30T04:02:17.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f614d600700 10 client.15762 handle_command_reply: tid=0 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 client.15762 shutdown 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 msgr2=0x55c3c9df10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f615c019fa0 tx=0x7f615c004b40 comp rx=0 tx=0).stop 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 shutdown_connections 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e03920 0x55c3c9e05d40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9df10b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 --2- 172.21.15.23:0/3485558484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f614d600700 0 client.15762 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:17.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 >> 172.21.15.23:0/3485558484 conn(0x55c3c9d486a0 msgr2=0x55c3c9dea6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 shutdown_connections 2024-03-30T04:02:17.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 1 -- 172.21.15.23:0/3485558484 wait complete. 2024-03-30T04:02:17.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.686+0000 7f617e427b80 20 client.15762 trim_cache size 0 max 16384 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 Processor -- start 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 -- start start 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9dfaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9dff160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9e060d0 con 0x55c3c9df3df0 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9de5660 con 0x55c3c9de8ef0 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.700+0000 7f617e427b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9dfaf50 con 0x55c3c9de85d0 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9dff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9dff160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49100/0 (socket says 172.21.15.23:49100) 2024-03-30T04:02:17.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6168a00700 1 -- 172.21.15.23:0/1634078126 learned_addr learned my addr 172.21.15.23:0/1634078126 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163e00700 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9dfaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 -- 172.21.15.23:0/1634078126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 msgr2=0x55c3c9dfaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9dfaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 -- 172.21.15.23:0/1634078126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 msgr2=0x55c3c9dff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9dff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 -- 172.21.15.23:0/1634078126 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3c9b68b60 con 0x55c3c9de8ef0 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163e00700 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9dfaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:17.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.701+0000 7f6163400700 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df32e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61500098c0 tx=0x7f615000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8908c3c898ba8990 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f6159c00700 1 -- 172.21.15.23:0/1634078126 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000fd70 con 0x55c3c9de8ef0 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f6159c00700 1 -- 172.21.15.23:0/1634078126 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61500050c0 con 0x55c3c9de8ef0 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f6159c00700 1 -- 172.21.15.23:0/1634078126 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615000b760 con 0x55c3c9de8ef0 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 -- 172.21.15.23:0/1634078126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 msgr2=0x55c3c9df32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df32e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61500098c0 tx=0x7f615000bf90 comp rx=0 tx=0).stop 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 -- 172.21.15.23:0/1634078126 shutdown_connections 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9dff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de85d0 0x55c3c9dfaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 --2- 172.21.15.23:0/1634078126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df32e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 -- 172.21.15.23:0/1634078126 >> 172.21.15.23:0/1634078126 conn(0x55c3c9d486a0 msgr2=0x55c3c9dddf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 -- 172.21.15.23:0/1634078126 shutdown_connections 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.702+0000 7f617e427b80 1 -- 172.21.15.23:0/1634078126 wait complete. 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.703+0000 7f617e427b80 1 -- start start 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.703+0000 7f617e427b80 1 Processor -- start 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9e56c90 con 0x55c3c9df3df0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9e56dd0 con 0x55c3c9de85d0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3c9e56f10 con 0x55c3c9de8ef0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f617e427b80 10 client.0 _ll_register_callbacks cb 0x55c3c9ce0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f6168a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49104/0 (socket says 172.21.15.23:49104) 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f6168a00700 1 -- 172.21.15.23:0/713035195 learned_addr learned my addr 172.21.15.23:0/713035195 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f6163e00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.704+0000 7f6163400700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 -- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 msgr2=0x55c3c9df9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 -- 172.21.15.23:0/713035195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 msgr2=0x55c3c9df8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 -- 172.21.15.23:0/713035195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6150009520 con 0x55c3c9df3df0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6163e00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f6168a00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f615c026da0 tx=0x7f615c009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4d1bd34387b4bb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f614d600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c02a050 con 0x55c3c9df3df0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f615001d040 con 0x55c3c9df3df0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f615c021040 con 0x55c3c9df3df0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f615c006000 con 0x55c3c9df3df0 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f617e427b80 10 client.15768 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.705+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c3c9bbc3b0 con 0x55c3c9df3df0 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f615001d040 con 0x55c3c9df3df0 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f615c01b450 con 0x55c3c9df3df0 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 10 client.15768 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 10 client.15768 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 20 client.15768 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e58620 0x55c3c9e5aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 4 client.15768 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c3c9b89cf0 con 0x55c3c9e58620 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.706+0000 7f6163e00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e58620 0x55c3c9e5aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f6163e00700 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e58620 0x55c3c9e5aa40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2cc158926037430e server_cookie=4decfcffcbf97888 in_seq=0 out_seq=0 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f614d600700 10 client.15768 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f614d600700 1 -- 172.21.15.23:0/713035195 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c3c9b89cf0 con 0x55c3c9e58620 2024-03-30T04:02:17.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f614d600700 10 client.15768 handle_command_reply: tid=0 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 client.15768 shutdown 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 msgr2=0x55c3c9e56750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f615c026da0 tx=0x7f615c009eb0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 shutdown_connections 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55c3c9e58620 0x55c3c9e5aa40 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c3c9df3df0 0x55c3c9e56750 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c3c9de85d0 0x55c3c9df8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 --2- 172.21.15.23:0/713035195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3c9de8ef0 0x55c3c9df9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f614d600700 0 client.15768 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 >> 172.21.15.23:0/713035195 conn(0x55c3c9d486a0 msgr2=0x55c3c9dff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 shutdown_connections 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.707+0000 7f617e427b80 1 -- 172.21.15.23:0/713035195 wait complete. 2024-03-30T04:02:17.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f617e427b80 20 client.15768 trim_cache size 0 max 16384 2024-03-30T04:02:17.712 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:17.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 msgr2=0x7f61480a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 0x7f61480a38f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f615c007b50 tx=0x7f615c009200 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 msgr2=0x7f6164199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f615400ec10 tx=0x7f615400ef20 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 shutdown_connections 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f616419dd40 0x7f616419e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6164100f60 0x7f6164199460 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61480a1430 0x7f61480a38f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 --2- 172.21.15.23:0/1682617169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6164106080 0x7f61641999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 >> 172.21.15.23:0/1682617169 conn(0x7f61640fc7c0 msgr2=0x7f61640fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 shutdown_connections 2024-03-30T04:02:17.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.708+0000 7f616aa00700 1 -- 172.21.15.23:0/1682617169 wait complete. 2024-03-30T04:02:17.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:02:17.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 -- 172.21.15.23:0/1424853087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed50106080 msgr2=0x7fed5010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 --2- 172.21.15.23:0/1424853087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed50106080 0x7fed5010dc00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fed4800b0a0 tx=0x7fed4800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 -- 172.21.15.23:0/1424853087 shutdown_connections 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 --2- 172.21.15.23:0/1424853087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed50106080 0x7fed5010dc00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 --2- 172.21.15.23:0/1424853087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 0x7fed50101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 --2- 172.21.15.23:0/1424853087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50101910 0x7fed50105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 -- 172.21.15.23:0/1424853087 >> 172.21.15.23:0/1424853087 conn(0x7fed500fc7c0 msgr2=0x7fed500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 -- 172.21.15.23:0/1424853087 shutdown_connections 2024-03-30T04:02:17.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.928+0000 7fed55000700 1 -- 172.21.15.23:0/1424853087 wait complete. 2024-03-30T04:02:17.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 Processor -- start 2024-03-30T04:02:17.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 -- start start 2024-03-30T04:02:17.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 0x7fed50199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50106080 0x7fed50199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019e8c0 con 0x7fed5019de40 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019ea30 con 0x7fed50106080 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.929+0000 7fed55000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5019ebf0 con 0x7fed50100f60 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49120/0 (socket says 172.21.15.23:49120) 2024-03-30T04:02:17.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 -- 172.21.15.23:0/1323723415 learned_addr learned my addr 172.21.15.23:0/1323723415 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4de00700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 0x7fed50199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4d400700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50106080 0x7fed50199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 -- 172.21.15.23:0/1323723415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 msgr2=0x7fed50199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 0x7fed50199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 -- 172.21.15.23:0/1323723415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50106080 msgr2=0x7fed50199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50106080 0x7fed50199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 -- 172.21.15.23:0/1323723415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed48009d00 con 0x7fed5019de40 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed4e800700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fed4800b070 tx=0x7fed48007cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed46a00700 1 -- 172.21.15.23:0/1323723415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4800e040 con 0x7fed5019de40 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.930+0000 7fed46a00700 1 -- 172.21.15.23:0/1323723415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed4800bdc0 con 0x7fed5019de40 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.931+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed501a60f0 con 0x7fed5019de40 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.931+0000 7fed46a00700 1 -- 172.21.15.23:0/1323723415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed480052a0 con 0x7fed5019de40 2024-03-30T04:02:17.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.931+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed501a6510 con 0x7fed5019de40 2024-03-30T04:02:17.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.934+0000 7fed46a00700 1 -- 172.21.15.23:0/1323723415 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fed4801d480 con 0x7fed5019de40 2024-03-30T04:02:17.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.935+0000 7fed46a00700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 0x7fed340a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.935+0000 7fed68a4fb80 1 Processor -- start 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.935+0000 7fed46a00700 1 -- 172.21.15.23:0/1323723415 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fed480ce3d0 con 0x7fed5019de40 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.935+0000 7fed68a4fb80 1 -- start start 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.935+0000 7fed4de00700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 0x7fed340a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a31b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 0x56323a3268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a3291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a274990 con 0x56323a31b5d0 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a272350 con 0x56323a326df0 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a26dc40 con 0x56323a31bef0 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a31b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a31b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49128/0 (socket says 172.21.15.23:49128) 2024-03-30T04:02:17.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4de00700 1 -- 172.21.15.23:0/2737747034 learned_addr learned my addr 172.21.15.23:0/2737747034 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4d400700 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 0x56323a3268b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4e800700 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a3291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.936+0000 7fed4d400700 1 -- 172.21.15.23:0/2737747034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 msgr2=0x56323a3291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4d400700 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a3291e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4de00700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 0x7fed340a38f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed40014660 tx=0x7fed40014a20 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4d400700 1 -- 172.21.15.23:0/2737747034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 msgr2=0x56323a31b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4d400700 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a31b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4d400700 1 -- 172.21.15.23:0/2737747034 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56323a0ef3b0 con 0x56323a31bef0 2024-03-30T04:02:17.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed4d400700 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 0x56323a3268b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed5006a380 tx=0x7fed3c009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecff67cb33f4655 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed3b400700 1 -- 172.21.15.23:0/2737747034 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c011c00 con 0x56323a31bef0 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed3b400700 1 -- 172.21.15.23:0/2737747034 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed3c004510 con 0x56323a31bef0 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.937+0000 7fed3b400700 1 -- 172.21.15.23:0/2737747034 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c005280 con 0x56323a31bef0 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 -- 172.21.15.23:0/2737747034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 msgr2=0x56323a3268b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 0x56323a3268b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed5006a380 tx=0x7fed3c009f60 comp rx=0 tx=0).stop 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 -- 172.21.15.23:0/2737747034 shutdown_connections 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a3291e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a31b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/2737747034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31bef0 0x56323a3268b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 -- 172.21.15.23:0/2737747034 >> 172.21.15.23:0/2737747034 conn(0x56323a27b6a0 msgr2=0x56323a317a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 -- 172.21.15.23:0/2737747034 shutdown_connections 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.938+0000 7fed68a4fb80 1 -- 172.21.15.23:0/2737747034 wait complete. 2024-03-30T04:02:17.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.939+0000 7fed68a4fb80 1 -- start start 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.939+0000 7fed68a4fb80 1 Processor -- start 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a31bef0 0x56323a323b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a326df0 0x56323a3242a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a209d80 con 0x56323a31b5d0 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a209a20 con 0x56323a31bef0 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a0f16a0 con 0x56323a326df0 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed68a4fb80 10 client.0 _ll_register_callbacks cb 0x56323a213470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed4de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed4de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49134/0 (socket says 172.21.15.23:49134) 2024-03-30T04:02:17.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed4de00700 1 -- 172.21.15.23:0/864164693 learned_addr learned my addr 172.21.15.23:0/864164693 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed4d400700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a31bef0 0x56323a323b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.940+0000 7fed4e800700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a326df0 0x56323a3242a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 -- 172.21.15.23:0/864164693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a326df0 msgr2=0x56323a3242a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a326df0 0x56323a3242a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 -- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a31bef0 msgr2=0x56323a323b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a31bef0 0x56323a323b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 -- 172.21.15.23:0/864164693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3c0098e0 con 0x56323a31b5d0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed4de00700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed40014270 tx=0x7fed4001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bfa398321d6c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40004050 con 0x56323a31b5d0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fed400041b0 con 0x56323a31b5d0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56323a22e380 con 0x56323a31b5d0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.941+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed40004370 con 0x56323a31b5d0 2024-03-30T04:02:17.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 10 client.15780 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56323a09bb60 con 0x56323a31b5d0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56323a22e380 con 0x56323a31b5d0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed40020440 con 0x56323a31b5d0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 10 client.15780 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 10 client.15780 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 20 client.15780 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a3369d0 0x56323a338df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 4 client.15780 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.942+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56323a266700 con 0x56323a3369d0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.943+0000 7fed4d400700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a3369d0 0x56323a338df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.943+0000 7fed4d400700 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a3369d0 0x56323a338df0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=608b4ca1307c34d0 server_cookie=a28d32debddb43be in_seq=0 out_seq=0 2024-03-30T04:02:17.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.943+0000 7fed2fe00700 10 client.15780 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed2fe00700 1 -- 172.21.15.23:0/864164693 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56323a266700 con 0x56323a3369d0 2024-03-30T04:02:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed2fe00700 10 client.15780 handle_command_reply: tid=0 2024-03-30T04:02:17.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 client.15780 shutdown 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 msgr2=0x56323a326820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed40014270 tx=0x7fed4001a000 comp rx=0 tx=0).stop 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 shutdown_connections 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a3369d0 0x56323a338df0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a326df0 0x56323a3242a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31b5d0 0x56323a326820 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/864164693 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a31bef0 0x56323a323b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed2fe00700 0 client.15780 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 >> 172.21.15.23:0/864164693 conn(0x56323a27b6a0 msgr2=0x56323a31d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 shutdown_connections 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 1 -- 172.21.15.23:0/864164693 wait complete. 2024-03-30T04:02:17.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.944+0000 7fed68a4fb80 20 client.15780 trim_cache size 0 max 16384 2024-03-30T04:02:17.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 Processor -- start 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 -- start start 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a310f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a318660 con 0x56323a31bef0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a32dfb0 con 0x56323a326df0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.958+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a31b3c0 con 0x56323a31b5d0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55182/0 (socket says 172.21.15.23:55182) 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 -- 172.21.15.23:0/1134368838 learned_addr learned my addr 172.21.15.23:0/1134368838 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4e800700 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a310f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 -- 172.21.15.23:0/1134368838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 msgr2=0x56323a310f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a310f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 -- 172.21.15.23:0/1134368838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 msgr2=0x56323a32da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 -- 172.21.15.23:0/1134368838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56323a233ea0 con 0x56323a31b5d0 2024-03-30T04:02:17.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.959+0000 7fed4de00700 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed40012a10 tx=0x7fed40012a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e8b40caf66af970 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed3be00700 1 -- 172.21.15.23:0/1134368838 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40014360 con 0x56323a31b5d0 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed3be00700 1 -- 172.21.15.23:0/1134368838 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed4001a070 con 0x56323a31b5d0 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed3be00700 1 -- 172.21.15.23:0/1134368838 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed400038e0 con 0x56323a31b5d0 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 -- 172.21.15.23:0/1134368838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 msgr2=0x56323a31e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed40012a10 tx=0x7fed40012a40 comp rx=0 tx=0).stop 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 -- 172.21.15.23:0/1134368838 shutdown_connections 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a310f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a31e610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/1134368838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 -- 172.21.15.23:0/1134368838 >> 172.21.15.23:0/1134368838 conn(0x56323a27b6a0 msgr2=0x56323a3105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 -- 172.21.15.23:0/1134368838 shutdown_connections 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.960+0000 7fed68a4fb80 1 -- 172.21.15.23:0/1134368838 wait complete. 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 -- start start 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 Processor -- start 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a32d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a32ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a32c160 con 0x56323a31bef0 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a32c2d0 con 0x56323a326df0 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56323a32c490 con 0x56323a31b5d0 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.961+0000 7fed68a4fb80 10 client.0 _ll_register_callbacks cb 0x56323a213470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:17.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a32d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49150/0 (socket says 172.21.15.23:49150) 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a32ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 -- 172.21.15.23:0/3895073224 learned_addr learned my addr 172.21.15.23:0/3895073224 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a32d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55190/0 (socket says 172.21.15.23:55190) 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 -- 172.21.15.23:0/3895073224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 msgr2=0x56323a32d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a32d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 -- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 msgr2=0x56323a32ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a32ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 -- 172.21.15.23:0/3895073224 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56323a09bb60 con 0x56323a31bef0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4e800700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a32ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed4d400700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed5006a380 tx=0x7fed3c0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54e96dbbfed48d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.962+0000 7fed2fe00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:17.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c02a070 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed3c004510 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fed3c0049a0 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed3c004ba0 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 10 client.15786 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56323a0f3cd0 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed3c004510 con 0x56323a31bef0 2024-03-30T04:02:17.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed3c005400 con 0x56323a31bef0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 10 client.15786 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 10 client.15786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 20 client.15786 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a38bac0 0x56323a38dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 4 client.15786 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.963+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56323a0bccf0 con 0x56323a38bac0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed4de00700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a38bac0 0x56323a38dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed4de00700 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a38bac0 0x56323a38dee0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8beeaf2c96a4c371 server_cookie=329cf787f40adc87 in_seq=0 out_seq=0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed2fe00700 10 client.15786 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed2fe00700 1 -- 172.21.15.23:0/3895073224 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56323a0bccf0 con 0x56323a38bac0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed2fe00700 10 client.15786 handle_command_reply: tid=0 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.964+0000 7fed68a4fb80 1 client.15786 shutdown 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 msgr2=0x56323a32d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fed5006a380 tx=0x7fed3c0048b0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 shutdown_connections 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56323a38bac0 0x56323a38dee0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56323a326df0 0x56323a32ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56323a31b5d0 0x56323a32d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 --2- 172.21.15.23:0/3895073224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56323a31bef0 0x56323a32d9d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 >> 172.21.15.23:0/3895073224 conn(0x56323a27b6a0 msgr2=0x56323a31ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed2fe00700 0 client.15786 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 shutdown_connections 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 1 -- 172.21.15.23:0/3895073224 wait complete. 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed68a4fb80 20 client.15786 trim_cache size 0 max 16384 2024-03-30T04:02:17.969 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 msgr2=0x7fed340a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.965+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 0x7fed340a38f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fed40014660 tx=0x7fed40014a20 comp rx=0 tx=0).stop 2024-03-30T04:02:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 msgr2=0x7fed5019e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:17.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fed4800b070 tx=0x7fed48007cf0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 shutdown_connections 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed5019de40 0x7fed5019e2c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed50100f60 0x7fed50199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed340a1430 0x7fed340a38f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 --2- 172.21.15.23:0/1323723415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed50106080 0x7fed50199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 >> 172.21.15.23:0/1323723415 conn(0x7fed500fc7c0 msgr2=0x7fed500fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 shutdown_connections 2024-03-30T04:02:17.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:17.966+0000 7fed55000700 1 -- 172.21.15.23:0/1323723415 wait complete. 2024-03-30T04:02:17.982 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:02:18.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.187+0000 7f7489200700 1 -- 172.21.15.23:0/374861989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 msgr2=0x7f748410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.187+0000 7f7489200700 1 --2- 172.21.15.23:0/374861989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f748410dc00 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b0a0 tx=0x7f747c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 -- 172.21.15.23:0/374861989 shutdown_connections 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 --2- 172.21.15.23:0/374861989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f748410dc00 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 --2- 172.21.15.23:0/374861989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484100f60 0x7f7484101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 --2- 172.21.15.23:0/374861989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484101910 0x7f7484105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 -- 172.21.15.23:0/374861989 >> 172.21.15.23:0/374861989 conn(0x7f74840fc7c0 msgr2=0x7f74840febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 -- 172.21.15.23:0/374861989 shutdown_connections 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.188+0000 7f7489200700 1 -- 172.21.15.23:0/374861989 wait complete. 2024-03-30T04:02:18.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 Processor -- start 2024-03-30T04:02:18.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 -- start start 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484100f60 0x7f74840730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 0x7f74841a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841a5b40 con 0x7f7484106080 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841a5c80 con 0x7f7484073b30 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.189+0000 7f7489200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74841a5e40 con 0x7f7484100f60 2024-03-30T04:02:18.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59222/0 (socket says 172.21.15.23:59222) 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 -- 172.21.15.23:0/379177117 learned_addr learned my addr 172.21.15.23:0/379177117 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7482800700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 0x7f74841a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481e00700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484100f60 0x7f74840730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 -- 172.21.15.23:0/379177117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484100f60 msgr2=0x7f74840730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484100f60 0x7f74840730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 -- 172.21.15.23:0/379177117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 msgr2=0x7f74841a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 0x7f74841a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 -- 172.21.15.23:0/379177117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c009d00 con 0x7f7484106080 2024-03-30T04:02:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7482800700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 0x7f74841a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.190+0000 7f7481400700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f747000d980 tx=0x7f747000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.191+0000 7f747aa00700 1 -- 172.21.15.23:0/379177117 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747000f890 con 0x7f7484106080 2024-03-30T04:02:18.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.191+0000 7f747aa00700 1 -- 172.21.15.23:0/379177117 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7470010460 con 0x7f7484106080 2024-03-30T04:02:18.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.191+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74841a6120 con 0x7f7484106080 2024-03-30T04:02:18.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.191+0000 7f747aa00700 1 -- 172.21.15.23:0/379177117 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747000e610 con 0x7f7484106080 2024-03-30T04:02:18.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.191+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74841a65e0 con 0x7f7484106080 2024-03-30T04:02:18.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.195+0000 7f747aa00700 1 -- 172.21.15.23:0/379177117 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f74700105d0 con 0x7f7484106080 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.196+0000 7f749cc81b80 1 Processor -- start 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.196+0000 7f749cc81b80 1 -- start start 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 0x55db59d089b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 0x55db59d138b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f747aa00700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 0x7f74680a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59c61990 con 0x55db59d13df0 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59c5f350 con 0x55db59d08ef0 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f749cc81b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59c5ac40 con 0x55db59d085d0 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f747aa00700 1 -- 172.21.15.23:0/379177117 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f74700ccd00 con 0x7f7484106080 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7482800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 0x55db59d089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7481400700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 0x7f74680a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7481e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 0x55db59d138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7482800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 0x55db59d089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37494/0 (socket says 172.21.15.23:37494) 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7482800700 1 -- 172.21.15.23:0/2868407230 learned_addr learned my addr 172.21.15.23:0/2868407230 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.197+0000 7f7482800700 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 -- 172.21.15.23:0/2868407230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 msgr2=0x55db59d089b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 0x55db59d089b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 -- 172.21.15.23:0/2868407230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 msgr2=0x55db59d161e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d161e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 -- 172.21.15.23:0/2868407230 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db59adc3b0 con 0x55db59d08ef0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7482800700 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481400700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 0x7f74680a3870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74700c9f70 tx=0x7f7470010bb0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f7481e00700 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 0x55db59d138b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7474009cb0 tx=0x7f7474011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cc9a350a0dc1802 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f746f400700 1 -- 172.21.15.23:0/2868407230 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474011d20 con 0x55db59d08ef0 2024-03-30T04:02:18.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f746f400700 1 -- 172.21.15.23:0/2868407230 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7474011e80 con 0x55db59d08ef0 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.198+0000 7f746f400700 1 -- 172.21.15.23:0/2868407230 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474005280 con 0x55db59d08ef0 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 -- 172.21.15.23:0/2868407230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 msgr2=0x55db59d138b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 0x55db59d138b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7474009cb0 tx=0x7f7474011440 comp rx=0 tx=0).stop 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 -- 172.21.15.23:0/2868407230 shutdown_connections 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d161e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d085d0 0x55db59d089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 --2- 172.21.15.23:0/2868407230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d08ef0 0x55db59d138b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 -- 172.21.15.23:0/2868407230 >> 172.21.15.23:0/2868407230 conn(0x55db59c686a0 msgr2=0x55db59d04a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 -- 172.21.15.23:0/2868407230 shutdown_connections 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.199+0000 7f749cc81b80 1 -- 172.21.15.23:0/2868407230 wait complete. 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 -- start start 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 Processor -- start 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d08ef0 0x55db59d10af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d13df0 0x55db59d11120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59bf6d80 con 0x55db59d08ef0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59bf6a20 con 0x55db59d085d0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.200+0000 7f749cc81b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59ade6a0 con 0x55db59d13df0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f749cc81b80 10 client.0 _ll_register_callbacks cb 0x55db59c00470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7481e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d08ef0 0x55db59d10af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43300/0 (socket says 172.21.15.23:43300) 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 -- 172.21.15.23:0/3505204996 learned_addr learned my addr 172.21.15.23:0/3505204996 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d13df0 0x55db59d11120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 -- 172.21.15.23:0/3505204996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d13df0 msgr2=0x55db59d11120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d13df0 0x55db59d11120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 -- 172.21.15.23:0/3505204996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d08ef0 msgr2=0x55db59d10af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d08ef0 0x55db59d10af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 -- 172.21.15.23:0/3505204996 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74740098e0 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7482800700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b780 tx=0x7f747c00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f4ccc4c98fcc52 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.201+0000 7f7463e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c00e070 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db59c53700 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f747c003a00 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f747c003bc0 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f749cc81b80 10 client.25805 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55db59a88b60 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.202+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db59c53700 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f747c00b0d0 con 0x55db59d085d0 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 10 client.25805 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 10 client.25805 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 20 client.25805 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:18.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d239e0 0x55db59d25dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 4 client.25805 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db59aa9cf0 con 0x55db59d239e0 2024-03-30T04:02:18.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.203+0000 7f7481e00700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d239e0 0x55db59d25dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.204+0000 7f7481e00700 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d239e0 0x55db59d25dd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc63f3359a5ed1e server_cookie=bdad9e0dac13a9cd in_seq=0 out_seq=0 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.204+0000 7f7463e00700 10 client.25805 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.204+0000 7f7463e00700 1 -- 172.21.15.23:0/3505204996 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55db59aa9cf0 con 0x55db59d239e0 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.204+0000 7f7463e00700 10 client.25805 handle_command_reply: tid=0 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.204+0000 7f749cc81b80 1 client.25805 shutdown 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 msgr2=0x55db59d13800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b780 tx=0x7f747c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 shutdown_connections 2024-03-30T04:02:18.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d239e0 0x55db59d25dd0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d13df0 0x55db59d11120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d13800 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3505204996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d08ef0 0x55db59d10af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f7463e00700 0 client.25805 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 >> 172.21.15.23:0/3505204996 conn(0x55db59c686a0 msgr2=0x55db59d0a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 shutdown_connections 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 1 -- 172.21.15.23:0/3505204996 wait complete. 2024-03-30T04:02:18.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.205+0000 7f749cc81b80 20 client.25805 trim_cache size 0 max 16384 2024-03-30T04:02:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.218+0000 7f749cc81b80 1 Processor -- start 2024-03-30T04:02:18.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.218+0000 7f749cc81b80 1 -- start start 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d1aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d13df0 0x55db59d13330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d09850 con 0x55db59d085d0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d114a0 con 0x55db59d13df0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f749cc81b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d11700 con 0x55db59d08ef0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f7482800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f7482800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59260/0 (socket says 172.21.15.23:59260) 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f7482800700 1 -- 172.21.15.23:0/156729591 learned_addr learned my addr 172.21.15.23:0/156729591 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.219+0000 7f7481e00700 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d1aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d13df0 0x55db59d13330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 -- 172.21.15.23:0/156729591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 msgr2=0x55db59d1aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d1aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 -- 172.21.15.23:0/156729591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d13df0 msgr2=0x55db59d13330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d13df0 0x55db59d13330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 -- 172.21.15.23:0/156729591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db59c20ea0 con 0x55db59d085d0 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f7482800700 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f747c008b00 tx=0x7f747c01f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ce0c6efdef2cc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f746fe00700 1 -- 172.21.15.23:0/156729591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c00e020 con 0x55db59d085d0 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f746fe00700 1 -- 172.21.15.23:0/156729591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f747c026070 con 0x55db59d085d0 2024-03-30T04:02:18.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.220+0000 7f746fe00700 1 -- 172.21.15.23:0/156729591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c0057a0 con 0x55db59d085d0 2024-03-30T04:02:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- 172.21.15.23:0/156729591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 msgr2=0x55db59d03f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f747c008b00 tx=0x7f747c01f480 comp rx=0 tx=0).stop 2024-03-30T04:02:18.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- 172.21.15.23:0/156729591 shutdown_connections 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d13df0 0x55db59d13330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d085d0 0x55db59d03f10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 --2- 172.21.15.23:0/156729591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d1aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- 172.21.15.23:0/156729591 >> 172.21.15.23:0/156729591 conn(0x55db59c686a0 msgr2=0x55db59cfd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- 172.21.15.23:0/156729591 shutdown_connections 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- 172.21.15.23:0/156729591 wait complete. 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 -- start start 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.221+0000 7f749cc81b80 1 Processor -- start 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d1a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d19250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d19790 con 0x55db59d13df0 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d19900 con 0x55db59d085d0 2024-03-30T04:02:18.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db59d19ac0 con 0x55db59d08ef0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f749cc81b80 10 client.0 _ll_register_callbacks cb 0x55db59c00470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f7482800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d19250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f7481e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f7481e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37516/0 (socket says 172.21.15.23:37516) 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f7481e00700 1 -- 172.21.15.23:0/3055663171 learned_addr learned my addr 172.21.15.23:0/3055663171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.222+0000 7f7482800700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d1a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 -- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 msgr2=0x55db59d1a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d1a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 -- 172.21.15.23:0/3055663171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 msgr2=0x55db59d19250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d19250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 -- 172.21.15.23:0/3055663171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db59a88b60 con 0x55db59d08ef0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7482800700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d1a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7481e00700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7474009cb0 tx=0x7f7474009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8d8a7c04c7bd089 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7463e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474004040 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.223+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db59c1b380 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f74740041a0 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7474004360 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 10 client.25798 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55db59ae0cd0 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db59c1b380 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7474025420 con 0x55db59d08ef0 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 10 client.25798 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 10 client.25798 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 20 client.25798 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d78c80 0x55db59d7b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 4 client.25798 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.224+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55db59adece0 con 0x55db59d78c80 2024-03-30T04:02:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f7482800700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d78c80 0x55db59d7b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f7482800700 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d78c80 0x55db59d7b0a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c61476f2bacbd544 server_cookie=3c8f7a29ecfd8f8f in_seq=0 out_seq=0 2024-03-30T04:02:18.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f7463e00700 10 client.25798 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f7463e00700 1 -- 172.21.15.23:0/3055663171 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55db59adece0 con 0x55db59d78c80 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f7463e00700 10 client.25798 handle_command_reply: tid=0 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f749cc81b80 1 client.25798 shutdown 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 msgr2=0x55db59d18d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.225+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7474009cb0 tx=0x7f7474009bf0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 shutdown_connections 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55db59d13df0 0x55db59d19250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55db59d085d0 0x55db59d1a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55db59d78c80 0x55db59d7b0a0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 --2- 172.21.15.23:0/3055663171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55db59d08ef0 0x55db59d18d10 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 >> 172.21.15.23:0/3055663171 conn(0x55db59c686a0 msgr2=0x55db59d044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f7463e00700 0 client.25798 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 shutdown_connections 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 1 -- 172.21.15.23:0/3055663171 wait complete. 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f749cc81b80 20 client.25798 trim_cache size 0 max 16384 2024-03-30T04:02:18.229 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 msgr2=0x7f74680a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 0x7f74680a3870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74700c9f70 tx=0x7f7470010bb0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 msgr2=0x7f74840735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.226+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f747000d980 tx=0x7f747000dc90 comp rx=0 tx=0).stop 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 shutdown_connections 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74680a13b0 0x7f74680a3870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7484073b30 0x7f74841a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7484100f60 0x7f74840730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 --2- 172.21.15.23:0/379177117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7484106080 0x7f74840735f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 >> 172.21.15.23:0/379177117 conn(0x7f74840fc7c0 msgr2=0x7f74840fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 shutdown_connections 2024-03-30T04:02:18.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.227+0000 7f7489200700 1 -- 172.21.15.23:0/379177117 wait complete. 2024-03-30T04:02:18.241 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:17 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/722186520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:18.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:17 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/722186520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:18.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:02:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:17 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/722186520' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 -- 172.21.15.23:0/2898853083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34101910 msgr2=0x7f6b34105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 --2- 172.21.15.23:0/2898853083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34101910 0x7f6b34105950 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6b20009990 tx=0x7f6b20009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 -- 172.21.15.23:0/2898853083 shutdown_connections 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 --2- 172.21.15.23:0/2898853083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b34106080 0x7f6b3410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 --2- 172.21.15.23:0/2898853083 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 0x7f6b34101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 --2- 172.21.15.23:0/2898853083 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34101910 0x7f6b34105950 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 -- 172.21.15.23:0/2898853083 >> 172.21.15.23:0/2898853083 conn(0x7f6b340fc7c0 msgr2=0x7f6b340febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 -- 172.21.15.23:0/2898853083 shutdown_connections 2024-03-30T04:02:18.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.448+0000 7f6b39200700 1 -- 172.21.15.23:0/2898853083 wait complete. 2024-03-30T04:02:18.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.449+0000 7f6b39200700 1 Processor -- start 2024-03-30T04:02:18.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.449+0000 7f6b39200700 1 -- start start 2024-03-30T04:02:18.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 0x7f6b34199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3419dcc0 0x7f6b3419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3419e740 con 0x7f6b34106080 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3419e8b0 con 0x7f6b3419dcc0 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b39200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3419ea70 con 0x7f6b34100f60 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59278/0 (socket says 172.21.15.23:59278) 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 -- 172.21.15.23:0/3074169639 learned_addr learned my addr 172.21.15.23:0/3074169639 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31e00700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 0x7f6b34199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b32800700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3419dcc0 0x7f6b3419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 -- 172.21.15.23:0/3074169639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 msgr2=0x7f6b34199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 0x7f6b34199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 -- 172.21.15.23:0/3074169639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3419dcc0 msgr2=0x7f6b3419e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3419dcc0 0x7f6b3419e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.450+0000 7f6b31400700 1 -- 172.21.15.23:0/3074169639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b200095f0 con 0x7f6b34106080 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b31400700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6b20000c00 tx=0x7f6b20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b2aa00700 1 -- 172.21.15.23:0/3074169639 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2001d070 con 0x7f6b34106080 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b2aa00700 1 -- 172.21.15.23:0/3074169639 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b20021410 con 0x7f6b34106080 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341a5f70 con 0x7f6b34106080 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b2aa00700 1 -- 172.21.15.23:0/3074169639 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b20017650 con 0x7f6b34106080 2024-03-30T04:02:18.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.451+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b341a6330 con 0x7f6b34106080 2024-03-30T04:02:18.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.456+0000 7f6b2aa00700 1 -- 172.21.15.23:0/3074169639 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f6b20021580 con 0x7f6b34106080 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 Processor -- start 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 -- start start 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b2aa00700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 0x7f6b180a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b2aa00700 1 -- 172.21.15.23:0/3074169639 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6b200ce5a0 con 0x7f6b34106080 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a8e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a911e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce39dc990 con 0x564ce3a8edf0 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce39da350 con 0x564ce3a83ef0 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce39d5c40 con 0x564ce3a835d0 2024-03-30T04:02:18.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b32800700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 0x7f6b180a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.457+0000 7f6b32800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a911e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37540/0 (socket says 172.21.15.23:37540) 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 -- 172.21.15.23:0/2990640860 learned_addr learned my addr 172.21.15.23:0/2990640860 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31400700 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a8e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 -- 172.21.15.23:0/2990640860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 msgr2=0x564ce3a8e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a8e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 -- 172.21.15.23:0/2990640860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 msgr2=0x564ce3a911e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a911e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 -- 172.21.15.23:0/2990640860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ce38573b0 con 0x564ce3a835d0 2024-03-30T04:02:18.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31400700 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a8e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:18.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b32800700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 0x7f6b180a3870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c007af0 tx=0x7f6b2c009bf0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.458+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d80 tx=0x7f6b24011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1e22ba447b8e0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b1f400700 1 -- 172.21.15.23:0/2990640860 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24004230 con 0x564ce3a835d0 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b1f400700 1 -- 172.21.15.23:0/2990640860 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b24004390 con 0x564ce3a835d0 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b1f400700 1 -- 172.21.15.23:0/2990640860 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24019860 con 0x564ce3a835d0 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2990640860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 msgr2=0x564ce3a839b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009d80 tx=0x7f6b24011720 comp rx=0 tx=0).stop 2024-03-30T04:02:18.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2990640860 shutdown_connections 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a911e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2990640860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a8e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2990640860 >> 172.21.15.23:0/2990640860 conn(0x564ce39e36a0 msgr2=0x564ce3a7fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2990640860 shutdown_connections 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.459+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2990640860 wait complete. 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.460+0000 7f6b4cc30b80 1 -- start start 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.460+0000 7f6b4cc30b80 1 Processor -- start 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a83ef0 0x564ce3a8ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a8edf0 0x564ce3a8bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3971d80 con 0x564ce3a835d0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3971a20 con 0x564ce3a8edf0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce38596a0 con 0x564ce3a83ef0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b4cc30b80 10 client.0 _ll_register_callbacks cb 0x564ce397b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59284/0 (socket says 172.21.15.23:59284) 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b31e00700 1 -- 172.21.15.23:0/2696920406 learned_addr learned my addr 172.21.15.23:0/2696920406 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.461+0000 7f6b31400700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a83ef0 0x564ce3a8ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b32800700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a8edf0 0x564ce3a8bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 -- 172.21.15.23:0/2696920406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a83ef0 msgr2=0x564ce3a8ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a83ef0 0x564ce3a8ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 -- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a8edf0 msgr2=0x564ce3a8bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a8edf0 0x564ce3a8bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 -- 172.21.15.23:0/2696920406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b240099e0 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6b24003f40 tx=0x7f6b240038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e49e191cfc9ad65 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b13e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.462+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24005450 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6b24005ce0 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ce3996380 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b24009ed0 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 10 client.15816 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ce3803b60 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564ce3996380 con 0x564ce3a835d0 2024-03-30T04:02:18.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6b24028020 con 0x564ce3a835d0 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 10 client.15816 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 20 client.15816 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3a9e780 0x564ce3aa0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 4 client.15816 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.463+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ce39ce700 con 0x564ce3a9e780 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.464+0000 7f6b31400700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3a9e780 0x564ce3aa0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.464+0000 7f6b31400700 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3a9e780 0x564ce3aa0b70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34eb0bdc1303bb2c server_cookie=b018c600947e3f77 in_seq=0 out_seq=0 2024-03-30T04:02:18.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.464+0000 7f6b13e00700 10 client.15816 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:18.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b13e00700 1 -- 172.21.15.23:0/2696920406 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564ce39ce700 con 0x564ce3a9e780 2024-03-30T04:02:18.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b13e00700 10 client.15816 handle_command_reply: tid=0 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 client.15816 shutdown 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 msgr2=0x564ce3a8e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6b24003f40 tx=0x7f6b240038b0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 shutdown_connections 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a8edf0 0x564ce3a8bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a8e710 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3a9e780 0x564ce3aa0b70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2696920406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a83ef0 0x564ce3a8ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 >> 172.21.15.23:0/2696920406 conn(0x564ce39e36a0 msgr2=0x564ce3a85580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b13e00700 0 client.15816 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.465+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 shutdown_connections 2024-03-30T04:02:18.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.466+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2696920406 wait complete. 2024-03-30T04:02:18.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.466+0000 7f6b4cc30b80 20 client.15816 trim_cache size 0 max 16384 2024-03-30T04:02:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.479+0000 7f6b4cc30b80 1 Processor -- start 2024-03-30T04:02:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.479+0000 7f6b4cc30b80 1 -- start start 2024-03-30T04:02:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a86d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 0x564ce3a87290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3a8c2f0 con 0x564ce3a835d0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3aa0ea0 con 0x564ce3a83ef0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3a8c590 con 0x564ce3a8edf0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59296/0 (socket says 172.21.15.23:59296) 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 -- 172.21.15.23:0/2192049455 learned_addr learned my addr 172.21.15.23:0/2192049455 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31400700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a86d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b32800700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 0x564ce3a87290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 -- 172.21.15.23:0/2192049455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 msgr2=0x564ce3a87290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 0x564ce3a87290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 -- 172.21.15.23:0/2192049455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 msgr2=0x564ce3a86d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a86d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b31e00700 1 -- 172.21.15.23:0/2192049455 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ce399bea0 con 0x564ce3a835d0 2024-03-30T04:02:18.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.480+0000 7f6b32800700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 0x564ce3a87290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6b24019490 tx=0x7f6b24019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee4e81c50f2718d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b1fe00700 1 -- 172.21.15.23:0/2192049455 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24011a70 con 0x564ce3a835d0 2024-03-30T04:02:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b1fe00700 1 -- 172.21.15.23:0/2192049455 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b24019700 con 0x564ce3a835d0 2024-03-30T04:02:18.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b1fe00700 1 -- 172.21.15.23:0/2192049455 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400f3c0 con 0x564ce3a835d0 2024-03-30T04:02:18.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2192049455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 msgr2=0x564ce3a839b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6b24019490 tx=0x7f6b24019630 comp rx=0 tx=0).stop 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2192049455 shutdown_connections 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a8edf0 0x564ce3a87290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a835d0 0x564ce3a839b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2192049455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a86d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2192049455 >> 172.21.15.23:0/2192049455 conn(0x564ce39e36a0 msgr2=0x564ce3aa0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.481+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2192049455 shutdown_connections 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.482+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2192049455 wait complete. 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.482+0000 7f6b4cc30b80 1 -- start start 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.482+0000 7f6b4cc30b80 1 Processor -- start 2024-03-30T04:02:18.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.482+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a93300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.482+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b4cc30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a93d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3af1580 con 0x564ce3a8edf0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3af16c0 con 0x564ce3a83ef0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b4cc30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ce3af1800 con 0x564ce3a835d0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b4cc30b80 10 client.0 _ll_register_callbacks cb 0x564ce397b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b32800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a93d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43378/0 (socket says 172.21.15.23:43378) 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 -- 172.21.15.23:0/2326860338 learned_addr learned my addr 172.21.15.23:0/2326860338 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a93300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 -- 172.21.15.23:0/2326860338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 msgr2=0x564ce3a93300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a93300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 -- 172.21.15.23:0/2326860338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 msgr2=0x564ce3a93d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a93d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 -- 172.21.15.23:0/2326860338 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2402f040 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b31400700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000f740 tx=0x7f6b20005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0e39539f7284157 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.483+0000 7f6b13e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2009b830 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b24004390 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6b2009b130 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b2009ef60 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 10 client.25829 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ce385bcd0 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b24004390 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6b2007c020 con 0x564ce3a83ef0 2024-03-30T04:02:18.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 10 client.25829 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 10 client.25829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 20 client.25829 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3af2bb0 0x564ce3af4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 4 client.25829 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.484+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564ce3824cf0 con 0x564ce3af2bb0 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3af2bb0 0x564ce3af4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b31e00700 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3af2bb0 0x564ce3af4fd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a9ae0cd91f6927 server_cookie=2c51a58a97506a4b in_seq=0 out_seq=0 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b13e00700 10 client.25829 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b13e00700 1 -- 172.21.15.23:0/2326860338 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564ce3824cf0 con 0x564ce3af2bb0 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b13e00700 10 client.25829 handle_command_reply: tid=0 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b4cc30b80 1 client.25829 shutdown 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.485+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 msgr2=0x564ce3a93840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6b2000f740 tx=0x7f6b20005c70 comp rx=0 tx=0).stop 2024-03-30T04:02:18.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 shutdown_connections 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564ce3a8edf0 0x564ce3a93d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564ce3af2bb0 0x564ce3af4fd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce3a835d0 0x564ce3a93300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 --2- 172.21.15.23:0/2326860338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564ce3a83ef0 0x564ce3a93840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b13e00700 0 client.25829 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 >> 172.21.15.23:0/2326860338 conn(0x564ce39e36a0 msgr2=0x564ce3a86d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 shutdown_connections 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 1 -- 172.21.15.23:0/2326860338 wait complete. 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b4cc30b80 20 client.25829 trim_cache size 0 max 16384 2024-03-30T04:02:18.489 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 msgr2=0x7f6b180a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 0x7f6b180a3870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c007af0 tx=0x7f6b2c009bf0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 msgr2=0x7f6b34199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.486+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6b20000c00 tx=0x7f6b20004730 comp rx=0 tx=0).stop 2024-03-30T04:02:18.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 shutdown_connections 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b180a13b0 0x7f6b180a3870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3419dcc0 0x7f6b3419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b34100f60 0x7f6b34199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 --2- 172.21.15.23:0/3074169639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b34106080 0x7f6b34199990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 >> 172.21.15.23:0/3074169639 conn(0x7f6b340fc7c0 msgr2=0x7f6b340feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 shutdown_connections 2024-03-30T04:02:18.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:18.487+0000 7f6b39200700 1 -- 172.21.15.23:0/3074169639 wait complete. 2024-03-30T04:02:19.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:18 smithi193 ceph-mon[79581]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-30T04:02:19.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:18 smithi023 ceph-mon[38254]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-30T04:02:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:18 smithi154 ceph-mon[79161]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 13 op/s 2024-03-30T04:02:19.504 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:02:19.504 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 -- 172.21.15.23:0/2353946775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc106080 msgr2=0x7f51cc10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 --2- 172.21.15.23:0/2353946775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc106080 0x7f51cc10dc00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f51c400b0a0 tx=0x7f51c400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 -- 172.21.15.23:0/2353946775 shutdown_connections 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 --2- 172.21.15.23:0/2353946775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc106080 0x7f51cc10dc00 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 --2- 172.21.15.23:0/2353946775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc100f60 0x7f51cc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 --2- 172.21.15.23:0/2353946775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc101910 0x7f51cc105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 -- 172.21.15.23:0/2353946775 >> 172.21.15.23:0/2353946775 conn(0x7f51cc0fc7c0 msgr2=0x7f51cc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 -- 172.21.15.23:0/2353946775 shutdown_connections 2024-03-30T04:02:19.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.721+0000 7f51d4000700 1 -- 172.21.15.23:0/2353946775 wait complete. 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 Processor -- start 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 -- start start 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc106080 0x7f51cc197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 0x7f51cc19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19c6a0 con 0x7f51cc100f60 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19c810 con 0x7f51cc19bc20 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.722+0000 7f51d4000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc19c9d0 con 0x7f51cc106080 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59314/0 (socket says 172.21.15.23:59314) 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 -- 172.21.15.23:0/1665072656 learned_addr learned my addr 172.21.15.23:0/1665072656 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d2000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 0x7f51cc19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d0c00700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc106080 0x7f51cc197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 -- 172.21.15.23:0/1665072656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc106080 msgr2=0x7f51cc197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc106080 0x7f51cc197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 -- 172.21.15.23:0/1665072656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 msgr2=0x7f51cc19c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 0x7f51cc19c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 -- 172.21.15.23:0/1665072656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c4009d00 con 0x7f51cc100f60 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d2000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 0x7f51cc19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.723+0000 7f51d1600700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51c000e9a0 tx=0x7f51c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.724+0000 7f51ca000700 1 -- 172.21.15.23:0/1665072656 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0009790 con 0x7f51cc100f60 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.724+0000 7f51ca000700 1 -- 172.21.15.23:0/1665072656 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c0010430 con 0x7f51cc100f60 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.724+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc1a3f30 con 0x7f51cc100f60 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.724+0000 7f51ca000700 1 -- 172.21.15.23:0/1665072656 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0018680 con 0x7f51cc100f60 2024-03-30T04:02:19.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.724+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc1a43b0 con 0x7f51cc100f60 2024-03-30T04:02:19.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.727+0000 7f51ca000700 1 -- 172.21.15.23:0/1665072656 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f51c00105a0 con 0x7f51cc100f60 2024-03-30T04:02:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.728+0000 7f51ca000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 0x7f51b40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.728+0000 7f51e7be3b80 1 Processor -- start 2024-03-30T04:02:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.728+0000 7f51ca000700 1 -- 172.21.15.23:0/1665072656 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f51c0014070 con 0x7f51cc100f60 2024-03-30T04:02:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.728+0000 7f51d0c00700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 0x7f51b40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.728+0000 7f51e7be3b80 1 -- start start 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 0x558523aeb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523aebe70 0x558523aee260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523a39a10 con 0x558523ae0f70 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523a373d0 con 0x558523aebe70 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523a32cc0 con 0x558523ae0650 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d0c00700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 0x7f51b40a3870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51cc198890 tx=0x7f51b8009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d0c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 0x558523aeb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d1600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d1600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37594/0 (socket says 172.21.15.23:37594) 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d1600700 1 -- 172.21.15.23:0/2420393107 learned_addr learned my addr 172.21.15.23:0/2420393107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.729+0000 7f51d2000700 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523aebe70 0x558523aee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 -- 172.21.15.23:0/2420393107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523aebe70 msgr2=0x558523aee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523aebe70 0x558523aee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 -- 172.21.15.23:0/2420393107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 msgr2=0x558523aeb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 0x558523aeb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 -- 172.21.15.23:0/2420393107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585238b4430 con 0x558523ae0650 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d0c00700 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 0x558523aeb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51d1600700 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f51c0099fb0 tx=0x7f51c0099a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=428c18a76d1de021 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51be600700 1 -- 172.21.15.23:0/2420393107 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c009a020 con 0x558523ae0650 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51be600700 1 -- 172.21.15.23:0/2420393107 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c009de50 con 0x558523ae0650 2024-03-30T04:02:19.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.730+0000 7f51be600700 1 -- 172.21.15.23:0/2420393107 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0099770 con 0x558523ae0650 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2420393107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 msgr2=0x558523ae0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f51c0099fb0 tx=0x7f51c0099a70 comp rx=0 tx=0).stop 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2420393107 shutdown_connections 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523aebe70 0x558523aee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523ae0a30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2420393107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523ae0f70 0x558523aeb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2420393107 >> 172.21.15.23:0/2420393107 conn(0x558523a40720 msgr2=0x558523adcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2420393107 shutdown_connections 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.731+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2420393107 wait complete. 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.732+0000 7f51e7be3b80 1 -- start start 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.732+0000 7f51e7be3b80 1 Processor -- start 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523aeb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585239cee00 con 0x558523aebe70 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585239ceaa0 con 0x558523ae0f70 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585238b6720 con 0x558523ae0650 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51e7be3b80 10 client.0 _ll_register_callbacks cb 0x5585239d84f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43410/0 (socket says 172.21.15.23:43410) 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d0c00700 1 -- 172.21.15.23:0/4074462209 learned_addr learned my addr 172.21.15.23:0/4074462209 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d1600700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523aeb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d0c00700 1 -- 172.21.15.23:0/4074462209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 msgr2=0x558523aeb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.733+0000 7f51d0c00700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523aeb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51d0c00700 1 -- 172.21.15.23:0/4074462209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 msgr2=0x558523ae9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51d0c00700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51d0c00700 1 -- 172.21.15.23:0/4074462209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b8017070 con 0x558523ae0f70 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51d0c00700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51b801f780 tx=0x7f51b80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bb1ad71cd1e8b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51b2a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b8004020 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585239f3400 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f51b8004180 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51b8004340 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51e7be3b80 10 client.25841 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.734+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f51cc1a43b0 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5585239f3400 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f51b800e7f0 con 0x558523ae0f70 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 10 client.25841 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 20 client.25841 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:19.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523afba10 0x558523afde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 4 client.25841 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558523a2b780 con 0x558523afba10 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.735+0000 7f51d1600700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523afba10 0x558523afde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.736+0000 7f51d1600700 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523afba10 0x558523afde30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8fb63b57cb47415 server_cookie=74a461f855a83b39 in_seq=0 out_seq=0 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.736+0000 7f51b2a00700 10 client.25841 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.737+0000 7f51b2a00700 1 -- 172.21.15.23:0/4074462209 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558523a2b780 con 0x558523afba10 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.737+0000 7f51b2a00700 10 client.25841 handle_command_reply: tid=0 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 client.25841 shutdown 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 msgr2=0x558523ae8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51b801f780 tx=0x7f51b80047c0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 shutdown_connections 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523afba10 0x558523afde30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0650 0x558523aeb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/4074462209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0f70 0x558523ae8b40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51b2a00700 0 client.25841 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 >> 172.21.15.23:0/4074462209 conn(0x558523a40720 msgr2=0x558523ae2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 shutdown_connections 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 1 -- 172.21.15.23:0/4074462209 wait complete. 2024-03-30T04:02:19.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.738+0000 7f51e7be3b80 20 client.25841 trim_cache size 0 max 16384 2024-03-30T04:02:19.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 Processor -- start 2024-03-30T04:02:19.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 -- start start 2024-03-30T04:02:19.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523aeb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523add6e0 con 0x558523aebe70 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523ae94b0 con 0x558523ae0650 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523ae7090 con 0x558523ae0f70 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51d2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51d2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59344/0 (socket says 172.21.15.23:59344) 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51d2000700 1 -- 172.21.15.23:0/2390678992 learned_addr learned my addr 172.21.15.23:0/2390678992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51d1600700 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.756+0000 7f51d0c00700 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523aeb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 -- 172.21.15.23:0/2390678992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 msgr2=0x558523aeb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523aeb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 -- 172.21.15.23:0/2390678992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 msgr2=0x558523af7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 -- 172.21.15.23:0/2390678992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558523b02a60 con 0x558523aebe70 2024-03-30T04:02:19.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51d2000700 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f51c400d010 tx=0x7f51c40094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c2857c168cc494 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51bf000700 1 -- 172.21.15.23:0/2390678992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c400e040 con 0x558523aebe70 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51bf000700 1 -- 172.21.15.23:0/2390678992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c4022070 con 0x558523aebe70 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.757+0000 7f51bf000700 1 -- 172.21.15.23:0/2390678992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40276d0 con 0x558523aebe70 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2390678992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 msgr2=0x558523ae5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f51c400d010 tx=0x7f51c40094b0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2390678992 shutdown_connections 2024-03-30T04:02:19.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523ae5dc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/2390678992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523aeb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2390678992 >> 172.21.15.23:0/2390678992 conn(0x558523a40720 msgr2=0x558523af7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2390678992 shutdown_connections 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- 172.21.15.23:0/2390678992 wait complete. 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.758+0000 7f51e7be3b80 1 -- start start 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 Processor -- start 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523b4f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523b4fd90 con 0x558523aebe70 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523b4ff00 con 0x558523ae0650 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558523b500c0 con 0x558523ae0f70 2024-03-30T04:02:19.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51e7be3b80 10 client.0 _ll_register_callbacks cb 0x5585239d84f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51d0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51d2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523b4f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51d0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37620/0 (socket says 172.21.15.23:37620) 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51d0c00700 1 -- 172.21.15.23:0/89367379 learned_addr learned my addr 172.21.15.23:0/89367379 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.759+0000 7f51d1600700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 -- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 msgr2=0x558523af19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 -- 172.21.15.23:0/89367379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 msgr2=0x558523b4f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523b4f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 -- 172.21.15.23:0/89367379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc10e470 con 0x558523ae0f70 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51d0c00700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f51b8016040 tx=0x7f51b801feb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90eff1bcae11b633 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51b2a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b80092d0 con 0x558523ae0f70 2024-03-30T04:02:19.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.760+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f51c009de50 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f51b8004040 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51b8004200 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 10 client.25834 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558523860be0 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f51c009de50 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f51b8005160 con 0x558523ae0f70 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 10 client.25834 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 10 client.25834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 20 client.25834 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523b518c0 0x558523b53ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 4 client.25834 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.761+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558523881d70 con 0x558523b518c0 2024-03-30T04:02:19.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.762+0000 7f51d1600700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523b518c0 0x558523b53ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.762+0000 7f51d1600700 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523b518c0 0x558523b53ce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cee59c23cd48f31f server_cookie=e8b3d0a3c7c1ba97 in_seq=0 out_seq=0 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.762+0000 7f51b2a00700 10 client.25834 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51b2a00700 1 -- 172.21.15.23:0/89367379 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558523881d70 con 0x558523b518c0 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51b2a00700 10 client.25834 handle_command_reply: tid=0 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 client.25834 shutdown 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 msgr2=0x558523af1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f51b8016040 tx=0x7f51b801feb0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 shutdown_connections 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558523aebe70 0x558523b4f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558523ae0650 0x558523af19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558523b518c0 0x558523b53ce0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 --2- 172.21.15.23:0/89367379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558523ae0f70 0x558523af1ee0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 >> 172.21.15.23:0/89367379 conn(0x558523a40720 msgr2=0x558523ae8080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.763+0000 7f51b2a00700 0 client.25834 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:19.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 shutdown_connections 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51e7be3b80 1 -- 172.21.15.23:0/89367379 wait complete. 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51e7be3b80 20 client.25834 trim_cache size 0 max 16384 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bfa8cff9-e63f-4988-91d7-fcf612317eaf", 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:02:19.766 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 msgr2=0x7f51b40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 0x7f51b40a3870 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f51cc198890 tx=0x7f51b8009260 comp rx=0 tx=0).stop 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 msgr2=0x7f51cc197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51c000e9a0 tx=0x7f51c000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 shutdown_connections 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f51b40a13b0 0x7f51b40a3870 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51cc19bc20 0x7f51cc19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51cc100f60 0x7f51cc197340 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 --2- 172.21.15.23:0/1665072656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51cc106080 0x7f51cc197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.764+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 >> 172.21.15.23:0/1665072656 conn(0x7f51cc0fc7c0 msgr2=0x7f51cc0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.765+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 shutdown_connections 2024-03-30T04:02:19.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.765+0000 7f51d4000700 1 -- 172.21.15.23:0/1665072656 wait complete. 2024-03-30T04:02:19.782 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:02:19.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.986+0000 7f7d08e00700 1 -- 172.21.15.23:0/1725739224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 msgr2=0x7f7d04101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.986+0000 7f7d08e00700 1 --2- 172.21.15.23:0/1725739224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04101340 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40098f0 tx=0x7f7cf4009c00 comp rx=0 tx=0).stop 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 -- 172.21.15.23:0/1725739224 shutdown_connections 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 --2- 172.21.15.23:0/1725739224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d0410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 --2- 172.21.15.23:0/1725739224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04101340 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 --2- 172.21.15.23:0/1725739224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d04101910 0x7f7d04105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 -- 172.21.15.23:0/1725739224 >> 172.21.15.23:0/1725739224 conn(0x7f7d040fc7c0 msgr2=0x7f7d040febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 -- 172.21.15.23:0/1725739224 shutdown_connections 2024-03-30T04:02:19.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.987+0000 7f7d08e00700 1 -- 172.21.15.23:0/1725739224 wait complete. 2024-03-30T04:02:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 Processor -- start 2024-03-30T04:02:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 -- start start 2024-03-30T04:02:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d0419dce0 0x7f7d0419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0419e760 con 0x7f7d04106080 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0419e8d0 con 0x7f7d04100f60 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.988+0000 7f7d08e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0419ea90 con 0x7f7d0419dce0 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59366/0 (socket says 172.21.15.23:59366) 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 -- 172.21.15.23:0/2283780300 learned_addr learned my addr 172.21.15.23:0/2283780300 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d02800700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d0419dce0 0x7f7d0419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 -- 172.21.15.23:0/2283780300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d0419dce0 msgr2=0x7f7d0419e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d0419dce0 0x7f7d0419e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 -- 172.21.15.23:0/2283780300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 msgr2=0x7f7d04199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 -- 172.21.15.23:0/2283780300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4009520 con 0x7f7d04106080 2024-03-30T04:02:19.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7d01400700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000e9f0 tx=0x7f7cf000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.989+0000 7f7cfaa00700 1 -- 172.21.15.23:0/2283780300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0009690 con 0x7f7d04106080 2024-03-30T04:02:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.990+0000 7f7cfaa00700 1 -- 172.21.15.23:0/2283780300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf000ce80 con 0x7f7d04106080 2024-03-30T04:02:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.990+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d041a5ff0 con 0x7f7d04106080 2024-03-30T04:02:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.990+0000 7f7cfaa00700 1 -- 172.21.15.23:0/2283780300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0018420 con 0x7f7d04106080 2024-03-30T04:02:19.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.990+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d041a6470 con 0x7f7d04106080 2024-03-30T04:02:19.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.993+0000 7f7cfaa00700 1 -- 172.21.15.23:0/2283780300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f7cf000c9c0 con 0x7f7d04106080 2024-03-30T04:02:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 Processor -- start 2024-03-30T04:02:19.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 -- start start 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7cfaa00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 0x7f7ce80a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7cfaa00700 1 -- 172.21.15.23:0/2283780300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7cf0014070 con 0x7f7d04106080 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d7c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d8a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.994+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7cd5990 con 0x562df7d7c5d0 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d02800700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 0x7f7ce80a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7cd3350 con 0x562df7d7cef0 2024-03-30T04:02:19.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7ccec40 con 0x562df7d87df0 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43460/0 (socket says 172.21.15.23:43460) 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 -- 172.21.15.23:0/1563951069 learned_addr learned my addr 172.21.15.23:0/1563951069 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01e00700 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d7c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d02800700 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d8a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:19.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 -- 172.21.15.23:0/1563951069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 msgr2=0x562df7d8a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d8a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 -- 172.21.15.23:0/1563951069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 msgr2=0x562df7d7c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d7c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d01400700 1 -- 172.21.15.23:0/1563951069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562df7b503b0 con 0x562df7d7cef0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.995+0000 7f7d02800700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 0x7f7ce80a3870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009ec0 tx=0x7f7cfc00fc40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7d01400700 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0096980 tx=0x7f7cf009a6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=192adbef34ef7d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7cef400700 1 -- 172.21.15.23:0/1563951069 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf009dd40 con 0x562df7d7cef0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7cef400700 1 -- 172.21.15.23:0/1563951069 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf00a2390 con 0x562df7d7cef0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7cef400700 1 -- 172.21.15.23:0/1563951069 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0098390 con 0x562df7d7cef0 2024-03-30T04:02:19.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1563951069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 msgr2=0x562df7d878b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.996+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0096980 tx=0x7f7cf009a6e0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1563951069 shutdown_connections 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d8a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d7c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1563951069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d878b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1563951069 >> 172.21.15.23:0/1563951069 conn(0x562df7cdc6a0 msgr2=0x562df7d78a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1563951069 shutdown_connections 2024-03-30T04:02:19.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1563951069 wait complete. 2024-03-30T04:02:20.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.997+0000 7f7d1c8b8b80 1 -- start start 2024-03-30T04:02:20.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 Processor -- start 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d84a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d85000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7c6ad80 con 0x562df7d7c5d0 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7c6aa20 con 0x562df7d7cef0 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.998+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7b526a0 con 0x562df7d87df0 2024-03-30T04:02:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d1c8b8b80 10 client.0 _ll_register_callbacks cb 0x562df7c74470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59382/0 (socket says 172.21.15.23:59382) 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 -- 172.21.15.23:0/2361747518 learned_addr learned my addr 172.21.15.23:0/2361747518 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01400700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d84a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d02800700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d85000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 -- 172.21.15.23:0/2361747518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 msgr2=0x562df7d85000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d85000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 -- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 msgr2=0x562df7d84a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d84a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 -- 172.21.15.23:0/2361747518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d041a6470 con 0x562df7d7c5d0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01400700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d84a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009230 tx=0x7f7cf400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79e1267ef1c9e2f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:19.999+0000 7f7ce3e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400b760 con 0x562df7d7c5d0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562df7c8f380 con 0x562df7d7c5d0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7cf400b8c0 con 0x562df7d7c5d0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cf400ba80 con 0x562df7d7c5d0 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7d1c8b8b80 10 client.15852 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:20.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562df7afcb60 con 0x562df7d7c5d0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562df7c8f380 con 0x562df7d7c5d0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.000+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cf4015020 con 0x562df7d7c5d0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 10 client.15852 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 20 client.15852 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7d978a0 0x562df7d99cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 4 client.15852 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562df7cc7700 con 0x562df7d978a0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.001+0000 7f7d01400700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7d978a0 0x562df7d99cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d01400700 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7d978a0 0x562df7d99cc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e8eca0866be69d server_cookie=c941330b09fac11b in_seq=0 out_seq=0 2024-03-30T04:02:20.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7ce3e00700 10 client.15852 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2361747518 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562df7cc7700 con 0x562df7d978a0 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7ce3e00700 10 client.15852 handle_command_reply: tid=0 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 client.15852 shutdown 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 msgr2=0x562df7d87760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009230 tx=0x7f7cf400f690 comp rx=0 tx=0).stop 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 shutdown_connections 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7d978a0 0x562df7d99cc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d85000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d87760 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.002+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2361747518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d84a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.003+0000 7f7ce3e00700 0 client.15852 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.003+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 >> 172.21.15.23:0/2361747518 conn(0x562df7cdc6a0 msgr2=0x562df7d7e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.003+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 shutdown_connections 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.003+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2361747518 wait complete. 2024-03-30T04:02:20.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.003+0000 7f7d1c8b8b80 20 client.15852 trim_cache size 0 max 16384 2024-03-30T04:02:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 Processor -- start 2024-03-30T04:02:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 -- start start 2024-03-30T04:02:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d83390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d838d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d86050 con 0x562df7d7c5d0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d7d850 con 0x562df7d7cef0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d9a080 con 0x562df7d87df0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d01e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d01e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59394/0 (socket says 172.21.15.23:59394) 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d01e00700 1 -- 172.21.15.23:0/1933887921 learned_addr learned my addr 172.21.15.23:0/1933887921 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d01400700 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d83390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.021+0000 7f7d02800700 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d838d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 -- 172.21.15.23:0/1933887921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 msgr2=0x562df7d838d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d838d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 -- 172.21.15.23:0/1933887921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 msgr2=0x562df7d83390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d83390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 -- 172.21.15.23:0/1933887921 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562df7c94ea0 con 0x562df7d7c5d0 2024-03-30T04:02:20.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7d01e00700 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40092a0 tx=0x7f7cf4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e692373178ac7ec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7cefe00700 1 -- 172.21.15.23:0/1933887921 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4039030 con 0x562df7d7c5d0 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7cefe00700 1 -- 172.21.15.23:0/1933887921 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf401d070 con 0x562df7d7c5d0 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.022+0000 7f7cefe00700 1 -- 172.21.15.23:0/1933887921 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40054b0 con 0x562df7d7c5d0 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1933887921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 msgr2=0x562df7d8ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40092a0 tx=0x7f7cf4009fd0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1933887921 shutdown_connections 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d87df0 0x562df7d838d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7c5d0 0x562df7d8ea10 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/1933887921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d7cef0 0x562df7d83390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1933887921 >> 172.21.15.23:0/1933887921 conn(0x562df7cdc6a0 msgr2=0x562df7d7fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1933887921 shutdown_connections 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/1933887921 wait complete. 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 -- start start 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.023+0000 7f7d1c8b8b80 1 Processor -- start 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d7c5d0 0x562df7d8e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d87df0 0x562df7d8d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d8d960 con 0x562df7d7cef0 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d8dad0 con 0x562df7d87df0 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562df7d8dc90 con 0x562df7d7c5d0 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d1c8b8b80 10 client.0 _ll_register_callbacks cb 0x562df7c74470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:20.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d01400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d01e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d7c5d0 0x562df7d8e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d01400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59406/0 (socket says 172.21.15.23:59406) 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d01400700 1 -- 172.21.15.23:0/2940918723 learned_addr learned my addr 172.21.15.23:0/2940918723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.024+0000 7f7d02800700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d87df0 0x562df7d8d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 -- 172.21.15.23:0/2940918723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d7c5d0 msgr2=0x562df7d8e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d7c5d0 0x562df7d8e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 -- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d87df0 msgr2=0x562df7d8d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d87df0 0x562df7d8d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 -- 172.21.15.23:0/2940918723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562df7afcb60 con 0x562df7d7cef0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d01400700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000c5b0 tx=0x7f7cf0099a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef02fae79d44c4b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7ce3e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:20.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00980d0 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7cf00982b0 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cf00a2390 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cf00b82b0 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d1c8b8b80 10 client.15858 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.025+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562df7b54cd0 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf00a2390 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cf00a2000 con 0x562df7d7cef0 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 10 client.15858 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 20 client.15858 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7decb90 0x562df7deefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:20.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 4 client.15858 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562df7b1dcf0 con 0x562df7decb90 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.026+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7decb90 0x562df7deefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d01e00700 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7decb90 0x562df7deefb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45d08668630f6045 server_cookie=3f14b9d75717c2cc in_seq=0 out_seq=0 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7ce3e00700 10 client.15858 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7ce3e00700 1 -- 172.21.15.23:0/2940918723 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562df7b1dcf0 con 0x562df7decb90 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7ce3e00700 10 client.15858 handle_command_reply: tid=0 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 client.15858 shutdown 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 msgr2=0x562df7d8ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000c5b0 tx=0x7f7cf0099a50 comp rx=0 tx=0).stop 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 shutdown_connections 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562df7d87df0 0x562df7d8d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562df7decb90 0x562df7deefb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562df7d7c5d0 0x562df7d8e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 --2- 172.21.15.23:0/2940918723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562df7d7cef0 0x562df7d8ce20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7ce3e00700 0 client.15858 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.027+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 >> 172.21.15.23:0/2940918723 conn(0x562df7cdc6a0 msgr2=0x562df7d83310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 shutdown_connections 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d1c8b8b80 1 -- 172.21.15.23:0/2940918723 wait complete. 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d1c8b8b80 20 client.15858 trim_cache size 0 max 16384 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:20.032 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: "bfa8cff9-e63f-4988-91d7-fcf612317eaf": { 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bfa8cff9-e63f-4988-91d7-fcf612317eaf", 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:02:20.033 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 msgr2=0x7f7ce80a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 0x7f7ce80a3870 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009ec0 tx=0x7f7cfc00fc40 comp rx=0 tx=0).stop 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 msgr2=0x7f7d04199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000e9f0 tx=0x7f7cf000ed00 comp rx=0 tx=0).stop 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 shutdown_connections 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d0419dce0 0x7f7d0419e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7ce80a13b0 0x7f7ce80a3870 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d04100f60 0x7f7d04199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 --2- 172.21.15.23:0/2283780300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d04106080 0x7f7d04199940 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 >> 172.21.15.23:0/2283780300 conn(0x7f7d040fc7c0 msgr2=0x7f7d040fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.028+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 shutdown_connections 2024-03-30T04:02:20.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:20.029+0000 7f7d08e00700 1 -- 172.21.15.23:0/2283780300 wait complete. 2024-03-30T04:02:20.045 INFO:tasks.cephfs.filesystem:scrub status for tag:bfa8cff9-e63f-4988-91d7-fcf612317eaf - {'path': '/', 'tag': 'bfa8cff9-e63f-4988-91d7-fcf612317eaf', 'options': 'recursive,force'} 2024-03-30T04:02:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:20 smithi154 ceph-mon[79161]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:20.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:20 smithi193 ceph-mon[79581]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:20 smithi023 ceph-mon[38254]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:21 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:02:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:21 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:21 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:02:21.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:21 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:21 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:02:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:21 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:21 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:02:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:21 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:21 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:02:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:21 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:21 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:02:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:21 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:22 smithi154 ceph-mon[79161]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:22.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:22 smithi193 ceph-mon[79581]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:22 smithi023 ceph-mon[38254]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:23 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:02:23.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:23 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:02:23.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:23 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:02:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:24 smithi154 ceph-mon[79161]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:02:24.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:24 smithi193 ceph-mon[79581]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:02:24.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:24 smithi023 ceph-mon[38254]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:02:27.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:26 smithi193 ceph-mon[79581]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:27.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:26 smithi023 ceph-mon[38254]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:26 smithi154 ceph-mon[79161]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:28 smithi154 ceph-mon[79161]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:28 smithi193 ceph-mon[79581]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:28 smithi023 ceph-mon[38254]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:30.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:30 smithi154 ceph-mon[79161]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:30 smithi193 ceph-mon[79581]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:30 smithi023 ceph-mon[38254]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:02:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:32 smithi154 ceph-mon[79161]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-30T04:02:32.666 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:32 smithi023 ceph-mon[38254]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-30T04:02:32.666 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:32 smithi193 ceph-mon[79581]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-03-30T04:02:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:34 smithi154 ceph-mon[79161]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2024-03-30T04:02:34.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:34 smithi193 ceph-mon[79581]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2024-03-30T04:02:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:34 smithi023 ceph-mon[38254]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 12 op/s 2024-03-30T04:02:37.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:36 smithi193 ceph-mon[79581]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:36 smithi023 ceph-mon[38254]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:36 smithi154 ceph-mon[79161]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:38 smithi154 ceph-mon[79161]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-30T04:02:38.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:38 smithi193 ceph-mon[79581]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-30T04:02:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:38 smithi023 ceph-mon[38254]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-30T04:02:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:40 smithi154 ceph-mon[79161]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-30T04:02:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:40 smithi193 ceph-mon[79581]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-30T04:02:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:40 smithi023 ceph-mon[38254]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-30T04:02:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:42 smithi154 ceph-mon[79161]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-30T04:02:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:42 smithi193 ceph-mon[79581]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-30T04:02:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:42 smithi023 ceph-mon[38254]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-30T04:02:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:44 smithi154 ceph-mon[79161]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-30T04:02:44.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:44 smithi193 ceph-mon[79581]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-30T04:02:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:44 smithi023 ceph-mon[38254]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-30T04:02:47.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:46 smithi193 ceph-mon[79581]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:47.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:46 smithi023 ceph-mon[38254]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:46 smithi154 ceph-mon[79161]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-30T04:02:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:48 smithi154 ceph-mon[79161]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-30T04:02:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:48 smithi193 ceph-mon[79581]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-30T04:02:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:48 smithi023 ceph-mon[38254]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-30T04:02:50.046 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:02:50.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 -- 172.21.15.23:0/3876958332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 msgr2=0x7fd20010c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 --2- 172.21.15.23:0/3876958332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20010c8f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009f90 tx=0x7fd1f800b330 comp rx=0 tx=0).stop 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 -- 172.21.15.23:0/3876958332 shutdown_connections 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 --2- 172.21.15.23:0/3876958332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20010c8f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 --2- 172.21.15.23:0/3876958332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd200105a30 0x7fd200105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 --2- 172.21.15.23:0/3876958332 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd2000ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 -- 172.21.15.23:0/3876958332 >> 172.21.15.23:0/3876958332 conn(0x7fd2000fb510 msgr2=0x7fd2000fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 -- 172.21.15.23:0/3876958332 shutdown_connections 2024-03-30T04:02:50.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.324+0000 7fd206c00700 1 -- 172.21.15.23:0/3876958332 wait complete. 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.325+0000 7fd206c00700 1 Processor -- start 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 -- start start 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20019f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd200105a30 0x7fd200199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200199b90 con 0x7fd2000ffa20 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200199d00 con 0x7fd200105a30 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.326+0000 7fd206c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd200199ec0 con 0x7fd2001003e0 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45476/0 (socket says 172.21.15.23:45476) 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 -- 172.21.15.23:0/1081107418 learned_addr learned my addr 172.21.15.23:0/1081107418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd204e00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20019f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 -- 172.21.15.23:0/1081107418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 msgr2=0x7fd20019f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20019f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 -- 172.21.15.23:0/1081107418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd200105a30 msgr2=0x7fd200199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd200105a30 0x7fd200199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 -- 172.21.15.23:0/1081107418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f8009c40 con 0x7fd2000ffa20 2024-03-30T04:02:50.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.327+0000 7fd205800700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400e9a0 tx=0x7fd1f400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.328+0000 7fd1fe000700 1 -- 172.21.15.23:0/1081107418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4009790 con 0x7fd2000ffa20 2024-03-30T04:02:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.328+0000 7fd1fe000700 1 -- 172.21.15.23:0/1081107418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1f400cbd0 con 0x7fd2000ffa20 2024-03-30T04:02:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.328+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd20019a140 con 0x7fd2000ffa20 2024-03-30T04:02:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.328+0000 7fd1fe000700 1 -- 172.21.15.23:0/1081107418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f400eee0 con 0x7fd2000ffa20 2024-03-30T04:02:50.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.328+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2001a5ee0 con 0x7fd2000ffa20 2024-03-30T04:02:50.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.333+0000 7fd1fe000700 1 -- 172.21.15.23:0/1081107418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fd1f400cd40 con 0x7fd2000ffa20 2024-03-30T04:02:50.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.334+0000 7fd21bd32b80 1 Processor -- start 2024-03-30T04:02:50.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.334+0000 7fd21bd32b80 1 -- start start 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c87a0 0x55eb612c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 0x55eb612d63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb61221990 con 0x55eb612c90c0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6121f350 con 0x55eb612d3fc0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6121ac40 con 0x55eb612c87a0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd205800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd205800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45492/0 (socket says 172.21.15.23:45492) 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd205800700 1 -- 172.21.15.23:0/3737292974 learned_addr learned my addr 172.21.15.23:0/3737292974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd204e00700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 0x55eb612d63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd206200700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c87a0 0x55eb612c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.335+0000 7fd1fe000700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 0x7fd1e80a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd1fe000700 1 -- 172.21.15.23:0/1081107418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd1f4014070 con 0x7fd2000ffa20 2024-03-30T04:02:50.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 -- 172.21.15.23:0/3737292974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c87a0 msgr2=0x55eb612c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c87a0 0x55eb612c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 -- 172.21.15.23:0/3737292974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 msgr2=0x55eb612d63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 0x55eb612d63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 -- 172.21.15.23:0/3737292974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb6109c3b0 con 0x55eb612c90c0 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd204e00700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 0x55eb612d63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd206200700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 0x7fd1e80a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.336+0000 7fd205800700 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd1f40ca690 tx=0x7fd1f40963a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d428e788c25172b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd1f2a00700 1 -- 172.21.15.23:0/3737292974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40d0e80 con 0x55eb612c90c0 2024-03-30T04:02:50.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd1f2a00700 1 -- 172.21.15.23:0/3737292974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1f4099db0 con 0x55eb612c90c0 2024-03-30T04:02:50.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd1f2a00700 1 -- 172.21.15.23:0/3737292974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40a2460 con 0x55eb612c90c0 2024-03-30T04:02:50.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd206200700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 0x7fd1e80a38f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009550 tx=0x7fd1f8009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3737292974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 msgr2=0x55eb612d3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.337+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd1f40ca690 tx=0x7fd1f40963a0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3737292974 shutdown_connections 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612d3fc0 0x55eb612d63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c87a0 0x55eb612c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3737292974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612d3a80 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3737292974 >> 172.21.15.23:0/3737292974 conn(0x55eb612286a0 msgr2=0x55eb612c4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3737292974 shutdown_connections 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3737292974 wait complete. 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.338+0000 7fd21bd32b80 1 -- start start 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 Processor -- start 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c87a0 0x55eb612d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c90c0 0x55eb612d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb611b6d80 con 0x55eb612c87a0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb611b6a20 con 0x55eb612c90c0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb6109e6a0 con 0x55eb612d3fc0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.339+0000 7fd21bd32b80 10 client.0 _ll_register_callbacks cb 0x55eb611c0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44142/0 (socket says 172.21.15.23:44142) 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 -- 172.21.15.23:0/3627615444 learned_addr learned my addr 172.21.15.23:0/3627615444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd205800700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c87a0 0x55eb612d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd204e00700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c90c0 0x55eb612d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 -- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c90c0 msgr2=0x55eb612d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c90c0 0x55eb612d0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 -- 172.21.15.23:0/3627615444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c87a0 msgr2=0x55eb612d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c87a0 0x55eb612d39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.340+0000 7fd206200700 1 -- 172.21.15.23:0/3627615444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec009920 con 0x55eb612d3fc0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd206200700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80060b0 tx=0x7fd1f8009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22ec7d261c67461b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd1e7400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f800e040 con 0x55eb612d3fc0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd1f8007830 con 0x55eb612d3fc0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb611db380 con 0x55eb612d3fc0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.341+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1f801d480 con 0x55eb612d3fc0 2024-03-30T04:02:50.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 10 client.25864 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fd2001a5ee0 con 0x55eb612d3fc0 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb611db380 con 0x55eb612d3fc0 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd1f8005160 con 0x55eb612d3fc0 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 10 client.25864 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 10 client.25864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 20 client.25864 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb612e3b30 0x55eb612e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 4 client.25864 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.342+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb61213700 con 0x55eb612e3b30 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.343+0000 7fd205800700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb612e3b30 0x55eb612e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.343+0000 7fd205800700 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb612e3b30 0x55eb612e5f50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56c6da33d7250ea4 server_cookie=8f6c26864cfc550a in_seq=0 out_seq=0 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.343+0000 7fd1e7400700 10 client.25864 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd1e7400700 1 -- 172.21.15.23:0/3627615444 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eb61213700 con 0x55eb612e3b30 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd1e7400700 10 client.25864 handle_command_reply: tid=0 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 client.25864 shutdown 2024-03-30T04:02:50.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 msgr2=0x55eb612d1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1f80060b0 tx=0x7fd1f8009b20 comp rx=0 tx=0).stop 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 shutdown_connections 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb612e3b30 0x55eb612e5f50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612d1340 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c87a0 0x55eb612d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/3627615444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c90c0 0x55eb612d0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.344+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 >> 172.21.15.23:0/3627615444 conn(0x55eb612286a0 msgr2=0x55eb612ca890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.345+0000 7fd1e7400700 0 client.25864 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:50.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.345+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 shutdown_connections 2024-03-30T04:02:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.345+0000 7fd21bd32b80 1 -- 172.21.15.23:0/3627615444 wait complete. 2024-03-30T04:02:50.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.345+0000 7fd21bd32b80 20 client.25864 trim_cache size 0 max 16384 2024-03-30T04:02:50.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.366+0000 7fd21bd32b80 1 Processor -- start 2024-03-30T04:02:50.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.366+0000 7fd21bd32b80 1 -- start start 2024-03-30T04:02:50.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612be010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb612d22c0 con 0x55eb612c90c0 2024-03-30T04:02:50.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb612ca6a0 con 0x55eb612c87a0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb612c9a20 con 0x55eb612d3fc0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd205800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd206200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612be010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd205800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52578/0 (socket says 172.21.15.23:52578) 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd205800700 1 -- 172.21.15.23:0/62746647 learned_addr learned my addr 172.21.15.23:0/62746647 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.367+0000 7fd204e00700 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 -- 172.21.15.23:0/62746647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 msgr2=0x55eb612be010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612be010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 -- 172.21.15.23:0/62746647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 msgr2=0x55eb612dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612dabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 -- 172.21.15.23:0/62746647 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb61048b60 con 0x55eb612c87a0 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd206200700 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612be010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:50.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd205800700 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4096490 tx=0x7fd1f40d0bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39709b8eba27e28e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd1f3400700 1 -- 172.21.15.23:0/62746647 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40d08a0 con 0x55eb612c87a0 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd1f3400700 1 -- 172.21.15.23:0/62746647 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1f40a2460 con 0x55eb612c87a0 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.368+0000 7fd1f3400700 1 -- 172.21.15.23:0/62746647 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40cd360 con 0x55eb612c87a0 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 -- 172.21.15.23:0/62746647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 msgr2=0x55eb612c06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4096490 tx=0x7fd1f40d0bb0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 -- 172.21.15.23:0/62746647 shutdown_connections 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612d3fc0 0x55eb612be010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612c06d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/62746647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612c90c0 0x55eb612dabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 -- 172.21.15.23:0/62746647 >> 172.21.15.23:0/62746647 conn(0x55eb612286a0 msgr2=0x55eb612cb7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 -- 172.21.15.23:0/62746647 shutdown_connections 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.369+0000 7fd21bd32b80 1 -- 172.21.15.23:0/62746647 wait complete. 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.370+0000 7fd21bd32b80 1 -- start start 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.370+0000 7fd21bd32b80 1 Processor -- start 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 0x55eb612d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb61330ea0 con 0x55eb612d3fc0 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb61331010 con 0x55eb612c87a0 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb613311d0 con 0x55eb612c90c0 2024-03-30T04:02:50.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd21bd32b80 10 client.0 _ll_register_callbacks cb 0x55eb611c0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd205800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd206200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd204e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 0x55eb612d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd204e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 0x55eb612d92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44166/0 (socket says 172.21.15.23:44166) 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd204e00700 1 -- 172.21.15.23:0/247455385 learned_addr learned my addr 172.21.15.23:0/247455385 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd206200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45508/0 (socket says 172.21.15.23:45508) 2024-03-30T04:02:50.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.371+0000 7fd205800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52586/0 (socket says 172.21.15.23:52586) 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 -- 172.21.15.23:0/247455385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 msgr2=0x55eb612d92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 0x55eb612d92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 -- 172.21.15.23:0/247455385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 msgr2=0x55eb61330960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 -- 172.21.15.23:0/247455385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2001a5ee0 con 0x55eb612c87a0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd206200700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd205800700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4096a50 tx=0x7fd1f4099ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7665ddf8da0a56ab server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.372+0000 7fd1e7400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40cd320 con 0x55eb612c87a0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd1f409de10 con 0x55eb612c87a0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd1f4099db0 con 0x55eb612c87a0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1f40d08a0 con 0x55eb612c87a0 2024-03-30T04:02:50.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 10 client.25880 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eb6109c3b0 con 0x55eb612c87a0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1f40994d0 con 0x55eb612c87a0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd1f40b2cb0 con 0x55eb612c87a0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 10 client.25880 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 10 client.25880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 20 client.25880 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb613385d0 0x55eb6133a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 4 client.25880 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.373+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fd1f4099db0 con 0x55eb613385d0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.374+0000 7fd204e00700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb613385d0 0x55eb6133a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.374+0000 7fd1e7400700 10 client.25880 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.374+0000 7fd204e00700 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb613385d0 0x55eb6133a9c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d14964330d8076e8 server_cookie=f5964e26677418ad in_seq=0 out_seq=0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd1e7400700 1 -- 172.21.15.23:0/247455385 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fd1f4099db0 con 0x55eb613385d0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd1e7400700 10 client.25880 handle_command_reply: tid=0 2024-03-30T04:02:50.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 client.25880 shutdown 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 msgr2=0x55eb612d8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4096a50 tx=0x7fd1f4099ba0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 shutdown_connections 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb612d3fc0 0x55eb61330960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb612c87a0 0x55eb612d8da0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eb613385d0 0x55eb6133a9c0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 --2- 172.21.15.23:0/247455385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb612c90c0 0x55eb612d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 >> 172.21.15.23:0/247455385 conn(0x55eb612286a0 msgr2=0x55eb612c46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.375+0000 7fd1e7400700 0 client.25880 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 shutdown_connections 2024-03-30T04:02:50.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd21bd32b80 1 -- 172.21.15.23:0/247455385 wait complete. 2024-03-30T04:02:50.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd21bd32b80 20 client.25880 trim_cache size 0 max 16384 2024-03-30T04:02:50.378 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:50.378 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:02:50.378 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:02:50.378 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:50.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 msgr2=0x7fd1e80a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 0x7fd1e80a38f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009550 tx=0x7fd1f8009920 comp rx=0 tx=0).stop 2024-03-30T04:02:50.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 msgr2=0x7fd20019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.376+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd1f400e9a0 tx=0x7fd1f400ed60 comp rx=0 tx=0).stop 2024-03-30T04:02:50.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 shutdown_connections 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd1e80a1430 0x7fd1e80a38f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd200105a30 0x7fd200199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd2000ffa20 0x7fd20019eea0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 --2- 172.21.15.23:0/1081107418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd2001003e0 0x7fd20019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 >> 172.21.15.23:0/1081107418 conn(0x7fd2000fb510 msgr2=0x7fd200068790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 shutdown_connections 2024-03-30T04:02:50.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.377+0000 7fd206c00700 1 -- 172.21.15.23:0/1081107418 wait complete. 2024-03-30T04:02:50.397 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:02:50.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:02:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:50 smithi154 ceph-mon[79161]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:50.461 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:50 smithi023 ceph-mon[38254]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:50 smithi193 ceph-mon[79581]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:02:50.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.666+0000 7fc8a0400700 1 -- 172.21.15.23:0/2587347898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc89810b1d0 msgr2=0x7fc89810d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.666+0000 7fc8a0400700 1 --2- 172.21.15.23:0/2587347898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc89810b1d0 0x7fc89810d690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc89000b0a0 tx=0x7fc89000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 -- 172.21.15.23:0/2587347898 shutdown_connections 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 --2- 172.21.15.23:0/2587347898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc89810b1d0 0x7fc89810d690 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 --2- 172.21.15.23:0/2587347898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc898101db0 0x7fc898102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 --2- 172.21.15.23:0/2587347898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8981026d0 0x7fc89810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 -- 172.21.15.23:0/2587347898 >> 172.21.15.23:0/2587347898 conn(0x7fc8980fb5f0 msgr2=0x7fc8980fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 -- 172.21.15.23:0/2587347898 shutdown_connections 2024-03-30T04:02:50.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.667+0000 7fc8a0400700 1 -- 172.21.15.23:0/2587347898 wait complete. 2024-03-30T04:02:50.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.668+0000 7fc8a0400700 1 Processor -- start 2024-03-30T04:02:50.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.668+0000 7fc8a0400700 1 -- start start 2024-03-30T04:02:50.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 0x7fc8980fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 0x7fc898100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981a5580 con 0x7fc898101db0 2024-03-30T04:02:50.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981a56f0 con 0x7fc8981026d0 2024-03-30T04:02:50.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc8a0400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8981a58b0 con 0x7fc89810b1d0 2024-03-30T04:02:50.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc89da00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc89da00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45538/0 (socket says 172.21.15.23:45538) 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc89da00700 1 -- 172.21.15.23:0/164305443 learned_addr learned my addr 172.21.15.23:0/164305443 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.669+0000 7fc89e400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 0x7fc898100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89d000700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 0x7fc8980fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 -- 172.21.15.23:0/164305443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 msgr2=0x7fc8980fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 0x7fc8980fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 -- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 msgr2=0x7fc8980ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 -- 172.21.15.23:0/164305443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc890009d00 con 0x7fc89810b1d0 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89d000700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 0x7fc8980fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89da00700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:50.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.670+0000 7fc89e400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 0x7fc898100480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc89000b070 tx=0x7fc890007860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.671+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89000e040 con 0x7fc89810b1d0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.671+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc89001d470 con 0x7fc89810b1d0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.671+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8981a5b30 con 0x7fc89810b1d0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.671+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc890005280 con 0x7fc89810b1d0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.671+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8981a6040 con 0x7fc89810b1d0 2024-03-30T04:02:50.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.672+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc898106300 con 0x7fc89810b1d0 2024-03-30T04:02:50.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.674+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fc89001d5e0 con 0x7fc89810b1d0 2024-03-30T04:02:50.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.676+0000 7fc896000700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 0x7fc8800a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:50.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.677+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc8900ce6f0 con 0x7fc89810b1d0 2024-03-30T04:02:50.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.677+0000 7fc89da00700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 0x7fc8800a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:50.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.677+0000 7fc89da00700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 0x7fc8800a38f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc88c011550 tx=0x7fc88c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:50.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.678+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc8900975c0 con 0x7fc89810b1d0 2024-03-30T04:02:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.966+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8980686d0 con 0x7fc89810b1d0 2024-03-30T04:02:50.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.967+0000 7fc896000700 1 -- 172.21.15.23:0/164305443 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fc89009abe0 con 0x7fc89810b1d0 2024-03-30T04:02:50.966 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:02:50.966 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:01:35.365141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 msgr2=0x7fc8800a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 0x7fc8800a38f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc88c011550 tx=0x7fc88c009260 comp rx=0 tx=0).stop 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 msgr2=0x7fc898100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 0x7fc898100480 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc89000b070 tx=0x7fc890007860 comp rx=0 tx=0).stop 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 shutdown_connections 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc89810b1d0 0x7fc898100480 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc898101db0 0x7fc8980ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8800a1430 0x7fc8800a38f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 --2- 172.21.15.23:0/164305443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8981026d0 0x7fc8980fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 >> 172.21.15.23:0/164305443 conn(0x7fc8980fb5f0 msgr2=0x7fc8980fce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 shutdown_connections 2024-03-30T04:02:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:50.970+0000 7fc8a0400700 1 -- 172.21.15.23:0/164305443 wait complete. 2024-03-30T04:02:50.970 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-30T04:02:50.993 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:02:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.211+0000 7f6fc0e00700 1 -- 172.21.15.23:0/2674633354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc069c10 msgr2=0x7f6fbc105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.211+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/2674633354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc069c10 0x7f6fbc105980 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80099c0 tx=0x7f6fa8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 -- 172.21.15.23:0/2674633354 shutdown_connections 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/2674633354 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/2674633354 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/2674633354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc069c10 0x7f6fbc105980 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 -- 172.21.15.23:0/2674633354 >> 172.21.15.23:0/2674633354 conn(0x7f6fbc076d80 msgr2=0x7f6fbc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 -- 172.21.15.23:0/2674633354 shutdown_connections 2024-03-30T04:02:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.212+0000 7f6fc0e00700 1 -- 172.21.15.23:0/2674633354 wait complete. 2024-03-30T04:02:51.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.213+0000 7f6fc0e00700 1 Processor -- start 2024-03-30T04:02:51.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.213+0000 7f6fc0e00700 1 -- start start 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc19dcc0 0x7f6fbc19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc19e740 con 0x7f6fbc1060b0 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc19e8b0 con 0x7f6fbc19dcc0 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fc0e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fbc19ea70 con 0x7f6fbc069260 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fb9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fb9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44198/0 (socket says 172.21.15.23:44198) 2024-03-30T04:02:51.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fb9e00700 1 -- 172.21.15.23:0/798120431 learned_addr learned my addr 172.21.15.23:0/798120431 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fba800700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc19dcc0 0x7f6fbc19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.214+0000 7f6fb9400700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 -- 172.21.15.23:0/798120431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc19dcc0 msgr2=0x7f6fbc19e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc19dcc0 0x7f6fbc19e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 -- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 msgr2=0x7f6fbc199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 -- 172.21.15.23:0/798120431 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa80095f0 con 0x7f6fbc069260 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9400700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00e9a0 tx=0x7f6fac00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb2a00700 1 -- 172.21.15.23:0/798120431 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac009790 con 0x7f6fbc069260 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb2a00700 1 -- 172.21.15.23:0/798120431 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac00cbd0 con 0x7f6fbc069260 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fb2a00700 1 -- 172.21.15.23:0/798120431 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac018680 con 0x7f6fbc069260 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.215+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc1a5fd0 con 0x7f6fbc069260 2024-03-30T04:02:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.216+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc1a6340 con 0x7f6fbc069260 2024-03-30T04:02:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.220+0000 7f6fb2a00700 1 -- 172.21.15.23:0/798120431 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f6fac00cd40 con 0x7f6fbc069260 2024-03-30T04:02:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.221+0000 7f6fd48d0b80 1 Processor -- start 2024-03-30T04:02:51.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.221+0000 7f6fd48d0b80 1 -- start start 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec06b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec06e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebfb9990 con 0x5605ec06bdf0 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebfb7350 con 0x5605ec060ef0 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebfb2c40 con 0x5605ec0605d0 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fba800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fb2a00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 0x7f6fa00a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fba800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44214/0 (socket says 172.21.15.23:44214) 2024-03-30T04:02:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.222+0000 7f6fba800700 1 -- 172.21.15.23:0/2087051573 learned_addr learned my addr 172.21.15.23:0/2087051573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec06b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb2a00700 1 -- 172.21.15.23:0/798120431 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6fac014070 con 0x7f6fbc069260 2024-03-30T04:02:51.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9400700 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec06e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fba800700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 0x7f6fa00a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2087051573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 msgr2=0x5605ec0609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2087051573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 msgr2=0x5605ec06e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec06e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2087051573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605ebe343b0 con 0x5605ec060ef0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.223+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec06b8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0ca740 tx=0x7f6fac0654c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f57e80a4b61f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fa7400700 1 -- 172.21.15.23:0/2087051573 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0659d0 con 0x5605ec060ef0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fa7400700 1 -- 172.21.15.23:0/2087051573 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac09a730 con 0x5605ec060ef0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fa7400700 1 -- 172.21.15.23:0/2087051573 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac065790 con 0x5605ec060ef0 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2087051573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 msgr2=0x5605ec06b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fba800700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 0x7f6fa00a3870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009390 tx=0x7f6fb4009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.224+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec06b8b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0ca740 tx=0x7f6fac0654c0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2087051573 shutdown_connections 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec06e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2087051573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec06b8b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2087051573 >> 172.21.15.23:0/2087051573 conn(0x5605ebfc06a0 msgr2=0x5605ec05ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2087051573 shutdown_connections 2024-03-30T04:02:51.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.225+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2087051573 wait complete. 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.226+0000 7f6fd48d0b80 1 -- start start 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.226+0000 7f6fd48d0b80 1 Processor -- start 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 0x5605ec06b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 0x5605ec068a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec0690b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebf4ed80 con 0x5605ec06bdf0 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebf4ea20 con 0x5605ec0605d0 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ebe366a0 con 0x5605ec060ef0 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fd48d0b80 10 client.0 _ll_register_callbacks cb 0x5605ebf58470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fb9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 0x5605ec068a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fb9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 0x5605ec068a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44216/0 (socket says 172.21.15.23:44216) 2024-03-30T04:02:51.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fb9400700 1 -- 172.21.15.23:0/4123675588 learned_addr learned my addr 172.21.15.23:0/4123675588 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.227+0000 7f6fba800700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec0690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 0x5605ec06b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 -- 172.21.15.23:0/4123675588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 msgr2=0x5605ec068a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 0x5605ec068a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 -- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 msgr2=0x5605ec06b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 0x5605ec06b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 -- 172.21.15.23:0/4123675588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc1a6340 con 0x5605ec06bdf0 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 0x5605ec06b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6fba800700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec0690b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4017e20 tx=0x7f6fb4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0339cd93831133e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.228+0000 7f6f9be00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:51.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400e040 con 0x5605ec06bdf0 2024-03-30T04:02:51.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6fb401f040 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5605ebf73380 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fb4016d40 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6fd48d0b80 10 client.15894 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.229+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5605ebde0b60 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.230+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5605ebf73380 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.230+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6fb402b070 con 0x5605ec06bdf0 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 10 client.15894 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 10 client.15894 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 20 client.15894 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec084070 0x5605ec086490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 4 client.15894 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:51.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605ebfab700 con 0x5605ec084070 2024-03-30T04:02:51.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.231+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec084070 0x5605ec086490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.232+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec084070 0x5605ec086490 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fa17b8d36e737a4e server_cookie=92246d0ee25894c9 in_seq=0 out_seq=0 2024-03-30T04:02:51.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.232+0000 7f6f9be00700 10 client.15894 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.232+0000 7f6f9be00700 1 -- 172.21.15.23:0/4123675588 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5605ebfab700 con 0x5605ec084070 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.232+0000 7f6f9be00700 10 client.15894 handle_command_reply: tid=0 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.232+0000 7f6fd48d0b80 1 client.15894 shutdown 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 msgr2=0x5605ec0690b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec0690b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4017e20 tx=0x7f6fb4004b40 comp rx=0 tx=0).stop 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 shutdown_connections 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec06bdf0 0x5605ec0690b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec0605d0 0x5605ec06b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec084070 0x5605ec086490 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4123675588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec060ef0 0x5605ec068a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 >> 172.21.15.23:0/4123675588 conn(0x5605ebfc06a0 msgr2=0x5605ec0625e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6f9be00700 0 client.15894 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 shutdown_connections 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4123675588 wait complete. 2024-03-30T04:02:51.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.233+0000 7f6fd48d0b80 20 client.15894 trim_cache size 0 max 16384 2024-03-30T04:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.247+0000 7f6fd48d0b80 1 Processor -- start 2024-03-30T04:02:51.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.247+0000 7f6fd48d0b80 1 -- start start 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec07b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec055f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec061850 con 0x5605ec0605d0 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec069990 con 0x5605ec060ef0 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec07b840 con 0x5605ec06bdf0 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fb9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fb9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45588/0 (socket says 172.21.15.23:45588) 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fb9e00700 1 -- 172.21.15.23:0/4218631256 learned_addr learned my addr 172.21.15.23:0/4218631256 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fb9400700 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec07b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fba800700 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec055f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.248+0000 7f6fb9e00700 1 -- 172.21.15.23:0/4218631256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 msgr2=0x5605ec055f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec055f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fb9e00700 1 -- 172.21.15.23:0/4218631256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 msgr2=0x5605ec07b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec07b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fb9e00700 1 -- 172.21.15.23:0/4218631256 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605ebf78ea0 con 0x5605ec0605d0 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6fac000c00 tx=0x7f6fac09a8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ed7a709e50b0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fa7e00700 1 -- 172.21.15.23:0/4218631256 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0b1070 con 0x5605ec0605d0 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fa7e00700 1 -- 172.21.15.23:0/4218631256 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac0a24b0 con 0x5605ec0605d0 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.249+0000 7f6fa7e00700 1 -- 172.21.15.23:0/4218631256 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac09a4f0 con 0x5605ec0605d0 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4218631256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 msgr2=0x5605ec0609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6fac000c00 tx=0x7f6fac09a8a0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4218631256 shutdown_connections 2024-03-30T04:02:51.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec055f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec0609b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/4218631256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec07b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4218631256 >> 172.21.15.23:0/4218631256 conn(0x5605ebfc06a0 msgr2=0x5605ec0555a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.250+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4218631256 shutdown_connections 2024-03-30T04:02:51.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.251+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/4218631256 wait complete. 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.251+0000 7f6fd48d0b80 1 -- start start 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.251+0000 7f6fd48d0b80 1 Processor -- start 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec07aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec0783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec078900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec078e40 con 0x5605ec0605d0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec078fb0 con 0x5605ec060ef0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605ec079170 con 0x5605ec06bdf0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fd48d0b80 10 client.0 _ll_register_callbacks cb 0x5605ebf58470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.252+0000 7f6fb9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec07aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fba800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec078900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fba800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec078900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44230/0 (socket says 172.21.15.23:44230) 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fba800700 1 -- 172.21.15.23:0/2459087254 learned_addr learned my addr 172.21.15.23:0/2459087254 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9400700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec0783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2459087254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 msgr2=0x5605ec078900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec078900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 msgr2=0x5605ec0783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec0783c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 -- 172.21.15.23:0/2459087254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605ebe343b0 con 0x5605ec0605d0 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6fb9e00700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec07aee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0a1f80 tx=0x7f6fac09a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4441f9597e7fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.253+0000 7f6f9be00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:51.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac0b1050 con 0x5605ec0605d0 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6fac09a730 con 0x5605ec0605d0 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6fac09de60 con 0x5605ec0605d0 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fac0b83b0 con 0x5605ec0605d0 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6fd48d0b80 10 client.15900 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5605ebe38cd0 con 0x5605ec0605d0 2024-03-30T04:02:51.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6fac09a730 con 0x5605ec0605d0 2024-03-30T04:02:51.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6fac09a290 con 0x5605ec0605d0 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6fd48d0b80 10 client.15900 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.254+0000 7f6fd48d0b80 10 client.15900 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fd48d0b80 20 client.15900 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec0d7e50 0x5605ec0da240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fd48d0b80 4 client.15900 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5605ebe01cf0 con 0x5605ec0d7e50 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fb9400700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec0d7e50 0x5605ec0da240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6fb9400700 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec0d7e50 0x5605ec0da240 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f3cdcfbf4ad0d813 server_cookie=18de3a4ce8129c6b in_seq=0 out_seq=0 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.255+0000 7f6f9be00700 10 client.15900 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6f9be00700 1 -- 172.21.15.23:0/2459087254 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5605ebe01cf0 con 0x5605ec0d7e50 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6f9be00700 10 client.15900 handle_command_reply: tid=0 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 client.15900 shutdown 2024-03-30T04:02:51.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 msgr2=0x5605ec07aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec07aee0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0a1f80 tx=0x7f6fac09a0b0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 shutdown_connections 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5605ec0d7e50 0x5605ec0da240 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5605ec06bdf0 0x5605ec078900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5605ec0605d0 0x5605ec07aee0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 --2- 172.21.15.23:0/2459087254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5605ec060ef0 0x5605ec0783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 >> 172.21.15.23:0/2459087254 conn(0x5605ebfc06a0 msgr2=0x5605ec063d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6f9be00700 0 client.15900 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 shutdown_connections 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.256+0000 7f6fd48d0b80 1 -- 172.21.15.23:0/2459087254 wait complete. 2024-03-30T04:02:51.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fd48d0b80 20 client.15900 trim_cache size 0 max 16384 2024-03-30T04:02:51.258 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 msgr2=0x7f6fa00a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 0x7f6fa00a3870 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009390 tx=0x7f6fb4009040 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 msgr2=0x7f6fbc1993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00e9a0 tx=0x7f6fac00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 shutdown_connections 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6fa00a13b0 0x7f6fa00a3870 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fbc19dcc0 0x7f6fbc19e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6fbc069260 0x7f6fbc1993e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 --2- 172.21.15.23:0/798120431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6fbc1060b0 0x7f6fbc199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.257+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 >> 172.21.15.23:0/798120431 conn(0x7f6fbc076d80 msgr2=0x7f6fbc10c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.258+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 shutdown_connections 2024-03-30T04:02:51.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.258+0000 7f6fc0e00700 1 -- 172.21.15.23:0/798120431 wait complete. 2024-03-30T04:02:51.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/164305443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:51.276 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:02:51.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/164305443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:51.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/164305443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.551+0000 7f8021e00700 1 -- 172.21.15.23:0/2539575436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c105ce0 msgr2=0x7f801c1060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 --2- 172.21.15.23:0/2539575436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c105ce0 0x7f801c1060c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f800c0099c0 tx=0x7f800c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 -- 172.21.15.23:0/2539575436 shutdown_connections 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 --2- 172.21.15.23:0/2539575436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 --2- 172.21.15.23:0/2539575436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c105ce0 0x7f801c1060c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 --2- 172.21.15.23:0/2539575436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c067e30 0x7f801c0682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 -- 172.21.15.23:0/2539575436 >> 172.21.15.23:0/2539575436 conn(0x7f801c076d40 msgr2=0x7f801c077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 -- 172.21.15.23:0/2539575436 shutdown_connections 2024-03-30T04:02:51.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.552+0000 7f8021e00700 1 -- 172.21.15.23:0/2539575436 wait complete. 2024-03-30T04:02:51.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.553+0000 7f8021e00700 1 Processor -- start 2024-03-30T04:02:51.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 -- start start 2024-03-30T04:02:51.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c105ce0 0x7f801c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c19e340 con 0x7f801c105ce0 2024-03-30T04:02:51.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c19e4b0 con 0x7f801c0687f0 2024-03-30T04:02:51.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.554+0000 7f8021e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801c19e670 con 0x7f801c067e30 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44260/0 (socket says 172.21.15.23:44260) 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 -- 172.21.15.23:0/527665610 learned_addr learned my addr 172.21.15.23:0/527665610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801b400700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c105ce0 0x7f801c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 -- 172.21.15.23:0/527665610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 msgr2=0x7f801c199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 -- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c105ce0 msgr2=0x7f801c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c105ce0 0x7f801c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 -- 172.21.15.23:0/527665610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c0095f0 con 0x7f801c067e30 2024-03-30T04:02:51.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801a000700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.555+0000 7f801aa00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f800c005b40 tx=0x7f800c00ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.556+0000 7f8020a00700 1 -- 172.21.15.23:0/527665610 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c01d070 con 0x7f801c067e30 2024-03-30T04:02:51.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.556+0000 7f8020a00700 1 -- 172.21.15.23:0/527665610 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f800c00be70 con 0x7f801c067e30 2024-03-30T04:02:51.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.556+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c19e8f0 con 0x7f801c067e30 2024-03-30T04:02:51.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.556+0000 7f8020a00700 1 -- 172.21.15.23:0/527665610 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c00fb70 con 0x7f801c067e30 2024-03-30T04:02:51.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.556+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c1a6190 con 0x7f801c067e30 2024-03-30T04:02:51.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.561+0000 7f8020a00700 1 -- 172.21.15.23:0/527665610 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f800c02b440 con 0x7f801c067e30 2024-03-30T04:02:51.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.562+0000 7f8020a00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 0x7f80000a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.562+0000 7f8020a00700 1 -- 172.21.15.23:0/527665610 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f800c0cdf30 con 0x7f801c067e30 2024-03-30T04:02:51.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.562+0000 7f803586fb80 1 Processor -- start 2024-03-30T04:02:51.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.562+0000 7f801a000700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 0x7f80000a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 -- start start 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f702990 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f700350 con 0x55bf1f7b1de0 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f803586fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f6fbc40 con 0x55bf1f7b2700 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.563+0000 7f801a000700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 0x7f80000a38f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f801c19a970 tx=0x7f8008009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44274/0 (socket says 172.21.15.23:44274) 2024-03-30T04:02:51.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 -- 172.21.15.23:0/4235502782 learned_addr learned my addr 172.21.15.23:0/4235502782 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801aa00700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801b400700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 -- 172.21.15.23:0/4235502782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 msgr2=0x55bf1f7b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 -- 172.21.15.23:0/4235502782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 msgr2=0x55bf1f7bf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 -- 172.21.15.23:0/4235502782 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf1f57d3b0 con 0x55bf1f7b2700 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801b400700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:51.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.564+0000 7f801a000700 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f800800e6d0 tx=0x7f8008014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe910dbdcffc591 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.565+0000 7f8007e00700 1 -- 172.21.15.23:0/4235502782 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800800ee70 con 0x55bf1f7b2700 2024-03-30T04:02:51.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.565+0000 7f8007e00700 1 -- 172.21.15.23:0/4235502782 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f800801e8e0 con 0x55bf1f7b2700 2024-03-30T04:02:51.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.565+0000 7f8007e00700 1 -- 172.21.15.23:0/4235502782 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800801d890 con 0x55bf1f7b2700 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 -- 172.21.15.23:0/4235502782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 msgr2=0x55bf1f7bd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f800800e6d0 tx=0x7f8008014000 comp rx=0 tx=0).stop 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 -- 172.21.15.23:0/4235502782 shutdown_connections 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 --2- 172.21.15.23:0/4235502782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b2700 0x55bf1f7bd0a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 -- 172.21.15.23:0/4235502782 >> 172.21.15.23:0/4235502782 conn(0x55bf1f7096a0 msgr2=0x55bf1f7a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 -- 172.21.15.23:0/4235502782 shutdown_connections 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.566+0000 7f803586fb80 1 -- 172.21.15.23:0/4235502782 wait complete. 2024-03-30T04:02:51.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.567+0000 7f803586fb80 1 -- start start 2024-03-30T04:02:51.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.567+0000 7f803586fb80 1 Processor -- start 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b2700 0x55bf1f7ba950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f697d80 con 0x55bf1f7b1de0 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f697a20 con 0x55bf1f7b2700 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f57f6a0 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f803586fb80 10 client.0 _ll_register_callbacks cb 0x55bf1f6a1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f801aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f801b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f801aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45636/0 (socket says 172.21.15.23:45636) 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.568+0000 7f801aa00700 1 -- 172.21.15.23:0/1489111166 learned_addr learned my addr 172.21.15.23:0/1489111166 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801a000700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b2700 0x55bf1f7ba950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 -- 172.21.15.23:0/1489111166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b2700 msgr2=0x55bf1f7ba950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b2700 0x55bf1f7ba950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 -- 172.21.15.23:0/1489111166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 msgr2=0x55bf1f7ba410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 -- 172.21.15.23:0/1489111166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8008017070 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801aa00700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f801b400700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bae90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80140040f0 tx=0x7f8014004400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc4702d6c2fda577 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.569+0000 7f8004c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80140079d0 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf1f6bc380 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8014007b30 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8014007cf0 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f803586fb80 10 client.25906 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.570+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55bf1f529b60 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bf1f6bc380 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8014014040 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 10 client.25906 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 10 client.25906 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 20 client.25906 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f7cd2f0 0x55bf1f7cf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 4 client.25906 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf1f6f4700 con 0x55bf1f7cd2f0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.571+0000 7f801aa00700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f7cd2f0 0x55bf1f7cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.572+0000 7f801aa00700 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f7cd2f0 0x55bf1f7cf710 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=61df1b6775854d68 server_cookie=9a1b13d966fe72fd in_seq=0 out_seq=0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.572+0000 7f8004c00700 10 client.25906 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f8004c00700 1 -- 172.21.15.23:0/1489111166 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bf1f6f4700 con 0x55bf1f7cd2f0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f8004c00700 10 client.25906 handle_command_reply: tid=0 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 client.25906 shutdown 2024-03-30T04:02:51.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 msgr2=0x55bf1f7bae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bae90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f80140040f0 tx=0x7f8014004400 comp rx=0 tx=0).stop 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 shutdown_connections 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f7cd2f0 0x55bf1f7cf710 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7bae90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7ba410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f8004c00700 0 client.25906 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 --2- 172.21.15.23:0/1489111166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b2700 0x55bf1f7ba950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 >> 172.21.15.23:0/1489111166 conn(0x55bf1f7096a0 msgr2=0x55bf1f7b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 shutdown_connections 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 1 -- 172.21.15.23:0/1489111166 wait complete. 2024-03-30T04:02:51.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.573+0000 7f803586fb80 20 client.25906 trim_cache size 0 max 16384 2024-03-30T04:02:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.599+0000 7f803586fb80 1 Processor -- start 2024-03-30T04:02:51.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 -- start start 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f79ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f7bfd90 con 0x55bf1f7b2700 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f7c9dc0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f803586fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f7bb9c0 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f801a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f801aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f801a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45638/0 (socket says 172.21.15.23:45638) 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f801aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52706/0 (socket says 172.21.15.23:52706) 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.600+0000 7f801a000700 1 -- 172.21.15.23:0/76389815 learned_addr learned my addr 172.21.15.23:0/76389815 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801b400700 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f79ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 -- 172.21.15.23:0/76389815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 msgr2=0x55bf1f79ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f79ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 -- 172.21.15.23:0/76389815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 msgr2=0x55bf1f7c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 -- 172.21.15.23:0/76389815 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf1f6c1ea0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801a000700 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f801aa00700 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f800c09aa40 tx=0x7f800c096f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3670dc6cfd6c39a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f8011200700 1 -- 172.21.15.23:0/76389815 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c09b200 con 0x55bf1f7b1de0 2024-03-30T04:02:51.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f8011200700 1 -- 172.21.15.23:0/76389815 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f800c09ab00 con 0x55bf1f7b1de0 2024-03-30T04:02:51.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.601+0000 7f8011200700 1 -- 172.21.15.23:0/76389815 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c0a2a60 con 0x55bf1f7b1de0 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 -- 172.21.15.23:0/76389815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 msgr2=0x55bf1f7cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f800c09aa40 tx=0x7f800c096f70 comp rx=0 tx=0).stop 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 -- 172.21.15.23:0/76389815 shutdown_connections 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f79ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7cf6e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 --2- 172.21.15.23:0/76389815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 -- 172.21.15.23:0/76389815 >> 172.21.15.23:0/76389815 conn(0x55bf1f7096a0 msgr2=0x55bf1f79e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.602+0000 7f803586fb80 1 -- 172.21.15.23:0/76389815 shutdown_connections 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.603+0000 7f803586fb80 1 -- 172.21.15.23:0/76389815 wait complete. 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.603+0000 7f803586fb80 1 -- start start 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.603+0000 7f803586fb80 1 Processor -- start 2024-03-30T04:02:51.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7c2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f820200 con 0x55bf1f7b2700 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f820340 con 0x55bf1f7bd5e0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf1f820500 con 0x55bf1f7b1de0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.604+0000 7f803586fb80 10 client.0 _ll_register_callbacks cb 0x55bf1f6a1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44306/0 (socket says 172.21.15.23:44306) 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 -- 172.21.15.23:0/2093895519 learned_addr learned my addr 172.21.15.23:0/2093895519 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801a000700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801b400700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 -- 172.21.15.23:0/2093895519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7bd5e0 msgr2=0x55bf1f7c2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7c2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 -- 172.21.15.23:0/2093895519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 msgr2=0x55bf1f7c2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 -- 172.21.15.23:0/2093895519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf1f57d3b0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f801aa00700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f800c097000 tx=0x7f800c09b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c315d1620e7704ed server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.605+0000 7f8004c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:51.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800c079de0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f800c09b200 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f800801e8e0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f800c0a2ef0 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f803586fb80 10 client.25912 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7f801c1a6190 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f800c0a4080 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.606+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f800c0b3120 con 0x55bf1f7b1de0 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 10 client.25912 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 10 client.25912 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 20 client.25912 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f821800 0x55bf1f823c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 4 client.25912 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bf1f54acf0 con 0x55bf1f821800 2024-03-30T04:02:51.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.607+0000 7f801a000700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f821800 0x55bf1f823c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f801a000700 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f821800 0x55bf1f823c20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3467e52ed05005a4 server_cookie=f05a4f0db0928b14 in_seq=0 out_seq=0 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f8004c00700 10 client.25912 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f8004c00700 1 -- 172.21.15.23:0/2093895519 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bf1f54acf0 con 0x55bf1f821800 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f8004c00700 10 client.25912 handle_command_reply: tid=0 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 client.25912 shutdown 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 msgr2=0x55bf1f7c1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f800c097000 tx=0x7f800c09b040 comp rx=0 tx=0).stop 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 shutdown_connections 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf1f821800 0x55bf1f823c20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf1f7bd5e0 0x55bf1f7c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf1f7b1de0 0x55bf1f7c1bf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f8004c00700 0 client.25912 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:02:51.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 --2- 172.21.15.23:0/2093895519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf1f7b2700 0x55bf1f7c2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.608+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 >> 172.21.15.23:0/2093895519 conn(0x55bf1f7096a0 msgr2=0x55bf1f79ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 shutdown_connections 2024-03-30T04:02:51.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f803586fb80 1 -- 172.21.15.23:0/2093895519 wait complete. 2024-03-30T04:02:51.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f803586fb80 20 client.25912 trim_cache size 0 max 16384 2024-03-30T04:02:51.611 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 msgr2=0x7f80000a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 0x7f80000a38f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f801c19a970 tx=0x7f8008009260 comp rx=0 tx=0).stop 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 msgr2=0x7f801c199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.609+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f800c005b40 tx=0x7f800c00ba30 comp rx=0 tx=0).stop 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 shutdown_connections 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f801c105ce0 0x7f801c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f801c067e30 0x7f801c199420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80000a1430 0x7f80000a38f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 --2- 172.21.15.23:0/527665610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f801c0687f0 0x7f801c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 >> 172.21.15.23:0/527665610 conn(0x7f801c076d40 msgr2=0x7f801c101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 shutdown_connections 2024-03-30T04:02:51.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.610+0000 7f8021e00700 1 -- 172.21.15.23:0/527665610 wait complete. 2024-03-30T04:02:51.627 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 -- 172.21.15.23:0/1150209228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 msgr2=0x7f7d14105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 --2- 172.21.15.23:0/1150209228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d14105e10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009990 tx=0x7f7d04009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 -- 172.21.15.23:0/1150209228 shutdown_connections 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 --2- 172.21.15.23:0/1150209228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d141003e0 0x7f7d1410c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 --2- 172.21.15.23:0/1150209228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d14105e10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 --2- 172.21.15.23:0/1150209228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d140ffa20 0x7f7d140ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 -- 172.21.15.23:0/1150209228 >> 172.21.15.23:0/1150209228 conn(0x7f7d140fb510 msgr2=0x7f7d140fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 -- 172.21.15.23:0/1150209228 shutdown_connections 2024-03-30T04:02:51.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.901+0000 7f7d13e00700 1 -- 172.21.15.23:0/1150209228 wait complete. 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.902+0000 7f7d13e00700 1 Processor -- start 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.902+0000 7f7d13e00700 1 -- start start 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d141003e0 0x7f7d14197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d1419b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1419be00 con 0x7f7d140ffa20 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1419bf70 con 0x7f7d141003e0 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1419c130 con 0x7f7d14105a30 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d12a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d12000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d141003e0 0x7f7d14197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d12a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45650/0 (socket says 172.21.15.23:45650) 2024-03-30T04:02:51.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d12a00700 1 -- 172.21.15.23:0/2153887212 learned_addr learned my addr 172.21.15.23:0/2153887212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.903+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d1419b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 -- 172.21.15.23:0/2153887212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d141003e0 msgr2=0x7f7d14197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d141003e0 0x7f7d14197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 -- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 msgr2=0x7f7d14196ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 -- 172.21.15.23:0/2153887212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d040095f0 con 0x7f7d14105a30 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d12a00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d1419b7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007f10 tx=0x7f7d0c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d0b400700 1 -- 172.21.15.23:0/2153887212 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c009ea0 con 0x7f7d14105a30 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d0b400700 1 -- 172.21.15.23:0/2153887212 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d0c00f040 con 0x7f7d14105a30 2024-03-30T04:02:51.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.904+0000 7f7d0b400700 1 -- 172.21.15.23:0/2153887212 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0043a0 con 0x7f7d14105a30 2024-03-30T04:02:51.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.905+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1419c410 con 0x7f7d14105a30 2024-03-30T04:02:51.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.905+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d141a3c50 con 0x7f7d14105a30 2024-03-30T04:02:51.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.908+0000 7f7d0b400700 1 -- 172.21.15.23:0/2153887212 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f7d0c00db20 con 0x7f7d14105a30 2024-03-30T04:02:51.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.910+0000 7f7d2d6e9b80 1 Processor -- start 2024-03-30T04:02:51.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.910+0000 7f7d2d6e9b80 1 -- start start 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 0x56115b960b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 0x56115b96ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b96e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b8b9990 con 0x56115b9607a0 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b8b7350 con 0x56115b9610c0 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b8b2c40 con 0x56115b96bfc0 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d0b400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 0x7f7cf80a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d0b400700 1 -- 172.21.15.23:0/2153887212 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7d0c0cca60 con 0x7f7d14105a30 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 0x7f7cf80a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d12000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 0x56115b960b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d12a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 0x56115b96ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d12000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 0x56115b960b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45654/0 (socket says 172.21.15.23:45654) 2024-03-30T04:02:51.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.911+0000 7f7d12000700 1 -- 172.21.15.23:0/2169092380 learned_addr learned my addr 172.21.15.23:0/2169092380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12000700 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b96e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 -- 172.21.15.23:0/2169092380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 msgr2=0x56115b96e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b96e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 -- 172.21.15.23:0/2169092380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 msgr2=0x56115b960b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 0x56115b960b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 -- 172.21.15.23:0/2169092380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56115b7343b0 con 0x56115b9610c0 2024-03-30T04:02:51.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d13400700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 0x7f7cf80a38f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0c9cd0 tx=0x7f7d0c007f40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.912+0000 7f7d12a00700 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 0x56115b96ba80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d040098e0 tx=0x7f7d04009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec2ada6cb22c6d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.913+0000 7f7cffe00700 1 -- 172.21.15.23:0/2169092380 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d040050d0 con 0x56115b9610c0 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.913+0000 7f7cffe00700 1 -- 172.21.15.23:0/2169092380 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d0401e040 con 0x56115b9610c0 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.913+0000 7f7cffe00700 1 -- 172.21.15.23:0/2169092380 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400f6c0 con 0x56115b9610c0 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/2169092380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 msgr2=0x56115b96ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 0x56115b96ba80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7d040098e0 tx=0x7f7d04009ee0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/2169092380 shutdown_connections 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b96e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9607a0 0x56115b960b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/2169092380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9610c0 0x56115b96ba80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/2169092380 >> 172.21.15.23:0/2169092380 conn(0x56115b8c06a0 msgr2=0x56115b95cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/2169092380 shutdown_connections 2024-03-30T04:02:51.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.914+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/2169092380 wait complete. 2024-03-30T04:02:51.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.915+0000 7f7d2d6e9b80 1 -- start start 2024-03-30T04:02:51.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.915+0000 7f7d2d6e9b80 1 Processor -- start 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b968c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b969180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b84ed80 con 0x56115b9610c0 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b84ea20 con 0x56115b9607a0 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b7366a0 con 0x56115b96bfc0 2024-03-30T04:02:51.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.916+0000 7f7d2d6e9b80 10 client.0 _ll_register_callbacks cb 0x56115b858470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.917+0000 7f7d12000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b969180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.917+0000 7f7d12000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.917+0000 7f7d12000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52748/0 (socket says 172.21.15.23:52748) 2024-03-30T04:02:51.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.917+0000 7f7d12000700 1 -- 172.21.15.23:0/321007592 learned_addr learned my addr 172.21.15.23:0/321007592 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.917+0000 7f7d12a00700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b968c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 -- 172.21.15.23:0/321007592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 msgr2=0x56115b969180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b969180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 -- 172.21.15.23:0/321007592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 msgr2=0x56115b968c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b968c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 -- 172.21.15.23:0/321007592 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d1410d1f0 con 0x56115b9607a0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d12000700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000ce10 tx=0x7f7d00011830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39cbb4aff2590975 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7cfcc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00004030 con 0x56115b9607a0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7d00004190 con 0x56115b9607a0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.918+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56115b873380 con 0x56115b9607a0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.919+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d00004350 con 0x56115b9607a0 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.919+0000 7f7d2d6e9b80 10 client.25928 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.919+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56115b6e0b60 con 0x56115b9607a0 2024-03-30T04:02:51.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.919+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56115b873380 con 0x56115b9607a0 2024-03-30T04:02:51.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7d0001e870 con 0x56115b9607a0 2024-03-30T04:02:51.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 10 client.25928 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 10 client.25928 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 20 client.25928 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b97baf0 0x56115b97dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 4 client.25928 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.920+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56115b8ab700 con 0x56115b97baf0 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.921+0000 7f7d12a00700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b97baf0 0x56115b97dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.921+0000 7f7d12a00700 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b97baf0 0x56115b97dee0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=89e1c280e1cc9544 server_cookie=48d8aeb6f00b7738 in_seq=0 out_seq=0 2024-03-30T04:02:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.921+0000 7f7cfcc00700 10 client.25928 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.922+0000 7f7cfcc00700 1 -- 172.21.15.23:0/321007592 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56115b8ab700 con 0x56115b97baf0 2024-03-30T04:02:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.922+0000 7f7cfcc00700 10 client.25928 handle_command_reply: tid=0 2024-03-30T04:02:51.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.922+0000 7f7d2d6e9b80 1 client.25928 shutdown 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.922+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 msgr2=0x56115b96b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000ce10 tx=0x7f7d00011830 comp rx=0 tx=0).stop 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 shutdown_connections 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b97baf0 0x56115b97dee0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b96bfc0 0x56115b969180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b96b910 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/321007592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b968c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 >> 172.21.15.23:0/321007592 conn(0x56115b8c06a0 msgr2=0x56115b962840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7cfcc00700 0 client.25928 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:51.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 shutdown_connections 2024-03-30T04:02:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/321007592 wait complete. 2024-03-30T04:02:51.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.923+0000 7f7d2d6e9b80 20 client.25928 trim_cache size 0 max 16384 2024-03-30T04:02:51.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.950+0000 7f7d2d6e9b80 1 Processor -- start 2024-03-30T04:02:51.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.950+0000 7f7d2d6e9b80 1 -- start start 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.950+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.950+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 0x56115b968bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.950+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b96bfc0 0x56115b972c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b961a20 con 0x56115b96bfc0 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b95c160 con 0x56115b9607a0 2024-03-30T04:02:51.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b97b8d0 con 0x56115b9610c0 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52762/0 (socket says 172.21.15.23:52762) 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12000700 1 -- 172.21.15.23:0/337890628 learned_addr learned my addr 172.21.15.23:0/337890628 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 0x56115b968bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12000700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b96bfc0 0x56115b972c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 -- 172.21.15.23:0/337890628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 msgr2=0x56115b9586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 -- 172.21.15.23:0/337890628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b96bfc0 msgr2=0x56115b972c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b96bfc0 0x56115b972c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12a00700 1 -- 172.21.15.23:0/337890628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56115b878ea0 con 0x56115b9610c0 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.951+0000 7f7d12000700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:51.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.952+0000 7f7d12a00700 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 0x56115b968bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a010 tx=0x7f7d04009380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a350914d6fa71bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.952+0000 7f7d08c00700 1 -- 172.21.15.23:0/337890628 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400be00 con 0x56115b9610c0 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.952+0000 7f7d08c00700 1 -- 172.21.15.23:0/337890628 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d04031070 con 0x56115b9610c0 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.952+0000 7f7d08c00700 1 -- 172.21.15.23:0/337890628 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400fd30 con 0x56115b9610c0 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/337890628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 msgr2=0x56115b968bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 0x56115b968bc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400a010 tx=0x7f7d04009380 comp rx=0 tx=0).stop 2024-03-30T04:02:51.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/337890628 shutdown_connections 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b96bfc0 0x56115b972c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b9607a0 0x56115b9586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/337890628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9610c0 0x56115b968bc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/337890628 >> 172.21.15.23:0/337890628 conn(0x56115b8c06a0 msgr2=0x56115b968490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/337890628 shutdown_connections 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.953+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/337890628 wait complete. 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.954+0000 7f7d2d6e9b80 1 -- start start 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.954+0000 7f7d2d6e9b80 1 Processor -- start 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b9702f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b96bfc0 0x56115b970a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b971110 con 0x56115b9610c0 2024-03-30T04:02:51.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b9ce8f0 con 0x56115b96bfc0 2024-03-30T04:02:51.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56115b9cea30 con 0x56115b9607a0 2024-03-30T04:02:51.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d2d6e9b80 10 client.0 _ll_register_callbacks cb 0x56115b858470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44358/0 (socket says 172.21.15.23:44358) 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 -- 172.21.15.23:0/650666476 learned_addr learned my addr 172.21.15.23:0/650666476 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b96bfc0 0x56115b970a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12a00700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b9702f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 -- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b96bfc0 msgr2=0x56115b970a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b96bfc0 0x56115b970a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 -- 172.21.15.23:0/650666476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 msgr2=0x56115b9702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b9702f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.955+0000 7f7d12000700 1 -- 172.21.15.23:0/650666476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56115b6e0b60 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7d12a00700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b9702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7d12000700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000cb30 tx=0x7f7d00005c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a22bcc54ca3b0176 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7cfcc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d00004920 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f7d00000e40 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.956+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d0401e040 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d00004a80 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 10 client.25930 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56115b7343b0 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d0401e040 con 0x56115b9607a0 2024-03-30T04:02:51.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7d00034420 con 0x56115b9607a0 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 10 client.25930 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 10 client.25930 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 20 client.25930 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b9d8990 0x56115b9dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 4 client.25930 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.957+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56115b701cf0 con 0x56115b9d8990 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.958+0000 7f7d12a00700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b9d8990 0x56115b9dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.958+0000 7f7d12a00700 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b9d8990 0x56115b9dadb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d3b605b357ac7b7 server_cookie=525a7202167d55a6 in_seq=0 out_seq=0 2024-03-30T04:02:51.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.958+0000 7f7cfcc00700 10 client.25930 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7cfcc00700 1 -- 172.21.15.23:0/650666476 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56115b701cf0 con 0x56115b9d8990 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7cfcc00700 10 client.25930 handle_command_reply: tid=0 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 client.25930 shutdown 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 msgr2=0x56115b972e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7d0000cb30 tx=0x7f7d00005c20 comp rx=0 tx=0).stop 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 shutdown_connections 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x56115b9d8990 0x56115b9dadb0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56115b96bfc0 0x56115b970a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56115b9607a0 0x56115b972e60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 --2- 172.21.15.23:0/650666476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56115b9610c0 0x56115b9702f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 >> 172.21.15.23:0/650666476 conn(0x56115b8c06a0 msgr2=0x56115b95c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7cfcc00700 0 client.25930 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:02:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 shutdown_connections 2024-03-30T04:02:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.959+0000 7f7d2d6e9b80 1 -- 172.21.15.23:0/650666476 wait complete. 2024-03-30T04:02:51.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.960+0000 7f7d2d6e9b80 20 client.25930 trim_cache size 0 max 16384 2024-03-30T04:02:51.962 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.960+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 msgr2=0x7f7cf80a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.960+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 0x7f7cf80a38f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0c9cd0 tx=0x7f7d0c007f40 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.960+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 msgr2=0x7f7d1419b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.960+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d1419b7c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007f10 tx=0x7f7d0c00d3b0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 shutdown_connections 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d14105a30 0x7f7d1419b7c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d140ffa20 0x7f7d14196ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cf80a1430 0x7f7cf80a38f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 --2- 172.21.15.23:0/2153887212 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d141003e0 0x7f7d14197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 >> 172.21.15.23:0/2153887212 conn(0x7f7d140fb510 msgr2=0x7f7d140fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 shutdown_connections 2024-03-30T04:02:51.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:51.961+0000 7f7d13e00700 1 -- 172.21.15.23:0/2153887212 wait complete. 2024-03-30T04:02:51.984 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:02:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 -- 172.21.15.23:0/3341762394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6308103b20 msgr2=0x7f6308107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 --2- 172.21.15.23:0/3341762394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6308103b20 0x7f6308107b60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009990 tx=0x7f62f4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 -- 172.21.15.23:0/3341762394 shutdown_connections 2024-03-30T04:02:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 --2- 172.21.15.23:0/3341762394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308108290 0x7f630810fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 --2- 172.21.15.23:0/3341762394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308103170 0x7f6308103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 --2- 172.21.15.23:0/3341762394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6308103b20 0x7f6308107b60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 -- 172.21.15.23:0/3341762394 >> 172.21.15.23:0/3341762394 conn(0x7f63080fe990 msgr2=0x7f6308100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 -- 172.21.15.23:0/3341762394 shutdown_connections 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.254+0000 7f630fe00700 1 -- 172.21.15.23:0/3341762394 wait complete. 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.256+0000 7f630fe00700 1 Processor -- start 2024-03-30T04:02:52.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.256+0000 7f630fe00700 1 -- start start 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.256+0000 7f630fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308108290 0x7f63081999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f630819dd00 0x7f630819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630fe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630819e780 con 0x7f6308103170 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630fe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630819e8f0 con 0x7f6308108290 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630819eab0 con 0x7f630819dd00 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45698/0 (socket says 172.21.15.23:45698) 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 -- 172.21.15.23:0/752374568 learned_addr learned my addr 172.21.15.23:0/752374568 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630ca00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308108290 0x7f63081999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630de00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f630819dd00 0x7f630819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 -- 172.21.15.23:0/752374568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f630819dd00 msgr2=0x7f630819e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f630819dd00 0x7f630819e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 -- 172.21.15.23:0/752374568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308108290 msgr2=0x7f63081999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308108290 0x7f63081999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.257+0000 7f630d400700 1 -- 172.21.15.23:0/752374568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40095f0 con 0x7f6308103170 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f630d400700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e9a0 tx=0x7f62fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f6306000700 1 -- 172.21.15.23:0/752374568 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc00cbd0 con 0x7f6308103170 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63081a5fb0 con 0x7f6308103170 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f6306000700 1 -- 172.21.15.23:0/752374568 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62fc00cd30 con 0x7f6308103170 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f6306000700 1 -- 172.21.15.23:0/752374568 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc018680 con 0x7f6308103170 2024-03-30T04:02:52.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.258+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63081a6480 con 0x7f6308103170 2024-03-30T04:02:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.263+0000 7f6306000700 1 -- 172.21.15.23:0/752374568 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f62fc010430 con 0x7f6308103170 2024-03-30T04:02:52.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.264+0000 7f63238f9b80 1 Processor -- start 2024-03-30T04:02:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.264+0000 7f63238f9b80 1 -- start start 2024-03-30T04:02:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9fc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9ff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a94a990 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a948350 con 0x55fd3a9f1ef0 2024-03-30T04:02:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f63238f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a943c40 con 0x55fd3a9f15d0 2024-03-30T04:02:52.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f6306000700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 0x7f62f00a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f6306000700 1 -- 172.21.15.23:0/752374568 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f62fc014070 con 0x7f6308103170 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f630de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f630d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9fc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f630d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9fc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52798/0 (socket says 172.21.15.23:52798) 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.265+0000 7f630d400700 1 -- 172.21.15.23:0/2998192317 learned_addr learned my addr 172.21.15.23:0/2998192317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630ca00700 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9ff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 -- 172.21.15.23:0/2998192317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 msgr2=0x55fd3a9fc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9fc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 -- 172.21.15.23:0/2998192317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 msgr2=0x55fd3a9ff1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9ff1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 -- 172.21.15.23:0/2998192317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd3a7c53b0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.266+0000 7f630de00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 0x7f62f00a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.267+0000 7f630de00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 0x7f62f00a38f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6300009510 tx=0x7f6300015000 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.267+0000 7f630de00700 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9f19b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6300009200 tx=0x7f6300007500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f40b3889493080b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.268+0000 7f62fa600700 1 -- 172.21.15.23:0/2998192317 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630000e070 con 0x55fd3a9f15d0 2024-03-30T04:02:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.268+0000 7f62fa600700 1 -- 172.21.15.23:0/2998192317 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f630000f040 con 0x55fd3a9f15d0 2024-03-30T04:02:52.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.268+0000 7f62fa600700 1 -- 172.21.15.23:0/2998192317 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630001fde0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.268+0000 7f63238f9b80 1 -- 172.21.15.23:0/2998192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 msgr2=0x55fd3a9f19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.268+0000 7f63238f9b80 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9f19b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6300009200 tx=0x7f6300007500 comp rx=0 tx=0).stop 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 -- 172.21.15.23:0/2998192317 shutdown_connections 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9f19b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 --2- 172.21.15.23:0/2998192317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 -- 172.21.15.23:0/2998192317 >> 172.21.15.23:0/2998192317 conn(0x55fd3a9516a0 msgr2=0x55fd3a9eda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 -- 172.21.15.23:0/2998192317 shutdown_connections 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 -- 172.21.15.23:0/2998192317 wait complete. 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 -- start start 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.269+0000 7f63238f9b80 1 Processor -- start 2024-03-30T04:02:52.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9fa0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a8dfd80 con 0x55fd3a9f15d0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a8dfa20 con 0x55fd3a9f1ef0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a7c76a0 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f63238f9b80 10 client.0 _ll_register_callbacks cb 0x55fd3a8e9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.270+0000 7f630d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45706/0 (socket says 172.21.15.23:45706) 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 -- 172.21.15.23:0/3055100936 learned_addr learned my addr 172.21.15.23:0/3055100936 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630de00700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9fa0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630ca00700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 -- 172.21.15.23:0/3055100936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 msgr2=0x55fd3a9fa0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9fa0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 -- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 msgr2=0x55fd3a9f9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 -- 172.21.15.23:0/3055100936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6300011040 con 0x55fd3a9f15d0 2024-03-30T04:02:52.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630ca00700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f630d400700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e970 tx=0x7f62fc0109c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c1c9a14a356852 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.271+0000 7f62eea00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0997c0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f62fc0cd3b0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fd3a904380 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62fc09a0d0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f63238f9b80 10 client.15945 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fd3a771b60 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.272+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fd3a904380 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f62fc095ec0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 10 client.15945 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:52.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 10 client.15945 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:02:52.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 20 client.15945 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:52.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa0c960 0x55fd3aa0ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 4 client.15945 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:52.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.273+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd3a93c700 con 0x55fd3aa0c960 2024-03-30T04:02:52.273 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:52 smithi023 ceph-mon[38254]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-30T04:02:52.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.274+0000 7f630ca00700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa0c960 0x55fd3aa0ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.275+0000 7f630ca00700 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa0c960 0x55fd3aa0ed50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8aa0a4e646d3b639 server_cookie=9e32f4f6ce17a237 in_seq=0 out_seq=0 2024-03-30T04:02:52.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.275+0000 7f62eea00700 10 client.15945 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f62eea00700 1 -- 172.21.15.23:0/3055100936 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fd3a93c700 con 0x55fd3aa0c960 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f62eea00700 10 client.15945 handle_command_reply: tid=0 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 client.15945 shutdown 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 msgr2=0x55fd3a9fc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e970 tx=0x7f62fc0109c0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 shutdown_connections 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa0c960 0x55fd3aa0ed50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9fa0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3a9fc7d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 --2- 172.21.15.23:0/3055100936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 >> 172.21.15.23:0/3055100936 conn(0x55fd3a9516a0 msgr2=0x55fd3a9f34f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f62eea00700 0 client.15945 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 shutdown_connections 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.276+0000 7f63238f9b80 1 -- 172.21.15.23:0/3055100936 wait complete. 2024-03-30T04:02:52.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.277+0000 7f63238f9b80 20 client.15945 trim_cache size 0 max 16384 2024-03-30T04:02:52.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.290+0000 7f63238f9b80 1 Processor -- start 2024-03-30T04:02:52.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 -- start start 2024-03-30T04:02:52.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a9f5500 con 0x55fd3a9f15d0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a9f2850 con 0x55fd3a9f1ef0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f63238f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3a9f43a0 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.291+0000 7f630d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44396/0 (socket says 172.21.15.23:44396) 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 -- 172.21.15.23:0/1135104807 learned_addr learned my addr 172.21.15.23:0/1135104807 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630ca00700 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 -- 172.21.15.23:0/1135104807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 msgr2=0x55fd3a9f8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 -- 172.21.15.23:0/1135104807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 msgr2=0x55fd3aa03a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 -- 172.21.15.23:0/1135104807 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd3a909ea0 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630d400700 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.292+0000 7f630de00700 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6300009200 tx=0x7f6300007cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd63fd52299c902 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.293+0000 7f62fb000700 1 -- 172.21.15.23:0/1135104807 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630000e040 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.293+0000 7f62fb000700 1 -- 172.21.15.23:0/1135104807 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f630002e070 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.293+0000 7f62fb000700 1 -- 172.21.15.23:0/1135104807 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63000336e0 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 -- 172.21.15.23:0/1135104807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 msgr2=0x55fd3a9f88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6300009200 tx=0x7f6300007cf0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 -- 172.21.15.23:0/1135104807 shutdown_connections 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9fcdf0 0x55fd3a9f88d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1135104807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f1ef0 0x55fd3a9f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 -- 172.21.15.23:0/1135104807 >> 172.21.15.23:0/1135104807 conn(0x55fd3a9516a0 msgr2=0x55fd3a9f4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 -- 172.21.15.23:0/1135104807 shutdown_connections 2024-03-30T04:02:52.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.294+0000 7f63238f9b80 1 -- 172.21.15.23:0/1135104807 wait complete. 2024-03-30T04:02:52.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.295+0000 7f63238f9b80 1 -- start start 2024-03-30T04:02:52.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.295+0000 7f63238f9b80 1 Processor -- start 2024-03-30T04:02:52.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.295+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f1ef0 0x55fd3aa01f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3aa02b80 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3aa02cf0 con 0x55fd3a9f15d0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd3aa60770 con 0x55fd3a9f1ef0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f63238f9b80 10 client.0 _ll_register_callbacks cb 0x55fd3a8e9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45716/0 (socket says 172.21.15.23:45716) 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 -- 172.21.15.23:0/1948101323 learned_addr learned my addr 172.21.15.23:0/1948101323 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630d400700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630ca00700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f1ef0 0x55fd3aa01f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 -- 172.21.15.23:0/1948101323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f1ef0 msgr2=0x55fd3aa01f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f1ef0 0x55fd3aa01f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 -- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f15d0 msgr2=0x55fd3aa03a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.296+0000 7f630de00700 1 -- 172.21.15.23:0/1948101323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd3a771b60 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.297+0000 7f630de00700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f63000091d0 tx=0x7f6300017b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d3fefdcac71b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.297+0000 7f62eea00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630000e050 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6300017e30 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f630000f040 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f630001f840 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f63238f9b80 10 client.15951 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.298+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fd3a7c53b0 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f630000f040 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6300045500 con 0x55fd3a9fcdf0 2024-03-30T04:02:52.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 10 client.15951 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 10 client.15951 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 20 client.15951 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa61a10 0x55fd3aa63e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 4 client.15951 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.299+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fd3a792cf0 con 0x55fd3aa61a10 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f630d400700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa61a10 0x55fd3aa63e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f630d400700 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa61a10 0x55fd3aa63e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eabb4cf8328aea99 server_cookie=453a7d2770f4b187 in_seq=0 out_seq=0 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f62eea00700 10 client.15951 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f62eea00700 1 -- 172.21.15.23:0/1948101323 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fd3a792cf0 con 0x55fd3aa61a10 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f62eea00700 10 client.15951 handle_command_reply: tid=0 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.300+0000 7f63238f9b80 1 client.15951 shutdown 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 msgr2=0x55fd3aa02490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f63000091d0 tx=0x7f6300017b90 comp rx=0 tx=0).stop 2024-03-30T04:02:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 shutdown_connections 2024-03-30T04:02:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fd3a9fcdf0 0x55fd3aa02490 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fd3a9f15d0 0x55fd3aa03a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55fd3aa61a10 0x55fd3aa63e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 --2- 172.21.15.23:0/1948101323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd3a9f1ef0 0x55fd3aa01f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 >> 172.21.15.23:0/1948101323 conn(0x55fd3a9516a0 msgr2=0x55fd3a9f8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f62eea00700 0 client.15951 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 shutdown_connections 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 1 -- 172.21.15.23:0/1948101323 wait complete. 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.301+0000 7f63238f9b80 20 client.15951 trim_cache size 0 max 16384 2024-03-30T04:02:52.302 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 msgr2=0x7f62f00a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 0x7f62f00a38f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6300009510 tx=0x7f6300015000 comp rx=0 tx=0).stop 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 msgr2=0x7f6308199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f62fc00e9a0 tx=0x7f62fc00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 shutdown_connections 2024-03-30T04:02:52.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f630819dd00 0x7f630819e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6308103170 0x7f6308199470 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f62f00a1430 0x7f62f00a38f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 --2- 172.21.15.23:0/752374568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6308108290 0x7f63081999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.302+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 >> 172.21.15.23:0/752374568 conn(0x7f63080fe990 msgr2=0x7f63080ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.303+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 shutdown_connections 2024-03-30T04:02:52.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.303+0000 7f630fe00700 1 -- 172.21.15.23:0/752374568 wait complete. 2024-03-30T04:02:52.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:02:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:52 smithi154 ceph-mon[79161]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-30T04:02:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:52 smithi193 ceph-mon[79581]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 85 B/s wr, 11 op/s 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 -- 172.21.15.23:0/504963717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c068370 msgr2=0x7fe56c0687f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 --2- 172.21.15.23:0/504963717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c068370 0x7fe56c0687f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe558009990 tx=0x7fe558009ca0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 -- 172.21.15.23:0/504963717 shutdown_connections 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 --2- 172.21.15.23:0/504963717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c103740 0x7fe56c103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 --2- 172.21.15.23:0/504963717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c105ca0 0x7fe56c067e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 --2- 172.21.15.23:0/504963717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c068370 0x7fe56c0687f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 -- 172.21.15.23:0/504963717 >> 172.21.15.23:0/504963717 conn(0x7fe56c076d40 msgr2=0x7fe56c077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 -- 172.21.15.23:0/504963717 shutdown_connections 2024-03-30T04:02:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.575+0000 7fe573000700 1 -- 172.21.15.23:0/504963717 wait complete. 2024-03-30T04:02:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.576+0000 7fe573000700 1 Processor -- start 2024-03-30T04:02:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.576+0000 7fe573000700 1 -- start start 2024-03-30T04:02:52.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 0x7fe56c1a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c105ca0 0x7fe56c19d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19dd10 con 0x7fe56c068370 2024-03-30T04:02:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19de80 con 0x7fe56c105ca0 2024-03-30T04:02:52.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe573000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56c19e040 con 0x7fe56c103740 2024-03-30T04:02:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe56be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe56be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45736/0 (socket says 172.21.15.23:45736) 2024-03-30T04:02:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe56be00700 1 -- 172.21.15.23:0/697891933 learned_addr learned my addr 172.21.15.23:0/697891933 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.577+0000 7fe56b400700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 0x7fe56c1a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe571000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c105ca0 0x7fe56c19d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 -- 172.21.15.23:0/697891933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 msgr2=0x7fe56c1a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 0x7fe56c1a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 -- 172.21.15.23:0/697891933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c105ca0 msgr2=0x7fe56c19d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c105ca0 0x7fe56c19d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 -- 172.21.15.23:0/697891933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580095f0 con 0x7fe56c068370 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56b400700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 0x7fe56c1a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe56be00700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e9a0 tx=0x7fe55c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.578+0000 7fe568c00700 1 -- 172.21.15.23:0/697891933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c009790 con 0x7fe56c068370 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.579+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe56c19e320 con 0x7fe56c068370 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.579+0000 7fe568c00700 1 -- 172.21.15.23:0/697891933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c010430 con 0x7fe56c068370 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.579+0000 7fe568c00700 1 -- 172.21.15.23:0/697891933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c018680 con 0x7fe56c068370 2024-03-30T04:02:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.579+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c1aa570 con 0x7fe56c068370 2024-03-30T04:02:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.583+0000 7fe568c00700 1 -- 172.21.15.23:0/697891933 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fe55c0105a0 con 0x7fe56c068370 2024-03-30T04:02:52.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.584+0000 7fe586bcbb80 1 Processor -- start 2024-03-30T04:02:52.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.584+0000 7fe586bcbb80 1 -- start start 2024-03-30T04:02:52.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd3249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd32fdf0 0x5573cd3321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd27d990 con 0x5573cd3245d0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd27b350 con 0x5573cd324ef0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd276c40 con 0x5573cd32fdf0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe568c00700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 0x7fe5500a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe568c00700 1 -- 172.21.15.23:0/697891933 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe55c014070 con 0x7fe56c068370 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe56be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe56be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52856/0 (socket says 172.21.15.23:52856) 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe56be00700 1 -- 172.21.15.23:0/4237749001 learned_addr learned my addr 172.21.15.23:0/4237749001 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe571000700 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd3249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.585+0000 7fe56b400700 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd32fdf0 0x5573cd3321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 -- 172.21.15.23:0/4237749001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd32fdf0 msgr2=0x5573cd3321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd32fdf0 0x5573cd3321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 -- 172.21.15.23:0/4237749001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 msgr2=0x5573cd3249b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd3249b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe571000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 0x7fe5500a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 -- 172.21.15.23:0/4237749001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573cd0f83b0 con 0x5573cd324ef0 2024-03-30T04:02:52.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe56be00700 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099cc0 tx=0x7fe55c099c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff7dca03b8c0495d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe561600700 1 -- 172.21.15.23:0/4237749001 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c09ddf0 con 0x5573cd324ef0 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe561600700 1 -- 172.21.15.23:0/4237749001 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c0a2440 con 0x5573cd324ef0 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe571000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 0x7fe5500a3870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe564009550 tx=0x7fe564009130 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.586+0000 7fe561600700 1 -- 172.21.15.23:0/4237749001 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c094bd0 con 0x5573cd324ef0 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 -- 172.21.15.23:0/4237749001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 msgr2=0x5573cd32f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099cc0 tx=0x7fe55c099c30 comp rx=0 tx=0).stop 2024-03-30T04:02:52.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 -- 172.21.15.23:0/4237749001 shutdown_connections 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd32fdf0 0x5573cd3321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd3249b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/4237749001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd324ef0 0x5573cd32f8b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 -- 172.21.15.23:0/4237749001 >> 172.21.15.23:0/4237749001 conn(0x5573cd2846a0 msgr2=0x5573cd320a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.587+0000 7fe586bcbb80 1 -- 172.21.15.23:0/4237749001 shutdown_connections 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.588+0000 7fe586bcbb80 1 -- 172.21.15.23:0/4237749001 wait complete. 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.588+0000 7fe586bcbb80 1 -- start start 2024-03-30T04:02:52.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 Processor -- start 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd32d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd212d80 con 0x5573cd3245d0 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd212a20 con 0x5573cd32fdf0 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd0fa6a0 con 0x5573cd324ef0 2024-03-30T04:02:52.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.589+0000 7fe586bcbb80 10 client.0 _ll_register_callbacks cb 0x5573cd21c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44438/0 (socket says 172.21.15.23:44438) 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56b400700 1 -- 172.21.15.23:0/2561089073 learned_addr learned my addr 172.21.15.23:0/2561089073 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe571000700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd32d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 -- 172.21.15.23:0/2561089073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 msgr2=0x5573cd32cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 -- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 msgr2=0x5573cd32d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd32d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.590+0000 7fe56be00700 1 -- 172.21.15.23:0/2561089073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56c1aa570 con 0x5573cd3245d0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe56be00700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32f830 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099cc0 tx=0x7fe55c010ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67c212fc4dff040 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe556000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c094bd0 con 0x5573cd3245d0 2024-03-30T04:02:52.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573cd237380 con 0x5573cd3245d0 2024-03-30T04:02:52.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe55c094d30 con 0x5573cd3245d0 2024-03-30T04:02:52.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe55c0b81a0 con 0x5573cd3245d0 2024-03-30T04:02:52.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe586bcbb80 10 client.15963 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:52.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5573cd0a4b60 con 0x5573cd3245d0 2024-03-30T04:02:52.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.591+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5573cd237380 con 0x5573cd3245d0 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe55c0ca9b0 con 0x5573cd3245d0 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 10 client.15963 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 20 client.15963 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd33f570 0x5573cd341990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 4 client.15963 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:52.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.592+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573cd26f700 con 0x5573cd33f570 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.593+0000 7fe56b400700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd33f570 0x5573cd341990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.594+0000 7fe556000700 10 client.15963 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.594+0000 7fe56b400700 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd33f570 0x5573cd341990 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8e4d3fd0ab6cb4 server_cookie=1178638027b79692 in_seq=0 out_seq=0 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe556000700 1 -- 172.21.15.23:0/2561089073 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5573cd26f700 con 0x5573cd33f570 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe556000700 10 client.15963 handle_command_reply: tid=0 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 client.15963 shutdown 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 msgr2=0x5573cd32f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32f830 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099cc0 tx=0x7fe55c010ab0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 shutdown_connections 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd32d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32f830 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd33f570 0x5573cd341990 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2561089073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe556000700 0 client.15963 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 >> 172.21.15.23:0/2561089073 conn(0x5573cd2846a0 msgr2=0x5573cd3266c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.595+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 shutdown_connections 2024-03-30T04:02:52.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.596+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2561089073 wait complete. 2024-03-30T04:02:52.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.596+0000 7fe586bcbb80 20 client.15963 trim_cache size 0 max 16384 2024-03-30T04:02:52.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 Processor -- start 2024-03-30T04:02:52.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 -- start start 2024-03-30T04:02:52.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd31ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd327930 con 0x5573cd3245d0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd33bd50 con 0x5573cd32fdf0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd321710 con 0x5573cd324ef0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.610+0000 7fe56be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45766/0 (socket says 172.21.15.23:45766) 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 -- 172.21.15.23:0/1327769290 learned_addr learned my addr 172.21.15.23:0/1327769290 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe571000700 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd31ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 -- 172.21.15.23:0/1327769290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 msgr2=0x5573cd32d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 -- 172.21.15.23:0/1327769290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 msgr2=0x5573cd31ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd31ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 -- 172.21.15.23:0/1327769290 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573cd23cea0 con 0x5573cd3245d0 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.611+0000 7fe56be00700 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099f50 tx=0x7fe55c0cd3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46d2596e1cc9d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe562000700 1 -- 172.21.15.23:0/1327769290 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c098420 con 0x5573cd3245d0 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe562000700 1 -- 172.21.15.23:0/1327769290 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c0ade90 con 0x5573cd3245d0 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe562000700 1 -- 172.21.15.23:0/1327769290 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0a2210 con 0x5573cd3245d0 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 -- 172.21.15.23:0/1327769290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 msgr2=0x5573cd32cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe55c099f50 tx=0x7fe55c0cd3e0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 -- 172.21.15.23:0/1327769290 shutdown_connections 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd31ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd3245d0 0x5573cd32cac0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/1327769290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd324ef0 0x5573cd32d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 -- 172.21.15.23:0/1327769290 >> 172.21.15.23:0/1327769290 conn(0x5573cd2846a0 msgr2=0x5573cd32c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 -- 172.21.15.23:0/1327769290 shutdown_connections 2024-03-30T04:02:52.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.612+0000 7fe586bcbb80 1 -- 172.21.15.23:0/1327769290 wait complete. 2024-03-30T04:02:52.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.613+0000 7fe586bcbb80 1 -- start start 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.613+0000 7fe586bcbb80 1 Processor -- start 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 0x5573cd3350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd392360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd3928a0 con 0x5573cd324ef0 2024-03-30T04:02:52.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd3929e0 con 0x5573cd32fdf0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573cd392b20 con 0x5573cd3245d0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe586bcbb80 10 client.0 _ll_register_callbacks cb 0x5573cd21c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe571000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd392360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe56be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe571000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd392360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52892/0 (socket says 172.21.15.23:52892) 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe56be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44458/0 (socket says 172.21.15.23:44458) 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.614+0000 7fe571000700 1 -- 172.21.15.23:0/2895347671 learned_addr learned my addr 172.21.15.23:0/2895347671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56b400700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 0x5573cd3350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 -- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 msgr2=0x5573cd392360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd392360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 -- 172.21.15.23:0/2895347671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 msgr2=0x5573cd3350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 0x5573cd3350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 -- 172.21.15.23:0/2895347671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573cd0a4b60 con 0x5573cd3245d0 2024-03-30T04:02:52.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56b400700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 0x5573cd3350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe56be00700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009230 tx=0x7fe55c0cabe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8af855a2bf9d4290 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe556000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.615+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0b2d70 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe55c098420 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe55c0a2440 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe55c0b8020 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 10 client.25966 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5573cd0fccd0 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe55c0a2440 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe55c098f20 con 0x5573cd3245d0 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 10 client.25966 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 10 client.25966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:52.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.616+0000 7fe586bcbb80 20 client.25966 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.617+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd393dc0 0x5573cd3961b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.617+0000 7fe586bcbb80 4 client.25966 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.617+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5573cd0c5cf0 con 0x5573cd393dc0 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.617+0000 7fe56b400700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd393dc0 0x5573cd3961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe556000700 10 client.25966 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe56b400700 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd393dc0 0x5573cd3961b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adedf3178a300ae4 server_cookie=cff58a3bc2ba0f26 in_seq=0 out_seq=0 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe556000700 1 -- 172.21.15.23:0/2895347671 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5573cd0c5cf0 con 0x5573cd393dc0 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe556000700 10 client.25966 handle_command_reply: tid=0 2024-03-30T04:02:52.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 client.25966 shutdown 2024-03-30T04:02:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 msgr2=0x5573cd334b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe55c009230 tx=0x7fe55c0cabe0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 shutdown_connections 2024-03-30T04:02:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573cd32fdf0 0x5573cd392360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573cd393dc0 0x5573cd3961b0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573cd3245d0 0x5573cd334b80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 --2- 172.21.15.23:0/2895347671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573cd324ef0 0x5573cd3350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 >> 172.21.15.23:0/2895347671 conn(0x5573cd2846a0 msgr2=0x5573cd327df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe556000700 0 client.25966 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.618+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 shutdown_connections 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe586bcbb80 1 -- 172.21.15.23:0/2895347671 wait complete. 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe586bcbb80 20 client.25966 trim_cache size 0 max 16384 2024-03-30T04:02:52.621 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 msgr2=0x7fe5500a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 0x7fe5500a3870 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe564009550 tx=0x7fe564009130 comp rx=0 tx=0).stop 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 msgr2=0x7fe56c1a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.619+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00e9a0 tx=0x7fe55c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 shutdown_connections 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe5500a13b0 0x7fe5500a3870 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe56c105ca0 0x7fe56c19d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe56c068370 0x7fe56c1a3580 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 --2- 172.21.15.23:0/697891933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe56c103740 0x7fe56c1a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 >> 172.21.15.23:0/697891933 conn(0x7fe56c076d40 msgr2=0x7fe56c100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 shutdown_connections 2024-03-30T04:02:52.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:52.620+0000 7fe573000700 1 -- 172.21.15.23:0/697891933 wait complete. 2024-03-30T04:02:53.645 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:02:53.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.884+0000 7f2101000700 1 -- 172.21.15.23:0/472689770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 msgr2=0x7f20fc0682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 --2- 172.21.15.23:0/472689770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0682b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f20e80099c0 tx=0x7f20e8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 -- 172.21.15.23:0/472689770 shutdown_connections 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 --2- 172.21.15.23:0/472689770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc0687f0 0x7f20fc10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 --2- 172.21.15.23:0/472689770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc105ce0 0x7f20fc1060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 --2- 172.21.15.23:0/472689770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0682b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 -- 172.21.15.23:0/472689770 >> 172.21.15.23:0/472689770 conn(0x7f20fc076d40 msgr2=0x7f20fc077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 -- 172.21.15.23:0/472689770 shutdown_connections 2024-03-30T04:02:53.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.885+0000 7f2101000700 1 -- 172.21.15.23:0/472689770 wait complete. 2024-03-30T04:02:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.886+0000 7f2101000700 1 Processor -- start 2024-03-30T04:02:53.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.886+0000 7f2101000700 1 -- start start 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc0687f0 0x7f20fc1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 0x7f20fc1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc1a5620 con 0x7f20fc0687f0 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc1a5790 con 0x7f20fc105ce0 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f2101000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20fc1a5950 con 0x7f20fc067e30 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44480/0 (socket says 172.21.15.23:44480) 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 -- 172.21.15.23:0/1486767760 learned_addr learned my addr 172.21.15.23:0/1486767760 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9400700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc0687f0 0x7f20fc1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20fa800700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 0x7f20fc1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 -- 172.21.15.23:0/1486767760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 msgr2=0x7f20fc1006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 0x7f20fc1006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 -- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc0687f0 msgr2=0x7f20fc1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.887+0000 7f20f9e00700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc0687f0 0x7f20fc1001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20f9e00700 1 -- 172.21.15.23:0/1486767760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e80095f0 con 0x7f20fc067e30 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20fa800700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 0x7f20fc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20f9e00700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9a0 tx=0x7f20ec00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20f2a00700 1 -- 172.21.15.23:0/1486767760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec009790 con 0x7f20fc067e30 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20f2a00700 1 -- 172.21.15.23:0/1486767760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20ec00cbd0 con 0x7f20fc067e30 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.888+0000 7f20f2a00700 1 -- 172.21.15.23:0/1486767760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec018680 con 0x7f20fc067e30 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.889+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc1a5c30 con 0x7f20fc067e30 2024-03-30T04:02:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.889+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc1a60c0 con 0x7f20fc067e30 2024-03-30T04:02:53.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.894+0000 7f20f2a00700 1 -- 172.21.15.23:0/1486767760 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f20ec00cd40 con 0x7f20fc067e30 2024-03-30T04:02:53.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.895+0000 7f2114b7db80 1 Processor -- start 2024-03-30T04:02:53.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.895+0000 7f2114b7db80 1 -- start start 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bb5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bc0e70 0x559805bc3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805b0ea10 con 0x559805bb5f70 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805b0c3d0 con 0x559805bc0e70 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f2114b7db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805b07cc0 con 0x559805bb5650 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f20f2a00700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 0x7f20e00a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.896+0000 7f20f2a00700 1 -- 172.21.15.23:0/1486767760 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f20ec014070 con 0x7f20fc067e30 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20fa800700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 0x7f20e00a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45810/0 (socket says 172.21.15.23:45810) 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 -- 172.21.15.23:0/2270426902 learned_addr learned my addr 172.21.15.23:0/2270426902 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9400700 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bc0e70 0x559805bc3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20fa800700 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bb5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 -- 172.21.15.23:0/2270426902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 msgr2=0x559805bb5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bb5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 -- 172.21.15.23:0/2270426902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bc0e70 msgr2=0x559805bc3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bc0e70 0x559805bc3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.897+0000 7f20f9e00700 1 -- 172.21.15.23:0/2270426902 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559805989430 con 0x559805bb5f70 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.898+0000 7f20f9e00700 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0a20b0 tx=0x7f20ec099d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6e4b1c465523e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.898+0000 7f20fa800700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 0x7f20e00a3970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009fd0 tx=0x7f20f4009050 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.898+0000 7f20e7400700 1 -- 172.21.15.23:0/2270426902 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0b1020 con 0x559805bb5f70 2024-03-30T04:02:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.898+0000 7f20e7400700 1 -- 172.21.15.23:0/2270426902 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20ec0cc040 con 0x559805bb5f70 2024-03-30T04:02:53.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.898+0000 7f20e7400700 1 -- 172.21.15.23:0/2270426902 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec09a160 con 0x559805bb5f70 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 -- 172.21.15.23:0/2270426902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 msgr2=0x559805bc0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0a20b0 tx=0x7f20ec099d20 comp rx=0 tx=0).stop 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 -- 172.21.15.23:0/2270426902 shutdown_connections 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bc0e70 0x559805bc3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bb5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 --2- 172.21.15.23:0/2270426902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bb5f70 0x559805bc0930 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 -- 172.21.15.23:0/2270426902 >> 172.21.15.23:0/2270426902 conn(0x559805b15720 msgr2=0x559805bb1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 -- 172.21.15.23:0/2270426902 shutdown_connections 2024-03-30T04:02:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.899+0000 7f2114b7db80 1 -- 172.21.15.23:0/2270426902 wait complete. 2024-03-30T04:02:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.900+0000 7f2114b7db80 1 -- start start 2024-03-30T04:02:53.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.900+0000 7f2114b7db80 1 Processor -- start 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bbdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bbe130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805aa3e00 con 0x559805bc0e70 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805aa3aa0 con 0x559805bb5f70 2024-03-30T04:02:53.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55980598b720 con 0x559805bb5650 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f2114b7db80 10 client.0 _ll_register_callbacks cb 0x559805aad4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.901+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44506/0 (socket says 172.21.15.23:44506) 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 -- 172.21.15.23:0/3160883977 learned_addr learned my addr 172.21.15.23:0/3160883977 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9400700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bbdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20fa800700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bbe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 -- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 msgr2=0x559805bbdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bbdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 -- 172.21.15.23:0/3160883977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 msgr2=0x559805bbe130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bbe130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 -- 172.21.15.23:0/3160883977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc1a60c0 con 0x559805bb5650 2024-03-30T04:02:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20fa800700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bbe130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20ec000c00 tx=0x7f20ec0965d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee3e77a0d54fa40 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.902+0000 7f20dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:02:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0b1040 con 0x559805bb5650 2024-03-30T04:02:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f20ec09a860 con 0x559805bb5650 2024-03-30T04:02:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559805ac8400 con 0x559805bb5650 2024-03-30T04:02:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20ec00f3c0 con 0x559805bb5650 2024-03-30T04:02:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f2114b7db80 10 client.25975 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:53.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559805935be0 con 0x559805bb5650 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.903+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559805ac8400 con 0x559805bb5650 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f20ec099650 con 0x559805bb5650 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 10 client.25975 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 10 client.25975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 20 client.25975 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805bd0660 0x559805bd2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 4 client.25975 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559805b00780 con 0x559805bd0660 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.904+0000 7f20f9400700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805bd0660 0x559805bd2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.905+0000 7f20f9400700 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805bd0660 0x559805bd2a50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74121067cb9fb55e server_cookie=919f6ea4adb689f7 in_seq=0 out_seq=0 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.905+0000 7f20dbe00700 10 client.25975 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.905+0000 7f20dbe00700 1 -- 172.21.15.23:0/3160883977 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559805b00780 con 0x559805bd0660 2024-03-30T04:02:53.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.905+0000 7f20dbe00700 10 client.25975 handle_command_reply: tid=0 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 client.25975 shutdown 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 msgr2=0x559805bc0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20ec000c00 tx=0x7f20ec0965d0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 shutdown_connections 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bbe130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805bd0660 0x559805bd2a50 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc0790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3160883977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bbdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 >> 172.21.15.23:0/3160883977 conn(0x559805b15720 msgr2=0x559805bb7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f20dbe00700 0 client.25975 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 shutdown_connections 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 1 -- 172.21.15.23:0/3160883977 wait complete. 2024-03-30T04:02:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.906+0000 7f2114b7db80 20 client.25975 trim_cache size 0 max 16384 2024-03-30T04:02:53.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.937+0000 7f2114b7db80 1 Processor -- start 2024-03-30T04:02:53.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.937+0000 7f2114b7db80 1 -- start start 2024-03-30T04:02:53.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5f70 0x559805bcc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bcc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805ac8400 con 0x559805bc0e70 2024-03-30T04:02:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805bb68d0 con 0x559805bb5650 2024-03-30T04:02:53.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f2114b7db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805bbe4e0 con 0x559805bb5f70 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52950/0 (socket says 172.21.15.23:52950) 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f20fa800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bcc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f20f9e00700 1 -- 172.21.15.23:0/3854602438 learned_addr learned my addr 172.21.15.23:0/3854602438 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.938+0000 7f20f9400700 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5f70 0x559805bcc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 -- 172.21.15.23:0/3854602438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5f70 msgr2=0x559805bcc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5f70 0x559805bcc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 -- 172.21.15.23:0/3854602438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 msgr2=0x559805bcc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bcc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 -- 172.21.15.23:0/3854602438 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559805bd75e0 con 0x559805bb5650 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20f9e00700 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e970 tx=0x7f20ec099b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cd1183153d4bf35 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.939+0000 7f20e7e00700 1 -- 172.21.15.23:0/3854602438 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec0b1070 con 0x559805bb5650 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f20e7e00700 1 -- 172.21.15.23:0/3854602438 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20ec0a21a0 con 0x559805bb5650 2024-03-30T04:02:53.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f20e7e00700 1 -- 172.21.15.23:0/3854602438 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20ec09a3c0 con 0x559805bb5650 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 -- 172.21.15.23:0/3854602438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 msgr2=0x559805bc7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e970 tx=0x7f20ec099b40 comp rx=0 tx=0).stop 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 -- 172.21.15.23:0/3854602438 shutdown_connections 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805bcc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5650 0x559805bc7b50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 --2- 172.21.15.23:0/3854602438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5f70 0x559805bcc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.940+0000 7f2114b7db80 1 -- 172.21.15.23:0/3854602438 >> 172.21.15.23:0/3854602438 conn(0x559805b15720 msgr2=0x559805bbc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.941+0000 7f2114b7db80 1 -- 172.21.15.23:0/3854602438 shutdown_connections 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.941+0000 7f2114b7db80 1 -- 172.21.15.23:0/3854602438 wait complete. 2024-03-30T04:02:53.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.941+0000 7f2114b7db80 1 -- start start 2024-03-30T04:02:53.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.941+0000 7f2114b7db80 1 Processor -- start 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bc6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805bc7310 con 0x559805bc0e70 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805c24910 con 0x559805bb5f70 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559805c24a50 con 0x559805bb5650 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f2114b7db80 10 client.0 _ll_register_callbacks cb 0x559805aad4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f20fa800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.942+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45848/0 (socket says 172.21.15.23:45848) 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44522/0 (socket says 172.21.15.23:44522) 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 -- 172.21.15.23:0/4194753668 learned_addr learned my addr 172.21.15.23:0/4194753668 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20f9400700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bc6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 -- 172.21.15.23:0/4194753668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 msgr2=0x559805bc64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 -- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 msgr2=0x559805bc6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bc6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 -- 172.21.15.23:0/4194753668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559805935be0 con 0x559805bc0e70 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20fa800700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f20f4018cb0 tx=0x7f20f4004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3e217df51a8da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:53.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.943+0000 7f20dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f400e050 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20ec0cc040 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f20f4017c50 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20f4017e10 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f2114b7db80 10 client.15987 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559805bab2c0 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.944+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20ec0cc040 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f20f402d440 con 0x559805bc0e70 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 10 client.15987 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:53.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 20 client.15987 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805c26240 0x559805c28660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 4 client.15987 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559805956d70 con 0x559805c26240 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.945+0000 7f20f9e00700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805c26240 0x559805c28660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.946+0000 7f20f9e00700 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805c26240 0x559805c28660 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=918c2e854e5623ee server_cookie=169e35fb7f5ba60d in_seq=0 out_seq=0 2024-03-30T04:02:53.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.946+0000 7f20dbe00700 10 client.15987 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f20dbe00700 1 -- 172.21.15.23:0/4194753668 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559805956d70 con 0x559805c26240 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f20dbe00700 10 client.15987 handle_command_reply: tid=0 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 client.15987 shutdown 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 msgr2=0x559805c243d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f20f4018cb0 tx=0x7f20f4004b40 comp rx=0 tx=0).stop 2024-03-30T04:02:53.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 shutdown_connections 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559805bc0e70 0x559805c243d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559805c26240 0x559805c28660 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559805bb5650 0x559805bc64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 --2- 172.21.15.23:0/4194753668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559805bb5f70 0x559805bc6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 >> 172.21.15.23:0/4194753668 conn(0x559805b15720 msgr2=0x559805bcc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.947+0000 7f20dbe00700 0 client.15987 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.948+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 shutdown_connections 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.948+0000 7f2114b7db80 1 -- 172.21.15.23:0/4194753668 wait complete. 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.948+0000 7f2114b7db80 20 client.15987 trim_cache size 0 max 16384 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:53.948 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "5bc6cb06-8c32-415a-a829-8da57fd7bee5", 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 msgr2=0x7f20e00a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 0x7f20e00a3970 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009fd0 tx=0x7f20f4009050 comp rx=0 tx=0).stop 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 msgr2=0x7f20fc0ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:53.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9a0 tx=0x7f20ec00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 shutdown_connections 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f20e00a14b0 0x7f20e00a3970 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20fc105ce0 0x7f20fc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20fc067e30 0x7f20fc0ffc70 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 --2- 172.21.15.23:0/1486767760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f20fc0687f0 0x7f20fc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 >> 172.21.15.23:0/1486767760 conn(0x7f20fc076d40 msgr2=0x7f20fc0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 shutdown_connections 2024-03-30T04:02:53.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:53.949+0000 7f2101000700 1 -- 172.21.15.23:0/1486767760 wait complete. 2024-03-30T04:02:53.968 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:02:54.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.218+0000 7fba14200700 1 -- 172.21.15.23:0/2729780476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c0698d0 msgr2=0x7fba0c10d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.218+0000 7fba14200700 1 --2- 172.21.15.23:0/2729780476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c0698d0 0x7fba0c10d210 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fb9f80098f0 tx=0x7fb9f8009c00 comp rx=0 tx=0).stop 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.219+0000 7fba14200700 1 -- 172.21.15.23:0/2729780476 shutdown_connections 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.219+0000 7fba14200700 1 --2- 172.21.15.23:0/2729780476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 0x7fba0c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.219+0000 7fba14200700 1 --2- 172.21.15.23:0/2729780476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c068fb0 0x7fba0c069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.219+0000 7fba14200700 1 --2- 172.21.15.23:0/2729780476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c0698d0 0x7fba0c10d210 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.219+0000 7fba14200700 1 -- 172.21.15.23:0/2729780476 >> 172.21.15.23:0/2729780476 conn(0x7fba0c076d80 msgr2=0x7fba0c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.220+0000 7fba14200700 1 -- 172.21.15.23:0/2729780476 shutdown_connections 2024-03-30T04:02:54.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.220+0000 7fba14200700 1 -- 172.21.15.23:0/2729780476 wait complete. 2024-03-30T04:02:54.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.221+0000 7fba14200700 1 Processor -- start 2024-03-30T04:02:54.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.221+0000 7fba14200700 1 -- start start 2024-03-30T04:02:54.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c0698d0 0x7fba0c10a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 0x7fba0c10a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10ad50 con 0x7fba0c068fb0 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c10aec0 con 0x7fba0c10d750 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba14200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c1a9da0 con 0x7fba0c0698d0 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba11800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba11800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45868/0 (socket says 172.21.15.23:45868) 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba11800700 1 -- 172.21.15.23:0/3341611763 learned_addr learned my addr 172.21.15.23:0/3341611763 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba10e00700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c0698d0 0x7fba0c10a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.222+0000 7fba12200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 0x7fba0c10a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 -- 172.21.15.23:0/3341611763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c0698d0 msgr2=0x7fba0c10a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c0698d0 0x7fba0c10a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 -- 172.21.15.23:0/3341611763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 msgr2=0x7fba0c10a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 0x7fba0c10a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 -- 172.21.15.23:0/3341611763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba000095f0 con 0x7fba0c068fb0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba11800700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fba0000ec40 tx=0x7fba0000ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba0a000700 1 -- 172.21.15.23:0/3341611763 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00009a10 con 0x7fba0c068fb0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba0a000700 1 -- 172.21.15.23:0/3341611763 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba000044e0 con 0x7fba0c068fb0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba0a000700 1 -- 172.21.15.23:0/3341611763 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00010430 con 0x7fba0c068fb0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.223+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f8009520 con 0x7fba0c068fb0 2024-03-30T04:02:54.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.224+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c1aa2a0 con 0x7fba0c068fb0 2024-03-30T04:02:54.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.227+0000 7fba0a000700 1 -- 172.21.15.23:0/3341611763 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fba00003f40 con 0x7fba0c068fb0 2024-03-30T04:02:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.229+0000 7fba0a000700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 0x7fb9f40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.229+0000 7fba0a000700 1 -- 172.21.15.23:0/3341611763 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fba00014070 con 0x7fba0c068fb0 2024-03-30T04:02:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.229+0000 7fba27dd9b80 1 Processor -- start 2024-03-30T04:02:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.229+0000 7fba27dd9b80 1 -- start start 2024-03-30T04:02:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba10e00700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 0x7fb9f40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b8575d0 0x559d9b8579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 0x559d9b8628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b8651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b7b0990 con 0x559d9b857ef0 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b7ae350 con 0x559d9b8575d0 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b7a9c40 con 0x559d9b862df0 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba10e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 0x559d9b8628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba10e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 0x559d9b8628b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45876/0 (socket says 172.21.15.23:45876) 2024-03-30T04:02:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba10e00700 1 -- 172.21.15.23:0/3818695381 learned_addr learned my addr 172.21.15.23:0/3818695381 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:54.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.230+0000 7fba11800700 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b8575d0 0x559d9b8579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b8651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 -- 172.21.15.23:0/3818695381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b8575d0 msgr2=0x559d9b8579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b8575d0 0x559d9b8579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 -- 172.21.15.23:0/3818695381 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 msgr2=0x559d9b8628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 0x559d9b8628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 -- 172.21.15.23:0/3818695381 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d9b62b3b0 con 0x559d9b862df0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba10e00700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 0x7fb9f40a3870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c010 tx=0x7fb9f800fed0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.231+0000 7fba12200700 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b8651e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fba0c067940 tx=0x7fba04009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f80fd335cf2a5e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fb9fea00700 1 -- 172.21.15.23:0/3818695381 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04019070 con 0x559d9b862df0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fb9fea00700 1 -- 172.21.15.23:0/3818695381 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0400f040 con 0x559d9b862df0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fb9fea00700 1 -- 172.21.15.23:0/3818695381 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007810 con 0x559d9b862df0 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3818695381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 msgr2=0x559d9b8651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b8651e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fba0c067940 tx=0x7fba04009ba0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3818695381 shutdown_connections 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b8651e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b8575d0 0x559d9b8579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.232+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3818695381 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b857ef0 0x559d9b8628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.233+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3818695381 >> 172.21.15.23:0/3818695381 conn(0x559d9b7b76a0 msgr2=0x559d9b853a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.233+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3818695381 shutdown_connections 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.233+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3818695381 wait complete. 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.233+0000 7fba27dd9b80 1 -- start start 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.234+0000 7fba27dd9b80 1 Processor -- start 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.234+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8626d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.234+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b85ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b745d80 con 0x559d9b862df0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b745a20 con 0x559d9b857ef0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b62d6a0 con 0x559d9b8575d0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba27dd9b80 10 client.0 _ll_register_callbacks cb 0x559d9b74f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba10e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba11800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba10e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:53012/0 (socket says 172.21.15.23:53012) 2024-03-30T04:02:54.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba10e00700 1 -- 172.21.15.23:0/2267054012 learned_addr learned my addr 172.21.15.23:0/2267054012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.235+0000 7fba12200700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b85ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 -- 172.21.15.23:0/2267054012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 msgr2=0x559d9b8626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8626d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 -- 172.21.15.23:0/2267054012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 msgr2=0x559d9b85ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b85ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 -- 172.21.15.23:0/2267054012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba04003ae0 con 0x559d9b857ef0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba11800700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba10e00700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005e90 tx=0x7fb9f8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f710fed71d557eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fb9f3400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f80054a0 con 0x559d9b857ef0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d9b76a380 con 0x559d9b857ef0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.236+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb9f8005600 con 0x559d9b857ef0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9f800b560 con 0x559d9b857ef0 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fba27dd9b80 10 client.26000 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:54.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d9b5d7b60 con 0x559d9b857ef0 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559d9b76a380 con 0x559d9b857ef0 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb9f8003d00 con 0x559d9b857ef0 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fba27dd9b80 10 client.26000 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.237+0000 7fba27dd9b80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.238+0000 7fba27dd9b80 20 client.26000 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:54.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.238+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b883760 0x559d9b885b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.238+0000 7fba27dd9b80 4 client.26000 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:02:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.238+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d9b7a2700 con 0x559d9b883760 2024-03-30T04:02:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.238+0000 7fba11800700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b883760 0x559d9b885b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.239+0000 7fb9f3400700 10 client.26000 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:54.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.239+0000 7fba11800700 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b883760 0x559d9b885b80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8df206956156c4dc server_cookie=209c2fd4599ff407 in_seq=0 out_seq=0 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fb9f3400700 1 -- 172.21.15.23:0/2267054012 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559d9b7a2700 con 0x559d9b883760 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fb9f3400700 10 client.26000 handle_command_reply: tid=0 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 client.26000 shutdown 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 msgr2=0x559d9b85f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8005e90 tx=0x7fb9f8009c00 comp rx=0 tx=0).stop 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 shutdown_connections 2024-03-30T04:02:54.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b85ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b883760 0x559d9b885b80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/2267054012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b85f9c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 >> 172.21.15.23:0/2267054012 conn(0x559d9b7b76a0 msgr2=0x559d9b859690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fb9f3400700 0 client.26000 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 shutdown_connections 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.240+0000 7fba27dd9b80 1 -- 172.21.15.23:0/2267054012 wait complete. 2024-03-30T04:02:54.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.241+0000 7fba27dd9b80 20 client.26000 trim_cache size 0 max 16384 2024-03-30T04:02:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.258+0000 7fba27dd9b80 1 Processor -- start 2024-03-30T04:02:54.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 -- start start 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b8575d0 0x559d9b885b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b869aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b85a830 con 0x559d9b8575d0 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b85a690 con 0x559d9b857ef0 2024-03-30T04:02:54.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b855300 con 0x559d9b862df0 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba12200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.259+0000 7fba12200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44554/0 (socket says 172.21.15.23:44554) 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 -- 172.21.15.23:0/3390175325 learned_addr learned my addr 172.21.15.23:0/3390175325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba10e00700 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b869aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 -- 172.21.15.23:0/3390175325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 msgr2=0x559d9b869aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b869aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 -- 172.21.15.23:0/3390175325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b8575d0 msgr2=0x559d9b885b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b8575d0 0x559d9b885b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 -- 172.21.15.23:0/3390175325 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d9b76fea0 con 0x559d9b862df0 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.260+0000 7fba12200700 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba04004e80 tx=0x7fba04003db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be8709edad20796c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.261+0000 7fb9ff400700 1 -- 172.21.15.23:0/3390175325 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04019040 con 0x559d9b862df0 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.261+0000 7fb9ff400700 1 -- 172.21.15.23:0/3390175325 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba04024070 con 0x559d9b862df0 2024-03-30T04:02:54.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.261+0000 7fb9ff400700 1 -- 172.21.15.23:0/3390175325 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04004af0 con 0x559d9b862df0 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3390175325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 msgr2=0x559d9b862370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fba04004e80 tx=0x7fba04003db0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3390175325 shutdown_connections 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b862df0 0x559d9b862370 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b8575d0 0x559d9b885b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3390175325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b869aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3390175325 >> 172.21.15.23:0/3390175325 conn(0x559d9b7b76a0 msgr2=0x559d9b84c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3390175325 shutdown_connections 2024-03-30T04:02:54.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.262+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3390175325 wait complete. 2024-03-30T04:02:54.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.263+0000 7fba27dd9b80 1 -- start start 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.263+0000 7fba27dd9b80 1 Processor -- start 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b867eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b868930 con 0x559d9b862df0 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b868aa0 con 0x559d9b857ef0 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d9b868c60 con 0x559d9b8575d0 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba27dd9b80 10 client.0 _ll_register_callbacks cb 0x559d9b74f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:02:54.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba12200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba12200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45902/0 (socket says 172.21.15.23:45902) 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba12200700 1 -- 172.21.15.23:0/3172822304 learned_addr learned my addr 172.21.15.23:0/3172822304 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba11800700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.264+0000 7fba10e00700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b867eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 -- 172.21.15.23:0/3172822304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 msgr2=0x559d9b8697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 -- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 msgr2=0x559d9b867eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b867eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 -- 172.21.15.23:0/3172822304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d9b62b3b0 con 0x559d9b862df0 2024-03-30T04:02:54.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba10e00700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b867eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:02:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fba12200700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba04009ef0 tx=0x7fba04004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c27073f5bff5e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:02:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fb9f3400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:02:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.265+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04007a10 con 0x559d9b862df0 2024-03-30T04:02:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fba04007b70 con 0x559d9b862df0 2024-03-30T04:02:54.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba0400f040 con 0x559d9b862df0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba04004490 con 0x559d9b862df0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fba27dd9b80 10 client.16002 fetch_fsmap learned FSMap version 28 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d9b62fcd0 con 0x559d9b862df0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0400f040 con 0x559d9b862df0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.266+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fba04032ca0 con 0x559d9b862df0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 10 client.16002 fetch_fsmap finished waiting for FSMap version 28 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 20 client.16002 populate_metadata read hostname 'smithi023' 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b8d85c0 0x559d9b8da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 4 client.16002 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559d9b5f8cf0 con 0x559d9b8d85c0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.267+0000 7fba11800700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b8d85c0 0x559d9b8da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba11800700 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b8d85c0 0x559d9b8da9b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15dfcef7d99eec9 server_cookie=be6e96bca31d1a29 in_seq=0 out_seq=0 2024-03-30T04:02:54.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fb9f3400700 10 client.16002 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fb9f3400700 1 -- 172.21.15.23:0/3172822304 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559d9b5f8cf0 con 0x559d9b8d85c0 2024-03-30T04:02:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fb9f3400700 10 client.16002 handle_command_reply: tid=0 2024-03-30T04:02:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 client.16002 shutdown 2024-03-30T04:02:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 msgr2=0x559d9b8683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fba04009ef0 tx=0x7fba04004ba0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 shutdown_connections 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d9b862df0 0x559d9b8683f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559d9b8d85c0 0x559d9b8da9b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d9b8575d0 0x559d9b8697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 --2- 172.21.15.23:0/3172822304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d9b857ef0 0x559d9b867eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 >> 172.21.15.23:0/3172822304 conn(0x559d9b7b76a0 msgr2=0x559d9b862370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fb9f3400700 0 client.16002 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 shutdown_connections 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.268+0000 7fba27dd9b80 1 -- 172.21.15.23:0/3172822304 wait complete. 2024-03-30T04:02:54.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba27dd9b80 20 client.16002 trim_cache size 0 max 16384 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "5bc6cb06-8c32-415a-a829-8da57fd7bee5": { 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "5bc6cb06-8c32-415a-a829-8da57fd7bee5", 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:02:54.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 msgr2=0x7fb9f40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 0x7fb9f40a3870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9f800c010 tx=0x7fb9f800fed0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 msgr2=0x7fba0c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fba0000ec40 tx=0x7fba0000ef50 comp rx=0 tx=0).stop 2024-03-30T04:02:54.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 shutdown_connections 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.269+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb9f40a13b0 0x7fb9f40a3870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba0c10d750 0x7fba0c10a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba0c068fb0 0x7fba0c10cfc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 --2- 172.21.15.23:0/3341611763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba0c0698d0 0x7fba0c10a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 >> 172.21.15.23:0/3341611763 conn(0x7fba0c076d80 msgr2=0x7fba0c10eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 shutdown_connections 2024-03-30T04:02:54.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:02:54.270+0000 7fba14200700 1 -- 172.21.15.23:0/3341611763 wait complete. 2024-03-30T04:02:54.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:54 smithi023 ceph-mon[38254]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:02:54.296 INFO:tasks.cephfs.filesystem:scrub status for tag:5bc6cb06-8c32-415a-a829-8da57fd7bee5 - {'path': '/', 'tag': '5bc6cb06-8c32-415a-a829-8da57fd7bee5', 'options': 'recursive,force'} 2024-03-30T04:02:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:54 smithi154 ceph-mon[79161]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:02:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:54 smithi193 ceph-mon[79581]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-30T04:02:55.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:55 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:02:55.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:55.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:02:55.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:55.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:55 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:02:55.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:55.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:02:55.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:55.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:55 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:02:55.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:02:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:02:57.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:56 smithi023 ceph-mon[38254]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:56 smithi154 ceph-mon[79161]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:56 smithi193 ceph-mon[79581]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:02:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:58 smithi154 ceph-mon[79161]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:58 smithi193 ceph-mon[79581]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:58 smithi023 ceph-mon[38254]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-30T04:02:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:02:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:02:59.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:02:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:02:59.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:02:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:03:00.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:00 smithi193 ceph-mon[79581]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:03:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:00 smithi023 ceph-mon[38254]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:03:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:00 smithi154 ceph-mon[79161]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-30T04:03:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:02 smithi154 ceph-mon[79161]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:02 smithi193 ceph-mon[79581]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:02.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:02 smithi023 ceph-mon[38254]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:03 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:03:03.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:03 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:03:03.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:03 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:03:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:03:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:03:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:03:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:04 smithi154 ceph-mon[79161]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-30T04:03:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:03:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:03:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:03:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:04 smithi193 ceph-mon[79581]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-30T04:03:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:03:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:03:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:03:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:04 smithi023 ceph-mon[38254]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 24 op/s 2024-03-30T04:03:07.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:06 smithi023 ceph-mon[38254]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:06 smithi154 ceph-mon[79161]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:07.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:06 smithi193 ceph-mon[79581]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:08 smithi154 ceph-mon[79161]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:08 smithi193 ceph-mon[79581]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:08 smithi023 ceph-mon[38254]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:10 smithi154 ceph-mon[79161]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:10 smithi193 ceph-mon[79581]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:10 smithi023 ceph-mon[38254]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-30T04:03:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:12 smithi154 ceph-mon[79161]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:12 smithi193 ceph-mon[79581]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:12 smithi023 ceph-mon[38254]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-30T04:03:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:14 smithi154 ceph-mon[79161]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:14 smithi193 ceph-mon[79581]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:14 smithi023 ceph-mon[38254]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:17 smithi023 ceph-mon[38254]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:17 smithi154 ceph-mon[79161]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:17 smithi193 ceph-mon[79581]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-30T04:03:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:18 smithi154 ceph-mon[79161]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-30T04:03:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:18 smithi193 ceph-mon[79581]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-30T04:03:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:18 smithi023 ceph-mon[38254]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-30T04:03:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:20 smithi154 ceph-mon[79161]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:03:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:20 smithi193 ceph-mon[79581]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:03:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:20 smithi023 ceph-mon[38254]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-30T04:03:21.183 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 61% (7536/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to '470100a4ccbcf7769052bf09a5413881ccf1b97f'. 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.004 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-30T04:03:22.005 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at 470100a4ccb DNM: trigger builds again 2024-03-30T04:03:22.019 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-30T04:03:22.073 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-30T04:03:22.078 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T04:03:22.078 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-30T04:03:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:22 smithi154 ceph-mon[79161]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:03:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:22 smithi193 ceph-mon[79581]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:03:22.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:22 smithi023 ceph-mon[38254]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-30T04:03:22.672 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-30T04:03:22.784 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-30T04:03:22.849 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-30T04:03:22.853 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T04:03:22.853 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-30T04:03:22.916 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-30T04:03:22.923 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T04:03:22.924 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-30T04:03:22.965 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-30T04:03:22.966 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-30T04:03:22.967 DEBUG:teuthology.orchestra.run.smithi023:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-30T04:03:23.074 INFO:tasks.workunit.client.0.smithi023.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-30T04:03:23.074 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-30T04:03:23.074 INFO:tasks.workunit.client.0.smithi023.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-30T04:03:23.085 INFO:tasks.workunit.client.0.smithi023.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-30T04:03:23.185 INFO:tasks.workunit.client.0.smithi023.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-30T04:03:23.186 INFO:tasks.workunit.client.0.smithi023.stdout:releasing clients 2024-03-30T04:03:24.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 365 13.04 MB/sec warmup 1 sec latency 810.360 ms 2024-03-30T04:03:24.297 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:03:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 -- 172.21.15.23:0/1574562546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 msgr2=0x7f4e3c101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 --2- 172.21.15.23:0/1574562546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c101330 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009990 tx=0x7f4e2c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 -- 172.21.15.23:0/1574562546 shutdown_connections 2024-03-30T04:03:24.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 --2- 172.21.15.23:0/1574562546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 0x7f4e3c10dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 --2- 172.21.15.23:0/1574562546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c101330 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 --2- 172.21.15.23:0/1574562546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c101900 0x7f4e3c105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 -- 172.21.15.23:0/1574562546 >> 172.21.15.23:0/1574562546 conn(0x7f4e3c0fc7f0 msgr2=0x7f4e3c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 -- 172.21.15.23:0/1574562546 shutdown_connections 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.518+0000 7f4e41600700 1 -- 172.21.15.23:0/1574562546 wait complete. 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.519+0000 7f4e41600700 1 Processor -- start 2024-03-30T04:03:24.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.519+0000 7f4e41600700 1 -- start start 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 0x7f4e3c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c073b30 0x7f4e3c1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c1a5a90 con 0x7f4e3c100f50 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c1a5c00 con 0x7f4e3c073b30 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e41600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3c1a5dc0 con 0x7f4e3c106070 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39366/0 (socket says 172.21.15.23:39366) 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3086834330 learned_addr learned my addr 172.21.15.23:0/3086834330 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3a000700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 0x7f4e3c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3b400700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c073b30 0x7f4e3c1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3086834330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 msgr2=0x7f4e3c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 0x7f4e3c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3086834330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c073b30 msgr2=0x7f4e3c1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c073b30 0x7f4e3c1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.520+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3086834330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0095f0 con 0x7f4e3c100f50 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c000c00 tx=0x7f4e2c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e33400700 1 -- 172.21.15.23:0/3086834330 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c01d070 con 0x7f4e3c100f50 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e33400700 1 -- 172.21.15.23:0/3086834330 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e2c022470 con 0x7f4e3c100f50 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e33400700 1 -- 172.21.15.23:0/3086834330 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c017650 con 0x7f4e3c100f50 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3c1a6040 con 0x7f4e3c100f50 2024-03-30T04:03:24.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.521+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3c1a6410 con 0x7f4e3c100f50 2024-03-30T04:03:24.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.524+0000 7f4e33400700 1 -- 172.21.15.23:0/3086834330 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f4e2c0225e0 con 0x7f4e3c100f50 2024-03-30T04:03:24.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.526+0000 7f4e55151b80 1 Processor -- start 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.526+0000 7f4e55151b80 1 -- start start 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e33400700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 0x7f4e200a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e33400700 1 -- 172.21.15.23:0/3086834330 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f4e2c0ce590 con 0x7f4e3c100f50 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3b400700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 0x7f4e200a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea1bb990 con 0x560cea26ddf0 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea1b9350 con 0x560cea2625d0 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e55151b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea1b4c40 con 0x560cea262ef0 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39372/0 (socket says 172.21.15.23:39372) 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3b400700 1 -- 172.21.15.23:0/4112872840 learned_addr learned my addr 172.21.15.23:0/4112872840 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.527+0000 7f4e3a000700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 -- 172.21.15.23:0/4112872840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 msgr2=0x560cea26d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 -- 172.21.15.23:0/4112872840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 msgr2=0x560cea2701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 -- 172.21.15.23:0/4112872840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cea0363b0 con 0x560cea2625d0 2024-03-30T04:03:24.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3b400700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3a000700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3b400700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 0x7f4e200a3870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009510 tx=0x7f4e34009300 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2629b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0cb7a0 tx=0x7f4e2c09aef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbd76fc6ca8f0d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e27400700 1 -- 172.21.15.23:0/4112872840 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c09d020 con 0x560cea2625d0 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e27400700 1 -- 172.21.15.23:0/4112872840 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e2c09b820 con 0x560cea2625d0 2024-03-30T04:03:24.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.528+0000 7f4e27400700 1 -- 172.21.15.23:0/4112872840 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c0a1070 con 0x560cea2625d0 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 -- 172.21.15.23:0/4112872840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 msgr2=0x560cea2629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2629b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0cb7a0 tx=0x7f4e2c09aef0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 -- 172.21.15.23:0/4112872840 shutdown_connections 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea2701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2629b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 --2- 172.21.15.23:0/4112872840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 -- 172.21.15.23:0/4112872840 >> 172.21.15.23:0/4112872840 conn(0x560cea1c26a0 msgr2=0x560cea25ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 -- 172.21.15.23:0/4112872840 shutdown_connections 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.529+0000 7f4e55151b80 1 -- 172.21.15.23:0/4112872840 wait complete. 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.530+0000 7f4e55151b80 1 -- start start 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.530+0000 7f4e55151b80 1 Processor -- start 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea26d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea150d80 con 0x560cea26ddf0 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea150a20 con 0x560cea2625d0 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea0386a0 con 0x560cea262ef0 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e55151b80 10 client.0 _ll_register_callbacks cb 0x560cea15a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:24.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e3b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39382/0 (socket says 172.21.15.23:39382) 2024-03-30T04:03:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e3b400700 1 -- 172.21.15.23:0/2466568327 learned_addr learned my addr 172.21.15.23:0/2466568327 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.531+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea26d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 -- 172.21.15.23:0/2466568327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 msgr2=0x560cea26aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 -- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 msgr2=0x560cea26d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea26d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 -- 172.21.15.23:0/2466568327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e34011040 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e3b400700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e34019f70 tx=0x7f4e3401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36971c421b24ac5f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e1be00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34006000 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cea175380 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.532+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f4e34021070 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e3401be80 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 10 client.16014 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f4e3c1a6410 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560cea175380 con 0x560cea26ddf0 2024-03-30T04:03:24.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f4e34025380 con 0x560cea26ddf0 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 20 client.16014 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea27d630 0x560cea27fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 4 client.16014 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.533+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cea1ad700 con 0x560cea27d630 2024-03-30T04:03:24.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.536+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea27d630 0x560cea27fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.539+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea27d630 0x560cea27fa50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=290c8689c677eda6 server_cookie=2ce2a9ec7679d534 in_seq=0 out_seq=0 2024-03-30T04:03:24.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.539+0000 7f4e1be00700 10 client.16014 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.540+0000 7f4e1be00700 1 -- 172.21.15.23:0/2466568327 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560cea1ad700 con 0x560cea27d630 2024-03-30T04:03:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.540+0000 7f4e1be00700 10 client.16014 handle_command_reply: tid=0 2024-03-30T04:03:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.540+0000 7f4e55151b80 1 client.16014 shutdown 2024-03-30T04:03:24.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.540+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 msgr2=0x560cea26b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.540+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e34019f70 tx=0x7f4e3401c000 comp rx=0 tx=0).stop 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 shutdown_connections 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea26ddf0 0x560cea26b000 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea26d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea27d630 0x560cea27fa50 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 --2- 172.21.15.23:0/2466568327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea262ef0 0x560cea26aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e1be00700 0 client.16014 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 >> 172.21.15.23:0/2466568327 conn(0x560cea1c26a0 msgr2=0x560cea264530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 shutdown_connections 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 1 -- 172.21.15.23:0/2466568327 wait complete. 2024-03-30T04:03:24.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.541+0000 7f4e55151b80 20 client.16014 trim_cache size 0 max 16384 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.558+0000 7f4e55151b80 1 Processor -- start 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.558+0000 7f4e55151b80 1 -- start start 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2747e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea25ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea263850 con 0x560cea262ef0 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea27fde0 con 0x560cea2625d0 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e55151b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea26b600 con 0x560cea26ddf0 2024-03-30T04:03:24.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39384/0 (socket says 172.21.15.23:39384) 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 -- 172.21.15.23:0/1847747688 learned_addr learned my addr 172.21.15.23:0/1847747688 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 -- 172.21.15.23:0/1847747688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 msgr2=0x560cea25ea70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea25ea70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 -- 172.21.15.23:0/1847747688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 msgr2=0x560cea2747e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2747e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.559+0000 7f4e3a000700 1 -- 172.21.15.23:0/1847747688 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ce9fe2b60 con 0x560cea262ef0 2024-03-30T04:03:24.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e3a000700 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e28007dc0 tx=0x7f4e28011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9eec142ebecf8df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e27e00700 1 -- 172.21.15.23:0/1847747688 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28016070 con 0x560cea262ef0 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e27e00700 1 -- 172.21.15.23:0/1847747688 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e280044f0 con 0x560cea262ef0 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e27e00700 1 -- 172.21.15.23:0/1847747688 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e28003bf0 con 0x560cea262ef0 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 -- 172.21.15.23:0/1847747688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 msgr2=0x560cea25e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f4e28007dc0 tx=0x7f4e28011720 comp rx=0 tx=0).stop 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 -- 172.21.15.23:0/1847747688 shutdown_connections 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea25ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea2625d0 0x560cea2747e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 --2- 172.21.15.23:0/1847747688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea262ef0 0x560cea25e530 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 -- 172.21.15.23:0/1847747688 >> 172.21.15.23:0/1847747688 conn(0x560cea1c26a0 msgr2=0x560cea269390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.560+0000 7f4e55151b80 1 -- 172.21.15.23:0/1847747688 shutdown_connections 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 -- 172.21.15.23:0/1847747688 wait complete. 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 -- start start 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 Processor -- start 2024-03-30T04:03:24.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 0x560cea273320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea273860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.561+0000 7f4e55151b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea274bb0 con 0x560cea2625d0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e55151b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea2d1490 con 0x560cea262ef0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e55151b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cea2d1650 con 0x560cea26ddf0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e55151b80 10 client.0 _ll_register_callbacks cb 0x560cea15a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39392/0 (socket says 172.21.15.23:39392) 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3945207664 learned_addr learned my addr 172.21.15.23:0/3945207664 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3a000700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 0x560cea273320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3b400700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea273860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3945207664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 msgr2=0x560cea273860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea273860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 msgr2=0x560cea273320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 0x560cea273320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 -- 172.21.15.23:0/3945207664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e28007a30 con 0x560cea2625d0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3a000700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 0x560cea273320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e3aa00700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00f740 tx=0x7f4e2c022cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d296fd5ba34e7773 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.562+0000 7f4e1be00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c09d040 con 0x560cea2625d0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f4e2c09b120 con 0x560cea2625d0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e2c09b820 con 0x560cea2625d0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e2c0a1040 con 0x560cea2625d0 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 10 client.16020 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:24.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560cea17aea0 con 0x560cea2625d0 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e2c09b820 con 0x560cea2625d0 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f4e2c07d070 con 0x560cea2625d0 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 20 client.16020 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea2d2a70 0x560cea2d4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 4 client.16020 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:03:24.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.563+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560cea003cf0 con 0x560cea2d2a70 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.564+0000 7f4e3a000700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea2d2a70 0x560cea2d4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.567+0000 7f4e3a000700 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea2d2a70 0x560cea2d4e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cebc240808f94a6 server_cookie=fe846f075757c2 in_seq=0 out_seq=0 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.567+0000 7f4e1be00700 10 client.16020 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e1be00700 1 -- 172.21.15.23:0/3945207664 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560cea003cf0 con 0x560cea2d2a70 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e1be00700 10 client.16020 handle_command_reply: tid=0 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 client.16020 shutdown 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 msgr2=0x560cea272de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00f740 tx=0x7f4e2c022cb0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 shutdown_connections 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560cea2d2a70 0x560cea2d4e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cea26ddf0 0x560cea273860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560cea2625d0 0x560cea272de0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 --2- 172.21.15.23:0/3945207664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560cea262ef0 0x560cea273320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e1be00700 0 client.16020 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 >> 172.21.15.23:0/3945207664 conn(0x560cea1c26a0 msgr2=0x560cea269300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 shutdown_connections 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.568+0000 7f4e55151b80 1 -- 172.21.15.23:0/3945207664 wait complete. 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e55151b80 20 client.16020 trim_cache size 0 max 16384 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:03:24.570 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:03:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 msgr2=0x7f4e200a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 0x7f4e200a3870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009510 tx=0x7f4e34009300 comp rx=0 tx=0).stop 2024-03-30T04:03:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 msgr2=0x7f4e3c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c000c00 tx=0x7f4e2c005740 comp rx=0 tx=0).stop 2024-03-30T04:03:24.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 shutdown_connections 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e200a13b0 0x7f4e200a3870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e3c073b30 0x7f4e3c1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e3c100f50 0x7f4e3c0730b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 --2- 172.21.15.23:0/3086834330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e3c106070 0x7f4e3c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 >> 172.21.15.23:0/3086834330 conn(0x7f4e3c0fc7f0 msgr2=0x7f4e3c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 shutdown_connections 2024-03-30T04:03:24.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.569+0000 7f4e41600700 1 -- 172.21.15.23:0/3086834330 wait complete. 2024-03-30T04:03:24.587 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:03:24.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:03:24.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:24 smithi023 ceph-mon[38254]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:03:24.645 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:24 smithi023 ceph-mon[38254]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:24 smithi154 ceph-mon[79161]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:03:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:24 smithi154 ceph-mon[79161]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:24.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:24 smithi193 ceph-mon[79581]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:03:24.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:24 smithi193 ceph-mon[79581]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 -- 172.21.15.23:0/3322722947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 msgr2=0x7f1a5010db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3322722947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a5010db00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800b0a0 tx=0x7f1a4800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 -- 172.21.15.23:0/3322722947 shutdown_connections 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3322722947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a5010db00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3322722947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a50100e60 0x7f1a50101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3322722947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50101810 0x7f1a50105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 -- 172.21.15.23:0/3322722947 >> 172.21.15.23:0/3322722947 conn(0x7f1a500fc760 msgr2=0x7f1a500feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 -- 172.21.15.23:0/3322722947 shutdown_connections 2024-03-30T04:03:24.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.787+0000 7f1a55c00700 1 -- 172.21.15.23:0/3322722947 wait complete. 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 Processor -- start 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 -- start start 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a50199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 0x7f1a5019e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5019e8e0 con 0x7f1a50105f80 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5019ea50 con 0x7f1a50100e60 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.788+0000 7f1a55c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5019ec10 con 0x7f1a5019de60 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a50199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:32954/0 (socket says 172.21.15.23:32954) 2024-03-30T04:03:24.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 -- 172.21.15.23:0/3577598986 learned_addr learned my addr 172.21.15.23:0/3577598986 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4f400700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 0x7f1a5019e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 -- 172.21.15.23:0/3577598986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 msgr2=0x7f1a5019e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 0x7f1a5019e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 -- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 msgr2=0x7f1a50199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a50199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 -- 172.21.15.23:0/3577598986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48009d00 con 0x7f1a50100e60 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4f400700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 0x7f1a5019e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4e000700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a50199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:24.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.789+0000 7f1a4ea00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000b530 tx=0x7f1a4000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.790+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40011830 con 0x7f1a50100e60 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.790+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a501a6170 con 0x7f1a50100e60 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.790+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a40011e50 con 0x7f1a50100e60 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.790+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000f2e0 con 0x7f1a50100e60 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.790+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a501a66a0 con 0x7f1a50100e60 2024-03-30T04:03:24.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.791+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a50102330 con 0x7f1a50100e60 2024-03-30T04:03:24.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.793+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f1a40010380 con 0x7f1a50100e60 2024-03-30T04:03:24.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.794+0000 7f1a47400700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 0x7f1a340a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:24.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.795+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1a400cd230 con 0x7f1a50100e60 2024-03-30T04:03:24.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.795+0000 7f1a4e000700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 0x7f1a340a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:24.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.795+0000 7f1a4e000700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 0x7f1a340a3870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a5019aad0 tx=0x7f1a3c009190 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:24.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:24.797+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1a40096250 con 0x7f1a50100e60 2024-03-30T04:03:25.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.059+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5019a8b0 con 0x7f1a50100e60 2024-03-30T04:03:25.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.061+0000 7f1a47400700 1 -- 172.21.15.23:0/3577598986 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+10925 (secure 0 0 0) 0x7f1a40099870 con 0x7f1a50100e60 2024-03-30T04:03:25.060 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:03:25.060 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:03:23.374313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 msgr2=0x7f1a340a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 0x7f1a340a3870 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a5019aad0 tx=0x7f1a3c009190 comp rx=0 tx=0).stop 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 msgr2=0x7f1a50199580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000b530 tx=0x7f1a4000b840 comp rx=0 tx=0).stop 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 shutdown_connections 2024-03-30T04:03:25.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a5019de60 0x7f1a5019e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1a340a13b0 0x7f1a340a3870 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1a50100e60 0x7f1a50199580 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 --2- 172.21.15.23:0/3577598986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a50105f80 0x7f1a50199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.062+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 >> 172.21.15.23:0/3577598986 conn(0x7f1a500fc760 msgr2=0x7f1a500fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.063+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 shutdown_connections 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.063+0000 7f1a55c00700 1 -- 172.21.15.23:0/3577598986 wait complete. 2024-03-30T04:03:25.063 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-30T04:03:25.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:03:25.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 365 6.52 MB/sec warmup 2 sec latency 1810.457 ms 2024-03-30T04:03:25.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 -- 172.21.15.23:0/2175835708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 msgr2=0x7f9510101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 --2- 172.21.15.23:0/2175835708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510101340 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9500009990 tx=0x7f9500009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 -- 172.21.15.23:0/2175835708 shutdown_connections 2024-03-30T04:03:25.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 --2- 172.21.15.23:0/2175835708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9510106080 0x7f951010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 --2- 172.21.15.23:0/2175835708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510101340 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 --2- 172.21.15.23:0/2175835708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510101910 0x7f9510105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 -- 172.21.15.23:0/2175835708 >> 172.21.15.23:0/2175835708 conn(0x7f95100fc7c0 msgr2=0x7f95100febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 -- 172.21.15.23:0/2175835708 shutdown_connections 2024-03-30T04:03:25.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.284+0000 7f9516e00700 1 -- 172.21.15.23:0/2175835708 wait complete. 2024-03-30T04:03:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 Processor -- start 2024-03-30T04:03:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 -- start start 2024-03-30T04:03:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510106080 0x7f9510197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f951019bbd0 0x7f951019c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019c650 con 0x7f9510100f60 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019c7c0 con 0x7f9510106080 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.286+0000 7f9516e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951019c980 con 0x7f951019bbd0 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39428/0 (socket says 172.21.15.23:39428) 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 -- 172.21.15.23:0/1450813314 learned_addr learned my addr 172.21.15.23:0/1450813314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950f400700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510106080 0x7f9510197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f9514e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f951019bbd0 0x7f951019c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 -- 172.21.15.23:0/1450813314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f951019bbd0 msgr2=0x7f951019c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f951019bbd0 0x7f951019c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 -- 172.21.15.23:0/1450813314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510106080 msgr2=0x7f9510197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510106080 0x7f9510197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 -- 172.21.15.23:0/1450813314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000095f0 con 0x7f9510100f60 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.287+0000 7f950fe00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9500005910 tx=0x7f9500009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.288+0000 7f950cc00700 1 -- 172.21.15.23:0/1450813314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950001d070 con 0x7f9510100f60 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.288+0000 7f950cc00700 1 -- 172.21.15.23:0/1450813314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f950000fcb0 con 0x7f9510100f60 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.288+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95101a3e80 con 0x7f9510100f60 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.288+0000 7f950cc00700 1 -- 172.21.15.23:0/1450813314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500022ea0 con 0x7f9510100f60 2024-03-30T04:03:25.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.288+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a4240 con 0x7f9510100f60 2024-03-30T04:03:25.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.291+0000 7f950cc00700 1 -- 172.21.15.23:0/1450813314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f950000f7f0 con 0x7f9510100f60 2024-03-30T04:03:25.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.292+0000 7f952a9ceb80 1 Processor -- start 2024-03-30T04:03:25.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.292+0000 7f950cc00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 0x7f94f40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.292+0000 7f952a9ceb80 1 -- start start 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.292+0000 7f950cc00700 1 -- 172.21.15.23:0/1450813314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f95000ce680 con 0x7f9510100f60 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950f400700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 0x7f94f40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da32e990 con 0x55d9da3d55d0 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da32c350 con 0x55d9da3d5ef0 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da327c40 con 0x55d9da3e5200 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39430/0 (socket says 172.21.15.23:39430) 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950fe00700 1 -- 172.21.15.23:0/3545261171 learned_addr learned my addr 172.21.15.23:0/3545261171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950f400700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f9514e00700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.293+0000 7f950f400700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 0x7f94f40a3870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9510198840 tx=0x7f94fc009190 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 -- 172.21.15.23:0/3545261171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 msgr2=0x55d9da3e75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 -- 172.21.15.23:0/3545261171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 msgr2=0x55d9da3e4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 -- 172.21.15.23:0/3545261171 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9da1a93b0 con 0x55d9da3d55d0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f9514e00700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f950fe00700 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9500006010 tx=0x7f950009add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f2d140b948bf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f9505600700 1 -- 172.21.15.23:0/3545261171 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950000fe20 con 0x55d9da3d55d0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f9505600700 1 -- 172.21.15.23:0/3545261171 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f950009b210 con 0x55d9da3d55d0 2024-03-30T04:03:25.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.294+0000 7f9505600700 1 -- 172.21.15.23:0/3545261171 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000ce260 con 0x55d9da3d55d0 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3545261171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 msgr2=0x55d9da3d59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9500006010 tx=0x7f950009add0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3545261171 shutdown_connections 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3d59b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3545261171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3545261171 >> 172.21.15.23:0/3545261171 conn(0x55d9da3356a0 msgr2=0x55d9da3d1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3545261171 shutdown_connections 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3545261171 wait complete. 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.295+0000 7f952a9ceb80 1 -- start start 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 Processor -- start 2024-03-30T04:03:25.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3d55d0 0x55d9da3e4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da2c3d80 con 0x55d9da3e5200 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da2c3a20 con 0x55d9da3d5ef0 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da1ab6a0 con 0x55d9da3d55d0 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f952a9ceb80 10 client.0 _ll_register_callbacks cb 0x55d9da2cd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f9514e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f9514e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39442/0 (socket says 172.21.15.23:39442) 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.296+0000 7f9514e00700 1 -- 172.21.15.23:0/2865072209 learned_addr learned my addr 172.21.15.23:0/2865072209 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 -- 172.21.15.23:0/2865072209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3d55d0 msgr2=0x55d9da3e4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3d55d0 0x55d9da3e4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 -- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 msgr2=0x55d9da3e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 -- 172.21.15.23:0/2865072209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a4240 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f9514e00700 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f950800abc0 tx=0x7f950800aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0e35669f703ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f94fa000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.297+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508011040 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f9508006e40 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9da2e8380 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9508007b50 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 10 client.16038 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d9da155b60 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9da2e8380 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f95080153e0 con 0x55d9da3e5200 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 10 client.16038 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 20 client.16038 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da3f4b60 0x55d9da3f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 4 client.16038 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:25.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.298+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9da320700 con 0x55d9da3f4b60 2024-03-30T04:03:25.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.299+0000 7f950fe00700 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da3f4b60 0x55d9da3f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.300+0000 7f950fe00700 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da3f4b60 0x55d9da3f6f80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=76f02dc748df284 server_cookie=b3bc58b255d7c0a7 in_seq=0 out_seq=0 2024-03-30T04:03:25.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.300+0000 7f94fa000700 10 client.16038 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f94fa000700 1 -- 172.21.15.23:0/2865072209 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d9da320700 con 0x55d9da3f4b60 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f94fa000700 10 client.16038 handle_command_reply: tid=0 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 client.16038 shutdown 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 msgr2=0x55d9da3e2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f950800abc0 tx=0x7f950800aca0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 shutdown_connections 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3e5200 0x55d9da3e2560 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da3f4b60 0x55d9da3f6f80 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3d55d0 0x55d9da3e4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2865072209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f94fa000700 0 client.16038 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 >> 172.21.15.23:0/2865072209 conn(0x55d9da3356a0 msgr2=0x55d9da3d7670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 shutdown_connections 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.301+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2865072209 wait complete. 2024-03-30T04:03:25.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.302+0000 7f952a9ceb80 20 client.16038 trim_cache size 0 max 16384 2024-03-30T04:03:25.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 Processor -- start 2024-03-30T04:03:25.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 -- start start 2024-03-30T04:03:25.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d55d0 0x55d9da3e2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3d6850 con 0x55d9da3d5ef0 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3f7310 con 0x55d9da3d55d0 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.315+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3e2b40 con 0x55d9da3e5200 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39452/0 (socket says 172.21.15.23:39452) 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 -- 172.21.15.23:0/3373824145 learned_addr learned my addr 172.21.15.23:0/3373824145 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f9514e00700 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950fe00700 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d55d0 0x55d9da3e2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 -- 172.21.15.23:0/3373824145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 msgr2=0x55d9da3e0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 -- 172.21.15.23:0/3373824145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d55d0 msgr2=0x55d9da3e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d55d0 0x55d9da3e2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.316+0000 7f950f400700 1 -- 172.21.15.23:0/3373824145 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9da2edea0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f950f400700 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e870 tx=0x7f94fc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338cad0263330266 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f9506000700 1 -- 172.21.15.23:0/3373824145 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00ecb0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f9506000700 1 -- 172.21.15.23:0/3373824145 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94fc00ee10 con 0x55d9da3d5ef0 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f9506000700 1 -- 172.21.15.23:0/3373824145 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc013970 con 0x55d9da3d5ef0 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3373824145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 msgr2=0x55d9da3ebbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00e870 tx=0x7f94fc004750 comp rx=0 tx=0).stop 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3373824145 shutdown_connections 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3e0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d55d0 0x55d9da3e2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/3373824145 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d5ef0 0x55d9da3ebbf0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3373824145 >> 172.21.15.23:0/3373824145 conn(0x55d9da3356a0 msgr2=0x55d9da3caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.317+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3373824145 shutdown_connections 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.318+0000 7f952a9ceb80 1 -- 172.21.15.23:0/3373824145 wait complete. 2024-03-30T04:03:25.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.318+0000 7f952a9ceb80 1 -- start start 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.318+0000 7f952a9ceb80 1 Processor -- start 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3eb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3ea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3ea6c0 con 0x55d9da3d55d0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3ea830 con 0x55d9da3d5ef0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9da3ea9f0 con 0x55d9da3e5200 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f952a9ceb80 10 client.0 _ll_register_callbacks cb 0x55d9da2cd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33026/0 (socket says 172.21.15.23:33026) 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 -- 172.21.15.23:0/2694265795 learned_addr learned my addr 172.21.15.23:0/2694265795 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3eb700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39468/0 (socket says 172.21.15.23:39468) 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f9514e00700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3ea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 -- 172.21.15.23:0/2694265795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 msgr2=0x55d9da3ea180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3ea180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 -- 172.21.15.23:0/2694265795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 msgr2=0x55d9da3eb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3eb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.319+0000 7f950f400700 1 -- 172.21.15.23:0/2694265795 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc01a040 con 0x55d9da3d5ef0 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f950f400700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00efc0 tx=0x7f94fc015b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9aae44cbe5c911 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f94fa000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc00ecb0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f950009b210 con 0x55d9da3d5ef0 2024-03-30T04:03:25.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f94fc013790 con 0x55d9da3d5ef0 2024-03-30T04:03:25.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94fc006bf0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f952a9ceb80 10 client.26036 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d9da1adcd0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.320+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f950009b210 con 0x55d9da3d5ef0 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f94fc0358f0 con 0x55d9da3d5ef0 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 10 client.26036 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 10 client.26036 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 20 client.26036 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da449d50 0x55d9da44c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 4 client.26036 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d9da176cf0 con 0x55d9da449d50 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.321+0000 7f950fe00700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da449d50 0x55d9da44c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f950fe00700 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da449d50 0x55d9da44c170 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8e68083ce64bf640 server_cookie=8867d38c8f4663a in_seq=0 out_seq=0 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f94fa000700 10 client.26036 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f94fa000700 1 -- 172.21.15.23:0/2694265795 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d9da176cf0 con 0x55d9da449d50 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f94fa000700 10 client.26036 handle_command_reply: tid=0 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 client.26036 shutdown 2024-03-30T04:03:25.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 msgr2=0x55d9da3e9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00efc0 tx=0x7f94fc015b30 comp rx=0 tx=0).stop 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 shutdown_connections 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d9da449d50 0x55d9da44c170 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9da3e5200 0x55d9da3ea180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9da3d55d0 0x55d9da3eb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f94fa000700 0 client.26036 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 --2- 172.21.15.23:0/2694265795 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9da3d5ef0 0x55d9da3e9c40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 >> 172.21.15.23:0/2694265795 conn(0x55d9da3356a0 msgr2=0x55d9da3e07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 shutdown_connections 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 1 -- 172.21.15.23:0/2694265795 wait complete. 2024-03-30T04:03:25.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.322+0000 7f952a9ceb80 20 client.26036 trim_cache size 0 max 16384 2024-03-30T04:03:25.327 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 msgr2=0x7f94f40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 0x7f94f40a3870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9510198840 tx=0x7f94fc009190 comp rx=0 tx=0).stop 2024-03-30T04:03:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 msgr2=0x7f9510197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9500005910 tx=0x7f9500009f90 comp rx=0 tx=0).stop 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 shutdown_connections 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f951019bbd0 0x7f951019c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9510100f60 0x7f9510197340 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f94f40a13b0 0x7f94f40a3870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 --2- 172.21.15.23:0/1450813314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9510106080 0x7f9510197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 >> 172.21.15.23:0/1450813314 conn(0x7f95100fc7c0 msgr2=0x7f95100fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 shutdown_connections 2024-03-30T04:03:25.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.323+0000 7f9516e00700 1 -- 172.21.15.23:0/1450813314 wait complete. 2024-03-30T04:03:25.341 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:03:25.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 -- 172.21.15.23:0/821423509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 msgr2=0x7f45b4102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 --2- 172.21.15.23:0/821423509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4102110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009990 tx=0x7f45a4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 -- 172.21.15.23:0/821423509 shutdown_connections 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 --2- 172.21.15.23:0/821423509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b410b150 0x7f45b410d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 --2- 172.21.15.23:0/821423509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4102110 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 --2- 172.21.15.23:0/821423509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b4102650 0x7f45b410ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 -- 172.21.15.23:0/821423509 >> 172.21.15.23:0/821423509 conn(0x7f45b40fb570 msgr2=0x7f45b40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 -- 172.21.15.23:0/821423509 shutdown_connections 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.543+0000 7f45b3e00700 1 -- 172.21.15.23:0/821423509 wait complete. 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.544+0000 7f45b3e00700 1 Processor -- start 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.544+0000 7f45b3e00700 1 -- start start 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b4102650 0x7f45b4199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b410b150 0x7f45b419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b419e120 con 0x7f45b4101d30 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b419e290 con 0x7f45b4102650 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b419e450 con 0x7f45b410b150 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39480/0 (socket says 172.21.15.23:39480) 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 -- 172.21.15.23:0/3431776085 learned_addr learned my addr 172.21.15.23:0/3431776085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b3400700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b410b150 0x7f45b419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 -- 172.21.15.23:0/3431776085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b410b150 msgr2=0x7f45b419dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b410b150 0x7f45b419dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 -- 172.21.15.23:0/3431776085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b4102650 msgr2=0x7f45b4199740 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b4102650 0x7f45b4199740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.545+0000 7f45b2a00700 1 -- 172.21.15.23:0/3431776085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40095f0 con 0x7f45b4101d30 2024-03-30T04:03:25.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005910 tx=0x7f45a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45ab400700 1 -- 172.21.15.23:0/3431776085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a401d070 con 0x7f45b4101d30 2024-03-30T04:03:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45ab400700 1 -- 172.21.15.23:0/3431776085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45a4022470 con 0x7f45b4101d30 2024-03-30T04:03:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45ab400700 1 -- 172.21.15.23:0/3431776085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4017650 con 0x7f45b4101d30 2024-03-30T04:03:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b419e6d0 con 0x7f45b4101d30 2024-03-30T04:03:25.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.546+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b419e950 con 0x7f45b4101d30 2024-03-30T04:03:25.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.549+0000 7f45ab400700 1 -- 172.21.15.23:0/3431776085 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7f45a40225e0 con 0x7f45b4101d30 2024-03-30T04:03:25.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.551+0000 7f45cd46db80 1 Processor -- start 2024-03-30T04:03:25.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.551+0000 7f45cd46db80 1 -- start start 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45ab400700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 0x7f45980a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 0x55b536eeda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45ab400700 1 -- 172.21.15.23:0/3431776085 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f45a40ce460 con 0x7f45b4101d30 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536e3b990 con 0x55b536ee27a0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536e39350 con 0x55b536ee30c0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45cd46db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536e34c40 con 0x55b536eedfc0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45b3400700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 0x7f45980a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 0x55b536eeda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39482/0 (socket says 172.21.15.23:39482) 2024-03-30T04:03:25.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.552+0000 7f45b2a00700 1 -- 172.21.15.23:0/3130052095 learned_addr learned my addr 172.21.15.23:0/3130052095 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 -- 172.21.15.23:0/3130052095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 msgr2=0x55b536ef03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 -- 172.21.15.23:0/3130052095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 msgr2=0x55b536ee2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 -- 172.21.15.23:0/3130052095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b536cb63b0 con 0x55b536ee30c0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45b3400700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 0x7f45980a3870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009ec0 tx=0x7f45ac009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45abe00700 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 0x55b536eeda80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f459c009b20 tx=0x7f459c009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f45ddf7aaed507 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45a3e00700 1 -- 172.21.15.23:0/3130052095 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c004230 con 0x55b536ee30c0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45a3e00700 1 -- 172.21.15.23:0/3130052095 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f459c004cf0 con 0x55b536ee30c0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.553+0000 7f45a3e00700 1 -- 172.21.15.23:0/3130052095 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c005610 con 0x55b536ee30c0 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 -- 172.21.15.23:0/3130052095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 msgr2=0x55b536eeda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 0x55b536eeda80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f459c009b20 tx=0x7f459c009e30 comp rx=0 tx=0).stop 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 -- 172.21.15.23:0/3130052095 shutdown_connections 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee27a0 0x55b536ee2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3130052095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee30c0 0x55b536eeda80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 -- 172.21.15.23:0/3130052095 >> 172.21.15.23:0/3130052095 conn(0x55b536e426a0 msgr2=0x55b536edec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 -- 172.21.15.23:0/3130052095 shutdown_connections 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.554+0000 7f45cd46db80 1 -- 172.21.15.23:0/3130052095 wait complete. 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 -- start start 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 Processor -- start 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536eed9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536eeb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536dd0d80 con 0x55b536ee30c0 2024-03-30T04:03:25.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536dd0a20 con 0x55b536ee27a0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.555+0000 7f45cd46db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536cb86a0 con 0x55b536eedfc0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45cd46db80 10 client.0 _ll_register_callbacks cb 0x55b536dda470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39496/0 (socket says 172.21.15.23:39496) 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 -- 172.21.15.23:0/1422600309 learned_addr learned my addr 172.21.15.23:0/1422600309 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45b3400700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536eeb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 -- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 msgr2=0x55b536eeb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536eeb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 -- 172.21.15.23:0/1422600309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 msgr2=0x55b536eed9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536eed9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.556+0000 7f45abe00700 1 -- 172.21.15.23:0/1422600309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b419e6d0 con 0x55b536ee30c0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45abe00700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f459c005520 tx=0x7f459c005550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1338d95fa92b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45a0c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:25.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c01c070 con 0x55b536ee30c0 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f459c014070 con 0x55b536ee30c0 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b536df5380 con 0x55b536ee30c0 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f459c01f040 con 0x55b536ee30c0 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45cd46db80 10 client.16056 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f459c009800 con 0x55b536ee30c0 2024-03-30T04:03:25.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.557+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b536df5380 con 0x55b536ee30c0 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f459c022030 con 0x55b536ee30c0 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 10 client.16056 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 20 client.16056 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536efdb60 0x55b536efff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 4 client.16056 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.558+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b536e2d700 con 0x55b536efdb60 2024-03-30T04:03:25.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.560+0000 7f45b2a00700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536efdb60 0x55b536efff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.562+0000 7f45b2a00700 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536efdb60 0x55b536efff80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e3ff8a882bc55f51 server_cookie=85df6c1406e5cea5 in_seq=0 out_seq=0 2024-03-30T04:03:25.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.562+0000 7f45a0c00700 10 client.16056 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45a0c00700 1 -- 172.21.15.23:0/1422600309 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b536e2d700 con 0x55b536efdb60 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45a0c00700 10 client.16056 handle_command_reply: tid=0 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 client.16056 shutdown 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 msgr2=0x55b536eead50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f459c005520 tx=0x7f459c005550 comp rx=0 tx=0).stop 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 shutdown_connections 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536efdb60 0x55b536efff80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536eeb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536eed9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45a0c00700 0 client.16056 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 --2- 172.21.15.23:0/1422600309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536eead50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 >> 172.21.15.23:0/1422600309 conn(0x55b536e426a0 msgr2=0x55b536ee4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 shutdown_connections 2024-03-30T04:03:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.563+0000 7f45cd46db80 1 -- 172.21.15.23:0/1422600309 wait complete. 2024-03-30T04:03:25.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.564+0000 7f45cd46db80 20 client.16056 trim_cache size 0 max 16384 2024-03-30T04:03:25.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 Processor -- start 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 -- start start 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536ee30c0 0x55b536ef4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536eec190 con 0x55b536eedfc0 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536ee3a20 con 0x55b536ee27a0 2024-03-30T04:03:25.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.577+0000 7f45cd46db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536ee4240 con 0x55b536ee30c0 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39506/0 (socket says 172.21.15.23:39506) 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33070/0 (socket says 172.21.15.23:33070) 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 -- 172.21.15.23:0/233832992 learned_addr learned my addr 172.21.15.23:0/233832992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45abe00700 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536ee30c0 0x55b536ef4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 -- 172.21.15.23:0/233832992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536ee30c0 msgr2=0x55b536ef4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536ee30c0 0x55b536ef4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 -- 172.21.15.23:0/233832992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 msgr2=0x55b536efff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 -- 172.21.15.23:0/233832992 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b536c62b60 con 0x55b536eedfc0 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b2a00700 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.578+0000 7f45b3400700 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009360 tx=0x7f45ac01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b17f518778d834 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45a8c00700 1 -- 172.21.15.23:0/233832992 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac026070 con 0x55b536eedfc0 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45a8c00700 1 -- 172.21.15.23:0/233832992 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45ac00f040 con 0x55b536eedfc0 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45a8c00700 1 -- 172.21.15.23:0/233832992 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac00e040 con 0x55b536eedfc0 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 -- 172.21.15.23:0/233832992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 msgr2=0x55b536ed8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009360 tx=0x7f45ac01a000 comp rx=0 tx=0).stop 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 -- 172.21.15.23:0/233832992 shutdown_connections 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536eedfc0 0x55b536ed8010 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536efff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 --2- 172.21.15.23:0/233832992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536ee30c0 0x55b536ef4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 -- 172.21.15.23:0/233832992 >> 172.21.15.23:0/233832992 conn(0x55b536e426a0 msgr2=0x55b536ee5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 -- 172.21.15.23:0/233832992 shutdown_connections 2024-03-30T04:03:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.579+0000 7f45cd46db80 1 -- 172.21.15.23:0/233832992 wait complete. 2024-03-30T04:03:25.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 -- start start 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 Processor -- start 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536ef4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536ef5030 con 0x55b536ee30c0 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536ef36d0 con 0x55b536ee27a0 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b536ef3890 con 0x55b536eedfc0 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.580+0000 7f45cd46db80 10 client.0 _ll_register_callbacks cb 0x55b536dda470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39514/0 (socket says 172.21.15.23:39514) 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 -- 172.21.15.23:0/3182489199 learned_addr learned my addr 172.21.15.23:0/3182489199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 -- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 msgr2=0x55b536ef3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 -- 172.21.15.23:0/3182489199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 msgr2=0x55b536ef4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536ef4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 -- 172.21.15.23:0/3182489199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a40095f0 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45abe00700 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55b536edf970 tx=0x7f459c00e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e25e0e1d9076a60 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.581+0000 7f45a0c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459c01c040 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f459c014070 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f459c004cf0 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f459c009350 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 10 client.16062 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f45ac009040 con 0x55b536ee30c0 2024-03-30T04:03:25.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f459c004cf0 con 0x55b536ee30c0 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f459c03b070 con 0x55b536ee30c0 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 20 client.16062 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536f52c50 0x55b536f55070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 4 client.16062 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.582+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b536c83cf0 con 0x55b536f52c50 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536f52c50 0x55b536f55070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45b2a00700 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536f52c50 0x55b536f55070 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26ddd9884272247a server_cookie=10f69531159f8637 in_seq=0 out_seq=0 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45a0c00700 10 client.16062 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45a0c00700 1 -- 172.21.15.23:0/3182489199 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b536c83cf0 con 0x55b536f52c50 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45a0c00700 10 client.16062 handle_command_reply: tid=0 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.583+0000 7f45cd46db80 1 client.16062 shutdown 2024-03-30T04:03:25.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 msgr2=0x55b536ef2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55b536edf970 tx=0x7f459c00e6e0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 shutdown_connections 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b536f52c50 0x55b536f55070 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b536eedfc0 0x55b536ef3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b536ee27a0 0x55b536ef4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45a0c00700 0 client.16062 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 --2- 172.21.15.23:0/3182489199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b536ee30c0 0x55b536ef2c20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 >> 172.21.15.23:0/3182489199 conn(0x55b536e426a0 msgr2=0x55b536eea290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 shutdown_connections 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 1 -- 172.21.15.23:0/3182489199 wait complete. 2024-03-30T04:03:25.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45cd46db80 20 client.16062 trim_cache size 0 max 16384 2024-03-30T04:03:25.589 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 msgr2=0x7f45980a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 0x7f45980a3870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45ac009ec0 tx=0x7f45ac009920 comp rx=0 tx=0).stop 2024-03-30T04:03:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 msgr2=0x7f45b4199200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.584+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005910 tx=0x7f45a4004730 comp rx=0 tx=0).stop 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 shutdown_connections 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b410b150 0x7f45b419dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45b4101d30 0x7f45b4199200 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f45980a13b0 0x7f45980a3870 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 --2- 172.21.15.23:0/3431776085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45b4102650 0x7f45b4199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 >> 172.21.15.23:0/3431776085 conn(0x7f45b40fb570 msgr2=0x7f45b40ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 shutdown_connections 2024-03-30T04:03:25.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.585+0000 7f45b3e00700 1 -- 172.21.15.23:0/3431776085 wait complete. 2024-03-30T04:03:25.599 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:25 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3577598986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:25.602 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:03:25.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:25 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3577598986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:25.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:25 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3577598986' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:25.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.802+0000 7ff44f800700 1 -- 172.21.15.23:0/988749726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448106080 msgr2=0x7ff44810dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.802+0000 7ff44f800700 1 --2- 172.21.15.23:0/988749726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448106080 0x7ff44810dc00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b0a0 tx=0x7ff44000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 -- 172.21.15.23:0/988749726 shutdown_connections 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 --2- 172.21.15.23:0/988749726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448106080 0x7ff44810dc00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 --2- 172.21.15.23:0/988749726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff448100f60 0x7ff448101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 --2- 172.21.15.23:0/988749726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448101910 0x7ff448105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 -- 172.21.15.23:0/988749726 >> 172.21.15.23:0/988749726 conn(0x7ff4480fc7c0 msgr2=0x7ff4480febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 -- 172.21.15.23:0/988749726 shutdown_connections 2024-03-30T04:03:25.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.803+0000 7ff44f800700 1 -- 172.21.15.23:0/988749726 wait complete. 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 Processor -- start 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 -- start start 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff448100f60 0x7ff4480730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448106080 0x7ff4480735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481a5b40 con 0x7ff448073b30 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481a5c80 con 0x7ff448106080 2024-03-30T04:03:25.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44f800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4481a5e40 con 0x7ff448100f60 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.804+0000 7ff44d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39532/0 (socket says 172.21.15.23:39532) 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 -- 172.21.15.23:0/3186936444 learned_addr learned my addr 172.21.15.23:0/3186936444 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff447e00700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448106080 0x7ff4480735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 -- 172.21.15.23:0/3186936444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff448100f60 msgr2=0x7ff4480730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff448100f60 0x7ff4480730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 -- 172.21.15.23:0/3186936444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448106080 msgr2=0x7ff4480735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448106080 0x7ff4480735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 -- 172.21.15.23:0/3186936444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff440009d00 con 0x7ff448073b30 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44d800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b070 tx=0x7ff44000bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff445600700 1 -- 172.21.15.23:0/3186936444 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000e040 con 0x7ff448073b30 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff445600700 1 -- 172.21.15.23:0/3186936444 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff44001d480 con 0x7ff448073b30 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4481a60c0 con 0x7ff448073b30 2024-03-30T04:03:25.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.805+0000 7ff445600700 1 -- 172.21.15.23:0/3186936444 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400052a0 con 0x7ff448073b30 2024-03-30T04:03:25.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.806+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481a64e0 con 0x7ff448073b30 2024-03-30T04:03:25.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.809+0000 7ff445600700 1 -- 172.21.15.23:0/3186936444 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7ff44001d5f0 con 0x7ff448073b30 2024-03-30T04:03:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff445600700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 0x7ff4300a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff463418b80 1 Processor -- start 2024-03-30T04:03:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff445600700 1 -- 172.21.15.23:0/3186936444 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff4400ce3e0 con 0x7ff448073b30 2024-03-30T04:03:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff463418b80 1 -- start start 2024-03-30T04:03:25.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff44ce00700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 0x7ff4300a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.810+0000 7ff463418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff463418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff463418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 0x5600990d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff463418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560099022990 con 0x5600990c9ef0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff463418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560099020350 con 0x5600990c95d0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff463418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56009901bc40 con 0x5600990d4df0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39542/0 (socket says 172.21.15.23:39542) 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 -- 172.21.15.23:0/2689891684 learned_addr learned my addr 172.21.15.23:0/2689891684 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff44d800700 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 0x5600990d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff44ce00700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 0x7ff4300a38f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff43c005950 tx=0x7ff43c0058e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 -- 172.21.15.23:0/2689891684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 msgr2=0x5600990d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 0x5600990d71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 -- 172.21.15.23:0/2689891684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990c95d0 msgr2=0x5600990c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff447e00700 1 -- 172.21.15.23:0/2689891684 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560098e9d3b0 con 0x5600990c9ef0 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.811+0000 7ff44d800700 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 0x5600990d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.812+0000 7ff447e00700 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff434009cb0 tx=0x7ff434011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d55c8c220895888 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.812+0000 7ff43a000700 1 -- 172.21.15.23:0/2689891684 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434011d20 con 0x5600990c9ef0 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.812+0000 7ff43a000700 1 -- 172.21.15.23:0/2689891684 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff434011e80 con 0x5600990c9ef0 2024-03-30T04:03:25.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.812+0000 7ff43a000700 1 -- 172.21.15.23:0/2689891684 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434005280 con 0x5600990c9ef0 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- 172.21.15.23:0/2689891684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 msgr2=0x5600990d48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff434009cb0 tx=0x7ff434011440 comp rx=0 tx=0).stop 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- 172.21.15.23:0/2689891684 shutdown_connections 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990d4df0 0x5600990d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 --2- 172.21.15.23:0/2689891684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990d48b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- 172.21.15.23:0/2689891684 >> 172.21.15.23:0/2689891684 conn(0x5600990296a0 msgr2=0x5600990c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- 172.21.15.23:0/2689891684 shutdown_connections 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- 172.21.15.23:0/2689891684 wait complete. 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.813+0000 7ff463418b80 1 -- start start 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 Processor -- start 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990d2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560098fb7d80 con 0x5600990c95d0 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560098fb7a20 con 0x5600990d4df0 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560098e9f6a0 con 0x5600990c9ef0 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff463418b80 10 client.0 _ll_register_callbacks cb 0x560098fc1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39546/0 (socket says 172.21.15.23:39546) 2024-03-30T04:03:25.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff44ce00700 1 -- 172.21.15.23:0/2249166711 learned_addr learned my addr 172.21.15.23:0/2249166711 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.814+0000 7ff447e00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 -- 172.21.15.23:0/2249166711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 msgr2=0x5600990d1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990d1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 -- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 msgr2=0x5600990d2030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990d2030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 -- 172.21.15.23:0/2249166711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4481a64e0 con 0x5600990c95d0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff43c01b6e0 tx=0x7ff43c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b759b42be739f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff42ea00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c020070 con 0x5600990c95d0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff43c010640 con 0x5600990c95d0 2024-03-30T04:03:25.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560099014700 con 0x5600990c95d0 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.815+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff43c010800 con 0x5600990c95d0 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 10 client.16074 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ff4340098e0 con 0x5600990c95d0 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560099014700 con 0x5600990c95d0 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff43c024b00 con 0x5600990c95d0 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 10 client.16074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:03:25.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 20 client.16074 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5600990e49c0 0x5600990e6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 4 client.16074 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:25.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.816+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560098e6acf0 con 0x5600990e49c0 2024-03-30T04:03:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.817+0000 7ff447e00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5600990e49c0 0x5600990e6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.820+0000 7ff447e00700 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5600990e49c0 0x5600990e6db0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba7b08975cf8cec4 server_cookie=925b57670c235a71 in_seq=0 out_seq=0 2024-03-30T04:03:25.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.820+0000 7ff42ea00700 10 client.16074 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:25.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff42ea00700 1 -- 172.21.15.23:0/2249166711 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560098e6acf0 con 0x5600990e49c0 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff42ea00700 10 client.16074 handle_command_reply: tid=0 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 client.16074 shutdown 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 msgr2=0x5600990d4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff43c01b6e0 tx=0x7ff43c004750 comp rx=0 tx=0).stop 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 shutdown_connections 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5600990e49c0 0x5600990e6db0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990d2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990d4770 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 --2- 172.21.15.23:0/2249166711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990d1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff42ea00700 0 client.16074 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.822+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 >> 172.21.15.23:0/2249166711 conn(0x5600990296a0 msgr2=0x5600990cb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.823+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 shutdown_connections 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.823+0000 7ff463418b80 1 -- 172.21.15.23:0/2249166711 wait complete. 2024-03-30T04:03:25.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.823+0000 7ff463418b80 20 client.16074 trim_cache size 0 max 16384 2024-03-30T04:03:25.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.836+0000 7ff463418b80 1 Processor -- start 2024-03-30T04:03:25.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.836+0000 7ff463418b80 1 -- start start 2024-03-30T04:03:25.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990dbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990be5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600990e14a0 con 0x5600990c95d0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600990d2630 con 0x5600990d4df0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff463418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600990d3030 con 0x5600990c9ef0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39562/0 (socket says 172.21.15.23:39562) 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 -- 172.21.15.23:0/2150529535 learned_addr learned my addr 172.21.15.23:0/2150529535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44d800700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990be5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff447e00700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990dbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 -- 172.21.15.23:0/2150529535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 msgr2=0x5600990dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990dbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 -- 172.21.15.23:0/2150529535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 msgr2=0x5600990be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990be5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44ce00700 1 -- 172.21.15.23:0/2150529535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560098e49b60 con 0x5600990c95d0 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.837+0000 7ff44d800700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff44ce00700 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0094a0 tx=0x7ff43c005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5871d0644e8b0bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff43aa00700 1 -- 172.21.15.23:0/2150529535 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c020030 con 0x5600990c95d0 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff43aa00700 1 -- 172.21.15.23:0/2150529535 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff43c004cf0 con 0x5600990c95d0 2024-03-30T04:03:25.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff43aa00700 1 -- 172.21.15.23:0/2150529535 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0108a0 con 0x5600990c95d0 2024-03-30T04:03:25.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 -- 172.21.15.23:0/2150529535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 msgr2=0x5600990c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0094a0 tx=0x7ff43c005a00 comp rx=0 tx=0).stop 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 -- 172.21.15.23:0/2150529535 shutdown_connections 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600990be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c95d0 0x5600990c99b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.838+0000 7ff463418b80 1 --2- 172.21.15.23:0/2150529535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c9ef0 0x5600990dbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.839+0000 7ff463418b80 1 -- 172.21.15.23:0/2150529535 >> 172.21.15.23:0/2150529535 conn(0x5600990296a0 msgr2=0x5600990bef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.839+0000 7ff463418b80 1 -- 172.21.15.23:0/2150529535 shutdown_connections 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.839+0000 7ff463418b80 1 -- 172.21.15.23:0/2150529535 wait complete. 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.839+0000 7ff463418b80 1 -- start start 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.839+0000 7ff463418b80 1 Processor -- start 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560099131d10 con 0x5600990c9ef0 2024-03-30T04:03:25.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560099131e80 con 0x5600990d4df0 2024-03-30T04:03:25.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560099132040 con 0x5600990c95d0 2024-03-30T04:03:25.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff463418b80 10 client.0 _ll_register_callbacks cb 0x560098fc1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:25.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff447e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33134/0 (socket says 172.21.15.23:33134) 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:46770/0 (socket says 172.21.15.23:46770) 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff447e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39564/0 (socket says 172.21.15.23:39564) 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 -- 172.21.15.23:0/1443966600 learned_addr learned my addr 172.21.15.23:0/1443966600 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 -- 172.21.15.23:0/1443966600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 msgr2=0x5600990d9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 -- 172.21.15.23:0/1443966600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 msgr2=0x5600990da1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44d800700 1 -- 172.21.15.23:0/1443966600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4340098e0 con 0x5600990d4df0 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff447e00700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:25.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.840+0000 7ff44ce00700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff44d800700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff440009fd0 tx=0x7ff440099520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8141898be65674f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff42ea00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400a10c0 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff44009b670 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560098fdc380 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff44009af70 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff463418b80 10 client.26060 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.841+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560098fe1ea0 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560098fdc380 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff44009a460 con 0x5600990d4df0 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 10 client.26060 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 10 client.26060 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:03:25.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 20 client.26060 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x560099139740 0x56009913bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 4 client.26060 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560098e9fce0 con 0x560099139740 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.842+0000 7ff44ce00700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x560099139740 0x56009913bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff44ce00700 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x560099139740 0x56009913bb60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1b526d5f13cbd9c9 server_cookie=9e0ad10d381c40d in_seq=0 out_seq=0 2024-03-30T04:03:25.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff42ea00700 10 client.26060 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:25.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff42ea00700 1 -- 172.21.15.23:0/1443966600 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560098e9fce0 con 0x560099139740 2024-03-30T04:03:25.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff42ea00700 10 client.26060 handle_command_reply: tid=0 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 client.26060 shutdown 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 msgr2=0x5600991317d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff440009fd0 tx=0x7ff440099520 comp rx=0 tx=0).stop 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 shutdown_connections 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x560099139740 0x56009913bb60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600990d4df0 0x5600991317d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff42ea00700 0 client.26060 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600990c95d0 0x5600990d9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 --2- 172.21.15.23:0/1443966600 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600990c9ef0 0x5600990da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.843+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 >> 172.21.15.23:0/1443966600 conn(0x5600990296a0 msgr2=0x5600990be5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 shutdown_connections 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff463418b80 1 -- 172.21.15.23:0/1443966600 wait complete. 2024-03-30T04:03:25.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff463418b80 20 client.26060 trim_cache size 0 max 16384 2024-03-30T04:03:25.850 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:25.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 msgr2=0x7ff4300a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 0x7ff4300a38f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff43c005950 tx=0x7ff43c0058e0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 msgr2=0x7ff4481a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b070 tx=0x7ff44000bda0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 shutdown_connections 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff448073b30 0x7ff4481a5570 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff448100f60 0x7ff4480730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4300a1430 0x7ff4300a38f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 --2- 172.21.15.23:0/3186936444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff448106080 0x7ff4480735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 >> 172.21.15.23:0/3186936444 conn(0x7ff4480fc7c0 msgr2=0x7ff4480fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.844+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 shutdown_connections 2024-03-30T04:03:25.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:25.845+0000 7ff44f800700 1 -- 172.21.15.23:0/3186936444 wait complete. 2024-03-30T04:03:25.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 -- 172.21.15.23:0/2149212118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf40073b00 msgr2=0x7fdf4010d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 --2- 172.21.15.23:0/2149212118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf40073b00 0x7fdf4010d000 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009990 tx=0x7fdf2c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 -- 172.21.15.23:0/2149212118 shutdown_connections 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 --2- 172.21.15.23:0/2149212118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf4010d540 0x7fdf4010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 --2- 172.21.15.23:0/2149212118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf400731e0 0x7fdf400735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 --2- 172.21.15.23:0/2149212118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf40073b00 0x7fdf4010d000 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 -- 172.21.15.23:0/2149212118 >> 172.21.15.23:0/2149212118 conn(0x7fdf400fc7f0 msgr2=0x7fdf400fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 -- 172.21.15.23:0/2149212118 shutdown_connections 2024-03-30T04:03:26.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.064+0000 7fdf47800700 1 -- 172.21.15.23:0/2149212118 wait complete. 2024-03-30T04:03:26.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.065+0000 7fdf47800700 1 Processor -- start 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.065+0000 7fdf47800700 1 -- start start 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.065+0000 7fdf47800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf400731e0 0x7fdf40199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf47800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf40073b00 0x7fdf40199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf47800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf47800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4019e330 con 0x7fdf4010d540 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf47800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4019e4a0 con 0x7fdf400731e0 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf47800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4019e660 con 0x7fdf40073b00 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39576/0 (socket says 172.21.15.23:39576) 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 -- 172.21.15.23:0/1845069513 learned_addr learned my addr 172.21.15.23:0/1845069513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf44e00700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf400731e0 0x7fdf40199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf40073b00 0x7fdf40199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 -- 172.21.15.23:0/1845069513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf40073b00 msgr2=0x7fdf40199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf40073b00 0x7fdf40199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 -- 172.21.15.23:0/1845069513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf400731e0 msgr2=0x7fdf40199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf400731e0 0x7fdf40199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.066+0000 7fdf45800700 1 -- 172.21.15.23:0/1845069513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0095f0 con 0x7fdf4010d540 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf45800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800ef30 tx=0x7fdf3800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf3d600700 1 -- 172.21.15.23:0/1845069513 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf38018070 con 0x7fdf4010d540 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf3d600700 1 -- 172.21.15.23:0/1845069513 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf3800f040 con 0x7fdf4010d540 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf3d600700 1 -- 172.21.15.23:0/1845069513 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3800cd00 con 0x7fdf4010d540 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4019e8e0 con 0x7fdf4010d540 2024-03-30T04:03:26.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.067+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf401a6180 con 0x7fdf4010d540 2024-03-30T04:03:26.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.071+0000 7fdf3d600700 1 -- 172.21.15.23:0/1845069513 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fdf38003a00 con 0x7fdf4010d540 2024-03-30T04:03:26.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 Processor -- start 2024-03-30T04:03:26.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf3d600700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 0x7fdf280a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf3d600700 1 -- 172.21.15.23:0/1845069513 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fdf380ccd90 con 0x7fdf4010d540 2024-03-30T04:03:26.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 -- start start 2024-03-30T04:03:26.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 0x7fdf280a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7abd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7abddf0 0x5593e7ac01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7a0b990 con 0x5593e7abddf0 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7a09350 con 0x5593e7ab2ef0 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.073+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7a04c40 con 0x5593e7ab25d0 2024-03-30T04:03:26.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf45800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7abddf0 0x5593e7ac01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:46778/0 (socket says 172.21.15.23:46778) 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 -- 172.21.15.23:0/3253861997 learned_addr learned my addr 172.21.15.23:0/3253861997 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7abd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 0x7fdf280a38f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf4019a960 tx=0x7fdf2c00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 -- 172.21.15.23:0/3253861997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 msgr2=0x5593e7abd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7abd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 -- 172.21.15.23:0/3253861997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7abddf0 msgr2=0x5593e7ac01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7abddf0 0x5593e7ac01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 -- 172.21.15.23:0/3253861997 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593e78863b0 con 0x5593e7ab25d0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7abd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.074+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009d80 tx=0x7fdf34011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644f876fb14ce3de server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf32000700 1 -- 172.21.15.23:0/3253861997 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34004230 con 0x5593e7ab25d0 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf32000700 1 -- 172.21.15.23:0/3253861997 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf34004390 con 0x5593e7ab25d0 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf32000700 1 -- 172.21.15.23:0/3253861997 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34019880 con 0x5593e7ab25d0 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3253861997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 msgr2=0x5593e7ab29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009d80 tx=0x7fdf34011720 comp rx=0 tx=0).stop 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3253861997 shutdown_connections 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7abddf0 0x5593e7ac01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ab29b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3253861997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7abd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.075+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3253861997 >> 172.21.15.23:0/3253861997 conn(0x5593e7a126a0 msgr2=0x5593e7aaea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.076+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3253861997 shutdown_connections 2024-03-30T04:03:26.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.076+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3253861997 wait complete. 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.076+0000 7fdf5b3f4b80 1 -- start start 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.076+0000 7fdf5b3f4b80 1 Processor -- start 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 0x5593e7abaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7abb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e79a0d80 con 0x5593e7ab25d0 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e79a0a20 con 0x5593e7abddf0 2024-03-30T04:03:26.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e78886a0 con 0x5593e7ab2ef0 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf5b3f4b80 10 client.0 _ll_register_callbacks cb 0x5593e79aa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf3fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 0x5593e7abaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39596/0 (socket says 172.21.15.23:39596) 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf44e00700 1 -- 172.21.15.23:0/3385543767 learned_addr learned my addr 172.21.15.23:0/3385543767 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.077+0000 7fdf45800700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7abb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 -- 172.21.15.23:0/3385543767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 msgr2=0x5593e7abaa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 0x5593e7abaa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 -- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 msgr2=0x5593e7abb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7abb050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 -- 172.21.15.23:0/3385543767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf340099e0 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 0x5593e7abaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf44e00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdf34005950 tx=0x7fdf340038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbe5237c9177ed5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf26a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34005240 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fdf34005ce0 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.078+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5593e79c5380 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf34009ed0 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 10 client.16092 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5593e7832b60 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5593e79c5380 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fdf34021040 con 0x5593e7ab25d0 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 10 client.16092 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:03:26.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 20 client.16092 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7acd810 0x5593e7acfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 4 client.16092 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.079+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5593e79fd700 con 0x5593e7acd810 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.080+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7acd810 0x5593e7acfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.080+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7acd810 0x5593e7acfc00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a920983f1156f9e2 server_cookie=f7a5af205e86110e in_seq=0 out_seq=0 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.080+0000 7fdf26a00700 10 client.16092 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf26a00700 1 -- 172.21.15.23:0/3385543767 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5593e79fd700 con 0x5593e7acd810 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf26a00700 10 client.16092 handle_command_reply: tid=0 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 client.16092 shutdown 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 msgr2=0x5593e7abd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fdf34005950 tx=0x7fdf340038b0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 shutdown_connections 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7abb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7abd7a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7acd810 0x5593e7acfc00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/3385543767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab2ef0 0x5593e7abaa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 >> 172.21.15.23:0/3385543767 conn(0x5593e7a126a0 msgr2=0x5593e7ab46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf26a00700 0 client.16092 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 shutdown_connections 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/3385543767 wait complete. 2024-03-30T04:03:26.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.081+0000 7fdf5b3f4b80 20 client.16092 trim_cache size 0 max 16384 2024-03-30T04:03:26.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.095+0000 7fdf5b3f4b80 1 Processor -- start 2024-03-30T04:03:26.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.095+0000 7fdf5b3f4b80 1 -- start start 2024-03-30T04:03:26.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.095+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.095+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.095+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7ab98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e79c5380 con 0x5593e7ab2ef0 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7abb410 con 0x5593e7abddf0 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7aca500 con 0x5593e7ab25d0 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39606/0 (socket says 172.21.15.23:39606) 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 -- 172.21.15.23:0/1149637491 learned_addr learned my addr 172.21.15.23:0/1149637491 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf44e00700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf45800700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7ab98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 -- 172.21.15.23:0/1149637491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 msgr2=0x5593e7ac4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 -- 172.21.15.23:0/1149637491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 msgr2=0x5593e7ab98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7ab98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 -- 172.21.15.23:0/1149637491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5593e79caea0 con 0x5593e7ab2ef0 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf44e00700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.096+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c000c00 tx=0x7fdf2c025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c70f547862ce92e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf32a00700 1 -- 172.21.15.23:0/1149637491 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c035350 con 0x5593e7ab2ef0 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf32a00700 1 -- 172.21.15.23:0/1149637491 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf2c01e040 con 0x5593e7ab2ef0 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf32a00700 1 -- 172.21.15.23:0/1149637491 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c025470 con 0x5593e7ab2ef0 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/1149637491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 msgr2=0x5593e7ab9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c000c00 tx=0x7fdf2c025000 comp rx=0 tx=0).stop 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/1149637491 shutdown_connections 2024-03-30T04:03:26.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7abddf0 0x5593e7ab98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/1149637491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab2ef0 0x5593e7ab9360 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/1149637491 >> 172.21.15.23:0/1149637491 conn(0x5593e7a126a0 msgr2=0x5593e7aa7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.097+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/1149637491 shutdown_connections 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.098+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/1149637491 wait complete. 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.098+0000 7fdf5b3f4b80 1 -- start start 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.098+0000 7fdf5b3f4b80 1 Processor -- start 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7ac4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7abddf0 0x5593e7ac2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7ac3400 con 0x5593e7ab25d0 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7ac3570 con 0x5593e7ab2ef0 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593e7ac3730 con 0x5593e7abddf0 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf5b3f4b80 10 client.0 _ll_register_callbacks cb 0x5593e79aa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:26.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39616/0 (socket says 172.21.15.23:39616) 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 -- 172.21.15.23:0/464007022 learned_addr learned my addr 172.21.15.23:0/464007022 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7ac4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf45800700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7abddf0 0x5593e7ac2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 -- 172.21.15.23:0/464007022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7abddf0 msgr2=0x5593e7ac2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7abddf0 0x5593e7ac2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 -- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 msgr2=0x5593e7ac4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7ac4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 -- 172.21.15.23:0/464007022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5593e7832b60 con 0x5593e7ab25d0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf44e00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009cd0 tx=0x7fdf34009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4f05a0c8b91d806 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.099+0000 7fdf26a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34019ca0 con 0x5593e7ab25d0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdf34004390 con 0x5593e7ab25d0 2024-03-30T04:03:26.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fdf34000e40 con 0x5593e7ab25d0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf34009ed0 con 0x5593e7ab25d0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf5b3f4b80 10 client.16098 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fdf40110670 con 0x5593e7ab25d0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf34004390 con 0x5593e7ab25d0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.100+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fdf34021070 con 0x5593e7ab25d0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 20 client.16098 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7b225e0 0x5593e7b24a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 4 client.16098 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5593e7853cf0 con 0x5593e7b225e0 2024-03-30T04:03:26.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.101+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7b225e0 0x5593e7b24a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.105+0000 7fdf3fe00700 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7b225e0 0x5593e7b24a00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf2496eb79d0f3d4 server_cookie=6c689fdb600d3f65 in_seq=0 out_seq=0 2024-03-30T04:03:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.105+0000 7fdf26a00700 10 client.16098 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:26.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf26a00700 1 -- 172.21.15.23:0/464007022 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5593e7853cf0 con 0x5593e7b225e0 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf26a00700 10 client.16098 handle_command_reply: tid=0 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 client.16098 shutdown 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 msgr2=0x5593e7ac4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdf34009cd0 tx=0x7fdf34009d00 comp rx=0 tx=0).stop 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 shutdown_connections 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5593e7b225e0 0x5593e7b24a00 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593e7abddf0 0x5593e7ac2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5593e7ab25d0 0x5593e7ac4700 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 --2- 172.21.15.23:0/464007022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5593e7ab2ef0 0x5593e7ac4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf26a00700 0 client.16098 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 >> 172.21.15.23:0/464007022 conn(0x5593e7a126a0 msgr2=0x5593e7ab9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 shutdown_connections 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.106+0000 7fdf5b3f4b80 1 -- 172.21.15.23:0/464007022 wait complete. 2024-03-30T04:03:26.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf5b3f4b80 20 client.16098 trim_cache size 0 max 16384 2024-03-30T04:03:26.109 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:26.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 msgr2=0x7fdf280a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 0x7fdf280a38f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdf4019a960 tx=0x7fdf2c00b260 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 msgr2=0x7fdf4019dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800ef30 tx=0x7fdf3800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 shutdown_connections 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdf4010d540 0x7fdf4019dcf0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdf280a1430 0x7fdf280a38f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf400731e0 0x7fdf40199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 --2- 172.21.15.23:0/1845069513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdf40073b00 0x7fdf40199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 >> 172.21.15.23:0/1845069513 conn(0x7fdf400fc7f0 msgr2=0x7fdf400fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 shutdown_connections 2024-03-30T04:03:26.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.107+0000 7fdf47800700 1 -- 172.21.15.23:0/1845069513 wait complete. 2024-03-30T04:03:26.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:03:26.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 365 4.35 MB/sec warmup 3 sec latency 2810.497 ms 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 -- 172.21.15.23:0/1739667444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc160067e30 msgr2=0x7fc1600682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 --2- 172.21.15.23:0/1739667444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc160067e30 0x7fc1600682b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0099c0 tx=0x7fc14c009cd0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 -- 172.21.15.23:0/1739667444 shutdown_connections 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 --2- 172.21.15.23:0/1739667444 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1600687f0 0x7fc16010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 --2- 172.21.15.23:0/1739667444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160105ce0 0x7fc1601060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 --2- 172.21.15.23:0/1739667444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc160067e30 0x7fc1600682b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 -- 172.21.15.23:0/1739667444 >> 172.21.15.23:0/1739667444 conn(0x7fc160076d40 msgr2=0x7fc160077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 -- 172.21.15.23:0/1739667444 shutdown_connections 2024-03-30T04:03:26.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.332+0000 7fc165c00700 1 -- 172.21.15.23:0/1739667444 wait complete. 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.333+0000 7fc165c00700 1 Processor -- start 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.333+0000 7fc165c00700 1 -- start start 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160067e30 0x7fc1601994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 0x7fc160199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16019e3b0 con 0x7fc160105ce0 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16019e520 con 0x7fc1600687f0 2024-03-30T04:03:26.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc165c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc16019e6e0 con 0x7fc160067e30 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39636/0 (socket says 172.21.15.23:39636) 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 -- 172.21.15.23:0/4203388753 learned_addr learned my addr 172.21.15.23:0/4203388753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15ea00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160067e30 0x7fc1601994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15e000700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 0x7fc160199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 -- 172.21.15.23:0/4203388753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160067e30 msgr2=0x7fc1601994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160067e30 0x7fc1601994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 -- 172.21.15.23:0/4203388753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 msgr2=0x7fc160199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 0x7fc160199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15f400700 1 -- 172.21.15.23:0/4203388753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158009c40 con 0x7fc160105ce0 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.334+0000 7fc15e000700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 0x7fc160199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc15f400700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c370 tx=0x7fc15800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc157400700 1 -- 172.21.15.23:0/4203388753 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158017070 con 0x7fc160105ce0 2024-03-30T04:03:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc157400700 1 -- 172.21.15.23:0/4203388753 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc15800f040 con 0x7fc160105ce0 2024-03-30T04:03:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c0095f0 con 0x7fc160105ce0 2024-03-30T04:03:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc157400700 1 -- 172.21.15.23:0/4203388753 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc15800e050 con 0x7fc160105ce0 2024-03-30T04:03:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.335+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1601a6020 con 0x7fc160105ce0 2024-03-30T04:03:26.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.338+0000 7fc157400700 1 -- 172.21.15.23:0/4203388753 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fc158004520 con 0x7fc160105ce0 2024-03-30T04:03:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.339+0000 7fc1797bab80 1 Processor -- start 2024-03-30T04:03:26.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.339+0000 7fc157400700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 0x7fc1440a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc157400700 1 -- 172.21.15.23:0/4203388753 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc1580cccd0 con 0x7fc160105ce0 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 -- start start 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc15e000700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 0x7fc1440a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba4878b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba48a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba3d5990 con 0x5573ba487df0 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba3d3350 con 0x5573ba47c5d0 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc1797bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba3cec40 con 0x5573ba47cef0 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc15ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc15ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33232/0 (socket says 172.21.15.23:33232) 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.340+0000 7fc15ea00700 1 -- 172.21.15.23:0/1607321657 learned_addr learned my addr 172.21.15.23:0/1607321657 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15f400700 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba48a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15e000700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 0x7fc1440a3870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009990 tx=0x7fc14c005ab0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 -- 172.21.15.23:0/1607321657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 msgr2=0x5573ba4878b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba4878b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 -- 172.21.15.23:0/1607321657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 msgr2=0x5573ba48a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba48a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 -- 172.21.15.23:0/1607321657 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573ba2503b0 con 0x5573ba47c5d0 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc15ea00700 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc150009d80 tx=0x7fc150011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3da2377653466e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc14be00700 1 -- 172.21.15.23:0/1607321657 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150004230 con 0x5573ba47c5d0 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc14be00700 1 -- 172.21.15.23:0/1607321657 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc150004390 con 0x5573ba47c5d0 2024-03-30T04:03:26.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.341+0000 7fc14be00700 1 -- 172.21.15.23:0/1607321657 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150019880 con 0x5573ba47c5d0 2024-03-30T04:03:26.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 -- 172.21.15.23:0/1607321657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 msgr2=0x5573ba47c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc150009d80 tx=0x7fc150011720 comp rx=0 tx=0).stop 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 -- 172.21.15.23:0/1607321657 shutdown_connections 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba48a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba47c9b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 --2- 172.21.15.23:0/1607321657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba4878b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 -- 172.21.15.23:0/1607321657 >> 172.21.15.23:0/1607321657 conn(0x5573ba3dc6a0 msgr2=0x5573ba478a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 -- 172.21.15.23:0/1607321657 shutdown_connections 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.342+0000 7fc1797bab80 1 -- 172.21.15.23:0/1607321657 wait complete. 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 -- start start 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 Processor -- start 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba4877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47cef0 0x5573ba484ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba36ad80 con 0x5573ba487df0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba36aa20 con 0x5573ba47cef0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.343+0000 7fc1797bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba2526a0 con 0x5573ba47c5d0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc1797bab80 10 client.0 _ll_register_callbacks cb 0x5573ba374470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39652/0 (socket says 172.21.15.23:39652) 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 -- 172.21.15.23:0/2522133340 learned_addr learned my addr 172.21.15.23:0/2522133340 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 -- 172.21.15.23:0/2522133340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 msgr2=0x5573ba4877f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba4877f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 -- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47cef0 msgr2=0x5573ba484ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47cef0 0x5573ba484ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 -- 172.21.15.23:0/2522133340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1500099e0 con 0x5573ba487df0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc15f400700 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc158082fd0 tx=0x7fc158099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4ee68384c5b33d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.344+0000 7fc148c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc158099ed0 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc1580997d0 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573ba38f380 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc15809d600 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 10 client.16110 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5573ba1fcb60 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5573ba38f380 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fc1580a17c0 con 0x5573ba487df0 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 20 client.16110 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.345+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4978d0 0x5573ba499cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.346+0000 7fc1797bab80 4 client.16110 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:26.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.346+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573ba3c7700 con 0x5573ba4978d0 2024-03-30T04:03:26.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.346+0000 7fc15ea00700 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4978d0 0x5573ba499cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.347+0000 7fc15ea00700 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4978d0 0x5573ba499cf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c3e90638e21fd4 server_cookie=152dfd0fa7ca6c5b in_seq=0 out_seq=0 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.347+0000 7fc148c00700 10 client.16110 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc148c00700 1 -- 172.21.15.23:0/2522133340 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5573ba3c7700 con 0x5573ba4978d0 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc148c00700 10 client.16110 handle_command_reply: tid=0 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 client.16110 shutdown 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 msgr2=0x5573ba4850f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc158082fd0 tx=0x7fc158099d10 comp rx=0 tx=0).stop 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 shutdown_connections 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4850f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4978d0 0x5573ba499cf0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba4877f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 --2- 172.21.15.23:0/2522133340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47cef0 0x5573ba484ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 >> 172.21.15.23:0/2522133340 conn(0x5573ba3dc6a0 msgr2=0x5573ba47e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc148c00700 0 client.16110 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 shutdown_connections 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 1 -- 172.21.15.23:0/2522133340 wait complete. 2024-03-30T04:03:26.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.349+0000 7fc1797bab80 20 client.16110 trim_cache size 0 max 16384 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 Processor -- start 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 -- start start 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba48ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba487df0 0x5573ba47fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba47a370 con 0x5573ba47cef0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba47d850 con 0x5573ba487df0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc1797bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba48efb0 con 0x5573ba47c5d0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc15e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc15e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39656/0 (socket says 172.21.15.23:39656) 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.364+0000 7fc15e000700 1 -- 172.21.15.23:0/4079897092 learned_addr learned my addr 172.21.15.23:0/4079897092 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15f400700 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba487df0 0x5573ba47fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15ea00700 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba48ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 -- 172.21.15.23:0/4079897092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 msgr2=0x5573ba48ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba48ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 -- 172.21.15.23:0/4079897092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba487df0 msgr2=0x5573ba47fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba487df0 0x5573ba47fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 -- 172.21.15.23:0/4079897092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573ba394ea0 con 0x5573ba47cef0 2024-03-30T04:03:26.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc15e000700 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0099c0 tx=0x7fc14c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf88ae3daee3298 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc154c00700 1 -- 172.21.15.23:0/4079897092 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00ba10 con 0x5573ba47cef0 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc154c00700 1 -- 172.21.15.23:0/4079897092 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc14c01e040 con 0x5573ba47cef0 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.365+0000 7fc154c00700 1 -- 172.21.15.23:0/4079897092 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c02f6b0 con 0x5573ba47cef0 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 -- 172.21.15.23:0/4079897092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 msgr2=0x5573ba478530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0099c0 tx=0x7fc14c01f000 comp rx=0 tx=0).stop 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 -- 172.21.15.23:0/4079897092 shutdown_connections 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba487df0 0x5573ba47fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47c5d0 0x5573ba48ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 --2- 172.21.15.23:0/4079897092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba47cef0 0x5573ba478530 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 -- 172.21.15.23:0/4079897092 >> 172.21.15.23:0/4079897092 conn(0x5573ba3dc6a0 msgr2=0x5573ba4998e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 -- 172.21.15.23:0/4079897092 shutdown_connections 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.366+0000 7fc1797bab80 1 -- 172.21.15.23:0/4079897092 wait complete. 2024-03-30T04:03:26.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.367+0000 7fc1797bab80 1 -- start start 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.367+0000 7fc1797bab80 1 Processor -- start 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.367+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba48de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba48e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba4eac40 con 0x5573ba487df0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba4ead80 con 0x5573ba47c5d0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ba4eaf10 con 0x5573ba47cef0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc1797bab80 10 client.0 _ll_register_callbacks cb 0x5573ba374470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba48de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39670/0 (socket says 172.21.15.23:39670) 2024-03-30T04:03:26.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15f400700 1 -- 172.21.15.23:0/3079495571 learned_addr learned my addr 172.21.15.23:0/3079495571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15e000700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba48e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 -- 172.21.15.23:0/3079495571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 msgr2=0x5573ba48e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba48e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 -- 172.21.15.23:0/3079495571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 msgr2=0x5573ba4ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15ea00700 1 -- 172.21.15.23:0/3079495571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16010e550 con 0x5573ba47c5d0 2024-03-30T04:03:26.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.368+0000 7fc15f400700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc15ea00700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba48de80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc150004640 tx=0x7fc150005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a7dd85fee450d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc148c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150029130 con 0x5573ba47c5d0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc150004390 con 0x5573ba47c5d0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc150029290 con 0x5573ba47c5d0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc150029450 con 0x5573ba47c5d0 2024-03-30T04:03:26.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc1797bab80 10 client.26093 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5573ba254cd0 con 0x5573ba47c5d0 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.369+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc150004390 con 0x5573ba47c5d0 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fc15002f020 con 0x5573ba47c5d0 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 10 client.26093 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 20 client.26093 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4ec730 0x5573ba4eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 4 client.26093 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5573ba21dcf0 con 0x5573ba4ec730 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.370+0000 7fc15e000700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4ec730 0x5573ba4eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc15e000700 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4ec730 0x5573ba4eeb50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfeef516e281036d server_cookie=d07324af2d118de4 in_seq=0 out_seq=0 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc148c00700 10 client.26093 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc148c00700 1 -- 172.21.15.23:0/3079495571 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5573ba21dcf0 con 0x5573ba4ec730 2024-03-30T04:03:26.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc148c00700 10 client.26093 handle_command_reply: tid=0 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 client.26093 shutdown 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 msgr2=0x5573ba48de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba48de80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc150004640 tx=0x7fc150005e40 comp rx=0 tx=0).stop 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 shutdown_connections 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5573ba487df0 0x5573ba4ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5573ba47c5d0 0x5573ba48de80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5573ba4ec730 0x5573ba4eeb50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 --2- 172.21.15.23:0/3079495571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5573ba47cef0 0x5573ba48e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 >> 172.21.15.23:0/3079495571 conn(0x5573ba3dc6a0 msgr2=0x5573ba47fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc148c00700 0 client.26093 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.371+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 shutdown_connections 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc1797bab80 1 -- 172.21.15.23:0/3079495571 wait complete. 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc1797bab80 20 client.26093 trim_cache size 0 max 16384 2024-03-30T04:03:26.376 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 msgr2=0x7fc1440a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 0x7fc1440a3870 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009990 tx=0x7fc14c005ab0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 msgr2=0x7fc16019dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc15800c370 tx=0x7fc15800c680 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 shutdown_connections 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc160105ce0 0x7fc16019dd70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc160067e30 0x7fc1601994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc1440a13b0 0x7fc1440a3870 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 --2- 172.21.15.23:0/4203388753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc1600687f0 0x7fc160199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:26.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 >> 172.21.15.23:0/4203388753 conn(0x7fc160076d40 msgr2=0x7fc1600fe920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:26.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.372+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 shutdown_connections 2024-03-30T04:03:26.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:26.373+0000 7fc165c00700 1 -- 172.21.15.23:0/4203388753 wait complete. 2024-03-30T04:03:27.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 2922 8.37 MB/sec warmup 4 sec latency 2964.683 ms 2024-03-30T04:03:27.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:26 smithi023 ceph-mon[38254]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:27.392 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:03:27.393 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:03:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:26 smithi154 ceph-mon[79161]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:27.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:27 smithi193 ceph-mon[79581]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 -- 172.21.15.23:0/293559202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069c10 msgr2=0x7fa284105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 --2- 172.21.15.23:0/293559202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069c10 0x7fa284105980 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa270009990 tx=0x7fa270009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 -- 172.21.15.23:0/293559202 shutdown_connections 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 --2- 172.21.15.23:0/293559202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa28410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 --2- 172.21.15.23:0/293559202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa284069260 0x7fa284069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 --2- 172.21.15.23:0/293559202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069c10 0x7fa284105980 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 -- 172.21.15.23:0/293559202 >> 172.21.15.23:0/293559202 conn(0x7fa284076d80 msgr2=0x7fa284077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 -- 172.21.15.23:0/293559202 shutdown_connections 2024-03-30T04:03:27.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.613+0000 7fa28a200700 1 -- 172.21.15.23:0/293559202 wait complete. 2024-03-30T04:03:27.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.614+0000 7fa28a200700 1 Processor -- start 2024-03-30T04:03:27.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.614+0000 7fa28a200700 1 -- start start 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 0x7fa2841994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa2841999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa28419dd80 0x7fa28419e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419e800 con 0x7fa2841060b0 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419e970 con 0x7fa284069260 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa28a200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419eb30 con 0x7fa28419dd80 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa283400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 0x7fa2841994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa2841999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa283400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 0x7fa2841994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33276/0 (socket says 172.21.15.23:33276) 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa283400700 1 -- 172.21.15.23:0/1146098508 learned_addr learned my addr 172.21.15.23:0/1146098508 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 -- 172.21.15.23:0/1146098508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa28419dd80 msgr2=0x7fa28419e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa28419dd80 0x7fa28419e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 -- 172.21.15.23:0/1146098508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 msgr2=0x7fa2841994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 0x7fa2841994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 -- 172.21.15.23:0/1146098508 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700095f0 con 0x7fa2841060b0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.615+0000 7fa282a00700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa2841999e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2700098e0 tx=0x7fa270005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.616+0000 7fa27be00700 1 -- 172.21.15.23:0/1146098508 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27001d070 con 0x7fa2841060b0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.616+0000 7fa27be00700 1 -- 172.21.15.23:0/1146098508 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa270022470 con 0x7fa2841060b0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.616+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2841a6030 con 0x7fa2841060b0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.616+0000 7fa27be00700 1 -- 172.21.15.23:0/1146098508 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270017650 con 0x7fa2841060b0 2024-03-30T04:03:27.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.616+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2841a64b0 con 0x7fa2841060b0 2024-03-30T04:03:27.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.619+0000 7fa27be00700 1 -- 172.21.15.23:0/1146098508 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fa2700225e0 con 0x7fa2841060b0 2024-03-30T04:03:27.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.620+0000 7fa27be00700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 0x7fa2680a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.620+0000 7fa27be00700 1 -- 172.21.15.23:0/1146098508 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fa2700ce500 con 0x7fa2841060b0 2024-03-30T04:03:27.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa283400700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 0x7fa2680a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 Processor -- start 2024-03-30T04:03:27.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 -- start start 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907c91a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c9ce70 0x55f907c9f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907beaa10 con 0x55f907c91650 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907be83d0 con 0x55f907c91f70 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.621+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907be3cc0 con 0x55f907c9ce70 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33278/0 (socket says 172.21.15.23:33278) 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa283400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907c91a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 -- 172.21.15.23:0/2186153714 learned_addr learned my addr 172.21.15.23:0/2186153714 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa283e00700 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c9ce70 0x55f907c9f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 -- 172.21.15.23:0/2186153714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c9ce70 msgr2=0x55f907c9f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c9ce70 0x55f907c9f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 -- 172.21.15.23:0/2186153714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 msgr2=0x55f907c91a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907c91a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 -- 172.21.15.23:0/2186153714 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f907a65430 con 0x55f907c91f70 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa283400700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 0x7fa2680a38f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa274009d20 tx=0x7fa274009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa282a00700 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa270009990 tx=0x7fa270097540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bc813c849f76abc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa26fe00700 1 -- 172.21.15.23:0/2186153714 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27009b0d0 con 0x55f907c91f70 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa26fe00700 1 -- 172.21.15.23:0/2186153714 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa27009ef00 con 0x55f907c91f70 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.622+0000 7fa26fe00700 1 -- 172.21.15.23:0/2186153714 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27009b590 con 0x55f907c91f70 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2186153714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 msgr2=0x55f907c9c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa270009990 tx=0x7fa270097540 comp rx=0 tx=0).stop 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2186153714 shutdown_connections 2024-03-30T04:03:27.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c9ce70 0x55f907c9f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907c91a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2186153714 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c91f70 0x55f907c9c930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2186153714 >> 172.21.15.23:0/2186153714 conn(0x55f907bf1720 msgr2=0x55f907c8db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2186153714 shutdown_connections 2024-03-30T04:03:27.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.623+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2186153714 wait complete. 2024-03-30T04:03:27.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.624+0000 7fa29dd5fb80 1 -- start start 2024-03-30T04:03:27.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.624+0000 7fa29dd5fb80 1 Processor -- start 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c9d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907ca82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907b7fe00 con 0x55f907c91f70 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907b7faa0 con 0x55f907c9ce70 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907a67720 con 0x55f907c91650 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa29dd5fb80 10 client.0 _ll_register_callbacks cb 0x55f907b894f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa283400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa283e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907ca82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa282a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c9d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa283400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:46910/0 (socket says 172.21.15.23:46910) 2024-03-30T04:03:27.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa283400700 1 -- 172.21.15.23:0/2460160001 learned_addr learned my addr 172.21.15.23:0/2460160001 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.625+0000 7fa283e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907ca82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33286/0 (socket says 172.21.15.23:33286) 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 -- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 msgr2=0x55f907ca82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907ca82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 -- 172.21.15.23:0/2460160001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 msgr2=0x55f907c9d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c9d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 -- 172.21.15.23:0/2460160001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2841a64b0 con 0x55f907c91650 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa283400700 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa27401ec10 tx=0x7fa2740047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3152ec8cb104fc01 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa26cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27400ec50 con 0x55f907c91650 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.626+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f907ba4400 con 0x55f907c91650 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fa27400edb0 con 0x55f907c91650 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa274004830 con 0x55f907c91650 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 10 client.26092 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:27.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f907a11be0 con 0x55f907c91650 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f907ba4400 con 0x55f907c91650 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fa27402e070 con 0x55f907c91650 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 10 client.26092 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 10 client.26092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 20 client.26092 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907cac600 0x55f907caea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 4 client.26092 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.627+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f907bdc780 con 0x55f907cac600 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.628+0000 7fa282a00700 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907cac600 0x55f907caea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.628+0000 7fa282a00700 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907cac600 0x55f907caea20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac8b029782850478 server_cookie=a8708631c534d8c6 in_seq=0 out_seq=0 2024-03-30T04:03:27.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.628+0000 7fa26cc00700 10 client.26092 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa26cc00700 1 -- 172.21.15.23:0/2460160001 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f907bdc780 con 0x55f907cac600 2024-03-30T04:03:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa26cc00700 10 client.26092 handle_command_reply: tid=0 2024-03-30T04:03:27.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 client.26092 shutdown 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 msgr2=0x55f907c9d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa27401ec10 tx=0x7fa2740047c0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 shutdown_connections 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907ca82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907cac600 0x55f907caea20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907c9d250 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/2460160001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c9d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 >> 172.21.15.23:0/2460160001 conn(0x55f907bf1720 msgr2=0x55f907c93740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa26cc00700 0 client.26092 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 shutdown_connections 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/2460160001 wait complete. 2024-03-30T04:03:27.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.629+0000 7fa29dd5fb80 20 client.26092 trim_cache size 0 max 16384 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.647+0000 7fa29dd5fb80 1 Processor -- start 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.647+0000 7fa29dd5fb80 1 -- start start 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907ca2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907c97920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907c928d0 con 0x55f907c91f70 2024-03-30T04:03:27.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907c8e6e0 con 0x55f907c9ce70 2024-03-30T04:03:27.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907c8f1e0 con 0x55f907c91650 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa283e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907c97920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39708/0 (socket says 172.21.15.23:39708) 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 -- 172.21.15.23:0/4021932866 learned_addr learned my addr 172.21.15.23:0/4021932866 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa283400700 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907ca2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 -- 172.21.15.23:0/4021932866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 msgr2=0x55f907ca2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907ca2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 -- 172.21.15.23:0/4021932866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 msgr2=0x55f907c97920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907c97920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.648+0000 7fa282a00700 1 -- 172.21.15.23:0/4021932866 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f907cb3650 con 0x55f907c91f70 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.649+0000 7fa282a00700 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa2700cb810 tx=0x7fa2700b5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43f1a75c4cb82b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.649+0000 7fa279200700 1 -- 172.21.15.23:0/4021932866 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700b9040 con 0x55f907c91f70 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.649+0000 7fa279200700 1 -- 172.21.15.23:0/4021932866 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa27009b0d0 con 0x55f907c91f70 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.649+0000 7fa279200700 1 -- 172.21.15.23:0/4021932866 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700b91a0 con 0x55f907c91f70 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/4021932866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 msgr2=0x55f907c973e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa2700cb810 tx=0x7fa2700b5040 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/4021932866 shutdown_connections 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907c97920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91650 0x55f907ca2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/4021932866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91f70 0x55f907c973e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/4021932866 >> 172.21.15.23:0/4021932866 conn(0x55f907bf1720 msgr2=0x55f907c9b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/4021932866 shutdown_connections 2024-03-30T04:03:27.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/4021932866 wait complete. 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 -- start start 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.650+0000 7fa29dd5fb80 1 Processor -- start 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907cfa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907cf45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907cf4b80 con 0x55f907c91650 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907cf4cf0 con 0x55f907c9ce70 2024-03-30T04:03:27.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f907cf4eb0 con 0x55f907c91f70 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa29dd5fb80 10 client.0 _ll_register_callbacks cb 0x55f907b894f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa282a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa283400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907cfa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa282a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:46928/0 (socket says 172.21.15.23:46928) 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa282a00700 1 -- 172.21.15.23:0/744441353 learned_addr learned my addr 172.21.15.23:0/744441353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.651+0000 7fa283e00700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907cf45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 -- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 msgr2=0x55f907cf45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907cf45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 -- 172.21.15.23:0/744441353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 msgr2=0x55f907cfa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907cfa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 -- 172.21.15.23:0/744441353 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f907a65430 con 0x55f907c91f70 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa283400700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907cfa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:27.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa283e00700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907cf45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa26cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa282a00700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa270099420 tx=0x7fa270009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b84698c164973ef server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700b9270 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fa2700b8040 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa27009ef00 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.652+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2700b93d0 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 10 client.26098 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f907ba9f20 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa27009ef00 con 0x55f907c91f70 2024-03-30T04:03:27.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fa27009a6e0 con 0x55f907c91f70 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 10 client.26098 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 10 client.26098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 20 client.26098 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907d01ba0 0x55f907d03f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 4 client.26098 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f907a32d70 con 0x55f907d01ba0 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.653+0000 7fa283400700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907d01ba0 0x55f907d03f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.654+0000 7fa283400700 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907d01ba0 0x55f907d03f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96f3d01665f81063 server_cookie=e1124e506d614c20 in_seq=0 out_seq=0 2024-03-30T04:03:27.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.654+0000 7fa26cc00700 10 client.26098 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.655+0000 7fa26cc00700 1 -- 172.21.15.23:0/744441353 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f907a32d70 con 0x55f907d01ba0 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.655+0000 7fa26cc00700 10 client.26098 handle_command_reply: tid=0 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.655+0000 7fa29dd5fb80 1 client.26098 shutdown 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 msgr2=0x55f907cfa680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa270099420 tx=0x7fa270009290 comp rx=0 tx=0).stop 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 shutdown_connections 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f907c9ce70 0x55f907cf45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f907c91650 0x55f907cfa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f907d01ba0 0x55f907d03f90 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 --2- 172.21.15.23:0/744441353 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f907c91f70 0x55f907cfa680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa26cc00700 0 client.26098 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 >> 172.21.15.23:0/744441353 conn(0x55f907bf1720 msgr2=0x55f907c8d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 shutdown_connections 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 1 -- 172.21.15.23:0/744441353 wait complete. 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.656+0000 7fa29dd5fb80 20 client.26098 trim_cache size 0 max 16384 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bb0800b9-2485-46a7-9d14-f167168571a1", 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:03:27.660 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:03:27.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 msgr2=0x7fa2680a38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 0x7fa2680a38f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa274009d20 tx=0x7fa274009260 comp rx=0 tx=0).stop 2024-03-30T04:03:27.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 msgr2=0x7fa2841999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa2841999e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fa2700098e0 tx=0x7fa270005740 comp rx=0 tx=0).stop 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 shutdown_connections 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa28419dd80 0x7fa28419e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa2680a1430 0x7fa2680a38f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa284069260 0x7fa2841994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 --2- 172.21.15.23:0/1146098508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa2841060b0 0x7fa2841999e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 >> 172.21.15.23:0/1146098508 conn(0x7fa284076d80 msgr2=0x7fa2840fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 shutdown_connections 2024-03-30T04:03:27.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.657+0000 7fa28a200700 1 -- 172.21.15.23:0/1146098508 wait complete. 2024-03-30T04:03:27.676 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:03:27.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 -- 172.21.15.23:0/2026944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c100f60 msgr2=0x7fe49c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/2026944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c100f60 0x7fe49c101340 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0098f0 tx=0x7fe48c009c00 comp rx=0 tx=0).stop 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 -- 172.21.15.23:0/2026944685 shutdown_connections 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/2026944685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c106080 0x7fe49c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/2026944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c100f60 0x7fe49c101340 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/2026944685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c101910 0x7fe49c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.880+0000 7fe4a2c00700 1 -- 172.21.15.23:0/2026944685 >> 172.21.15.23:0/2026944685 conn(0x7fe49c0fc7c0 msgr2=0x7fe49c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.881+0000 7fe4a2c00700 1 -- 172.21.15.23:0/2026944685 shutdown_connections 2024-03-30T04:03:27.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.881+0000 7fe4a2c00700 1 -- 172.21.15.23:0/2026944685 wait complete. 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 Processor -- start 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 -- start start 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c106080 0x7fe49c1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c19dd50 0x7fe49c19e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c19e7d0 con 0x7fe49c106080 2024-03-30T04:03:27.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c19e940 con 0x7fe49c100f60 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe4a2c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c19eb00 con 0x7fe49c19dd50 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.882+0000 7fe49be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33334/0 (socket says 172.21.15.23:33334) 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c106080 0x7fe49c1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 -- 172.21.15.23:0/482930236 learned_addr learned my addr 172.21.15.23:0/482930236 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c19dd50 0x7fe49c19e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 -- 172.21.15.23:0/482930236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c19dd50 msgr2=0x7fe49c19e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c19dd50 0x7fe49c19e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 -- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c106080 msgr2=0x7fe49c1999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c106080 0x7fe49c1999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 -- 172.21.15.23:0/482930236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880095f0 con 0x7fe49c100f60 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe49be00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe48c005950 tx=0x7fe48c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.883+0000 7fe498c00700 1 -- 172.21.15.23:0/482930236 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c01c070 con 0x7fe49c100f60 2024-03-30T04:03:27.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.884+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c009520 con 0x7fe49c100f60 2024-03-30T04:03:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.884+0000 7fe498c00700 1 -- 172.21.15.23:0/482930236 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe48c00b9c0 con 0x7fe49c100f60 2024-03-30T04:03:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.884+0000 7fe498c00700 1 -- 172.21.15.23:0/482930236 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0038e0 con 0x7fe49c100f60 2024-03-30T04:03:27.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.884+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c1a62a0 con 0x7fe49c100f60 2024-03-30T04:03:27.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.887+0000 7fe498c00700 1 -- 172.21.15.23:0/482930236 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 26) v1 ==== 136098+0+0 (secure 0 0 0) 0x7fe48c020030 con 0x7fe49c100f60 2024-03-30T04:03:27.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.888+0000 7fe4b6790b80 1 Processor -- start 2024-03-30T04:03:27.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.888+0000 7fe4b6790b80 1 -- start start 2024-03-30T04:03:27.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.888+0000 7fe498c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 0x7fe4800a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe498c00700 1 -- 172.21.15.23:0/482930236 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe48c0ce080 con 0x7fe49c100f60 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6db1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff626990 con 0x55d6ff6cdef0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff624350 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff61fc40 con 0x55d6ff6d8df0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 0x7fe4800a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39744/0 (socket says 172.21.15.23:39744) 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 -- 172.21.15.23:0/146024265 learned_addr learned my addr 172.21.15.23:0/146024265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6db1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 -- 172.21.15.23:0/146024265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 msgr2=0x55d6ff6db1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6db1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 -- 172.21.15.23:0/146024265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cd5d0 msgr2=0x55d6ff6cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe49b400700 1 -- 172.21.15.23:0/146024265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ff4a13b0 con 0x55d6ff6cdef0 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.889+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6db1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.890+0000 7fe49b400700 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe488009f50 tx=0x7fe488011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670daeb94a5ede6d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.890+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 0x7fe4800a3870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe494009ec0 tx=0x7fe494009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.890+0000 7fe491200700 1 -- 172.21.15.23:0/146024265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488004310 con 0x55d6ff6cdef0 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.890+0000 7fe491200700 1 -- 172.21.15.23:0/146024265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe488004470 con 0x55d6ff6cdef0 2024-03-30T04:03:27.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.890+0000 7fe491200700 1 -- 172.21.15.23:0/146024265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488003680 con 0x55d6ff6cdef0 2024-03-30T04:03:27.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- 172.21.15.23:0/146024265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 msgr2=0x55d6ff6d88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe488009f50 tx=0x7fe488011720 comp rx=0 tx=0).stop 2024-03-30T04:03:27.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- 172.21.15.23:0/146024265 shutdown_connections 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6db1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/146024265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d88b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- 172.21.15.23:0/146024265 >> 172.21.15.23:0/146024265 conn(0x55d6ff62d6a0 msgr2=0x55d6ff6c9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- 172.21.15.23:0/146024265 shutdown_connections 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- 172.21.15.23:0/146024265 wait complete. 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.891+0000 7fe4b6790b80 1 -- start start 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 Processor -- start 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff5bbd80 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff5bba20 con 0x55d6ff6d8df0 2024-03-30T04:03:27.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff4a36a0 con 0x55d6ff6cdef0 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe4b6790b80 10 client.0 _ll_register_callbacks cb 0x55d6ff5c5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe49be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe49b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe49be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39758/0 (socket says 172.21.15.23:39758) 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe49b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:46968/0 (socket says 172.21.15.23:46968) 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.892+0000 7fe49be00700 1 -- 172.21.15.23:0/103106260 learned_addr learned my addr 172.21.15.23:0/103106260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 -- 172.21.15.23:0/103106260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 msgr2=0x55d6ff6d5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 -- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 msgr2=0x55d6ff6d60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 -- 172.21.15.23:0/103106260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe488009b80 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe49be00700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe48c09aba0 tx=0x7fe48c09a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d89937cf03effad server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe485600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0b2070 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe48c09ac10 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.893+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d6ff5e0380 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe48c09ea40 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 10 client.16146 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d6ff44db60 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe48c064080 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fe48c09ae70 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 10 client.16146 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 20 client.16146 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff6e8550 0x55d6ff6ea940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 4 client.16146 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:27.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.894+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6ff618700 con 0x55d6ff6e8550 2024-03-30T04:03:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.896+0000 7fe49b400700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff6e8550 0x55d6ff6ea940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.899+0000 7fe49b400700 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff6e8550 0x55d6ff6ea940 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b670d7107ad1033 server_cookie=cd918b31664ff429 in_seq=0 out_seq=0 2024-03-30T04:03:27.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.899+0000 7fe485600700 10 client.16146 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe485600700 1 -- 172.21.15.23:0/103106260 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d6ff618700 con 0x55d6ff6e8550 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe485600700 10 client.16146 handle_command_reply: tid=0 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 client.16146 shutdown 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 msgr2=0x55d6ff6d87a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe48c09aba0 tx=0x7fe48c09a840 comp rx=0 tx=0).stop 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 shutdown_connections 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6d87a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff6e8550 0x55d6ff6ea940 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/103106260 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe485600700 0 client.16146 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 >> 172.21.15.23:0/103106260 conn(0x55d6ff62d6a0 msgr2=0x55d6ff6cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 shutdown_connections 2024-03-30T04:03:27.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.900+0000 7fe4b6790b80 1 -- 172.21.15.23:0/103106260 wait complete. 2024-03-30T04:03:27.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.901+0000 7fe4b6790b80 20 client.16146 trim_cache size 0 max 16384 2024-03-30T04:03:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 Processor -- start 2024-03-30T04:03:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 -- start start 2024-03-30T04:03:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6dfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff6d6480 con 0x55d6ff6cdef0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff6ce880 con 0x55d6ff6d8df0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff6d66a0 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe49b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.920+0000 7fe4a0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39764/0 (socket says 172.21.15.23:39764) 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 -- 172.21.15.23:0/63561780 learned_addr learned my addr 172.21.15.23:0/63561780 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33374/0 (socket says 172.21.15.23:33374) 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49be00700 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6dfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 -- 172.21.15.23:0/63561780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cd5d0 msgr2=0x55d6ff6dfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6dfa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 -- 172.21.15.23:0/63561780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 msgr2=0x55d6ff6d1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 -- 172.21.15.23:0/63561780 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6ff5e5ea0 con 0x55d6ff6cdef0 2024-03-30T04:03:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe49b400700 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe488006010 tx=0x7fe4880099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1738ba0ebd9bcecf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe491c00700 1 -- 172.21.15.23:0/63561780 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488029070 con 0x55d6ff6cdef0 2024-03-30T04:03:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe491c00700 1 -- 172.21.15.23:0/63561780 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe488005a00 con 0x55d6ff6cdef0 2024-03-30T04:03:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.921+0000 7fe491c00700 1 -- 172.21.15.23:0/63561780 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488024680 con 0x55d6ff6cdef0 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 -- 172.21.15.23:0/63561780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 msgr2=0x55d6ff6d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe488006010 tx=0x7fe4880099f0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 -- 172.21.15.23:0/63561780 shutdown_connections 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6d1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6dfa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/63561780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6d0df0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 -- 172.21.15.23:0/63561780 >> 172.21.15.23:0/63561780 conn(0x55d6ff62d6a0 msgr2=0x55d6ff6c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 -- 172.21.15.23:0/63561780 shutdown_connections 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.922+0000 7fe4b6790b80 1 -- 172.21.15.23:0/63561780 wait complete. 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 -- start start 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 Processor -- start 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6decc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff73c320 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff73c460 con 0x55d6ff6cdef0 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6ff73c620 con 0x55d6ff6d8df0 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.923+0000 7fe4b6790b80 10 client.0 _ll_register_callbacks cb 0x55d6ff5c5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39776/0 (socket says 172.21.15.23:39776) 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 -- 172.21.15.23:0/1503531894 learned_addr learned my addr 172.21.15.23:0/1503531894 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:27.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe4a0c00700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6decc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 -- 172.21.15.23:0/1503531894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 msgr2=0x55d6ff6decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6decc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 -- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cdef0 msgr2=0x55d6ff6de780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6de780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 -- 172.21.15.23:0/1503531894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48801d040 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe49be00700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe48c02b000 tx=0x7fe48c0054e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90c28fff6824920 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.924+0000 7fe485600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0b2030 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe48c0b3070 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe488004470 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe48c0cb1d0 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 10 client.16152 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:27.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d6ff4a5cd0 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe488004470 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fe48c0cb570 con 0x55d6ff6cd5d0 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 10 client.16152 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 20 client.16152 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff73d920 0x55d6ff73fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 4 client.16152 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:03:27.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.925+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d6ff46ecf0 con 0x55d6ff73d920 2024-03-30T04:03:27.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.928+0000 7fe49b400700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff73d920 0x55d6ff73fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:27.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.929+0000 7fe49b400700 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff73d920 0x55d6ff73fd10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2638360fd18b74 server_cookie=99314344fdac29a0 in_seq=0 out_seq=0 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.929+0000 7fe485600700 10 client.16152 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.929+0000 7fe485600700 1 -- 172.21.15.23:0/1503531894 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d6ff46ecf0 con 0x55d6ff73d920 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.929+0000 7fe485600700 10 client.16152 handle_command_reply: tid=0 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.929+0000 7fe4b6790b80 1 client.16152 shutdown 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 msgr2=0x55d6ff6de240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fe48c02b000 tx=0x7fe48c0054e0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 shutdown_connections 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d6ff73d920 0x55d6ff73fd10 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d6ff6d8df0 0x55d6ff6decc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d6ff6cd5d0 0x55d6ff6de240 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe485600700 0 client.16152 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 --2- 172.21.15.23:0/1503531894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d6ff6cdef0 0x55d6ff6de780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 >> 172.21.15.23:0/1503531894 conn(0x55d6ff62d6a0 msgr2=0x55d6ff6d0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 shutdown_connections 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 1 -- 172.21.15.23:0/1503531894 wait complete. 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4b6790b80 20 client.16152 trim_cache size 0 max 16384 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout: "bb0800b9-2485-46a7-9d14-f167168571a1": { 2024-03-30T04:03:27.932 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:03:27.933 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bb0800b9-2485-46a7-9d14-f167168571a1", 2024-03-30T04:03:27.933 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:03:27.933 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:03:27.933 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:03:27.933 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 msgr2=0x7fe4800a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.930+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 0x7fe4800a3870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe494009ec0 tx=0x7fe494009920 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 msgr2=0x7fe49c199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe48c005950 tx=0x7fe48c00fa00 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 shutdown_connections 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe49c19dd50 0x7fe49c19e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4800a13b0 0x7fe4800a3870 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe49c100f60 0x7fe49c199470 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 --2- 172.21.15.23:0/482930236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe49c106080 0x7fe49c1999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 >> 172.21.15.23:0/482930236 conn(0x7fe49c0fc7c0 msgr2=0x7fe49c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:27.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 shutdown_connections 2024-03-30T04:03:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:27.931+0000 7fe4a2c00700 1 -- 172.21.15.23:0/482930236 wait complete. 2024-03-30T04:03:27.949 INFO:tasks.cephfs.filesystem:scrub status for tag:bb0800b9-2485-46a7-9d14-f167168571a1 - {'path': '/', 'tag': 'bb0800b9-2485-46a7-9d14-f167168571a1', 'options': 'recursive,force'} 2024-03-30T04:03:28.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 4433 8.14 MB/sec warmup 5 sec latency 205.534 ms 2024-03-30T04:03:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:28 smithi154 ceph-mon[79161]: pgmap v49: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 130 KiB/s wr, 28 op/s 2024-03-30T04:03:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:28 smithi154 ceph-mon[79161]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-30T04:03:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:28 smithi193 ceph-mon[79581]: pgmap v49: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 130 KiB/s wr, 28 op/s 2024-03-30T04:03:28.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:28 smithi193 ceph-mon[79581]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-30T04:03:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:28 smithi023 ceph-mon[38254]: pgmap v49: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 130 KiB/s wr, 28 op/s 2024-03-30T04:03:28.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:28 smithi023 ceph-mon[38254]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-03-30T04:03:29.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 7419 9.81 MB/sec warmup 6 sec latency 6.504 ms 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:29 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:03:29.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:03:29.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:03:29.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:29 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:29 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:03:30.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 9685 9.60 MB/sec warmup 7 sec latency 10.230 ms 2024-03-30T04:03:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:30 smithi193 ceph-mon[79581]: pgmap v50: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 130 KiB/s wr, 24 op/s 2024-03-30T04:03:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:30 smithi023 ceph-mon[38254]: pgmap v50: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 130 KiB/s wr, 24 op/s 2024-03-30T04:03:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:30 smithi154 ceph-mon[79161]: pgmap v50: 129 pgs: 129 active+clean; 2.7 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 130 KiB/s wr, 24 op/s 2024-03-30T04:03:31.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 11585 9.89 MB/sec warmup 8 sec latency 17.739 ms 2024-03-30T04:03:32.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 14574 10.82 MB/sec warmup 9 sec latency 6.578 ms 2024-03-30T04:03:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:32 smithi154 ceph-mon[79161]: pgmap v51: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 712 KiB/s wr, 97 op/s 2024-03-30T04:03:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:32 smithi193 ceph-mon[79581]: pgmap v51: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 712 KiB/s wr, 97 op/s 2024-03-30T04:03:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:32 smithi023 ceph-mon[38254]: pgmap v51: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 712 KiB/s wr, 97 op/s 2024-03-30T04:03:33.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 17199 11.00 MB/sec warmup 10 sec latency 7.587 ms 2024-03-30T04:03:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:33 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:03:33.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:33 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:03:33.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:33 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:03:34.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 18566 10.64 MB/sec warmup 11 sec latency 203.491 ms 2024-03-30T04:03:34.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:34 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:03:34.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:34 smithi154 ceph-mon[79161]: pgmap v52: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:34 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:03:34.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:34 smithi193 ceph-mon[79581]: pgmap v52: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:34.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:34 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:03:34.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:34 smithi023 ceph-mon[38254]: pgmap v52: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:35.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 21589 11.28 MB/sec warmup 12 sec latency 7.228 ms 2024-03-30T04:03:36.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 23742 10.99 MB/sec warmup 13 sec latency 17.864 ms 2024-03-30T04:03:37.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 25801 11.12 MB/sec warmup 14 sec latency 7.914 ms 2024-03-30T04:03:37.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:36 smithi023 ceph-mon[38254]: pgmap v53: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:36 smithi154 ceph-mon[79161]: pgmap v53: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:37.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:36 smithi193 ceph-mon[79581]: pgmap v53: 129 pgs: 129 active+clean; 8.3 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 712 KiB/s wr, 93 op/s 2024-03-30T04:03:38.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 28727 11.58 MB/sec warmup 15 sec latency 9.343 ms 2024-03-30T04:03:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:38 smithi154 ceph-mon[79161]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-03-30T04:03:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:38 smithi193 ceph-mon[79581]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-03-30T04:03:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:38 smithi023 ceph-mon[38254]: pgmap v54: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 1.3 MiB/s wr, 167 op/s 2024-03-30T04:03:39.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 30886 11.35 MB/sec warmup 16 sec latency 23.197 ms 2024-03-30T04:03:40.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 32964 11.41 MB/sec warmup 17 sec latency 7.667 ms 2024-03-30T04:03:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:40 smithi154 ceph-mon[79161]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-03-30T04:03:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:40 smithi193 ceph-mon[79581]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-03-30T04:03:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:40 smithi023 ceph-mon[38254]: pgmap v55: 129 pgs: 129 active+clean; 14 MiB data, 448 MiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 1.1 MiB/s wr, 147 op/s 2024-03-30T04:03:41.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 35787 11.77 MB/sec warmup 18 sec latency 27.812 ms 2024-03-30T04:03:42.186 INFO:tasks.workunit.client.0.smithi023.stdout: 1 38266 11.75 MB/sec warmup 19 sec latency 6.946 ms 2024-03-30T04:03:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:42 smithi154 ceph-mon[79161]: pgmap v56: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-03-30T04:03:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:42 smithi193 ceph-mon[79581]: pgmap v56: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-03-30T04:03:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:42 smithi023 ceph-mon[38254]: pgmap v56: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.7 MiB/s wr, 225 op/s 2024-03-30T04:03:43.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 40242 11.67 MB/sec warmup 20 sec latency 42.530 ms 2024-03-30T04:03:44.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 42827 11.91 MB/sec warmup 21 sec latency 84.243 ms 2024-03-30T04:03:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:44 smithi154 ceph-mon[79161]: pgmap v57: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:44.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:44 smithi193 ceph-mon[79581]: pgmap v57: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:44 smithi023 ceph-mon[38254]: pgmap v57: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:45.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 44944 11.70 MB/sec warmup 22 sec latency 14.463 ms 2024-03-30T04:03:46.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 46829 11.73 MB/sec warmup 23 sec latency 9.919 ms 2024-03-30T04:03:47.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 49065 11.75 MB/sec warmup 24 sec latency 208.720 ms 2024-03-30T04:03:47.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:47 smithi023 ceph-mon[38254]: pgmap v58: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:47 smithi154 ceph-mon[79161]: pgmap v58: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:47 smithi193 ceph-mon[79581]: pgmap v58: 129 pgs: 129 active+clean; 19 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 1.2 MiB/s wr, 151 op/s 2024-03-30T04:03:48.187 INFO:tasks.workunit.client.0.smithi023.stdout: 1 50773 11.63 MB/sec warmup 25 sec latency 10.532 ms 2024-03-30T04:03:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:48 smithi154 ceph-mon[79161]: pgmap v59: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 240 op/s 2024-03-30T04:03:48.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:48 smithi193 ceph-mon[79581]: pgmap v59: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 240 op/s 2024-03-30T04:03:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:48 smithi023 ceph-mon[38254]: pgmap v59: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 240 op/s 2024-03-30T04:03:49.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 53380 11.79 MB/sec warmup 26 sec latency 128.544 ms 2024-03-30T04:03:50.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 54274 11.47 MB/sec warmup 27 sec latency 143.109 ms 2024-03-30T04:03:50.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:50 smithi154 ceph-mon[79161]: pgmap v60: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-03-30T04:03:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:50 smithi193 ceph-mon[79581]: pgmap v60: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-03-30T04:03:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:50 smithi023 ceph-mon[38254]: pgmap v60: 129 pgs: 129 active+clean; 44 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 3.0 MiB/s wr, 166 op/s 2024-03-30T04:03:51.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 56892 11.63 MB/sec warmup 28 sec latency 13.760 ms 2024-03-30T04:03:52.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 58920 11.47 MB/sec warmup 29 sec latency 23.692 ms 2024-03-30T04:03:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:52 smithi154 ceph-mon[79161]: pgmap v61: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 321 op/s 2024-03-30T04:03:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:52 smithi193 ceph-mon[79581]: pgmap v61: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 321 op/s 2024-03-30T04:03:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:52 smithi023 ceph-mon[38254]: pgmap v61: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 6.1 MiB/s wr, 321 op/s 2024-03-30T04:03:53.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 60995 11.54 MB/sec warmup 30 sec latency 7.025 ms 2024-03-30T04:03:54.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 63970 11.73 MB/sec warmup 31 sec latency 13.927 ms 2024-03-30T04:03:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:54 smithi154 ceph-mon[79161]: pgmap v62: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:54 smithi193 ceph-mon[79581]: pgmap v62: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:54 smithi023 ceph-mon[38254]: pgmap v62: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:55.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 66167 11.62 MB/sec warmup 32 sec latency 12.455 ms 2024-03-30T04:03:56.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 68109 11.65 MB/sec warmup 33 sec latency 14.163 ms 2024-03-30T04:03:57.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 71034 11.82 MB/sec warmup 34 sec latency 11.723 ms 2024-03-30T04:03:57.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:57 smithi023 ceph-mon[38254]: pgmap v63: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:57.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:57 smithi154 ceph-mon[79161]: pgmap v63: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:57 smithi193 ceph-mon[79581]: pgmap v63: 129 pgs: 129 active+clean; 44 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.5 MiB/s wr, 243 op/s 2024-03-30T04:03:57.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:03:58.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 -- 172.21.15.23:0/3754267315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c100f60 msgr2=0x7f606c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 --2- 172.21.15.23:0/3754267315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c100f60 0x7f606c101340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f605c009990 tx=0x7f605c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 -- 172.21.15.23:0/3754267315 shutdown_connections 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 --2- 172.21.15.23:0/3754267315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 0x7f606c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 --2- 172.21.15.23:0/3754267315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c100f60 0x7f606c101340 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 --2- 172.21.15.23:0/3754267315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c101910 0x7f606c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.185+0000 7f6071e00700 1 -- 172.21.15.23:0/3754267315 >> 172.21.15.23:0/3754267315 conn(0x7f606c0fc7c0 msgr2=0x7f606c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.186+0000 7f6071e00700 1 -- 172.21.15.23:0/3754267315 shutdown_connections 2024-03-30T04:03:58.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.186+0000 7f6071e00700 1 -- 172.21.15.23:0/3754267315 wait complete. 2024-03-30T04:03:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.186+0000 7f6071e00700 1 Processor -- start 2024-03-30T04:03:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 -- start start 2024-03-30T04:03:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 0x7f606c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c19ddc0 0x7f606c19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19e840 con 0x7f606c106080 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19e9b0 con 0x7f606c100f60 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f6071e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c19eb70 con 0x7f606c19ddc0 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f606aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f606aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56552/0 (socket says 172.21.15.23:56552) 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f606aa00700 1 -- 172.21.15.23:0/1815050517 learned_addr learned my addr 172.21.15.23:0/1815050517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f606a000700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 0x7f606c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.187+0000 7f606b400700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c19ddc0 0x7f606c19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 -- 172.21.15.23:0/1815050517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c19ddc0 msgr2=0x7f606c19e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c19ddc0 0x7f606c19e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 -- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 msgr2=0x7f606c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 0x7f606c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 -- 172.21.15.23:0/1815050517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c0095f0 con 0x7f606c100f60 2024-03-30T04:03:58.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f606aa00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f605c000c00 tx=0x7f605c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f6070a00700 1 -- 172.21.15.23:0/1815050517 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c01d070 con 0x7f606c100f60 2024-03-30T04:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f6070a00700 1 -- 172.21.15.23:0/1815050517 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f605c022470 con 0x7f606c100f60 2024-03-30T04:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c1a6070 con 0x7f606c100f60 2024-03-30T04:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f6070a00700 1 -- 172.21.15.23:0/1815050517 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c017650 con 0x7f606c100f60 2024-03-30T04:03:58.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.188+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c1a6490 con 0x7f606c100f60 2024-03-30T04:03:58.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 72655 11.68 MB/sec warmup 35 sec latency 13.906 ms 2024-03-30T04:03:58.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.191+0000 7f6070a00700 1 -- 172.21.15.23:0/1815050517 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f605c0225e0 con 0x7f606c100f60 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 Processor -- start 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f6070a00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 0x7f60500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f6070a00700 1 -- 172.21.15.23:0/1815050517 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f605c0ce1f0 con 0x7f606c100f60 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 -- start start 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f606b400700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 0x7f60500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9d6c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9cb7990 con 0x55d9b9d5eef0 2024-03-30T04:03:58.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9cb5350 con 0x55d9b9d69df0 2024-03-30T04:03:58.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.193+0000 7f60859c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9cb0c40 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43400/0 (socket says 172.21.15.23:43400) 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 -- 172.21.15.23:0/224296059 learned_addr learned my addr 172.21.15.23:0/224296059 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606b400700 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9d6c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 -- 172.21.15.23:0/224296059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 msgr2=0x55d9b9d6c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9d6c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 -- 172.21.15.23:0/224296059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 msgr2=0x55d9b9d698b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d698b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 -- 172.21.15.23:0/224296059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9b9b323b0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606b400700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 0x7f60500a3750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f60640093d0 tx=0x7f606400b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.194+0000 7f606aa00700 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f605c0a2e90 tx=0x7f605c009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e929673b95ce87c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f6057e00700 1 -- 172.21.15.23:0/224296059 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c0ce030 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f6057e00700 1 -- 172.21.15.23:0/224296059 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f605c09b6d0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f6057e00700 1 -- 172.21.15.23:0/224296059 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c0a3400 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 -- 172.21.15.23:0/224296059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 msgr2=0x55d9b9d5e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f605c0a2e90 tx=0x7f605c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 -- 172.21.15.23:0/224296059 shutdown_connections 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9d6c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d5e9b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 --2- 172.21.15.23:0/224296059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d698b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 -- 172.21.15.23:0/224296059 >> 172.21.15.23:0/224296059 conn(0x55d9b9cbe6a0 msgr2=0x55d9b9d5aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.195+0000 7f60859c8b80 1 -- 172.21.15.23:0/224296059 shutdown_connections 2024-03-30T04:03:58.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.196+0000 7f60859c8b80 1 -- 172.21.15.23:0/224296059 wait complete. 2024-03-30T04:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.196+0000 7f60859c8b80 1 -- start start 2024-03-30T04:03:58.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.196+0000 7f60859c8b80 1 Processor -- start 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d69780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d66b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9c4cd80 con 0x55d9b9d5eef0 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9c4ca20 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9b346a0 con 0x55d9b9d69df0 2024-03-30T04:03:58.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f60859c8b80 10 client.0 _ll_register_callbacks cb 0x55d9b9c56470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d66b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43402/0 (socket says 172.21.15.23:43402) 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 -- 172.21.15.23:0/321574019 learned_addr learned my addr 172.21.15.23:0/321574019 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 -- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 msgr2=0x55d9b9d69780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d69780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 -- 172.21.15.23:0/321574019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 msgr2=0x55d9b9d66b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d66b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.197+0000 7f606b400700 1 -- 172.21.15.23:0/321574019 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c1a6490 con 0x55d9b9d69df0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f606b400700 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6064015df0 tx=0x7f6064004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d66339993f2b9f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f6054c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606400e050 con 0x55d9b9d69df0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f6064017e20 con 0x55d9b9d69df0 2024-03-30T04:03:58.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9b9c71380 con 0x55d9b9d69df0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6064021470 con 0x55d9b9d69df0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f60859c8b80 10 client.26125 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.198+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d9b9adeb60 con 0x55d9b9d69df0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9b9c71380 con 0x55d9b9d69df0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f6064020600 con 0x55d9b9d69df0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 10 client.26125 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 20 client.26125 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9d79710 0x55d9b9d7bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 4 client.26125 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9b9ca9700 con 0x55d9b9d79710 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.199+0000 7f606aa00700 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9d79710 0x55d9b9d7bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.200+0000 7f6054c00700 10 client.26125 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:58.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.200+0000 7f606aa00700 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9d79710 0x55d9b9d7bb30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f4dbd11146e751 server_cookie=1f8272345c2642a6 in_seq=0 out_seq=0 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f6054c00700 1 -- 172.21.15.23:0/321574019 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d9b9ca9700 con 0x55d9b9d79710 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f6054c00700 10 client.26125 handle_command_reply: tid=0 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 client.26125 shutdown 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 msgr2=0x55d9b9d67080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6064015df0 tx=0x7f6064004ad0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 shutdown_connections 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9d79710 0x55d9b9d7bb30 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d69df0 0x55d9b9d67080 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d69780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f6054c00700 0 client.26125 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 --2- 172.21.15.23:0/321574019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d66b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 >> 172.21.15.23:0/321574019 conn(0x55d9b9cbe6a0 msgr2=0x55d9b9d606c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 shutdown_connections 2024-03-30T04:03:58.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 1 -- 172.21.15.23:0/321574019 wait complete. 2024-03-30T04:03:58.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.201+0000 7f60859c8b80 20 client.26125 trim_cache size 0 max 16384 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.219+0000 7f60859c8b80 1 Processor -- start 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.219+0000 7f60859c8b80 1 -- start start 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.219+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.219+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d707e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d69df0 0x55d9b9d653d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f60859c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9d5e3f0 con 0x55d9b9d69df0 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f60859c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9d61910 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f60859c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9d70d20 con 0x55d9b9d5eef0 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56574/0 (socket says 172.21.15.23:56574) 2024-03-30T04:03:58.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 -- 172.21.15.23:0/3179223735 learned_addr learned my addr 172.21.15.23:0/3179223735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606a000700 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606b400700 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d69df0 0x55d9b9d653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 -- 172.21.15.23:0/3179223735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5eef0 msgr2=0x55d9b9d707e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d707e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 -- 172.21.15.23:0/3179223735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d69df0 msgr2=0x55d9b9d653d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d69df0 0x55d9b9d653d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.220+0000 7f606aa00700 1 -- 172.21.15.23:0/3179223735 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9b9c76ea0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f606aa00700 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f605c097a40 tx=0x7f605c00bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17ece508a3ae29cc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f6061200700 1 -- 172.21.15.23:0/3179223735 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c0a10b0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f6061200700 1 -- 172.21.15.23:0/3179223735 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f605c0a33c0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f6061200700 1 -- 172.21.15.23:0/3179223735 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f605c095be0 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 -- 172.21.15.23:0/3179223735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 msgr2=0x55d9b9d67050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f605c097a40 tx=0x7f605c00bfc0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 -- 172.21.15.23:0/3179223735 shutdown_connections 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d69df0 0x55d9b9d653d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d67050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 --2- 172.21.15.23:0/3179223735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d707e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.221+0000 7f60859c8b80 1 -- 172.21.15.23:0/3179223735 >> 172.21.15.23:0/3179223735 conn(0x55d9b9cbe6a0 msgr2=0x55d9b9d535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.222+0000 7f60859c8b80 1 -- 172.21.15.23:0/3179223735 shutdown_connections 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.222+0000 7f60859c8b80 1 -- 172.21.15.23:0/3179223735 wait complete. 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.222+0000 7f60859c8b80 1 -- start start 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.222+0000 7f60859c8b80 1 Processor -- start 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d6e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9dc65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9dc6ae0 con 0x55d9b9d5eef0 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9dc6c50 con 0x55d9b9d69df0 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9b9dc6e10 con 0x55d9b9d5e5d0 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f60859c8b80 10 client.0 _ll_register_callbacks cb 0x55d9b9c56470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:58.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60098/0 (socket says 172.21.15.23:60098) 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 -- 172.21.15.23:0/699080402 learned_addr learned my addr 172.21.15.23:0/699080402 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606b400700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9dc65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606aa00700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d6e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 -- 172.21.15.23:0/699080402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 msgr2=0x55d9b9d6e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d6e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 -- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 msgr2=0x55d9b9dc65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9dc65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606a000700 1 -- 172.21.15.23:0/699080402 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9b9b323b0 con 0x55d9b9d5eef0 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.223+0000 7f606aa00700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d6e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:58.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f606a000700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f606c06a380 tx=0x7f6058010ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5af6df949ff3e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f6054c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6058021070 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f605c09b6d0 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f605800f7b0 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6058004020 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f60859c8b80 10 client.16170 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.224+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d9b9b36cd0 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6058028020 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f6058024030 con 0x55d9b9d5eef0 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 10 client.16170 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 20 client.16170 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9dce880 0x55d9b9dd0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 4 client.16170 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d9b9affcf0 con 0x55d9b9dce880 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.225+0000 7f606aa00700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9dce880 0x55d9b9dd0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f606aa00700 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9dce880 0x55d9b9dd0ca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d993207eae2bd161 server_cookie=3600779f5959ed4c in_seq=0 out_seq=0 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f6054c00700 10 client.16170 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f6054c00700 1 -- 172.21.15.23:0/699080402 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d9b9affcf0 con 0x55d9b9dce880 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f6054c00700 10 client.16170 handle_command_reply: tid=0 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 client.16170 shutdown 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 msgr2=0x55d9b9d6ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f606c06a380 tx=0x7f6058010ad0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 shutdown_connections 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9b9d69df0 0x55d9b9dc65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d9b9dce880 0x55d9b9dd0ca0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9b9d5e5d0 0x55d9b9d6e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 --2- 172.21.15.23:0/699080402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9b9d5eef0 0x55d9b9d6ef10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f6054c00700 0 client.16170 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:03:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 >> 172.21.15.23:0/699080402 conn(0x55d9b9cbe6a0 msgr2=0x55d9b9d65340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 shutdown_connections 2024-03-30T04:03:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.226+0000 7f60859c8b80 1 -- 172.21.15.23:0/699080402 wait complete. 2024-03-30T04:03:58.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f60859c8b80 20 client.16170 trim_cache size 0 max 16384 2024-03-30T04:03:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:03:58.231 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:03:58.231 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:03:58.231 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:03:58.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 msgr2=0x7f60500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 0x7f60500a3750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f60640093d0 tx=0x7f606400b040 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 msgr2=0x7f606c1994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f605c000c00 tx=0x7f605c005740 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 shutdown_connections 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f606c19ddc0 0x7f606c19e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f60500a1290 0x7f60500a3750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f606c100f60 0x7f606c1994e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 --2- 172.21.15.23:0/1815050517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f606c106080 0x7f606c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 >> 172.21.15.23:0/1815050517 conn(0x7f606c0fc7c0 msgr2=0x7f606c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 shutdown_connections 2024-03-30T04:03:58.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.227+0000 7f6071e00700 1 -- 172.21.15.23:0/1815050517 wait complete. 2024-03-30T04:03:58.244 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:03:58.245 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:03:58.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:58 smithi023 ceph-mon[38254]: pgmap v64: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-30T04:03:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:58 smithi154 ceph-mon[79161]: pgmap v64: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.460+0000 7f8c79a00700 1 -- 172.21.15.23:0/530490610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74101810 msgr2=0x7f8c74105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.460+0000 7f8c79a00700 1 --2- 172.21.15.23:0/530490610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74101810 0x7f8c74105850 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8c60009990 tx=0x7f8c60009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 -- 172.21.15.23:0/530490610 shutdown_connections 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 --2- 172.21.15.23:0/530490610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c7410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 --2- 172.21.15.23:0/530490610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74100e60 0x7f8c74101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 --2- 172.21.15.23:0/530490610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74101810 0x7f8c74105850 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 -- 172.21.15.23:0/530490610 >> 172.21.15.23:0/530490610 conn(0x7f8c740fc700 msgr2=0x7f8c740feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 -- 172.21.15.23:0/530490610 shutdown_connections 2024-03-30T04:03:58.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.461+0000 7f8c79a00700 1 -- 172.21.15.23:0/530490610 wait complete. 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.462+0000 7f8c79a00700 1 Processor -- start 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.462+0000 7f8c79a00700 1 -- start start 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74100e60 0x7f8c740730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741a58e0 con 0x7f8c74105f80 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741a5a20 con 0x7f8c74100e60 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c79a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c741a5be0 con 0x7f8c74073b30 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c73400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c72000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c72000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60110/0 (socket says 172.21.15.23:60110) 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c72000700 1 -- 172.21.15.23:0/1148655503 learned_addr learned my addr 172.21.15.23:0/1148655503 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c73400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43444/0 (socket says 172.21.15.23:43444) 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.463+0000 7f8c72a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74100e60 0x7f8c740730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 -- 172.21.15.23:0/1148655503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 msgr2=0x7f8c741a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 -- 172.21.15.23:0/1148655503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74100e60 msgr2=0x7f8c740730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74100e60 0x7f8c740730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 -- 172.21.15.23:0/1148655503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c600095f0 con 0x7f8c74105f80 2024-03-30T04:03:58.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c73400700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:58.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c72000700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c00 tx=0x7f8c60009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6001d070 con 0x7f8c74105f80 2024-03-30T04:03:58.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c741a5e60 con 0x7f8c74105f80 2024-03-30T04:03:58.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c60021410 con 0x7f8c74105f80 2024-03-30T04:03:58.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c60017650 con 0x7f8c74105f80 2024-03-30T04:03:58.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.464+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c741a6290 con 0x7f8c74105f80 2024-03-30T04:03:58.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.466+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c74109250 con 0x7f8c74105f80 2024-03-30T04:03:58.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.467+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8c60021580 con 0x7f8c74105f80 2024-03-30T04:03:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.469+0000 7f8c6b400700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 0x7f8c580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.469+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8c600ce290 con 0x7f8c74105f80 2024-03-30T04:03:58.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.469+0000 7f8c72a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 0x7f8c580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:58.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.470+0000 7f8c72a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 0x7f8c580a3750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009c60 tx=0x7f8c64009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:58.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.473+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8c600973b0 con 0x7f8c74105f80 2024-03-30T04:03:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:58 smithi193 ceph-mon[79581]: pgmap v64: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 6.1 MiB/s wr, 325 op/s 2024-03-30T04:03:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.766+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c74101810 con 0x7f8c74105f80 2024-03-30T04:03:58.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.766+0000 7f8c6b400700 1 -- 172.21.15.23:0/1148655503 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+10925 (secure 0 0 0) 0x7f8c6009a9d0 con 0x7f8c74105f80 2024-03-30T04:03:58.765 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:03:58.765 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:03:23.374313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:03:58.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 msgr2=0x7f8c580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 0x7f8c580a3750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009c60 tx=0x7f8c64009260 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 msgr2=0x7f8c740735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c60000c00 tx=0x7f8c60009580 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 shutdown_connections 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c74073b30 0x7f8c741a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8c580a1290 0x7f8c580a3750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c74100e60 0x7f8c740730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 --2- 172.21.15.23:0/1148655503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c74105f80 0x7f8c740735f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 >> 172.21.15.23:0/1148655503 conn(0x7f8c740fc700 msgr2=0x7f8c740fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 shutdown_connections 2024-03-30T04:03:58.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:58.769+0000 7f8c79a00700 1 -- 172.21.15.23:0/1148655503 wait complete. 2024-03-30T04:03:58.769 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-30T04:03:58.787 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:03:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 -- 172.21.15.23:0/1552089810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290106080 msgr2=0x7fc29010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 --2- 172.21.15.23:0/1552089810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290106080 0x7fc29010dc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc28800b0a0 tx=0x7fc28800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 -- 172.21.15.23:0/1552089810 shutdown_connections 2024-03-30T04:03:59.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 --2- 172.21.15.23:0/1552089810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290106080 0x7fc29010dc00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 --2- 172.21.15.23:0/1552089810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc290100f60 0x7fc290101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 --2- 172.21.15.23:0/1552089810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290101910 0x7fc290105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 -- 172.21.15.23:0/1552089810 >> 172.21.15.23:0/1552089810 conn(0x7fc2900fc7c0 msgr2=0x7fc2900febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.032+0000 7fc297a00700 1 -- 172.21.15.23:0/1552089810 shutdown_connections 2024-03-30T04:03:59.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.033+0000 7fc297a00700 1 -- 172.21.15.23:0/1552089810 wait complete. 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.033+0000 7fc297a00700 1 Processor -- start 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 -- start start 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290106080 0x7fc290197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc29019b9c0 0x7fc29019be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29019c440 con 0x7fc29019b9c0 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29019c5b0 con 0x7fc290100f60 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc297a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29019c770 con 0x7fc290106080 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc295000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc295a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc29019b9c0 0x7fc29019be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc295000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56614/0 (socket says 172.21.15.23:56614) 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc295000700 1 -- 172.21.15.23:0/1345609118 learned_addr learned my addr 172.21.15.23:0/1345609118 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.034+0000 7fc28fe00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290106080 0x7fc290197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 -- 172.21.15.23:0/1345609118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290106080 msgr2=0x7fc290197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290106080 0x7fc290197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 -- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc29019b9c0 msgr2=0x7fc29019be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc29019b9c0 0x7fc29019be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 -- 172.21.15.23:0/1345609118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc288009d00 con 0x7fc290100f60 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc295000700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e9a0 tx=0x7fc28400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc28d600700 1 -- 172.21.15.23:0/1345609118 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284009790 con 0x7fc290100f60 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc28d600700 1 -- 172.21.15.23:0/1345609118 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc284010430 con 0x7fc290100f60 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2901a3cd0 con 0x7fc290100f60 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc28d600700 1 -- 172.21.15.23:0/1345609118 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc284018680 con 0x7fc290100f60 2024-03-30T04:03:59.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.035+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2901a41b0 con 0x7fc290100f60 2024-03-30T04:03:59.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.038+0000 7fc28d600700 1 -- 172.21.15.23:0/1345609118 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc2840105a0 con 0x7fc290100f60 2024-03-30T04:03:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc28d600700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 0x7fc2780a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 Processor -- start 2024-03-30T04:03:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc28d600700 1 -- 172.21.15.23:0/1345609118 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc284014070 con 0x7fc290100f60 2024-03-30T04:03:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 -- start start 2024-03-30T04:03:59.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc28fe00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 0x7fc2780a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0b975d0 0x5599c0b979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0af0990 con 0x5599c0b97ef0 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0aee350 con 0x5599c0ba7200 2024-03-30T04:03:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.040+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0ae9c40 con 0x5599c0b975d0 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0b975d0 0x5599c0b979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56624/0 (socket says 172.21.15.23:56624) 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 -- 172.21.15.23:0/1593769903 learned_addr learned my addr 172.21.15.23:0/1593769903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc28fe00700 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 -- 172.21.15.23:0/1593769903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0b975d0 msgr2=0x5599c0b979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0b975d0 0x5599c0b979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 -- 172.21.15.23:0/1593769903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 msgr2=0x5599c0ba6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc28fe00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 0x7fc2780a3750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc290198630 tx=0x7fc27c0073d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 -- 172.21.15.23:0/1593769903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599c096b3b0 con 0x5599c0ba7200 2024-03-30T04:03:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.041+0000 7fc295a00700 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc288009580 tx=0x7fc2880095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1bc4a2c1a65ae67 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc282000700 1 -- 172.21.15.23:0/1593769903 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28800e040 con 0x5599c0ba7200 2024-03-30T04:03:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc282000700 1 -- 172.21.15.23:0/1593769903 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc288006040 con 0x5599c0ba7200 2024-03-30T04:03:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc282000700 1 -- 172.21.15.23:0/1593769903 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc288007910 con 0x5599c0ba7200 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/1593769903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 msgr2=0x5599c0ba95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc288009580 tx=0x7fc2880095b0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/1593769903 shutdown_connections 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0ba7200 0x5599c0ba95f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0b975d0 0x5599c0b979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/1593769903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/1593769903 >> 172.21.15.23:0/1593769903 conn(0x5599c0af76a0 msgr2=0x5599c0b93a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.042+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/1593769903 shutdown_connections 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.043+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/1593769903 wait complete. 2024-03-30T04:03:59.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.043+0000 7fc2ab4d5b80 1 -- start start 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.043+0000 7fc2ab4d5b80 1 Processor -- start 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0a85d80 con 0x5599c0b97ef0 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0a85a20 con 0x5599c0b975d0 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c096d6a0 con 0x5599c0ba7200 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc2ab4d5b80 10 client.0 _ll_register_callbacks cb 0x5599c0a8f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56630/0 (socket says 172.21.15.23:56630) 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 -- 172.21.15.23:0/3231981431 learned_addr learned my addr 172.21.15.23:0/3231981431 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295a00700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc28fe00700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 -- 172.21.15.23:0/3231981431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 msgr2=0x5599c0ba43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 -- 172.21.15.23:0/3231981431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 msgr2=0x5599c0ba3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.044+0000 7fc295000700 1 -- 172.21.15.23:0/3231981431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29010e470 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc295000700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc284096a20 tx=0x7fc284099fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9abe1f32051b688 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840996a0 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc2840a2430 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599c0aaa380 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2840a1fa0 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 10 client.26156 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5599c0917b60 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5599c0aaa380 con 0x5599c0b975d0 2024-03-30T04:03:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fc284094bc0 con 0x5599c0b975d0 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 10 client.26156 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 10 client.26156 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 20 client.26156 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0bb6e10 0x5599c0bb9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 4 client.26156 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.045+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599c0ae2700 con 0x5599c0bb6e10 2024-03-30T04:03:59.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.046+0000 7fc28fe00700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0bb6e10 0x5599c0bb9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.047+0000 7fc28fe00700 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0bb6e10 0x5599c0bb9230 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=579bbded05368df8 server_cookie=5f9ddcf3ebb42335 in_seq=0 out_seq=0 2024-03-30T04:03:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.047+0000 7fc276a00700 10 client.26156 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc276a00700 1 -- 172.21.15.23:0/3231981431 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5599c0ae2700 con 0x5599c0bb6e10 2024-03-30T04:03:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc276a00700 10 client.26156 handle_command_reply: tid=0 2024-03-30T04:03:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 client.26156 shutdown 2024-03-30T04:03:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 msgr2=0x5599c0ba6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc284096a20 tx=0x7fc284099fb0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 shutdown_connections 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0bb6e10 0x5599c0bb9230 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0b975d0 0x5599c0ba6ba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/3231981431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0ba3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 >> 172.21.15.23:0/3231981431 conn(0x5599c0af76a0 msgr2=0x5599c0b996c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc276a00700 0 client.26156 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 shutdown_connections 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.048+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/3231981431 wait complete. 2024-03-30T04:03:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.049+0000 7fc2ab4d5b80 20 client.26156 trim_cache size 0 max 16384 2024-03-30T04:03:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 Processor -- start 2024-03-30T04:03:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 -- start start 2024-03-30T04:03:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0bade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0ba3170 con 0x5599c0b97ef0 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0bae390 con 0x5599c0bbdb50 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.062+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0ba49c0 con 0x5599c0ba7200 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0bade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56638/0 (socket says 172.21.15.23:56638) 2024-03-30T04:03:59.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295a00700 1 -- 172.21.15.23:0/2073368884 learned_addr learned my addr 172.21.15.23:0/2073368884 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 -- 172.21.15.23:0/2073368884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 msgr2=0x5599c0ba2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 -- 172.21.15.23:0/2073368884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 msgr2=0x5599c0bade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0bade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 -- 172.21.15.23:0/2073368884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599c0aafea0 con 0x5599c0ba7200 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295a00700 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc295000700 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0bade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:59.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.063+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba26f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5599c0bb9ad0 tx=0x7fc27c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c32ab4bd44f2f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc282a00700 1 -- 172.21.15.23:0/2073368884 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c024040 con 0x5599c0ba7200 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc282a00700 1 -- 172.21.15.23:0/2073368884 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc27c015070 con 0x5599c0ba7200 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc282a00700 1 -- 172.21.15.23:0/2073368884 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c003e40 con 0x5599c0ba7200 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2073368884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 msgr2=0x5599c0ba26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba26f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5599c0bb9ad0 tx=0x7fc27c013000 comp rx=0 tx=0).stop 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2073368884 shutdown_connections 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0ba2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0bade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2073368884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0ba26f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2073368884 >> 172.21.15.23:0/2073368884 conn(0x5599c0af76a0 msgr2=0x5599c0b8cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2073368884 shutdown_connections 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.064+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2073368884 wait complete. 2024-03-30T04:03:59.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 -- start start 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 Processor -- start 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0bac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0bac7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0bacec0 con 0x5599c0b97ef0 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0bad030 con 0x5599c0bbdb50 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599c0c0ac90 con 0x5599c0ba7200 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.065+0000 7fc2ab4d5b80 10 client.0 _ll_register_callbacks cb 0x5599c0a8f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0bac7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0bac7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56644/0 (socket says 172.21.15.23:56644) 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60158/0 (socket says 172.21.15.23:60158) 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295a00700 1 -- 172.21.15.23:0/2245198369 learned_addr learned my addr 172.21.15.23:0/2245198369 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0bac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 -- 172.21.15.23:0/2245198369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 msgr2=0x5599c0bac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0bac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 -- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 msgr2=0x5599c0bac7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0bac7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 -- 172.21.15.23:0/2245198369 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599c0917b60 con 0x5599c0b97ef0 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc295000700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e970 tx=0x7fc284097fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78750f66b8bd421 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.066+0000 7fc276a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2840b8340 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc284099290 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc288006040 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc28409a020 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 10 client.16194 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5599c096b3b0 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc288006040 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fc2840994a0 con 0x5599c0b97ef0 2024-03-30T04:03:59.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 10 client.16194 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 10 client.16194 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 20 client.16194 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0c0bfb0 0x5599c0c0e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 4 client.16194 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.067+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5599c0938cf0 con 0x5599c0c0bfb0 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.068+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0c0bfb0 0x5599c0c0e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc276a00700 10 client.16194 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc28fe00700 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0c0bfb0 0x5599c0c0e3a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e27f6a6eac4f09e8 server_cookie=d95f2f088a98616a in_seq=0 out_seq=0 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc276a00700 1 -- 172.21.15.23:0/2245198369 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5599c0938cf0 con 0x5599c0c0bfb0 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc276a00700 10 client.16194 handle_command_reply: tid=0 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 client.16194 shutdown 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 msgr2=0x5599c0badd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e970 tx=0x7fc284097fb0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 shutdown_connections 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599c0bbdb50 0x5599c0bac7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599c0b97ef0 0x5599c0badd40 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5599c0c0bfb0 0x5599c0c0e3a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 --2- 172.21.15.23:0/2245198369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599c0ba7200 0x5599c0bac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 >> 172.21.15.23:0/2245198369 conn(0x5599c0af76a0 msgr2=0x5599c0ba2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.069+0000 7fc276a00700 0 client.16194 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 shutdown_connections 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc2ab4d5b80 1 -- 172.21.15.23:0/2245198369 wait complete. 2024-03-30T04:03:59.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc2ab4d5b80 20 client.16194 trim_cache size 0 max 16384 2024-03-30T04:03:59.074 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:59.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 msgr2=0x7fc2780a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 0x7fc2780a3750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc290198630 tx=0x7fc27c0073d0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 msgr2=0x7fc290197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc28400e9a0 tx=0x7fc28400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 shutdown_connections 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc29019b9c0 0x7fc29019be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.070+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc2780a1290 0x7fc2780a3750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.071+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc290100f60 0x7fc290197130 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.071+0000 7fc297a00700 1 --2- 172.21.15.23:0/1345609118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc290106080 0x7fc290197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.071+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 >> 172.21.15.23:0/1345609118 conn(0x7fc2900fc7c0 msgr2=0x7fc2900fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.071+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 shutdown_connections 2024-03-30T04:03:59.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.071+0000 7fc297a00700 1 -- 172.21.15.23:0/1345609118 wait complete. 2024-03-30T04:03:59.088 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:03:59.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 75069 11.74 MB/sec warmup 36 sec latency 14.089 ms 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 -- 172.21.15.23:0/2101591090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069c10 msgr2=0x7f1b68105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 --2- 172.21.15.23:0/2101591090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069c10 0x7f1b68105980 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f1b540099c0 tx=0x7f1b54009cd0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 -- 172.21.15.23:0/2101591090 shutdown_connections 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 --2- 172.21.15.23:0/2101591090 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 --2- 172.21.15.23:0/2101591090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b68069260 0x7f1b68069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 --2- 172.21.15.23:0/2101591090 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069c10 0x7f1b68105980 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 -- 172.21.15.23:0/2101591090 >> 172.21.15.23:0/2101591090 conn(0x7f1b68076d80 msgr2=0x7f1b68077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 -- 172.21.15.23:0/2101591090 shutdown_connections 2024-03-30T04:03:59.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.320+0000 7f1b70000700 1 -- 172.21.15.23:0/2101591090 wait complete. 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.321+0000 7f1b70000700 1 Processor -- start 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.321+0000 7f1b70000700 1 -- start start 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069260 0x7f1b6810d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 0x7f1b6810b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6810df20 con 0x7f1b68069260 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b681aa010 con 0x7f1b681060b0 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b70000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b681aa150 con 0x7f1b6810ada0 2024-03-30T04:03:59.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 0x7f1b6810b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56666/0 (socket says 172.21.15.23:56666) 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b6cc00700 1 -- 172.21.15.23:0/317890157 learned_addr learned my addr 172.21.15.23:0/317890157 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.322+0000 7f1b6d600700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069260 0x7f1b6810d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 -- 172.21.15.23:0/317890157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 msgr2=0x7f1b6810a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 -- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069260 msgr2=0x7f1b6810d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069260 0x7f1b6810d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 -- 172.21.15.23:0/317890157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b540095f0 con 0x7f1b6810ada0 2024-03-30T04:03:59.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b6e000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 0x7f1b6810b250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000efc0 tx=0x7f1b6000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b66000700 1 -- 172.21.15.23:0/317890157 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60018070 con 0x7f1b6810ada0 2024-03-30T04:03:59.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b681aa380 con 0x7f1b6810ada0 2024-03-30T04:03:59.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b66000700 1 -- 172.21.15.23:0/317890157 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6000f040 con 0x7f1b6810ada0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b66000700 1 -- 172.21.15.23:0/317890157 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6000cd40 con 0x7f1b6810ada0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.323+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b681aa880 con 0x7f1b6810ada0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.326+0000 7f1b66000700 1 -- 172.21.15.23:0/317890157 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1b60003a00 con 0x7f1b6810ada0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 Processor -- start 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 -- start start 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e496e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b66000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 0x7f1b500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e49798b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 0x5647e497c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.328+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e48c7990 con 0x5647e4979df0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e48c5350 con 0x5647e496eef0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e48c0c40 con 0x5647e496e5d0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b66000700 1 -- 172.21.15.23:0/317890157 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1b600ccc20 con 0x7f1b6810ada0 2024-03-30T04:03:59.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 0x5647e497c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e49798b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e49798b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56674/0 (socket says 172.21.15.23:56674) 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6cc00700 1 -- 172.21.15.23:0/2673263112 learned_addr learned my addr 172.21.15.23:0/2673263112 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 0x7f1b500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6d600700 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e496e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 -- 172.21.15.23:0/2673263112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 msgr2=0x5647e496e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e496e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 -- 172.21.15.23:0/2673263112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 msgr2=0x5647e49798b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e49798b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.329+0000 7f1b6e000700 1 -- 172.21.15.23:0/2673263112 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e47423b0 con 0x5647e4979df0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b6d600700 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e496e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b6e000700 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 0x5647e497c1e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1b600c9fb0 tx=0x7f1b60095710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b343cfb004e3ddf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 0x7f1b500a3750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1b54005cd0 tx=0x7f1b54005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b5a600700 1 -- 172.21.15.23:0/2673263112 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6001b320 con 0x5647e4979df0 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b5a600700 1 -- 172.21.15.23:0/2673263112 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6000b2d0 con 0x5647e4979df0 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.330+0000 7f1b5a600700 1 -- 172.21.15.23:0/2673263112 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6009d830 con 0x5647e4979df0 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- 172.21.15.23:0/2673263112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 msgr2=0x5647e497c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 0x5647e497c1e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1b600c9fb0 tx=0x7f1b60095710 comp rx=0 tx=0).stop 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- 172.21.15.23:0/2673263112 shutdown_connections 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e4979df0 0x5647e497c1e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e496e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/2673263112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e49798b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- 172.21.15.23:0/2673263112 >> 172.21.15.23:0/2673263112 conn(0x5647e48ce6a0 msgr2=0x5647e496aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- 172.21.15.23:0/2673263112 shutdown_connections 2024-03-30T04:03:59.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- 172.21.15.23:0/2673263112 wait complete. 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.331+0000 7f1b83b82b80 1 -- start start 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 Processor -- start 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496e5d0 0x5647e4979830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e4976b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e485cd80 con 0x5647e496e5d0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e485ca20 con 0x5647e496eef0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e47446a0 con 0x5647e4979df0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b83b82b80 10 client.0 _ll_register_callbacks cb 0x5647e4866470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b6d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496e5d0 0x5647e4979830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43548/0 (socket says 172.21.15.23:43548) 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.332+0000 7f1b6e000700 1 -- 172.21.15.23:0/1167472094 learned_addr learned my addr 172.21.15.23:0/1167472094 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 -- 172.21.15.23:0/1167472094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 msgr2=0x5647e4976b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e4976b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 -- 172.21.15.23:0/1167472094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496e5d0 msgr2=0x5647e4979830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496e5d0 0x5647e4979830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 -- 172.21.15.23:0/1167472094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b681aa880 con 0x5647e4979df0 2024-03-30T04:03:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b6e000700 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b60096470 tx=0x7f1b60099e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c151fb9ed5782e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60004ad0 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1b6001b320 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647e4881380 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b6009a100 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b83b82b80 10 client.26164 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.333+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5647e46eeb60 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5647e4881380 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f1b60098ef0 con 0x5647e4979df0 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 10 client.26164 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 10 client.26164 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 20 client.26164 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e4989460 0x5647e498b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 4 client.26164 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.334+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647e48b9700 con 0x5647e4989460 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.335+0000 7f1b6d600700 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e4989460 0x5647e498b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.336+0000 7f1b6d600700 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e4989460 0x5647e498b880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ee78aa15adedf176 server_cookie=77b61b44ca9a35b in_seq=0 out_seq=0 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.336+0000 7f1b4ea00700 10 client.26164 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b4ea00700 1 -- 172.21.15.23:0/1167472094 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5647e48b9700 con 0x5647e4989460 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b4ea00700 10 client.26164 handle_command_reply: tid=0 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 client.26164 shutdown 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 msgr2=0x5647e4977260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1b60096470 tx=0x7f1b60099e90 comp rx=0 tx=0).stop 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 shutdown_connections 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e4989460 0x5647e498b880 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e4977260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b4ea00700 0 client.26164 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:59.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496e5d0 0x5647e4979830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/1167472094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496eef0 0x5647e4976b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 >> 172.21.15.23:0/1167472094 conn(0x5647e48ce6a0 msgr2=0x5647e4970670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 shutdown_connections 2024-03-30T04:03:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 1 -- 172.21.15.23:0/1167472094 wait complete. 2024-03-30T04:03:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.337+0000 7f1b83b82b80 20 client.26164 trim_cache size 0 max 16384 2024-03-30T04:03:59.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.352+0000 7f1b83b82b80 1 Processor -- start 2024-03-30T04:03:59.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.352+0000 7f1b83b82b80 1 -- start start 2024-03-30T04:03:59.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.352+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e4980a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.352+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e4979df0 0x5647e49798a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e496f850 con 0x5647e496eef0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e496f2d0 con 0x5647e4979df0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e4980fb0 con 0x5647e496e5d0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e4980a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e4979df0 0x5647e49798a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60196/0 (socket says 172.21.15.23:60196) 2024-03-30T04:03:59.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6cc00700 1 -- 172.21.15.23:0/4075243706 learned_addr learned my addr 172.21.15.23:0/4075243706 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 -- 172.21.15.23:0/4075243706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e4979df0 msgr2=0x5647e49798a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e4979df0 0x5647e49798a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 -- 172.21.15.23:0/4075243706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 msgr2=0x5647e4979360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6d600700 1 -- 172.21.15.23:0/4075243706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e4886ea0 con 0x5647e496e5d0 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.353+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b6d600700 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e4980a70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x5647e4989840 tx=0x7f1b5c01dc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73f90d8cf7eed0f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b5b000700 1 -- 172.21.15.23:0/4075243706 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c017040 con 0x5647e496e5d0 2024-03-30T04:03:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b5b000700 1 -- 172.21.15.23:0/4075243706 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b5c021070 con 0x5647e496e5d0 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b5b000700 1 -- 172.21.15.23:0/4075243706 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c005280 con 0x5647e496e5d0 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b83b82b80 1 -- 172.21.15.23:0/4075243706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 msgr2=0x5647e4980a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.354+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e4980a70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x5647e4989840 tx=0x7f1b5c01dc50 comp rx=0 tx=0).stop 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 -- 172.21.15.23:0/4075243706 shutdown_connections 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e4979df0 0x5647e49798a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e496e5d0 0x5647e4980a70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/4075243706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e4979360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 -- 172.21.15.23:0/4075243706 >> 172.21.15.23:0/4075243706 conn(0x5647e48ce6a0 msgr2=0x5647e4971e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 -- 172.21.15.23:0/4075243706 shutdown_connections 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 -- 172.21.15.23:0/4075243706 wait complete. 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 -- start start 2024-03-30T04:03:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.355+0000 7f1b83b82b80 1 Processor -- start 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e497ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e497f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e497f670 con 0x5647e496eef0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e497f7b0 con 0x5647e496e5d0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e497f970 con 0x5647e4979df0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b83b82b80 10 client.0 _ll_register_callbacks cb 0x5647e4866470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6d600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e497f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6d600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56712/0 (socket says 172.21.15.23:56712) 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6cc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e497ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e497f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43564/0 (socket says 172.21.15.23:43564) 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.356+0000 7f1b6d600700 1 -- 172.21.15.23:0/3789905880 learned_addr learned my addr 172.21.15.23:0/3789905880 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 -- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 msgr2=0x5647e497f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e497f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 -- 172.21.15.23:0/3789905880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 msgr2=0x5647e497ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e497ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 -- 172.21.15.23:0/3789905880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5401d040 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b6d600700 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006010 tx=0x7f1b5c00bae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7add9988e9cb92e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b4ea00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5c017040 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1b5c0047c0 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.357+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b6000b2d0 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b5c004980 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 10 client.26177 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f1b5c0044c0 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b6000b2d0 con 0x5647e496e5d0 2024-03-30T04:03:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f1b5c003680 con 0x5647e496e5d0 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 10 client.26177 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 10 client.26177 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 20 client.26177 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e49de5e0 0x5647e49e09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 4 client.26177 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5647e470fcf0 con 0x5647e49de5e0 2024-03-30T04:03:59.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.358+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e49de5e0 0x5647e49e09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.359+0000 7f1b6cc00700 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e49de5e0 0x5647e49e09d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=698743e7b8dacbe8 server_cookie=42a5525f5cf1b76b in_seq=0 out_seq=0 2024-03-30T04:03:59.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.359+0000 7f1b4ea00700 10 client.26177 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:59.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.359+0000 7f1b4ea00700 1 -- 172.21.15.23:0/3789905880 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5647e470fcf0 con 0x5647e49de5e0 2024-03-30T04:03:59.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.359+0000 7f1b4ea00700 10 client.26177 handle_command_reply: tid=0 2024-03-30T04:03:59.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.359+0000 7f1b83b82b80 1 client.26177 shutdown 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 msgr2=0x5647e4980700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c006010 tx=0x7f1b5c00bae0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 shutdown_connections 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5647e49de5e0 0x5647e49e09d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647e4979df0 0x5647e497f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5647e496e5d0 0x5647e4980700 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 --2- 172.21.15.23:0/3789905880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647e496eef0 0x5647e497ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 >> 172.21.15.23:0/3789905880 conn(0x5647e48ce6a0 msgr2=0x5647e49763b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b4ea00700 0 client.26177 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 shutdown_connections 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 1 -- 172.21.15.23:0/3789905880 wait complete. 2024-03-30T04:03:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.360+0000 7f1b83b82b80 20 client.26177 trim_cache size 0 max 16384 2024-03-30T04:03:59.364 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 msgr2=0x7f1b500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 0x7f1b500a3750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1b54005cd0 tx=0x7f1b54005fd0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 msgr2=0x7f1b6810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 0x7f1b6810b250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1b6000efc0 tx=0x7f1b6000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 shutdown_connections 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b6810ada0 0x7f1b6810b250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b68069260 0x7f1b6810d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1b500a1290 0x7f1b500a3750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 --2- 172.21.15.23:0/317890157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b681060b0 0x7f1b6810a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 >> 172.21.15.23:0/317890157 conn(0x7f1b68076d80 msgr2=0x7f1b680fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 shutdown_connections 2024-03-30T04:03:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.361+0000 7f1b70000700 1 -- 172.21.15.23:0/317890157 wait complete. 2024-03-30T04:03:59.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:03:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1148655503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:59.378 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:03:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:03:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1148655503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:59.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:03:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1148655503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:03:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 -- 172.21.15.23:0/1909590095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 msgr2=0x7efdc0101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 --2- 172.21.15.23:0/1909590095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc0101340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009990 tx=0x7efdb4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 -- 172.21.15.23:0/1909590095 shutdown_connections 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 --2- 172.21.15.23:0/1909590095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0106080 0x7efdc010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 --2- 172.21.15.23:0/1909590095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc0101340 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 --2- 172.21.15.23:0/1909590095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0101910 0x7efdc0105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 -- 172.21.15.23:0/1909590095 >> 172.21.15.23:0/1909590095 conn(0x7efdc00fc7c0 msgr2=0x7efdc00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 -- 172.21.15.23:0/1909590095 shutdown_connections 2024-03-30T04:03:59.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.592+0000 7efdc7400700 1 -- 172.21.15.23:0/1909590095 wait complete. 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.593+0000 7efdc7400700 1 Processor -- start 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.593+0000 7efdc7400700 1 -- start start 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0106080 0x7efdc00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0073b30 0x7efdc01a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01a5980 con 0x7efdc0073b30 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01a5af0 con 0x7efdc0100f60 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc7400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc01a5cb0 con 0x7efdc0106080 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56732/0 (socket says 172.21.15.23:56732) 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc4a00700 1 -- 172.21.15.23:0/986311709 learned_addr learned my addr 172.21.15.23:0/986311709 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc5400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0073b30 0x7efdc01a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdbfe00700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0106080 0x7efdc00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc4a00700 1 -- 172.21.15.23:0/986311709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0106080 msgr2=0x7efdc00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.594+0000 7efdc4a00700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0106080 0x7efdc00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc4a00700 1 -- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0073b30 msgr2=0x7efdc01a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc4a00700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0073b30 0x7efdc01a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc4a00700 1 -- 172.21.15.23:0/986311709 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb40095f0 con 0x7efdc0100f60 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc4a00700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efdb4004760 tx=0x7efdb4004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdbd600700 1 -- 172.21.15.23:0/986311709 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb401d070 con 0x7efdc0100f60 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc01a5f30 con 0x7efdc0100f60 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdbd600700 1 -- 172.21.15.23:0/986311709 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdb4022470 con 0x7efdc0100f60 2024-03-30T04:03:59.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdbd600700 1 -- 172.21.15.23:0/986311709 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb4017650 con 0x7efdc0100f60 2024-03-30T04:03:59.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.595+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc01a6360 con 0x7efdc0100f60 2024-03-30T04:03:59.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.598+0000 7efdbd600700 1 -- 172.21.15.23:0/986311709 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7efdb40225e0 con 0x7efdc0100f60 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.599+0000 7efddaebeb80 1 Processor -- start 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efdbd600700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 0x7efda80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 -- start start 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efdbd600700 1 -- 172.21.15.23:0/986311709 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efdb40ce120 con 0x7efdc0100f60 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efdc5400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 0x7efda80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604cecf9ef0 0x5604ced048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604ced04df0 0x5604ced071e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cec52990 con 0x5604cecf9ef0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cec50350 con 0x5604cecf95d0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.600+0000 7efddaebeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cec4bc40 con 0x5604ced04df0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdbfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604cecf9ef0 0x5604ced048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56738/0 (socket says 172.21.15.23:56738) 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 -- 172.21.15.23:0/3197282078 learned_addr learned my addr 172.21.15.23:0/3197282078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc5400700 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604ced04df0 0x5604ced071e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc5400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 0x7efda80a37d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdb800f590 tx=0x7efdb800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 -- 172.21.15.23:0/3197282078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604ced04df0 msgr2=0x5604ced071e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604ced04df0 0x5604ced071e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 -- 172.21.15.23:0/3197282078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604cecf9ef0 msgr2=0x5604ced048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604cecf9ef0 0x5604ced048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 -- 172.21.15.23:0/3197282078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604ceacd3b0 con 0x5604cecf95d0 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdc4a00700 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c010 tx=0x7efdb409ad50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ec5f4ac116f1bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdb1c00700 1 -- 172.21.15.23:0/3197282078 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb409b600 con 0x5604cecf95d0 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.601+0000 7efdb1c00700 1 -- 172.21.15.23:0/3197282078 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdb409af00 con 0x5604cecf95d0 2024-03-30T04:03:59.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efdb1c00700 1 -- 172.21.15.23:0/3197282078 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb409b3c0 con 0x5604cecf95d0 2024-03-30T04:03:59.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 -- 172.21.15.23:0/3197282078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 msgr2=0x5604cecf99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c010 tx=0x7efdb409ad50 comp rx=0 tx=0).stop 2024-03-30T04:03:59.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 -- 172.21.15.23:0/3197282078 shutdown_connections 2024-03-30T04:03:59.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604ced04df0 0x5604ced071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604cecf99b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 --2- 172.21.15.23:0/3197282078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604cecf9ef0 0x5604ced048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 -- 172.21.15.23:0/3197282078 >> 172.21.15.23:0/3197282078 conn(0x5604cec596a0 msgr2=0x5604cecf5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 -- 172.21.15.23:0/3197282078 shutdown_connections 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.602+0000 7efddaebeb80 1 -- 172.21.15.23:0/3197282078 wait complete. 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 -- start start 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 Processor -- start 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced05710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced10250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cebe7d80 con 0x5604ced04df0 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cebe7a20 con 0x5604cecf95d0 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.603+0000 7efddaebeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ceacf6a0 con 0x5604cecf9ef0 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efddaebeb80 10 client.0 _ll_register_callbacks cb 0x5604cebf1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56752/0 (socket says 172.21.15.23:56752) 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 -- 172.21.15.23:0/2971981687 learned_addr learned my addr 172.21.15.23:0/2971981687 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc5400700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced10250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdbfe00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced05710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 -- 172.21.15.23:0/2971981687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 msgr2=0x5604ced05710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced05710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 -- 172.21.15.23:0/2971981687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 msgr2=0x5604ced10250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced10250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 -- 172.21.15.23:0/2971981687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc01a6360 con 0x5604cecf95d0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efdc4a00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdb4004730 tx=0x7efdb40a10c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a648ef136375372 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efda6000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.604+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb409b600 con 0x5604cecf95d0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efdb409ed30 con 0x5604cecf95d0 2024-03-30T04:03:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604cec0c380 con 0x5604cecf95d0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdb40a32f0 con 0x5604cecf95d0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 10 client.26189 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5604cea79b60 con 0x5604cecf95d0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604cec0c380 con 0x5604cecf95d0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efdb4096d80 con 0x5604cecf95d0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 10 client.26189 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 10 client.26189 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 20 client.26189 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced14130 0x5604ced16550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 4 client.26189 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.605+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604cec44700 con 0x5604ced14130 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.606+0000 7efdbfe00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced14130 0x5604ced16550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.606+0000 7efdbfe00700 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced14130 0x5604ced16550 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cb061d9fd0f08f58 server_cookie=62dc187cb4287723 in_seq=0 out_seq=0 2024-03-30T04:03:59.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.606+0000 7efda6000700 10 client.26189 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efda6000700 1 -- 172.21.15.23:0/2971981687 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5604cec44700 con 0x5604ced14130 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efda6000700 10 client.26189 handle_command_reply: tid=0 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 client.26189 shutdown 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 msgr2=0x5604ced051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efdb4004730 tx=0x7efdb40a10c0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 shutdown_connections 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced14130 0x5604ced16550 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced10250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced051d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 --2- 172.21.15.23:0/2971981687 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced05710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efda6000700 0 client.26189 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 >> 172.21.15.23:0/2971981687 conn(0x5604cec596a0 msgr2=0x5604cecfb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 shutdown_connections 2024-03-30T04:03:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.607+0000 7efddaebeb80 1 -- 172.21.15.23:0/2971981687 wait complete. 2024-03-30T04:03:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.608+0000 7efddaebeb80 20 client.26189 trim_cache size 0 max 16384 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 Processor -- start 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 -- start start 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf95d0 0x5604ced0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 0x5604cecfe860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604cecfa850 con 0x5604ced04df0 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ced10ac0 con 0x5604cecf9ef0 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efddaebeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ced0a650 con 0x5604cecf95d0 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.622+0000 7efdc5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdc5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60232/0 (socket says 172.21.15.23:60232) 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdc5400700 1 -- 172.21.15.23:0/1829043713 learned_addr learned my addr 172.21.15.23:0/1829043713 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 0x5604cecfe860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdc4a00700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf95d0 0x5604ced0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 -- 172.21.15.23:0/1829043713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf95d0 msgr2=0x5604ced0a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf95d0 0x5604ced0a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 -- 172.21.15.23:0/1829043713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 msgr2=0x5604ced051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 -- 172.21.15.23:0/1829043713 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604cecf9440 con 0x5604cecf9ef0 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdc5400700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.623+0000 7efdbfe00700 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 0x5604cecfe860 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5604ced16e20 tx=0x7efdac014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e54734c1b69eaca server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efdb2600700 1 -- 172.21.15.23:0/1829043713 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac017030 con 0x5604cecf9ef0 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efdb2600700 1 -- 172.21.15.23:0/1829043713 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdac018040 con 0x5604cecf9ef0 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efdb2600700 1 -- 172.21.15.23:0/1829043713 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac003680 con 0x5604cecf9ef0 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 -- 172.21.15.23:0/1829043713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 msgr2=0x5604cecfe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 0x5604cecfe860 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x5604ced16e20 tx=0x7efdac014040 comp rx=0 tx=0).stop 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 -- 172.21.15.23:0/1829043713 shutdown_connections 2024-03-30T04:03:59.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf95d0 0x5604ced0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 --2- 172.21.15.23:0/1829043713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf9ef0 0x5604cecfe860 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 -- 172.21.15.23:0/1829043713 >> 172.21.15.23:0/1829043713 conn(0x5604cec596a0 msgr2=0x5604ceceef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 -- 172.21.15.23:0/1829043713 shutdown_connections 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.624+0000 7efddaebeb80 1 -- 172.21.15.23:0/1829043713 wait complete. 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 -- start start 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 Processor -- start 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced60840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced5beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ced5c590 con 0x5604ced04df0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ced5c700 con 0x5604cecf95d0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ced5c8c0 con 0x5604cecf9ef0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.625+0000 7efddaebeb80 10 client.0 _ll_register_callbacks cb 0x5604cebf1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56764/0 (socket says 172.21.15.23:56764) 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 -- 172.21.15.23:0/33440689 learned_addr learned my addr 172.21.15.23:0/33440689 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc5400700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced5beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdbfe00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced60840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 -- 172.21.15.23:0/33440689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 msgr2=0x5604ced60840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced60840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 -- 172.21.15.23:0/33440689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 msgr2=0x5604ced5beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced5beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 -- 172.21.15.23:0/33440689 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb800fe50 con 0x5604cecf95d0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdbfe00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced60840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efdc4a00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efdb409ac00 tx=0x7efdb40a1130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a711f1e9f9fd6434 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.626+0000 7efda6000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb40a3060 con 0x5604cecf95d0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efdb409b600 con 0x5604cecf95d0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdb409af00 con 0x5604cecf95d0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdb40a32f0 con 0x5604cecf95d0 2024-03-30T04:03:59.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efddaebeb80 10 client.26195 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.627+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7efdac010430 con 0x5604cecf95d0 2024-03-30T04:03:59.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdb409af00 con 0x5604cecf95d0 2024-03-30T04:03:59.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efdb409b160 con 0x5604cecf95d0 2024-03-30T04:03:59.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 10 client.26195 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 10 client.26195 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 20 client.26195 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced68d70 0x5604ced6b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 4 client.26195 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.628+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5604cea9acf0 con 0x5604ced68d70 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.629+0000 7efdbfe00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced68d70 0x5604ced6b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.629+0000 7efdbfe00700 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced68d70 0x5604ced6b160 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83ba74167d4f64ab server_cookie=9a82062c32c0f865 in_seq=0 out_seq=0 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.629+0000 7efda6000700 10 client.26195 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efda6000700 1 -- 172.21.15.23:0/33440689 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5604cea9acf0 con 0x5604ced68d70 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efda6000700 10 client.26195 handle_command_reply: tid=0 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 client.26195 shutdown 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 msgr2=0x5604ced60300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efdb409ac00 tx=0x7efdb40a1130 comp rx=0 tx=0).stop 2024-03-30T04:03:59.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 shutdown_connections 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5604ced68d70 0x5604ced6b160 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604ced04df0 0x5604ced5beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efda6000700 0 client.26195 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604cecf95d0 0x5604ced60300 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 --2- 172.21.15.23:0/33440689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604cecf9ef0 0x5604ced60840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 >> 172.21.15.23:0/33440689 conn(0x5604cec596a0 msgr2=0x5604ced051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.630+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 shutdown_connections 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efddaebeb80 1 -- 172.21.15.23:0/33440689 wait complete. 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efddaebeb80 20 client.26195 trim_cache size 0 max 16384 2024-03-30T04:03:59.633 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 msgr2=0x7efda80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 0x7efda80a37d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdb800f590 tx=0x7efdb800b040 comp rx=0 tx=0).stop 2024-03-30T04:03:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 msgr2=0x7efdc00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7efdb4004760 tx=0x7efdb4004790 comp rx=0 tx=0).stop 2024-03-30T04:03:59.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 shutdown_connections 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdc0073b30 0x7efdc01a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efda80a1310 0x7efda80a37d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdc0100f60 0x7efdc00730b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 --2- 172.21.15.23:0/986311709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdc0106080 0x7efdc00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.631+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 >> 172.21.15.23:0/986311709 conn(0x7efdc00fc7c0 msgr2=0x7efdc00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.632+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 shutdown_connections 2024-03-30T04:03:59.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.632+0000 7efdc7400700 1 -- 172.21.15.23:0/986311709 wait complete. 2024-03-30T04:03:59.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:03:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4156083709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a0100f50 msgr2=0x7fd6a0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4156083709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a0100f50 0x7fd6a0101330 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd694009990 tx=0x7fd694009ca0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4156083709 shutdown_connections 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4156083709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4156083709 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a0100f50 0x7fd6a0101330 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4156083709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0101900 0x7fd6a0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.868+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4156083709 >> 172.21.15.23:0/4156083709 conn(0x7fd6a00fc7f0 msgr2=0x7fd6a00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.869+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4156083709 shutdown_connections 2024-03-30T04:03:59.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.869+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4156083709 wait complete. 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 Processor -- start 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 -- start start 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 0x7fd6a01972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a019bb80 0x7fd6a019c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a019c600 con 0x7fd6a019bb80 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a019c770 con 0x7fd6a0106070 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a7a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a019c930 con 0x7fd6a0100f50 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 0x7fd6a01972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a019bb80 0x7fd6a019c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 0x7fd6a01972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43642/0 (socket says 172.21.15.23:43642) 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56780/0 (socket says 172.21.15.23:56780) 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.870+0000 7fd6a5000700 1 -- 172.21.15.23:0/4265015113 learned_addr learned my addr 172.21.15.23:0/4265015113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 -- 172.21.15.23:0/4265015113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 msgr2=0x7fd6a01972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 0x7fd6a01972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 -- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a019bb80 msgr2=0x7fd6a019c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a019bb80 0x7fd6a019c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 -- 172.21.15.23:0/4265015113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6940095f0 con 0x7fd6a0106070 2024-03-30T04:03:59.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69fe00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00e9c0 tx=0x7fd68c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69d600700 1 -- 172.21.15.23:0/4265015113 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c00c9c0 con 0x7fd6a0106070 2024-03-30T04:03:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a01a3e90 con 0x7fd6a0106070 2024-03-30T04:03:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69d600700 1 -- 172.21.15.23:0/4265015113 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd68c010430 con 0x7fd6a0106070 2024-03-30T04:03:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.871+0000 7fd69d600700 1 -- 172.21.15.23:0/4265015113 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c018680 con 0x7fd6a0106070 2024-03-30T04:03:59.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.872+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a01a4360 con 0x7fd6a0106070 2024-03-30T04:03:59.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.875+0000 7fd69d600700 1 -- 172.21.15.23:0/4265015113 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd68c00cb20 con 0x7fd6a0106070 2024-03-30T04:03:59.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.876+0000 7fd69d600700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 0x7fd6880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.876+0000 7fd69d600700 1 -- 172.21.15.23:0/4265015113 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd68c014070 con 0x7fd6a0106070 2024-03-30T04:03:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.876+0000 7fd6a5000700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 0x7fd6880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6bb4e3b80 1 Processor -- start 2024-03-30T04:03:59.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6bb4e3b80 1 -- start start 2024-03-30T04:03:59.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6a5000700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 0x7fd6880a3750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd694006010 tx=0x7fd69400b280 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 0x563b563c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.877+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b56310990 con 0x563b563c2df0 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b5630e350 con 0x563b563b75d0 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b56309c40 con 0x563b563b7ef0 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6a5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6a5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60274/0 (socket says 172.21.15.23:60274) 2024-03-30T04:03:59.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6a5a00700 1 -- 172.21.15.23:0/4075692904 learned_addr learned my addr 172.21.15.23:0/4075692904 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd69fe00700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 0x563b563c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.878+0000 7fd6a5000700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 -- 172.21.15.23:0/4075692904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 msgr2=0x563b563c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 0x563b563c28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 -- 172.21.15.23:0/4075692904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 msgr2=0x563b563b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563b79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 -- 172.21.15.23:0/4075692904 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b5618b3b0 con 0x563b563c2df0 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd69fe00700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 0x563b563c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd6a5a00700 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd698011680 tx=0x7fd698009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4fe87929127f03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.879+0000 7fd692000700 1 -- 172.21.15.23:0/4075692904 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800e040 con 0x563b563c2df0 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd692000700 1 -- 172.21.15.23:0/4075692904 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd69800f040 con 0x563b563c2df0 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd692000700 1 -- 172.21.15.23:0/4075692904 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980048c0 con 0x563b563c2df0 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/4075692904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 msgr2=0x563b563c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd698011680 tx=0x7fd698009920 comp rx=0 tx=0).stop 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/4075692904 shutdown_connections 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563c51e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563b79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/4075692904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b7ef0 0x563b563c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.880+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/4075692904 >> 172.21.15.23:0/4075692904 conn(0x563b563176a0 msgr2=0x563b563b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.881+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/4075692904 shutdown_connections 2024-03-30T04:03:59.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.881+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/4075692904 wait complete. 2024-03-30T04:03:59.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.881+0000 7fd6bb4e3b80 1 -- start start 2024-03-30T04:03:59.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.881+0000 7fd6bb4e3b80 1 Processor -- start 2024-03-30T04:03:59.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 0x563b563bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b563c00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b562a5d80 con 0x563b563b7ef0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b562a5a20 con 0x563b563b75d0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b5618d6a0 con 0x563b563c2df0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6bb4e3b80 10 client.0 _ll_register_callbacks cb 0x563b562af470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 0x563b563bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56806/0 (socket says 172.21.15.23:56806) 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6a5000700 1 -- 172.21.15.23:0/3973033838 learned_addr learned my addr 172.21.15.23:0/3973033838 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.882+0000 7fd6a5a00700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b563c00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 -- 172.21.15.23:0/3973033838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 msgr2=0x563b563c00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b563c00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 -- 172.21.15.23:0/3973033838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 msgr2=0x563b563bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 0x563b563bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 -- 172.21.15.23:0/3973033838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd698011330 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 0x563b563bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6a5000700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd69400be50 tx=0x7fd69401a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=825af871bb703690 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd686a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69400be80 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd694004c70 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.883+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b562ca380 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd694004e30 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 10 client.26207 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563b56137b60 con 0x563b563b75d0 2024-03-30T04:03:59.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563b562ca380 con 0x563b563b75d0 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd6940245a0 con 0x563b563b75d0 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 10 client.26207 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 10 client.26207 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 20 client.26207 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b563d2920 0x563b563d4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 4 client.26207 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b56302700 con 0x563b563d2920 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.884+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b563d2920 0x563b563d4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.885+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b563d2920 0x563b563d4d40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b23dbd461db64c41 server_cookie=e42c738b33fd435d in_seq=0 out_seq=0 2024-03-30T04:03:59.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.885+0000 7fd686a00700 10 client.26207 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.885+0000 7fd686a00700 1 -- 172.21.15.23:0/3973033838 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563b56302700 con 0x563b563d2920 2024-03-30T04:03:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.885+0000 7fd686a00700 10 client.26207 handle_command_reply: tid=0 2024-03-30T04:03:59.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.885+0000 7fd6bb4e3b80 1 client.26207 shutdown 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 msgr2=0x563b563c2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd69400be50 tx=0x7fd69401a050 comp rx=0 tx=0).stop 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 shutdown_connections 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b563d2920 0x563b563d4d40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b563c00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b75d0 0x563b563c2750 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3973033838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b7ef0 0x563b563bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd686a00700 0 client.26207 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 >> 172.21.15.23:0/3973033838 conn(0x563b563176a0 msgr2=0x563b563b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 shutdown_connections 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3973033838 wait complete. 2024-03-30T04:03:59.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.886+0000 7fd6bb4e3b80 20 client.26207 trim_cache size 0 max 16384 2024-03-30T04:03:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 Processor -- start 2024-03-30T04:03:59.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 -- start start 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 0x563b563d4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b563b73f0 con 0x563b563c2df0 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b563c9fb0 con 0x563b563b7ef0 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.902+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b563acae0 con 0x563b563b75d0 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd6a5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 0x563b563d4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 0x563b563d4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43676/0 (socket says 172.21.15.23:43676) 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd6a5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563ac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60294/0 (socket says 172.21.15.23:60294) 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd6a5000700 1 -- 172.21.15.23:0/1901079049 learned_addr learned my addr 172.21.15.23:0/1901079049 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56820/0 (socket says 172.21.15.23:56820) 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 -- 172.21.15.23:0/1901079049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 msgr2=0x563b563d4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 0x563b563d4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 -- 172.21.15.23:0/1901079049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 msgr2=0x563b563ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.903+0000 7fd69fe00700 1 -- 172.21.15.23:0/1901079049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b562cfea0 con 0x563b563b7ef0 2024-03-30T04:03:59.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd69fe00700 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd68c0ca510 tx=0x7fd68c010c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad1bd4f448072949 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd692a00700 1 -- 172.21.15.23:0/1901079049 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c099ed0 con 0x563b563b7ef0 2024-03-30T04:03:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd692a00700 1 -- 172.21.15.23:0/1901079049 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd68c09dd00 con 0x563b563b7ef0 2024-03-30T04:03:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd692a00700 1 -- 172.21.15.23:0/1901079049 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c010860 con 0x563b563b7ef0 2024-03-30T04:03:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/1901079049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 msgr2=0x563b563c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.904+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd68c0ca510 tx=0x7fd68c010c30 comp rx=0 tx=0).stop 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/1901079049 shutdown_connections 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563c2df0 0x563b563ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563b75d0 0x563b563d4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/1901079049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c9a70 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/1901079049 >> 172.21.15.23:0/1901079049 conn(0x563b563176a0 msgr2=0x563b563d96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/1901079049 shutdown_connections 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/1901079049 wait complete. 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 -- start start 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.905+0000 7fd6bb4e3b80 1 Processor -- start 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b75d0 0x563b563c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b5641f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b5641fd80 con 0x563b563b75d0 2024-03-30T04:03:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b5641fef0 con 0x563b563b7ef0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b564200b0 con 0x563b563c2df0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6bb4e3b80 10 client.0 _ll_register_callbacks cb 0x563b562af470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6a5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b75d0 0x563b563c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56822/0 (socket says 172.21.15.23:56822) 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 -- 172.21.15.23:0/3922726171 learned_addr learned my addr 172.21.15.23:0/3922726171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd6a5a00700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b5641f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 -- 172.21.15.23:0/3922726171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 msgr2=0x563b5641f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b5641f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 -- 172.21.15.23:0/3922726171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b75d0 msgr2=0x563b563c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b75d0 0x563b563c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.906+0000 7fd69fe00700 1 -- 172.21.15.23:0/3922726171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a01a4360 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd69fe00700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd68c096940 tx=0x7fd68c0a2350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74d0ed82adf070f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0b8280 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd68c099ed0 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd69800f040 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd68c010860 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd6bb4e3b80 10 client.26213 fetch_fsmap learned FSMap version 29 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563b5618fcd0 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd69800f040 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.907+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd68c0a1fc0 con 0x563b563b7ef0 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 10 client.26213 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 10 client.26213 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:03:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 20 client.26213 populate_metadata read hostname 'smithi023' 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b564274e0 0x563b56429900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 4 client.26213 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563b56158cf0 con 0x563b564274e0 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.908+0000 7fd6a5000700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b564274e0 0x563b56429900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6a5000700 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b564274e0 0x563b56429900 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf08495d15f330bc server_cookie=80074dd1bef101ed in_seq=0 out_seq=0 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd686a00700 10 client.26213 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd686a00700 1 -- 172.21.15.23:0/3922726171 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563b56158cf0 con 0x563b564274e0 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd686a00700 10 client.26213 handle_command_reply: tid=0 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 client.26213 shutdown 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 msgr2=0x563b563c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd68c096940 tx=0x7fd68c0a2350 comp rx=0 tx=0).stop 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 shutdown_connections 2024-03-30T04:03:59.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563b564274e0 0x563b56429900 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b563c2df0 0x563b5641f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b563b75d0 0x563b563c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd686a00700 0 client.26213 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 --2- 172.21.15.23:0/3922726171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b563b7ef0 0x563b563c8180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 >> 172.21.15.23:0/3922726171 conn(0x563b563176a0 msgr2=0x563b563acf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 shutdown_connections 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.909+0000 7fd6bb4e3b80 1 -- 172.21.15.23:0/3922726171 wait complete. 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6bb4e3b80 20 client.26213 trim_cache size 0 max 16384 2024-03-30T04:03:59.913 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:03:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 msgr2=0x7fd6880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 0x7fd6880a3750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd694006010 tx=0x7fd69400b280 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 msgr2=0x7fd6a01977e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00e9c0 tx=0x7fd68c00ecd0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 shutdown_connections 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6a019bb80 0x7fd6a019c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a0100f50 0x7fd6a01972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6880a1290 0x7fd6880a3750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 --2- 172.21.15.23:0/4265015113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a0106070 0x7fd6a01977e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 >> 172.21.15.23:0/4265015113 conn(0x7fd6a00fc7f0 msgr2=0x7fd6a00fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 shutdown_connections 2024-03-30T04:03:59.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:03:59.910+0000 7fd6a7a00700 1 -- 172.21.15.23:0/4265015113 wait complete. 2024-03-30T04:03:59.927 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 -- 172.21.15.23:0/3652680971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 msgr2=0x7efee810d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 --2- 172.21.15.23:0/3652680971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee810d040 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efed40098f0 tx=0x7efed4009c00 comp rx=0 tx=0).stop 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 -- 172.21.15.23:0/3652680971 shutdown_connections 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 --2- 172.21.15.23:0/3652680971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 0x7efee810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 --2- 172.21.15.23:0/3652680971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 0x7efee8073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 --2- 172.21.15.23:0/3652680971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee810d040 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 -- 172.21.15.23:0/3652680971 >> 172.21.15.23:0/3652680971 conn(0x7efee80fc7f0 msgr2=0x7efee80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 -- 172.21.15.23:0/3652680971 shutdown_connections 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.162+0000 7efef0600700 1 -- 172.21.15.23:0/3652680971 wait complete. 2024-03-30T04:04:00.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.163+0000 7efef0600700 1 Processor -- start 2024-03-30T04:04:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 -- start start 2024-03-30T04:04:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 0x7efee81994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 0x7efee819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819e3f0 con 0x7efee8073180 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819e560 con 0x7efee8073aa0 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efef0600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee819e720 con 0x7efee810d580 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efeed200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efeed200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56846/0 (socket says 172.21.15.23:56846) 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efeed200700 1 -- 172.21.15.23:0/1886428377 learned_addr learned my addr 172.21.15.23:0/1886428377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efeee600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 0x7efee819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.164+0000 7efeedc00700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 0x7efee81994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 -- 172.21.15.23:0/1886428377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 msgr2=0x7efee819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 0x7efee819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 -- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 msgr2=0x7efee81994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 0x7efee81994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 -- 172.21.15.23:0/1886428377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed4009520 con 0x7efee8073aa0 2024-03-30T04:04:00.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efeed200700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efed4005950 tx=0x7efed400fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efee6000700 1 -- 172.21.15.23:0/1886428377 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed401c070 con 0x7efee8073aa0 2024-03-30T04:04:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efee6000700 1 -- 172.21.15.23:0/1886428377 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efed400bb20 con 0x7efee8073aa0 2024-03-30T04:04:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee819e9a0 con 0x7efee8073aa0 2024-03-30T04:04:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efee6000700 1 -- 172.21.15.23:0/1886428377 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed40038e0 con 0x7efee8073aa0 2024-03-30T04:04:00.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.165+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee81a6240 con 0x7efee8073aa0 2024-03-30T04:04:00.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.168+0000 7efee6000700 1 -- 172.21.15.23:0/1886428377 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7efed4020030 con 0x7efee8073aa0 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.170+0000 7efee6000700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 0x7efed00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.170+0000 7efeedc00700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 0x7efed00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.170+0000 7efee6000700 1 -- 172.21.15.23:0/1886428377 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efed40cdec0 con 0x7efee8073aa0 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.170+0000 7eff0411bb80 1 Processor -- start 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 -- start start 2024-03-30T04:04:00.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7efeedc00700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 0x7efed00a37d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efedc005fd0 tx=0x7efedc009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed9418b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed88f990 con 0x5559ed936ef0 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed88d350 con 0x5559ed9365d0 2024-03-30T04:04:00.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7eff0411bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed888c40 con 0x5559ed941df0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7efeedc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7efeedc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56852/0 (socket says 172.21.15.23:56852) 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7efeedc00700 1 -- 172.21.15.23:0/1164717777 learned_addr learned my addr 172.21.15.23:0/1164717777 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.171+0000 7efeed200700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed9418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeee600700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 -- 172.21.15.23:0/1164717777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 msgr2=0x5559ed9441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 -- 172.21.15.23:0/1164717777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 msgr2=0x5559ed9418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed9418b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 -- 172.21.15.23:0/1164717777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559ed70a3b0 con 0x5559ed9365d0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeee600700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efeedc00700 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efedc005b40 tx=0x7efedc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aed610fc1499be6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efedaa00700 1 -- 172.21.15.23:0/1164717777 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc012dc0 con 0x5559ed9365d0 2024-03-30T04:04:00.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efedaa00700 1 -- 172.21.15.23:0/1164717777 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efedc016070 con 0x5559ed9365d0 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.172+0000 7efedaa00700 1 -- 172.21.15.23:0/1164717777 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc01d610 con 0x5559ed9365d0 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 -- 172.21.15.23:0/1164717777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 msgr2=0x5559ed9369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efedc005b40 tx=0x7efedc014000 comp rx=0 tx=0).stop 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 -- 172.21.15.23:0/1164717777 shutdown_connections 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed9369b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 --2- 172.21.15.23:0/1164717777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed9418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 -- 172.21.15.23:0/1164717777 >> 172.21.15.23:0/1164717777 conn(0x5559ed8966a0 msgr2=0x5559ed932a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 -- 172.21.15.23:0/1164717777 shutdown_connections 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.173+0000 7eff0411bb80 1 -- 172.21.15.23:0/1164717777 wait complete. 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 -- start start 2024-03-30T04:04:00.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 Processor -- start 2024-03-30T04:04:00.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed936ef0 0x5559ed93eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed93f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed824d80 con 0x5559ed9365d0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed824a20 con 0x5559ed936ef0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed70c6a0 con 0x5559ed941df0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.174+0000 7eff0411bb80 10 client.0 _ll_register_callbacks cb 0x5559ed82e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60322/0 (socket says 172.21.15.23:60322) 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 -- 172.21.15.23:0/473583601 learned_addr learned my addr 172.21.15.23:0/473583601 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeee600700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed93f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeed200700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed936ef0 0x5559ed93eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 -- 172.21.15.23:0/473583601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 msgr2=0x5559ed93f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed93f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 -- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed936ef0 msgr2=0x5559ed93eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed936ef0 0x5559ed93eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 -- 172.21.15.23:0/473583601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc011b40 con 0x5559ed9365d0 2024-03-30T04:04:00.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efeedc00700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efedc01f680 tx=0x7efedc01f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8864404239deb086 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.175+0000 7efecf400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc02c070 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efedc018070 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559ed849380 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efedc0119e0 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7eff0411bb80 10 client.16260 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.176+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5559ed6b6b60 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5559ed849380 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efedc012450 con 0x5559ed9365d0 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 10 client.16260 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 10 client.16260 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 20 client.16260 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed951880 0x5559ed953c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 4 client.16260 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559ed881700 con 0x5559ed951880 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.177+0000 7efeed200700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed951880 0x5559ed953c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.178+0000 7efeed200700 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed951880 0x5559ed953c70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d2cd169fe1f7edf server_cookie=334e945b0aa22b3b in_seq=0 out_seq=0 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.178+0000 7efecf400700 10 client.16260 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.178+0000 7efecf400700 1 -- 172.21.15.23:0/473583601 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5559ed881700 con 0x5559ed951880 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7efecf400700 10 client.16260 handle_command_reply: tid=0 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 client.16260 shutdown 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 msgr2=0x5559ed9417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7efedc01f680 tx=0x7efedc01f6b0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 shutdown_connections 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed951880 0x5559ed953c70 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed93f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed9365d0 0x5559ed9417e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 --2- 172.21.15.23:0/473583601 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed936ef0 0x5559ed93eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 >> 172.21.15.23:0/473583601 conn(0x5559ed8966a0 msgr2=0x5559ed938530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7efecf400700 0 client.16260 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:00.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 shutdown_connections 2024-03-30T04:04:00.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 1 -- 172.21.15.23:0/473583601 wait complete. 2024-03-30T04:04:00.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.179+0000 7eff0411bb80 20 client.16260 trim_cache size 0 max 16384 2024-03-30T04:04:00.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 77711 11.78 MB/sec warmup 37 sec latency 11.634 ms 2024-03-30T04:04:00.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.194+0000 7eff0411bb80 1 Processor -- start 2024-03-30T04:04:00.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.194+0000 7eff0411bb80 1 -- start start 2024-03-30T04:04:00.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 0x5559ed948a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed93d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed940120 con 0x5559ed936ef0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed93f500 con 0x5559ed941df0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7eff0411bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed9378a0 con 0x5559ed9365d0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeed200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed93d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56874/0 (socket says 172.21.15.23:56874) 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeedc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 0x5559ed948a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 -- 172.21.15.23:0/708102018 learned_addr learned my addr 172.21.15.23:0/708102018 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 -- 172.21.15.23:0/708102018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 msgr2=0x5559ed948a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 0x5559ed948a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 -- 172.21.15.23:0/708102018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 msgr2=0x5559ed93d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed93d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeee600700 1 -- 172.21.15.23:0/708102018 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559ed84eea0 con 0x5559ed941df0 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.195+0000 7efeedc00700 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 0x5559ed948a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.196+0000 7efeee600700 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efee0007710 tx=0x7efee000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ed678f70a02244 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.196+0000 7efedb400700 1 -- 172.21.15.23:0/708102018 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000e040 con 0x5559ed941df0 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.196+0000 7efedb400700 1 -- 172.21.15.23:0/708102018 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee000f040 con 0x5559ed941df0 2024-03-30T04:04:00.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.196+0000 7efedb400700 1 -- 172.21.15.23:0/708102018 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0017a80 con 0x5559ed941df0 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- 172.21.15.23:0/708102018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 msgr2=0x5559ed93d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efee0007710 tx=0x7efee000b040 comp rx=0 tx=0).stop 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- 172.21.15.23:0/708102018 shutdown_connections 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed941df0 0x5559ed93d950 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed9365d0 0x5559ed948a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 --2- 172.21.15.23:0/708102018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed93d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- 172.21.15.23:0/708102018 >> 172.21.15.23:0/708102018 conn(0x5559ed8966a0 msgr2=0x5559ed92bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- 172.21.15.23:0/708102018 shutdown_connections 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- 172.21.15.23:0/708102018 wait complete. 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 -- start start 2024-03-30T04:04:00.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.197+0000 7eff0411bb80 1 Processor -- start 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9a4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed9a4bc0 con 0x5559ed936ef0 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed9a4d00 con 0x5559ed9365d0 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559ed9a4e40 con 0x5559ed941df0 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7eff0411bb80 10 client.0 _ll_register_callbacks cb 0x5559ed82e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:00.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeedc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeed200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeed200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60346/0 (socket says 172.21.15.23:60346) 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeed200700 1 -- 172.21.15.23:0/3655528059 learned_addr learned my addr 172.21.15.23:0/3655528059 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeedc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56886/0 (socket says 172.21.15.23:56886) 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.198+0000 7efeee600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 -- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 msgr2=0x5559ed946c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 -- 172.21.15.23:0/3655528059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 msgr2=0x5559ed947180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 -- 172.21.15.23:0/3655528059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559ed6b6b60 con 0x5559ed941df0 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeedc00700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeed200700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efeee600700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9a4680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efee0009960 tx=0x7efee0017930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7e2c897f43731df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:00.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efecf400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000e050 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efee0028480 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efedc016070 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efee0028680 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7eff0411bb80 10 client.26221 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.199+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7efee0009300 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efedc016070 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efee0005410 con 0x5559ed941df0 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 10 client.26221 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 10 client.26221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 20 client.26221 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed9a64c0 0x5559ed9a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 4 client.26221 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:00.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5559ed6d7cf0 con 0x5559ed9a64c0 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.200+0000 7efeedc00700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed9a64c0 0x5559ed9a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7efeedc00700 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed9a64c0 0x5559ed9a88e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4976e1b091dbe079 server_cookie=bdd220dc65912523 in_seq=0 out_seq=0 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7efecf400700 10 client.26221 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7efecf400700 1 -- 172.21.15.23:0/3655528059 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5559ed6d7cf0 con 0x5559ed9a64c0 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7efecf400700 10 client.26221 handle_command_reply: tid=0 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 client.26221 shutdown 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 msgr2=0x5559ed9a4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9a4680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efee0009960 tx=0x7efee0017930 comp rx=0 tx=0).stop 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 shutdown_connections 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5559ed9a64c0 0x5559ed9a88e0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559ed941df0 0x5559ed9a4680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7efecf400700 0 client.26221 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5559ed9365d0 0x5559ed946c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 --2- 172.21.15.23:0/3655528059 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5559ed936ef0 0x5559ed947180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 >> 172.21.15.23:0/3655528059 conn(0x5559ed8966a0 msgr2=0x5559ed93d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 shutdown_connections 2024-03-30T04:04:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 1 -- 172.21.15.23:0/3655528059 wait complete. 2024-03-30T04:04:00.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.201+0000 7eff0411bb80 20 client.26221 trim_cache size 0 max 16384 2024-03-30T04:04:00.206 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 msgr2=0x7efed00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 0x7efed00a37d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efedc005fd0 tx=0x7efedc009310 comp rx=0 tx=0).stop 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 msgr2=0x7efee8199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efed4005950 tx=0x7efed400fa00 comp rx=0 tx=0).stop 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 shutdown_connections 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efee810d580 0x7efee819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efee8073180 0x7efee81994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efed00a1310 0x7efed00a37d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 --2- 172.21.15.23:0/1886428377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efee8073aa0 0x7efee8199a10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 >> 172.21.15.23:0/1886428377 conn(0x7efee80fc7f0 msgr2=0x7efee80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 shutdown_connections 2024-03-30T04:04:00.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:00.202+0000 7efef0600700 1 -- 172.21.15.23:0/1886428377 wait complete. 2024-03-30T04:04:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:00 smithi193 ceph-mon[79581]: pgmap v65: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-03-30T04:04:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:00 smithi023 ceph-mon[38254]: pgmap v65: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-03-30T04:04:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:00 smithi154 ceph-mon[79161]: pgmap v65: 129 pgs: 129 active+clean; 49 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.7 MiB/s wr, 236 op/s 2024-03-30T04:04:01.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 79101 11.65 MB/sec warmup 38 sec latency 69.683 ms 2024-03-30T04:04:01.220 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:04:01.221 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:04:01.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 -- 172.21.15.23:0/227257639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280100f70 msgr2=0x7fb280101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 --2- 172.21.15.23:0/227257639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280100f70 0x7fb280101350 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb274009990 tx=0x7fb274009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 -- 172.21.15.23:0/227257639 shutdown_connections 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 --2- 172.21.15.23:0/227257639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 0x7fb28010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 --2- 172.21.15.23:0/227257639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280100f70 0x7fb280101350 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 --2- 172.21.15.23:0/227257639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280101920 0x7fb280105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.444+0000 7fb287a00700 1 -- 172.21.15.23:0/227257639 >> 172.21.15.23:0/227257639 conn(0x7fb2800fc7f0 msgr2=0x7fb2800fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.445+0000 7fb287a00700 1 -- 172.21.15.23:0/227257639 shutdown_connections 2024-03-30T04:04:01.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.445+0000 7fb287a00700 1 -- 172.21.15.23:0/227257639 wait complete. 2024-03-30T04:04:01.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 Processor -- start 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 -- start start 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 0x7fb2800735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280073b30 0x7fb2801a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801a5a00 con 0x7fb280106090 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801a5b40 con 0x7fb280100f70 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.446+0000 7fb287a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2801a5d00 con 0x7fb280073b30 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb27fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 0x7fb2800735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56908/0 (socket says 172.21.15.23:56908) 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 -- 172.21.15.23:0/2204750390 learned_addr learned my addr 172.21.15.23:0/2204750390 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280073b30 0x7fb2801a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 -- 172.21.15.23:0/2204750390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280073b30 msgr2=0x7fb2801a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280073b30 0x7fb2801a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 -- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 msgr2=0x7fb2800735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 0x7fb2800735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 -- 172.21.15.23:0/2204750390 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c009520 con 0x7fb280100f70 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.447+0000 7fb285000700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb274005910 tx=0x7fb274004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.448+0000 7fb27d600700 1 -- 172.21.15.23:0/2204750390 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27401d070 con 0x7fb280100f70 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.448+0000 7fb27d600700 1 -- 172.21.15.23:0/2204750390 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb274022470 con 0x7fb280100f70 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.448+0000 7fb27d600700 1 -- 172.21.15.23:0/2204750390 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb274017650 con 0x7fb280100f70 2024-03-30T04:04:01.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.448+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2740095f0 con 0x7fb280100f70 2024-03-30T04:04:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.448+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2801a61c0 con 0x7fb280100f70 2024-03-30T04:04:01.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.451+0000 7fb27d600700 1 -- 172.21.15.23:0/2204750390 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb2740225e0 con 0x7fb280100f70 2024-03-30T04:04:01.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.453+0000 7fb29b47bb80 1 Processor -- start 2024-03-30T04:04:01.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.453+0000 7fb29b47bb80 1 -- start start 2024-03-30T04:04:01.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.453+0000 7fb27d600700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 0x7fb2680a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4e9da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb27d600700 1 -- 172.21.15.23:0/2204750390 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb2740ce280 con 0x7fb280100f70 2024-03-30T04:04:01.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eab260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4df6a10 con 0x5615b4e9df70 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4df43d0 con 0x5615b4e9d650 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4defcc0 con 0x5615b4ea8e70 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb285a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 0x7fb2680a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb27fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb285a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb27fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60388/0 (socket says 172.21.15.23:60388) 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb27fe00700 1 -- 172.21.15.23:0/1567047432 learned_addr learned my addr 172.21.15.23:0/1567047432 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.454+0000 7fb285000700 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4e9da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 -- 172.21.15.23:0/1567047432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 msgr2=0x5615b4eab260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eab260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 -- 172.21.15.23:0/1567047432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 msgr2=0x5615b4e9da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4e9da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 -- 172.21.15.23:0/1567047432 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615b4c71430 con 0x5615b4e9df70 2024-03-30T04:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb285000700 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4e9da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb285a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 0x7fb2680a37d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb278009ec0 tx=0x7fb2780092f0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb27fe00700 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009e80 tx=0x7fb26c011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b77ada6ce8be84 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb272000700 1 -- 172.21.15.23:0/1567047432 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c004250 con 0x5615b4e9df70 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb272000700 1 -- 172.21.15.23:0/1567047432 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb26c0043b0 con 0x5615b4e9df70 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.455+0000 7fb272000700 1 -- 172.21.15.23:0/1567047432 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c019850 con 0x5615b4e9df70 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 -- 172.21.15.23:0/1567047432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 msgr2=0x5615b4ea8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009e80 tx=0x7fb26c011720 comp rx=0 tx=0).stop 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 -- 172.21.15.23:0/1567047432 shutdown_connections 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eab260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4e9da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/1567047432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea8930 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 -- 172.21.15.23:0/1567047432 >> 172.21.15.23:0/1567047432 conn(0x5615b4dfd720 msgr2=0x5615b4e99b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 -- 172.21.15.23:0/1567047432 shutdown_connections 2024-03-30T04:04:01.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.456+0000 7fb29b47bb80 1 -- 172.21.15.23:0/1567047432 wait complete. 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 -- start start 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 Processor -- start 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 0x5615b4ea5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4ea8e70 0x5615b4ea61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4d8be00 con 0x5615b4ea8e70 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4d8baa0 con 0x5615b4e9d650 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.457+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4c73720 con 0x5615b4e9df70 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb29b47bb80 10 client.0 _ll_register_callbacks cb 0x5615b4d954f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb27fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 0x5615b4ea5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56924/0 (socket says 172.21.15.23:56924) 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 -- 172.21.15.23:0/3139668697 learned_addr learned my addr 172.21.15.23:0/3139668697 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285a00700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4ea8e70 0x5615b4ea61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 -- 172.21.15.23:0/3139668697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 msgr2=0x5615b4ea5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 0x5615b4ea5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 -- 172.21.15.23:0/3139668697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4ea8e70 msgr2=0x5615b4ea61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4ea8e70 0x5615b4ea61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 -- 172.21.15.23:0/3139668697 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2801a61c0 con 0x5615b4e9d650 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb27fe00700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 0x5615b4ea5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb285000700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb27409aff0 tx=0x7fb274009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=645463d62f7d0d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.458+0000 7fb266a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:01.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b2070 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb27409b760 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615b4db0400 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2740aeea0 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 10 client.26243 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fb26c009ab0 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5615b4db0400 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb27407c050 con 0x5615b4e9d650 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 10 client.26243 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 10 client.26243 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 20 client.26243 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4eb86e0 0x5615b4ebaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 4 client.26243 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:01.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.459+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615b4de8780 con 0x5615b4eb86e0 2024-03-30T04:04:01.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.460+0000 7fb27fe00700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4eb86e0 0x5615b4ebaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.461+0000 7fb27fe00700 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4eb86e0 0x5615b4ebaad0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c02d7582d2026a server_cookie=d51e2fae8d237f09 in_seq=0 out_seq=0 2024-03-30T04:04:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.461+0000 7fb266a00700 10 client.26243 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.461+0000 7fb266a00700 1 -- 172.21.15.23:0/3139668697 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5615b4de8780 con 0x5615b4eb86e0 2024-03-30T04:04:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.461+0000 7fb266a00700 10 client.26243 handle_command_reply: tid=0 2024-03-30T04:04:01.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.461+0000 7fb29b47bb80 1 client.26243 shutdown 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 msgr2=0x5615b4ea8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb27409aff0 tx=0x7fb274009310 comp rx=0 tx=0).stop 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 shutdown_connections 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4ea8e70 0x5615b4ea61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4ea8870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4eb86e0 0x5615b4ebaad0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/3139668697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4e9df70 0x5615b4ea5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb266a00700 0 client.26243 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 >> 172.21.15.23:0/3139668697 conn(0x5615b4dfd720 msgr2=0x5615b4e9f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 shutdown_connections 2024-03-30T04:04:01.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 1 -- 172.21.15.23:0/3139668697 wait complete. 2024-03-30T04:04:01.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.462+0000 7fb29b47bb80 20 client.26243 trim_cache size 0 max 16384 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.480+0000 7fb29b47bb80 1 Processor -- start 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.480+0000 7fb29b47bb80 1 -- start start 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4ea4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4ea67c0 con 0x5615b4e9df70 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4ea7160 con 0x5615b4e9d650 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4e9e8d0 con 0x5615b4ea8e70 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56936/0 (socket says 172.21.15.23:56936) 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285000700 1 -- 172.21.15.23:0/248097044 learned_addr learned my addr 172.21.15.23:0/248097044 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb27fe00700 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285a00700 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4ea4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285000700 1 -- 172.21.15.23:0/248097044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 msgr2=0x5615b4ea4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.481+0000 7fb285000700 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4ea4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb285000700 1 -- 172.21.15.23:0/248097044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 msgr2=0x5615b4ea44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb285000700 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb285000700 1 -- 172.21.15.23:0/248097044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615b4ebf700 con 0x5615b4e9d650 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb285000700 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2740b0040 tx=0x7fb2740094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99ff33679b157d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb272a00700 1 -- 172.21.15.23:0/248097044 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b2070 con 0x5615b4e9d650 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb272a00700 1 -- 172.21.15.23:0/248097044 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb27409a550 con 0x5615b4e9d650 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.482+0000 7fb272a00700 1 -- 172.21.15.23:0/248097044 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740aeea0 con 0x5615b4e9d650 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- 172.21.15.23:0/248097044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 msgr2=0x5615b4eafb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2740b0040 tx=0x7fb2740094b0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- 172.21.15.23:0/248097044 shutdown_connections 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4ea4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafb20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/248097044 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4ea44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- 172.21.15.23:0/248097044 >> 172.21.15.23:0/248097044 conn(0x5615b4dfd720 msgr2=0x5615b4ea0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- 172.21.15.23:0/248097044 shutdown_connections 2024-03-30T04:04:01.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- 172.21.15.23:0/248097044 wait complete. 2024-03-30T04:04:01.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.483+0000 7fb29b47bb80 1 -- start start 2024-03-30T04:04:01.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 Processor -- start 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4eadf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4eaeb20 con 0x5615b4e9df70 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4eaec90 con 0x5615b4e9d650 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615b4f0c4f0 con 0x5615b4ea8e70 2024-03-30T04:04:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb29b47bb80 10 client.0 _ll_register_callbacks cb 0x5615b4d954f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb285a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb285a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eae450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43796/0 (socket says 172.21.15.23:43796) 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb285000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56940/0 (socket says 172.21.15.23:56940) 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.484+0000 7fb285a00700 1 -- 172.21.15.23:0/2174616540 learned_addr learned my addr 172.21.15.23:0/2174616540 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb27fe00700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4eadf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 -- 172.21.15.23:0/2174616540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 msgr2=0x5615b4eae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 -- 172.21.15.23:0/2174616540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 msgr2=0x5615b4eadf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4eadf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 -- 172.21.15.23:0/2174616540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c009ab0 con 0x5615b4e9d650 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb27fe00700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4eadf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb285000700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb274006010 tx=0x7fb274009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7966c965b0c72e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb266a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2740b2050 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb27409b060 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.485+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb26c0043b0 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2740aeea0 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 10 client.26249 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5615b4c1dbe0 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb26c0043b0 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb27407c070 con 0x5615b4e9d650 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 10 client.26249 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 10 client.26249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 20 client.26249 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4f0d720 0x5615b4f0fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 4 client.26249 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:04:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5615b4c3ed70 con 0x5615b4f0d720 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.486+0000 7fb27fe00700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4f0d720 0x5615b4f0fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.487+0000 7fb27fe00700 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4f0d720 0x5615b4f0fb10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a684c85e94ecfed0 server_cookie=f8fbfdd9ebc1c08c in_seq=0 out_seq=0 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.487+0000 7fb266a00700 10 client.26249 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb266a00700 1 -- 172.21.15.23:0/2174616540 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5615b4c3ed70 con 0x5615b4f0d720 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb266a00700 10 client.26249 handle_command_reply: tid=0 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 client.26249 shutdown 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 msgr2=0x5615b4eafad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb274006010 tx=0x7fb274009310 comp rx=0 tx=0).stop 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 shutdown_connections 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5615b4f0d720 0x5615b4f0fb10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615b4ea8e70 0x5615b4eae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5615b4e9d650 0x5615b4eafad0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 --2- 172.21.15.23:0/2174616540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5615b4e9df70 0x5615b4eadf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb266a00700 0 client.26249 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 >> 172.21.15.23:0/2174616540 conn(0x5615b4dfd720 msgr2=0x5615b4ea40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.488+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 shutdown_connections 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb29b47bb80 1 -- 172.21.15.23:0/2174616540 wait complete. 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb29b47bb80 20 client.26249 trim_cache size 0 max 16384 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "364a12c0-9302-4c85-8e3d-95bcdb4380c7", 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:04:01.492 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:04:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 msgr2=0x7fb2680a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 0x7fb2680a37d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb278009ec0 tx=0x7fb2780092f0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 msgr2=0x7fb2800730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb274005910 tx=0x7fb274004730 comp rx=0 tx=0).stop 2024-03-30T04:04:01.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 shutdown_connections 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb280073b30 0x7fb2801a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2680a1310 0x7fb2680a37d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb280100f70 0x7fb2800730b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 --2- 172.21.15.23:0/2204750390 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb280106090 0x7fb2800735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.489+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 >> 172.21.15.23:0/2204750390 conn(0x7fb2800fc7f0 msgr2=0x7fb2800fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.490+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 shutdown_connections 2024-03-30T04:04:01.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.490+0000 7fb287a00700 1 -- 172.21.15.23:0/2204750390 wait complete. 2024-03-30T04:04:01.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:04:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 -- 172.21.15.23:0/506631747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a01060b0 msgr2=0x7f27a010dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 --2- 172.21.15.23:0/506631747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a01060b0 0x7f27a010dc30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f279800b0d0 tx=0x7f279800b3e0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 -- 172.21.15.23:0/506631747 shutdown_connections 2024-03-30T04:04:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 --2- 172.21.15.23:0/506631747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a01060b0 0x7f27a010dc30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 --2- 172.21.15.23:0/506631747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a0069240 0x7f27a0069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 --2- 172.21.15.23:0/506631747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a0069bf0 0x7f27a0105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 -- 172.21.15.23:0/506631747 >> 172.21.15.23:0/506631747 conn(0x7f27a0076d80 msgr2=0x7f27a0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 -- 172.21.15.23:0/506631747 shutdown_connections 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.741+0000 7f27a8400700 1 -- 172.21.15.23:0/506631747 wait complete. 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.742+0000 7f27a8400700 1 Processor -- start 2024-03-30T04:04:01.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.742+0000 7f27a8400700 1 -- start start 2024-03-30T04:04:01.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a01060b0 0x7f27a0199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a019ddd0 0x7f27a019e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a019e850 con 0x7f27a01060b0 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a019e9c0 con 0x7f27a019ddd0 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a8400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a019eb80 con 0x7f27a0069240 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43816/0 (socket says 172.21.15.23:43816) 2024-03-30T04:04:01.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a5a00700 1 -- 172.21.15.23:0/3967890609 learned_addr learned my addr 172.21.15.23:0/3967890609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a6400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a019ddd0 0x7f27a019e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.743+0000 7f27a5000700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a01060b0 0x7f27a0199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 -- 172.21.15.23:0/3967890609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a019ddd0 msgr2=0x7f27a019e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a019ddd0 0x7f27a019e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 -- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a01060b0 msgr2=0x7f27a0199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a01060b0 0x7f27a0199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 -- 172.21.15.23:0/3967890609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2798009d00 con 0x7f27a0069240 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a5a00700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f279400e9a0 tx=0x7f279400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f279e000700 1 -- 172.21.15.23:0/3967890609 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794009790 con 0x7f27a0069240 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f279e000700 1 -- 172.21.15.23:0/3967890609 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279400cbd0 con 0x7f27a0069240 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f279e000700 1 -- 172.21.15.23:0/3967890609 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2794018680 con 0x7f27a0069240 2024-03-30T04:04:01.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a01a60e0 con 0x7f27a0069240 2024-03-30T04:04:01.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.744+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a01a6570 con 0x7f27a0069240 2024-03-30T04:04:01.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.748+0000 7f279e000700 1 -- 172.21.15.23:0/3967890609 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f279400cd40 con 0x7f27a0069240 2024-03-30T04:04:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.749+0000 7f279e000700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 0x7f27880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.749+0000 7f279e000700 1 -- 172.21.15.23:0/3967890609 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2794014070 con 0x7f27a0069240 2024-03-30T04:04:01.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 Processor -- start 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27a5000700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 0x7f27880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 -- start start 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27a5000700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 0x7f27880a3750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f27a019aa40 tx=0x7f278c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f3468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3491e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f294990 con 0x56491f33bef0 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f292350 con 0x56491f346df0 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.750+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f28dc40 con 0x56491f33b5d0 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f3468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43822/0 (socket says 172.21.15.23:43822) 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 -- 172.21.15.23:0/4287043939 learned_addr learned my addr 172.21.15.23:0/4287043939 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a6400700 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 -- 172.21.15.23:0/4287043939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 msgr2=0x56491f3491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 -- 172.21.15.23:0/4287043939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 msgr2=0x56491f3468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f3468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 -- 172.21.15.23:0/4287043939 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56491f10f3b0 con 0x56491f33b5d0 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2794096ac0 tx=0x7f2794099a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4861e0eb8264d76b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f2792a00700 1 -- 172.21.15.23:0/4287043939 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279409a050 con 0x56491f33b5d0 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f2792a00700 1 -- 172.21.15.23:0/4287043939 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279409de80 con 0x56491f33b5d0 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.751+0000 7f2792a00700 1 -- 172.21.15.23:0/4287043939 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940a2040 con 0x56491f33b5d0 2024-03-30T04:04:01.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4287043939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 msgr2=0x56491f33b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2794096ac0 tx=0x7f2794099a30 comp rx=0 tx=0).stop 2024-03-30T04:04:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4287043939 shutdown_connections 2024-03-30T04:04:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f33b9b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4287043939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f3468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.752+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4287043939 >> 172.21.15.23:0/4287043939 conn(0x56491f29b6a0 msgr2=0x56491f337a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.753+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4287043939 shutdown_connections 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.753+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4287043939 wait complete. 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.753+0000 7f27bbf30b80 1 -- start start 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.753+0000 7f27bbf30b80 1 Processor -- start 2024-03-30T04:04:01.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f343b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3442b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f229d80 con 0x56491f33bef0 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f229a20 con 0x56491f346df0 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f1116a0 con 0x56491f33b5d0 2024-03-30T04:04:01.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27bbf30b80 10 client.0 _ll_register_callbacks cb 0x56491f233470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43838/0 (socket says 172.21.15.23:43838) 2024-03-30T04:04:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27a5a00700 1 -- 172.21.15.23:0/1605280103 learned_addr learned my addr 172.21.15.23:0/1605280103 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27a5000700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f343b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.754+0000 7f27a6400700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 -- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 msgr2=0x56491f3442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 -- 172.21.15.23:0/1605280103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 msgr2=0x56491f343b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f343b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 -- 172.21.15.23:0/1605280103 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a01a6570 con 0x56491f33b5d0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27a5a00700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f279400c5b0 tx=0x7f27940997f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=709c4d0488994c8e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f2787400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940a24d0 con 0x56491f33b5d0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f279409a050 con 0x56491f33b5d0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56491f24e380 con 0x56491f33b5d0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f279407a070 con 0x56491f33b5d0 2024-03-30T04:04:01.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27bbf30b80 10 client.26251 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.755+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56491f0bbb60 con 0x56491f33b5d0 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56491f24e380 con 0x56491f33b5d0 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f2794099150 con 0x56491f33b5d0 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 10 client.26251 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 10 client.26251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 20 client.26251 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f356670 0x56491f358a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 4 client.26251 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:01.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56491f286700 con 0x56491f356670 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.756+0000 7f27a5000700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f356670 0x56491f358a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.757+0000 7f27a5000700 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f356670 0x56491f358a90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a9d7bf0c7f490ce server_cookie=631323576e9f6a04 in_seq=0 out_seq=0 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.757+0000 7f2787400700 10 client.26251 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.757+0000 7f2787400700 1 -- 172.21.15.23:0/1605280103 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56491f286700 con 0x56491f356670 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.757+0000 7f2787400700 10 client.26251 handle_command_reply: tid=0 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.757+0000 7f27bbf30b80 1 client.26251 shutdown 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 msgr2=0x56491f346880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f279400c5b0 tx=0x7f27940997f0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 shutdown_connections 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f3442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f356670 0x56491f358a90 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f346880 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/1605280103 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f343b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f2787400700 0 client.26251 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 >> 172.21.15.23:0/1605280103 conn(0x56491f29b6a0 msgr2=0x56491f33d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 shutdown_connections 2024-03-30T04:04:01.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 1 -- 172.21.15.23:0/1605280103 wait complete. 2024-03-30T04:04:01.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.758+0000 7f27bbf30b80 20 client.26251 trim_cache size 0 max 16384 2024-03-30T04:04:01.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.775+0000 7f27bbf30b80 1 Processor -- start 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 -- start start 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f33eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f33f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f33c340 con 0x56491f33bef0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f339190 con 0x56491f346df0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.776+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f345170 con 0x56491f33b5d0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f33eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43846/0 (socket says 172.21.15.23:43846) 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a6400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f33f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 -- 172.21.15.23:0/4047014344 learned_addr learned my addr 172.21.15.23:0/4047014344 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 -- 172.21.15.23:0/4047014344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 msgr2=0x56491f33f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f33f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 -- 172.21.15.23:0/4047014344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 msgr2=0x56491f33eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f33eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 -- 172.21.15.23:0/4047014344 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56491f253ea0 con 0x56491f33b5d0 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2794099d50 tx=0x7f2794096390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e865c1dd3fabe1b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f2793400700 1 -- 172.21.15.23:0/4047014344 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279407a070 con 0x56491f33b5d0 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f2793400700 1 -- 172.21.15.23:0/4047014344 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279409a050 con 0x56491f33b5d0 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.777+0000 7f2793400700 1 -- 172.21.15.23:0/4047014344 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940b81c0 con 0x56491f33b5d0 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4047014344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 msgr2=0x56491f34da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2794099d50 tx=0x7f2794096390 comp rx=0 tx=0).stop 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4047014344 shutdown_connections 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f346df0 0x56491f33f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33b5d0 0x56491f34da70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4047014344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f33bef0 0x56491f33eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4047014344 >> 172.21.15.23:0/4047014344 conn(0x56491f29b6a0 msgr2=0x56491f3305a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4047014344 shutdown_connections 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.778+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4047014344 wait complete. 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.779+0000 7f27bbf30b80 1 -- start start 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.779+0000 7f27bbf30b80 1 Processor -- start 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33bef0 0x56491f34c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f346df0 0x56491f3a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f34c860 con 0x56491f346df0 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f3a99e0 con 0x56491f33b5d0 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56491f3a9b50 con 0x56491f33bef0 2024-03-30T04:04:01.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27bbf30b80 10 client.0 _ll_register_callbacks cb 0x56491f233470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56982/0 (socket says 172.21.15.23:56982) 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 -- 172.21.15.23:0/4277704113 learned_addr learned my addr 172.21.15.23:0/4277704113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a6400700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f346df0 0x56491f3a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5000700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33bef0 0x56491f34c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 -- 172.21.15.23:0/4277704113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33bef0 msgr2=0x56491f34c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33bef0 0x56491f34c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 -- 172.21.15.23:0/4277704113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f346df0 msgr2=0x56491f3a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:01.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f346df0 0x56491f3a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.780+0000 7f27a5a00700 1 -- 172.21.15.23:0/4277704113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56491f0bbb60 con 0x56491f33b5d0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f27a5a00700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27940cac90 tx=0x7f279400e8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd51b88acb170db server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279407a070 con 0x56491f33b5d0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f27940a24d0 con 0x56491f33b5d0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f279409de80 con 0x56491f33b5d0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27940b81c0 con 0x56491f33b5d0 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f27bbf30b80 10 client.26267 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:01.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56491f113cd0 con 0x56491f33b5d0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f279409de80 con 0x56491f33b5d0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.781+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f279409aa70 con 0x56491f33b5d0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 10 client.26267 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 10 client.26267 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 20 client.26267 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f3aaea0 0x56491f3ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 4 client.26267 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.782+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56491f0dccf0 con 0x56491f3aaea0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.783+0000 7f27a5000700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f3aaea0 0x56491f3ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.785+0000 7f27a5000700 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f3aaea0 0x56491f3ad2c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a624522b9c8ebaaf server_cookie=76d01e83d7c82046 in_seq=0 out_seq=0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.785+0000 7f2787400700 10 client.26267 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f2787400700 1 -- 172.21.15.23:0/4277704113 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56491f0dccf0 con 0x56491f3aaea0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f2787400700 10 client.26267 handle_command_reply: tid=0 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 client.26267 shutdown 2024-03-30T04:04:01.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 msgr2=0x56491f34bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f27940cac90 tx=0x7f279400e8f0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 shutdown_connections 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56491f346df0 0x56491f3a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56491f33b5d0 0x56491f34bc10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56491f3aaea0 0x56491f3ad2c0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 --2- 172.21.15.23:0/4277704113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56491f33bef0 0x56491f34c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 >> 172.21.15.23:0/4277704113 conn(0x56491f29b6a0 msgr2=0x56491f33eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f2787400700 0 client.26267 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 shutdown_connections 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.786+0000 7f27bbf30b80 1 -- 172.21.15.23:0/4277704113 wait complete. 2024-03-30T04:04:01.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27bbf30b80 20 client.26267 trim_cache size 0 max 16384 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "364a12c0-9302-4c85-8e3d-95bcdb4380c7": { 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "364a12c0-9302-4c85-8e3d-95bcdb4380c7", 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:04:01.794 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:04:01.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 msgr2=0x7f27880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 0x7f27880a3750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f27a019aa40 tx=0x7f278c009260 comp rx=0 tx=0).stop 2024-03-30T04:04:01.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 msgr2=0x7f27a01994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:01.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f279400e9a0 tx=0x7f279400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 shutdown_connections 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27880a1290 0x7f27880a3750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27a019ddd0 0x7f27a019e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27a0069240 0x7f27a01994f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 --2- 172.21.15.23:0/3967890609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27a01060b0 0x7f27a0199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 >> 172.21.15.23:0/3967890609 conn(0x7f27a0076d80 msgr2=0x7f27a010c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 shutdown_connections 2024-03-30T04:04:01.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:01.787+0000 7f27a8400700 1 -- 172.21.15.23:0/3967890609 wait complete. 2024-03-30T04:04:01.803 INFO:tasks.cephfs.filesystem:scrub status for tag:364a12c0-9302-4c85-8e3d-95bcdb4380c7 - {'path': '/', 'tag': '364a12c0-9302-4c85-8e3d-95bcdb4380c7', 'options': 'recursive,force'} 2024-03-30T04:04:02.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 80541 11.51 MB/sec warmup 39 sec latency 57.408 ms 2024-03-30T04:04:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:02 smithi154 ceph-mon[79161]: pgmap v66: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-30T04:04:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:02 smithi193 ceph-mon[79581]: pgmap v66: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-30T04:04:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:02 smithi023 ceph-mon[38254]: pgmap v66: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.3 MiB/s wr, 338 op/s 2024-03-30T04:04:03.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 82396 11.53 MB/sec warmup 40 sec latency 8.884 ms 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:03 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:04:03.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:04:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:04:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:03 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:04:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:04:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:04:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:03.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:03 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:04.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 85298 11.67 MB/sec warmup 41 sec latency 7.739 ms 2024-03-30T04:04:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:04:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:04 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:04 smithi154 ceph-mon[79161]: pgmap v67: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:04:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:04 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:04 smithi193 ceph-mon[79581]: pgmap v67: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:04:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:04 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:04 smithi023 ceph-mon[38254]: pgmap v67: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:05.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 87364 11.57 MB/sec warmup 42 sec latency 13.106 ms 2024-03-30T04:04:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:05 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:04:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:05 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:04:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:05 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:05 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:04:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:05 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:04:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:05 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:05 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:04:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:05 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:04:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:05 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:04:06.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 89392 11.61 MB/sec warmup 43 sec latency 36.328 ms 2024-03-30T04:04:07.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 92340 11.74 MB/sec warmup 44 sec latency 8.544 ms 2024-03-30T04:04:07.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:07 smithi023 ceph-mon[38254]: pgmap v68: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:07 smithi154 ceph-mon[79161]: pgmap v68: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:07.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:07 smithi193 ceph-mon[79581]: pgmap v68: 129 pgs: 129 active+clean; 54 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.2 MiB/s wr, 183 op/s 2024-03-30T04:04:08.188 INFO:tasks.workunit.client.0.smithi023.stdout: 1 94130 11.64 MB/sec warmup 45 sec latency 17.277 ms 2024-03-30T04:04:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:08 smithi154 ceph-mon[79161]: pgmap v69: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-30T04:04:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:08 smithi193 ceph-mon[79581]: pgmap v69: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-30T04:04:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:08 smithi023 ceph-mon[38254]: pgmap v69: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 262 op/s 2024-03-30T04:04:09.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 96378 11.68 MB/sec warmup 46 sec latency 9.251 ms 2024-03-30T04:04:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:04:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:04:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:04:09.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:04:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:04:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:04:10.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 99338 11.80 MB/sec warmup 47 sec latency 6.333 ms 2024-03-30T04:04:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:10 smithi193 ceph-mon[79581]: pgmap v70: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-30T04:04:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:10 smithi023 ceph-mon[38254]: pgmap v70: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-30T04:04:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:10 smithi154 ceph-mon[79161]: pgmap v70: 129 pgs: 129 active+clean; 61 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-03-30T04:04:11.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 100525 11.64 MB/sec warmup 48 sec latency 144.375 ms 2024-03-30T04:04:12.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 103287 11.75 MB/sec warmup 49 sec latency 13.309 ms 2024-03-30T04:04:12.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:12 smithi154 ceph-mon[79161]: pgmap v71: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-30T04:04:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:12 smithi193 ceph-mon[79581]: pgmap v71: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-30T04:04:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:12 smithi023 ceph-mon[38254]: pgmap v71: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-03-30T04:04:13.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 105882 11.76 MB/sec warmup 50 sec latency 7.290 ms 2024-03-30T04:04:14.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 107628 11.70 MB/sec warmup 51 sec latency 14.855 ms 2024-03-30T04:04:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:14 smithi154 ceph-mon[79161]: pgmap v72: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:14 smithi193 ceph-mon[79581]: pgmap v72: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:14 smithi023 ceph-mon[38254]: pgmap v72: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:15.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 110321 11.80 MB/sec warmup 52 sec latency 23.918 ms 2024-03-30T04:04:16.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 112478 11.73 MB/sec warmup 53 sec latency 46.128 ms 2024-03-30T04:04:17.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 114329 11.73 MB/sec warmup 54 sec latency 12.186 ms 2024-03-30T04:04:17.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:17 smithi023 ceph-mon[38254]: pgmap v73: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:17 smithi154 ceph-mon[79161]: pgmap v73: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:17 smithi193 ceph-mon[79581]: pgmap v73: 129 pgs: 129 active+clean; 67 MiB data, 587 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-03-30T04:04:18.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 117186 11.83 MB/sec warmup 55 sec latency 7.549 ms 2024-03-30T04:04:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:18 smithi154 ceph-mon[79161]: pgmap v74: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-30T04:04:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:18 smithi193 ceph-mon[79581]: pgmap v74: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-30T04:04:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:18 smithi023 ceph-mon[38254]: pgmap v74: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.1 MiB/s wr, 242 op/s 2024-03-30T04:04:19.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 119294 11.75 MB/sec warmup 56 sec latency 11.808 ms 2024-03-30T04:04:20.189 INFO:tasks.workunit.client.0.smithi023.stdout: 1 121241 11.77 MB/sec warmup 57 sec latency 10.192 ms 2024-03-30T04:04:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:20 smithi154 ceph-mon[79161]: pgmap v75: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-03-30T04:04:20.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:20 smithi193 ceph-mon[79581]: pgmap v75: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-03-30T04:04:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:20 smithi023 ceph-mon[38254]: pgmap v75: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.4 MiB/s wr, 162 op/s 2024-03-30T04:04:21.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 123653 11.79 MB/sec warmup 58 sec latency 65.856 ms 2024-03-30T04:04:22.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 125275 11.73 MB/sec warmup 59 sec latency 12.720 ms 2024-03-30T04:04:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:22 smithi154 ceph-mon[79161]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-03-30T04:04:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:22 smithi193 ceph-mon[79581]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-03-30T04:04:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:22 smithi023 ceph-mon[38254]: pgmap v76: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 263 op/s 2024-03-30T04:04:23.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 128054 11.82 MB/sec warmup 60 sec latency 10.033 ms 2024-03-30T04:04:24.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 130229 11.76 MB/sec warmup 61 sec latency 8.731 ms 2024-03-30T04:04:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:24 smithi154 ceph-mon[79161]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:24 smithi193 ceph-mon[79581]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:24 smithi023 ceph-mon[38254]: pgmap v77: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:25.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 132086 11.76 MB/sec warmup 62 sec latency 8.280 ms 2024-03-30T04:04:26.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 134645 11.79 MB/sec warmup 63 sec latency 76.221 ms 2024-03-30T04:04:27.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 136230 11.76 MB/sec warmup 64 sec latency 17.421 ms 2024-03-30T04:04:27.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:27 smithi023 ceph-mon[38254]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:27 smithi154 ceph-mon[79161]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:27.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:27 smithi193 ceph-mon[79581]: pgmap v78: 129 pgs: 129 active+clean; 78 MiB data, 699 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 186 op/s 2024-03-30T04:04:28.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 138776 11.80 MB/sec warmup 65 sec latency 11.105 ms 2024-03-30T04:04:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:28 smithi154 ceph-mon[79161]: pgmap v79: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 293 op/s 2024-03-30T04:04:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:28 smithi193 ceph-mon[79581]: pgmap v79: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 293 op/s 2024-03-30T04:04:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:28 smithi023 ceph-mon[38254]: pgmap v79: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 293 op/s 2024-03-30T04:04:29.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 141348 11.81 MB/sec warmup 66 sec latency 13.308 ms 2024-03-30T04:04:30.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 143053 11.76 MB/sec warmup 67 sec latency 10.199 ms 2024-03-30T04:04:30.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:30 smithi154 ceph-mon[79161]: pgmap v80: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:30 smithi193 ceph-mon[79581]: pgmap v80: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:30.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:30 smithi023 ceph-mon[38254]: pgmap v80: 129 pgs: 129 active+clean; 83 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:31.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 145614 11.82 MB/sec warmup 68 sec latency 55.607 ms 2024-03-30T04:04:31.804 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:04:32.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 -- 172.21.15.23:0/2613944407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae40687f0 msgr2=0x7ffae410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2613944407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae40687f0 0x7ffae410db50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009f90 tx=0x7ffadc00b330 comp rx=0 tx=0).stop 2024-03-30T04:04:32.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 -- 172.21.15.23:0/2613944407 shutdown_connections 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2613944407 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae40687f0 0x7ffae410db50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2613944407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae4105ce0 0x7ffae41060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2613944407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae40682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.025+0000 7ffaeb200700 1 -- 172.21.15.23:0/2613944407 >> 172.21.15.23:0/2613944407 conn(0x7ffae4076d40 msgr2=0x7ffae4077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.026+0000 7ffaeb200700 1 -- 172.21.15.23:0/2613944407 shutdown_connections 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.026+0000 7ffaeb200700 1 -- 172.21.15.23:0/2613944407 wait complete. 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.027+0000 7ffaeb200700 1 Processor -- start 2024-03-30T04:04:32.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.027+0000 7ffaeb200700 1 -- start start 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae4105ce0 0x7ffae419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae419e460 con 0x7ffae4105ce0 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae419e5d0 con 0x7ffae40687f0 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffaeb200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae419e790 con 0x7ffae4067e30 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58516/0 (socket says 172.21.15.23:58516) 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50364/0 (socket says 172.21.15.23:50364) 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae3400700 1 -- 172.21.15.23:0/2877476535 learned_addr learned my addr 172.21.15.23:0/2877476535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.028+0000 7ffae9200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae4105ce0 0x7ffae419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 -- 172.21.15.23:0/2877476535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 msgr2=0x7ffae4199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 -- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae4105ce0 msgr2=0x7ffae419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae4105ce0 0x7ffae419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 -- 172.21.15.23:0/2877476535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00095f0 con 0x7ffae40687f0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3e00700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae3400700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffad000ec10 tx=0x7ffad000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae0c00700 1 -- 172.21.15.23:0/2877476535 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad000cc90 con 0x7ffae40687f0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae0c00700 1 -- 172.21.15.23:0/2877476535 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffad00044e0 con 0x7ffae40687f0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc009c40 con 0x7ffae40687f0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffae0c00700 1 -- 172.21.15.23:0/2877476535 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0010430 con 0x7ffae40687f0 2024-03-30T04:04:32.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.029+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae41a60d0 con 0x7ffae40687f0 2024-03-30T04:04:32.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.032+0000 7ffae0c00700 1 -- 172.21.15.23:0/2877476535 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ffad0003fb0 con 0x7ffae40687f0 2024-03-30T04:04:32.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.033+0000 7ffae0c00700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 0x7ffac80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.034+0000 7ffae0c00700 1 -- 172.21.15.23:0/2877476535 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ffad0014070 con 0x7ffae40687f0 2024-03-30T04:04:32.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.034+0000 7ffae3e00700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 0x7ffac80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.034+0000 7ffafedbeb80 1 Processor -- start 2024-03-30T04:04:32.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.034+0000 7ffae3e00700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 0x7ffac80a3750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffad4011490 tx=0x7ffad4009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 -- start start 2024-03-30T04:04:32.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06ed1990 con 0x55df06f8c600 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06ecf350 con 0x55df06f78ef0 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06ecac40 con 0x55df06f785d0 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae9200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58530/0 (socket says 172.21.15.23:58530) 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae3400700 1 -- 172.21.15.23:0/1575535649 learned_addr learned my addr 172.21.15.23:0/1575535649 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae9200700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54614/0 (socket says 172.21.15.23:54614) 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.035+0000 7ffae3e00700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 -- 172.21.15.23:0/1575535649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 msgr2=0x55df06f789b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f789b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 -- 172.21.15.23:0/1575535649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 msgr2=0x55df06f8e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 -- 172.21.15.23:0/1575535649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df06d4c3b0 con 0x55df06f78ef0 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3e00700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae9200700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffae3400700 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005d70 tx=0x7ffad009a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d714eb17e05f3a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffad9600700 1 -- 172.21.15.23:0/1575535649 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad009aa20 con 0x55df06f78ef0 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffad9600700 1 -- 172.21.15.23:0/1575535649 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffad009a320 con 0x55df06f78ef0 2024-03-30T04:04:32.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.036+0000 7ffad9600700 1 -- 172.21.15.23:0/1575535649 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad0096a30 con 0x55df06f78ef0 2024-03-30T04:04:32.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1575535649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 msgr2=0x55df06f8c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffad0005d70 tx=0x7ffad009a1c0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1575535649 shutdown_connections 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f8e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1575535649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f8c0c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1575535649 >> 172.21.15.23:0/1575535649 conn(0x55df06ed86a0 msgr2=0x55df06f74a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1575535649 shutdown_connections 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1575535649 wait complete. 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.037+0000 7ffafedbeb80 1 -- start start 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.038+0000 7ffafedbeb80 1 Processor -- start 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f8c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06e66d80 con 0x55df06f8c600 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06e66a20 con 0x55df06f785d0 2024-03-30T04:04:32.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06d4e6a0 con 0x55df06f78ef0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffafedbeb80 10 client.0 _ll_register_callbacks cb 0x55df06e70470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae9200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f8c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50384/0 (socket says 172.21.15.23:50384) 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae3400700 1 -- 172.21.15.23:0/292814898 learned_addr learned my addr 172.21.15.23:0/292814898 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.039+0000 7ffae3400700 1 -- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 msgr2=0x55df06f8c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffae3400700 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f8c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffae3400700 1 -- 172.21.15.23:0/292814898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 msgr2=0x55df06f89ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffae3400700 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffae3400700 1 -- 172.21.15.23:0/292814898 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae410e550 con 0x55df06f78ef0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffae3400700 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffad00caee0 tx=0x7ffad009aa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94276be4ff9dc2b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7fface000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:32.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad00a27a0 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ffad009e150 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df06e8b380 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffad0004260 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffafedbeb80 10 client.26269 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7ffae41a60d0 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.040+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55df06e8b380 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ffad0099810 con 0x55df06f78ef0 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 10 client.26269 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 10 client.26269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 20 client.26269 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06f9be60 0x55df06f9e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 4 client.26269 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:32.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df06ec3700 con 0x55df06f9be60 2024-03-30T04:04:32.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.041+0000 7ffae3e00700 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06f9be60 0x55df06f9e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.042+0000 7fface000700 10 client.26269 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.042+0000 7ffae3e00700 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06f9be60 0x55df06f9e280 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20d40f84813deec6 server_cookie=2c8cdc05754f1b16 in_seq=0 out_seq=0 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.042+0000 7fface000700 1 -- 172.21.15.23:0/292814898 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55df06ec3700 con 0x55df06f9be60 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.042+0000 7fface000700 10 client.26269 handle_command_reply: tid=0 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 client.26269 shutdown 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 msgr2=0x55df06f89390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7ffad00caee0 tx=0x7ffad009aa90 comp rx=0 tx=0).stop 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 shutdown_connections 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f8c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06f9be60 0x55df06f9e280 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/292814898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f89390 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7fface000700 0 client.26269 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 >> 172.21.15.23:0/292814898 conn(0x55df06ed86a0 msgr2=0x55df06f7a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 shutdown_connections 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 1 -- 172.21.15.23:0/292814898 wait complete. 2024-03-30T04:04:32.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.043+0000 7ffafedbeb80 20 client.26269 trim_cache size 0 max 16384 2024-03-30T04:04:32.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 Processor -- start 2024-03-30T04:04:32.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 -- start start 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f88c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06f7a140 con 0x55df06f8c600 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06f75660 con 0x55df06f785d0 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06f89e50 con 0x55df06f78ef0 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffae3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f88c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58546/0 (socket says 172.21.15.23:58546) 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffae3e00700 1 -- 172.21.15.23:0/3320756810 learned_addr learned my addr 172.21.15.23:0/3320756810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.063+0000 7ffae9200700 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 -- 172.21.15.23:0/3320756810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 msgr2=0x55df06f88c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f88c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 -- 172.21.15.23:0/3320756810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 msgr2=0x55df06f89160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 -- 172.21.15.23:0/3320756810 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df06cf8b60 con 0x55df06f785d0 2024-03-30T04:04:32.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffae3e00700 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffad4013010 tx=0x7ffad4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3934f4bca28c339d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffada000700 1 -- 172.21.15.23:0/3320756810 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4024040 con 0x55df06f785d0 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffada000700 1 -- 172.21.15.23:0/3320756810 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffad40045a0 con 0x55df06f785d0 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.064+0000 7ffada000700 1 -- 172.21.15.23:0/3320756810 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad401f870 con 0x55df06f785d0 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 -- 172.21.15.23:0/3320756810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 msgr2=0x55df06f93280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffad4013010 tx=0x7ffad4014000 comp rx=0 tx=0).stop 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 -- 172.21.15.23:0/3320756810 shutdown_connections 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f89160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f785d0 0x55df06f93280 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/3320756810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f78ef0 0x55df06f88c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 -- 172.21.15.23:0/3320756810 >> 172.21.15.23:0/3320756810 conn(0x55df06ed86a0 msgr2=0x55df06f84660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 -- 172.21.15.23:0/3320756810 shutdown_connections 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.065+0000 7ffafedbeb80 1 -- 172.21.15.23:0/3320756810 wait complete. 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 -- start start 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 Processor -- start 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f91590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f91ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06f921e0 con 0x55df06f8c600 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06f92350 con 0x55df06f78ef0 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df06fefc60 con 0x55df06f785d0 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.066+0000 7ffafedbeb80 10 client.0 _ll_register_callbacks cb 0x55df06e70470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50404/0 (socket says 172.21.15.23:50404) 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 -- 172.21.15.23:0/1537197239 learned_addr learned my addr 172.21.15.23:0/1537197239 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae9200700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f91ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3400700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f91590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 -- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 msgr2=0x55df06f91590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f91590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 -- 172.21.15.23:0/1537197239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 msgr2=0x55df06f91ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f91ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 -- 172.21.15.23:0/1537197239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad4017070 con 0x55df06f785d0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7ffae3e00700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffad40040e0 tx=0x7ffad402a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac3a639da9f1e77 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7fface000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.067+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4024050 con 0x55df06f785d0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffad009a320 con 0x55df06f785d0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ffad4005510 con 0x55df06f785d0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffad40056d0 con 0x55df06f785d0 2024-03-30T04:04:32.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 10 client.26275 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55df06e90ea0 con 0x55df06f785d0 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffad009a320 con 0x55df06f785d0 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ffad4003b30 con 0x55df06f785d0 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 10 client.26275 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 10 client.26275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 20 client.26275 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.068+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06ff1280 0x55df06ff36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7ffafedbeb80 4 client.26275 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55df06d19cf0 con 0x55df06ff1280 2024-03-30T04:04:32.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7ffae3400700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06ff1280 0x55df06ff36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7ffae3400700 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06ff1280 0x55df06ff36a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feb83b97e7c3d95a server_cookie=3fcb0379d3773400 in_seq=0 out_seq=0 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7fface000700 10 client.26275 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7fface000700 1 -- 172.21.15.23:0/1537197239 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55df06d19cf0 con 0x55df06ff1280 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7fface000700 10 client.26275 handle_command_reply: tid=0 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.069+0000 7ffafedbeb80 1 client.26275 shutdown 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 msgr2=0x55df06f930a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffad40040e0 tx=0x7ffad402a4b0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 shutdown_connections 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55df06f8c600 0x55df06f91ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55df06ff1280 0x55df06ff36a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df06f785d0 0x55df06f930a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 --2- 172.21.15.23:0/1537197239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55df06f78ef0 0x55df06f91590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7fface000700 0 client.26275 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 >> 172.21.15.23:0/1537197239 conn(0x55df06ed86a0 msgr2=0x55df06f7a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 shutdown_connections 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 1 -- 172.21.15.23:0/1537197239 wait complete. 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.070+0000 7ffafedbeb80 20 client.26275 trim_cache size 0 max 16384 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:04:32.073 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 msgr2=0x7ffac80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 0x7ffac80a3750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ffad4011490 tx=0x7ffad4009260 comp rx=0 tx=0).stop 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 msgr2=0x7ffae4199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffad000ec10 tx=0x7ffad000ef20 comp rx=0 tx=0).stop 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 shutdown_connections 2024-03-30T04:04:32.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffae4105ce0 0x7ffae419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffae4067e30 0x7ffae4199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffac80a1290 0x7ffac80a3750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 --2- 172.21.15.23:0/2877476535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffae40687f0 0x7ffae4199a80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 >> 172.21.15.23:0/2877476535 conn(0x7ffae4076d40 msgr2=0x7ffae410c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 shutdown_connections 2024-03-30T04:04:32.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.071+0000 7ffaeb200700 1 -- 172.21.15.23:0/2877476535 wait complete. 2024-03-30T04:04:32.089 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:04:32.089 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:04:32.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 147321 11.75 MB/sec warmup 69 sec latency 13.149 ms 2024-03-30T04:04:32.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 -- 172.21.15.23:0/655536225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570100e60 msgr2=0x7fa570101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 --2- 172.21.15.23:0/655536225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570100e60 0x7fa570101240 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa564009990 tx=0x7fa564009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 -- 172.21.15.23:0/655536225 shutdown_connections 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 --2- 172.21.15.23:0/655536225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570105f80 0x7fa57010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 --2- 172.21.15.23:0/655536225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570100e60 0x7fa570101240 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 --2- 172.21.15.23:0/655536225 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570101810 0x7fa570105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.330+0000 7fa578600700 1 -- 172.21.15.23:0/655536225 >> 172.21.15.23:0/655536225 conn(0x7fa5700fc700 msgr2=0x7fa5700feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.331+0000 7fa578600700 1 -- 172.21.15.23:0/655536225 shutdown_connections 2024-03-30T04:04:32.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.331+0000 7fa578600700 1 -- 172.21.15.23:0/655536225 wait complete. 2024-03-30T04:04:32.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 Processor -- start 2024-03-30T04:04:32.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 -- start start 2024-03-30T04:04:32.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570100e60 0x7fa5700730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 0x7fa5701a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701a58c0 con 0x7fa570105f80 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701a5a00 con 0x7fa570073b30 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.332+0000 7fa578600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5701a5bc0 con 0x7fa570100e60 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa576600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 0x7fa5701a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54670/0 (socket says 172.21.15.23:54670) 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 -- 172.21.15.23:0/1624383628 learned_addr learned my addr 172.21.15.23:0/1624383628 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575c00700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570100e60 0x7fa5700730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 -- 172.21.15.23:0/1624383628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570100e60 msgr2=0x7fa5700730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570100e60 0x7fa5700730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 -- 172.21.15.23:0/1624383628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 msgr2=0x7fa5701a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 0x7fa5701a52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 -- 172.21.15.23:0/1624383628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5640095f0 con 0x7fa570105f80 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa576600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 0x7fa5701a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:32.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.333+0000 7fa575200700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e970 tx=0x7fa55c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.334+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009790 con 0x7fa570105f80 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.334+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa55c00c7e0 con 0x7fa570105f80 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.334+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5701a5ea0 con 0x7fa570105f80 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.334+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c0054e0 con 0x7fa570105f80 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.334+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5701a6320 con 0x7fa570105f80 2024-03-30T04:04:32.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.335+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5701091a0 con 0x7fa570105f80 2024-03-30T04:04:32.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.337+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa55c00c950 con 0x7fa570105f80 2024-03-30T04:04:32.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.339+0000 7fa56e000700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 0x7fa5580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.339+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fa55c014070 con 0x7fa570105f80 2024-03-30T04:04:32.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.339+0000 7fa575c00700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 0x7fa5580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.340+0000 7fa575c00700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 0x7fa5580a3750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa56400b2e0 tx=0x7fa564005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.344+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa55c096360 con 0x7fa570105f80 2024-03-30T04:04:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:32 smithi193 ceph-mon[79581]: pgmap v81: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-30T04:04:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:32 smithi023 ceph-mon[38254]: pgmap v81: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-30T04:04:32.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.652+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa570102330 con 0x7fa570105f80 2024-03-30T04:04:32.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.653+0000 7fa56e000700 1 -- 172.21.15.23:0/1624383628 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+10925 (secure 0 0 0) 0x7fa55c099980 con 0x7fa570105f80 2024-03-30T04:04:32.651 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:04:32.651 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:03:23.374313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 msgr2=0x7fa5580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 0x7fa5580a3750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa56400b2e0 tx=0x7fa564005fd0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 msgr2=0x7fa5700735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e970 tx=0x7fa55c00ec80 comp rx=0 tx=0).stop 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 shutdown_connections 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5580a1290 0x7fa5580a3750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa570073b30 0x7fa5701a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa570100e60 0x7fa5700730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 --2- 172.21.15.23:0/1624383628 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa570105f80 0x7fa5700735f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 >> 172.21.15.23:0/1624383628 conn(0x7fa5700fc700 msgr2=0x7fa5700fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 shutdown_connections 2024-03-30T04:04:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.655+0000 7fa578600700 1 -- 172.21.15.23:0/1624383628 wait complete. 2024-03-30T04:04:32.655 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-30T04:04:32.674 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:04:32.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:32 smithi154 ceph-mon[79161]: pgmap v81: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 311 op/s 2024-03-30T04:04:32.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.891+0000 7f857ca00700 1 -- 172.21.15.23:0/1830937811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 msgr2=0x7f85781060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.891+0000 7f857ca00700 1 --2- 172.21.15.23:0/1830937811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f85781060c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f85680099c0 tx=0x7f8568009cd0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 -- 172.21.15.23:0/1830937811 shutdown_connections 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 --2- 172.21.15.23:0/1830937811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85780687f0 0x7f857810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 --2- 172.21.15.23:0/1830937811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f85781060c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 --2- 172.21.15.23:0/1830937811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8578067e30 0x7f85780682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 -- 172.21.15.23:0/1830937811 >> 172.21.15.23:0/1830937811 conn(0x7f8578076d40 msgr2=0x7f8578077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 -- 172.21.15.23:0/1830937811 shutdown_connections 2024-03-30T04:04:32.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.892+0000 7f857ca00700 1 -- 172.21.15.23:0/1830937811 wait complete. 2024-03-30T04:04:32.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 Processor -- start 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 -- start start 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8578067e30 0x7f85781994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85780687f0 0x7f85781999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857819e330 con 0x7f8578067e30 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857819e4a0 con 0x7f8578105ce0 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.893+0000 7f857ca00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857819e660 con 0x7f85780687f0 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58594/0 (socket says 172.21.15.23:58594) 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 -- 172.21.15.23:0/45632614 learned_addr learned my addr 172.21.15.23:0/45632614 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8575e00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8578067e30 0x7f85781994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8575400700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85780687f0 0x7f85781999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 -- 172.21.15.23:0/45632614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85780687f0 msgr2=0x7f85781999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85780687f0 0x7f85781999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 -- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8578067e30 msgr2=0x7f85781994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8578067e30 0x7f85781994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 -- 172.21.15.23:0/45632614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85680095f0 con 0x7f8578105ce0 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.894+0000 7f8576800700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f857000ef30 tx=0x7f857000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.895+0000 7f856ea00700 1 -- 172.21.15.23:0/45632614 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570018070 con 0x7f8578105ce0 2024-03-30T04:04:32.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.895+0000 7f856ea00700 1 -- 172.21.15.23:0/45632614 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857000f040 con 0x7f8578105ce0 2024-03-30T04:04:32.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.895+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857819e8e0 con 0x7f8578105ce0 2024-03-30T04:04:32.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.895+0000 7f856ea00700 1 -- 172.21.15.23:0/45632614 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000cdb0 con 0x7f8578105ce0 2024-03-30T04:04:32.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.895+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85781a6180 con 0x7f8578105ce0 2024-03-30T04:04:32.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.899+0000 7f856ea00700 1 -- 172.21.15.23:0/45632614 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8570003a00 con 0x7f8578105ce0 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.900+0000 7f85904d3b80 1 Processor -- start 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 -- start start 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f856ea00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 0x7f855c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7999b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f856ea00700 1 -- 172.21.15.23:0/45632614 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f85700ccbe0 con 0x7f8578105ce0 2024-03-30T04:04:32.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d6f2990 con 0x56067d7995d0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d6f0350 con 0x56067d7a4df0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f85904d3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d6ebc40 con 0x56067d799ef0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8575400700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 0x7f855c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8575e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58602/0 (socket says 172.21.15.23:58602) 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8575e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54686/0 (socket says 172.21.15.23:54686) 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8576800700 1 -- 172.21.15.23:0/1079591627 learned_addr learned my addr 172.21.15.23:0/1079591627 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.901+0000 7f8575400700 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 -- 172.21.15.23:0/1079591627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 msgr2=0x56067d7a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 -- 172.21.15.23:0/1079591627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 msgr2=0x56067d7999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 -- 172.21.15.23:0/1079591627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56067d56d3b0 con 0x56067d7a4df0 2024-03-30T04:04:32.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8576800700 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85700a1880 tx=0x7f8570099860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2465e2a2a8a0de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8575400700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 0x7f855c0a37d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f857819aa00 tx=0x7f8564009b60 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8563000700 1 -- 172.21.15.23:0/1079591627 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570004ad0 con 0x56067d7a4df0 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8563000700 1 -- 172.21.15.23:0/1079591627 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857001b320 con 0x56067d7a4df0 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.902+0000 7f8563000700 1 -- 172.21.15.23:0/1079591627 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570099c20 con 0x56067d7a4df0 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 -- 172.21.15.23:0/1079591627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 msgr2=0x56067d7a71e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f85700a1880 tx=0x7f8570099860 comp rx=0 tx=0).stop 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 -- 172.21.15.23:0/1079591627 shutdown_connections 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a71e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 --2- 172.21.15.23:0/1079591627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 -- 172.21.15.23:0/1079591627 >> 172.21.15.23:0/1079591627 conn(0x56067d6f96a0 msgr2=0x56067d795a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 -- 172.21.15.23:0/1079591627 shutdown_connections 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.903+0000 7f85904d3b80 1 -- 172.21.15.23:0/1079591627 wait complete. 2024-03-30T04:04:32.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 -- start start 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 Processor -- start 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d687d80 con 0x56067d7995d0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d687a20 con 0x56067d7a4df0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d56f6a0 con 0x56067d799ef0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.904+0000 7f85904d3b80 10 client.0 _ll_register_callbacks cb 0x56067d691470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8575400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58616/0 (socket says 172.21.15.23:58616) 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 -- 172.21.15.23:0/768685843 learned_addr learned my addr 172.21.15.23:0/768685843 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8575e00700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 -- 172.21.15.23:0/768685843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 msgr2=0x56067d7a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 -- 172.21.15.23:0/768685843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 msgr2=0x56067d7a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 -- 172.21.15.23:0/768685843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85781a6180 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8576800700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85700ca4f0 tx=0x7f8570099160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6d166a90012f091 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.905+0000 7f8557400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570004ad0 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f85700999c0 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56067d6ac380 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f857009a0c0 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 10 client.26303 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56067d519b60 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56067d6ac380 con 0x56067d7a4df0 2024-03-30T04:04:32.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f85700a1920 con 0x56067d7a4df0 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 10 client.26303 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 10 client.26303 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 20 client.26303 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d7b4460 0x56067d7b6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 4 client.26303 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.906+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56067d6e4700 con 0x56067d7b4460 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.907+0000 7f8575e00700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d7b4460 0x56067d7b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.907+0000 7f8575e00700 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d7b4460 0x56067d7b6850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b320744ae2e6c39c server_cookie=785d0a6a673b45fe in_seq=0 out_seq=0 2024-03-30T04:04:32.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.907+0000 7f8557400700 10 client.26303 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:04:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f8557400700 1 -- 172.21.15.23:0/768685843 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56067d6e4700 con 0x56067d7b4460 2024-03-30T04:04:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f8557400700 10 client.26303 handle_command_reply: tid=0 2024-03-30T04:04:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 client.26303 shutdown 2024-03-30T04:04:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 msgr2=0x56067d7a20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f85700ca4f0 tx=0x7f8570099160 comp rx=0 tx=0).stop 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 shutdown_connections 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7a20b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d7995d0 0x56067d7a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d7b4460 0x56067d7b6850 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 --2- 172.21.15.23:0/768685843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d799ef0 0x56067d7a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 >> 172.21.15.23:0/768685843 conn(0x56067d6f96a0 msgr2=0x56067d79b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f8557400700 0 client.26303 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.908+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 shutdown_connections 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.909+0000 7f85904d3b80 1 -- 172.21.15.23:0/768685843 wait complete. 2024-03-30T04:04:32.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.909+0000 7f85904d3b80 20 client.26303 trim_cache size 0 max 16384 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.924+0000 7f85904d3b80 1 Processor -- start 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.924+0000 7f85904d3b80 1 -- start start 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d795530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7a4df0 0x56067d795a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d79b140 con 0x56067d799ef0 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d79a850 con 0x56067d7995d0 2024-03-30T04:04:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f85904d3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d7a26b0 con 0x56067d7a4df0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d795530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58622/0 (socket says 172.21.15.23:58622) 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 -- 172.21.15.23:0/2334272532 learned_addr learned my addr 172.21.15.23:0/2334272532 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8576800700 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7a4df0 0x56067d795a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 -- 172.21.15.23:0/2334272532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7a4df0 msgr2=0x56067d795a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7a4df0 0x56067d795a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 -- 172.21.15.23:0/2334272532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 msgr2=0x56067d795530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d795530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.925+0000 7f8575e00700 1 -- 172.21.15.23:0/2334272532 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56067d6b1ea0 con 0x56067d7995d0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.926+0000 7f8575e00700 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8568000c00 tx=0x7f8568005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8f819250ca1cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.926+0000 7f8563a00700 1 -- 172.21.15.23:0/2334272532 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85680055e0 con 0x56067d7995d0 2024-03-30T04:04:32.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.926+0000 7f8563a00700 1 -- 172.21.15.23:0/2334272532 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f856801e030 con 0x56067d7995d0 2024-03-30T04:04:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.926+0000 7f8563a00700 1 -- 172.21.15.23:0/2334272532 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856800b9f0 con 0x56067d7995d0 2024-03-30T04:04:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- 172.21.15.23:0/2334272532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 msgr2=0x56067d7aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8568000c00 tx=0x7f8568005a90 comp rx=0 tx=0).stop 2024-03-30T04:04:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- 172.21.15.23:0/2334272532 shutdown_connections 2024-03-30T04:04:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7a4df0 0x56067d795a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7995d0 0x56067d7aba70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2334272532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d795530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- 172.21.15.23:0/2334272532 >> 172.21.15.23:0/2334272532 conn(0x56067d6f96a0 msgr2=0x56067d78e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- 172.21.15.23:0/2334272532 shutdown_connections 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- 172.21.15.23:0/2334272532 wait complete. 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 -- start start 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.927+0000 7f85904d3b80 1 Processor -- start 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7995d0 0x56067d7ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d7a9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d7aa910 con 0x56067d799ef0 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d7aaa80 con 0x56067d7a4df0 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56067d808200 con 0x56067d7995d0 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f85904d3b80 10 client.0 _ll_register_callbacks cb 0x56067d691470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:32.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f8575e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7995d0 0x56067d7ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f8576800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58636/0 (socket says 172.21.15.23:58636) 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f8576800700 1 -- 172.21.15.23:0/2885442266 learned_addr learned my addr 172.21.15.23:0/2885442266 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.928+0000 7f8575400700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d7a9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 -- 172.21.15.23:0/2885442266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7995d0 msgr2=0x56067d7ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7995d0 0x56067d7ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 -- 172.21.15.23:0/2885442266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 msgr2=0x56067d7a9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d7a9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 -- 172.21.15.23:0/2885442266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857810e550 con 0x56067d7a4df0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8576800700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f857000c7a0 tx=0x7f85700990c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97461df19b99a305 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8557400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8570099790 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8570004ad0 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f857001b320 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f857009a0c0 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f85904d3b80 10 client.26309 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.929+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56067d571cd0 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857001b320 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f8570098920 con 0x56067d7a4df0 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 10 client.26309 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 10 client.26309 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 20 client.26309 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d8093e0 0x56067d80b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 4 client.26309 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56067d53acf0 con 0x56067d8093e0 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.930+0000 7f8575e00700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d8093e0 0x56067d80b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f8575e00700 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d8093e0 0x56067d80b7d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39ab9d76fd3027da server_cookie=94b0640cbe31f6d2 in_seq=0 out_seq=0 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f8557400700 10 client.26309 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f8557400700 1 -- 172.21.15.23:0/2885442266 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56067d53acf0 con 0x56067d8093e0 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f8557400700 10 client.26309 handle_command_reply: tid=0 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 client.26309 shutdown 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 msgr2=0x56067d7aa200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f857000c7a0 tx=0x7f85700990c0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 shutdown_connections 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56067d7a4df0 0x56067d7aa200 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56067d8093e0 0x56067d80b7d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56067d7995d0 0x56067d7ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 --2- 172.21.15.23:0/2885442266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56067d799ef0 0x56067d7a9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 >> 172.21.15.23:0/2885442266 conn(0x56067d6f96a0 msgr2=0x56067d7952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f8557400700 0 client.26309 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 shutdown_connections 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.931+0000 7f85904d3b80 1 -- 172.21.15.23:0/2885442266 wait complete. 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f85904d3b80 20 client.26309 trim_cache size 0 max 16384 2024-03-30T04:04:32.934 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 msgr2=0x7f855c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 0x7f855c0a37d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f857819aa00 tx=0x7f8564009b60 comp rx=0 tx=0).stop 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 msgr2=0x7f857819dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f857000ef30 tx=0x7f857000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 shutdown_connections 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f855c0a1310 0x7f855c0a37d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8578105ce0 0x7f857819dcf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8578067e30 0x7f85781994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 --2- 172.21.15.23:0/45632614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85780687f0 0x7f85781999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:32.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 >> 172.21.15.23:0/45632614 conn(0x7f8578076d40 msgr2=0x7f8578101000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:32.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 shutdown_connections 2024-03-30T04:04:32.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:32.932+0000 7f857ca00700 1 -- 172.21.15.23:0/45632614 wait complete. 2024-03-30T04:04:32.950 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 -- 172.21.15.23:0/1955733346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4101910 msgr2=0x7f56a4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 --2- 172.21.15.23:0/1955733346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4101910 0x7f56a4105950 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5690009990 tx=0x7f5690009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 -- 172.21.15.23:0/1955733346 shutdown_connections 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 --2- 172.21.15.23:0/1955733346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4106080 0x7f56a410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 --2- 172.21.15.23:0/1955733346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a4100f60 0x7f56a4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 --2- 172.21.15.23:0/1955733346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4101910 0x7f56a4105950 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 -- 172.21.15.23:0/1955733346 >> 172.21.15.23:0/1955733346 conn(0x7f56a40fc7c0 msgr2=0x7f56a40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 -- 172.21.15.23:0/1955733346 shutdown_connections 2024-03-30T04:04:33.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.163+0000 7f56a9a00700 1 -- 172.21.15.23:0/1955733346 wait complete. 2024-03-30T04:04:33.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.164+0000 7f56a9a00700 1 Processor -- start 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.164+0000 7f56a9a00700 1 -- start start 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.164+0000 7f56a9a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 0x7f56a4197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a9a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4106080 0x7f56a41978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a9a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a9a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a419c6c0 con 0x7f56a419bc40 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a9a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a419c830 con 0x7f56a4100f60 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a9a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a419c9f0 con 0x7f56a4106080 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 0x7f56a4197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54724/0 (socket says 172.21.15.23:54724) 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 -- 172.21.15.23:0/973724683 learned_addr learned my addr 172.21.15.23:0/973724683 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a2000700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4106080 0x7f56a41978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 -- 172.21.15.23:0/973724683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4106080 msgr2=0x7f56a41978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4106080 0x7f56a41978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 -- 172.21.15.23:0/973724683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 msgr2=0x7f56a4197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 0x7f56a4197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a3400700 1 -- 172.21.15.23:0/973724683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56900095f0 con 0x7f56a419bc40 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.165+0000 7f56a2a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 0x7f56a4197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f56a3400700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f569c00ef30 tx=0x7f569c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f569b400700 1 -- 172.21.15.23:0/973724683 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c018070 con 0x7f56a419bc40 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f569b400700 1 -- 172.21.15.23:0/973724683 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f569c00f040 con 0x7f56a419bc40 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a41a3ef0 con 0x7f56a419bc40 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f569b400700 1 -- 172.21.15.23:0/973724683 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c00cd00 con 0x7f56a419bc40 2024-03-30T04:04:33.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.166+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a41a43d0 con 0x7f56a419bc40 2024-03-30T04:04:33.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.169+0000 7f569b400700 1 -- 172.21.15.23:0/973724683 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f569c003a00 con 0x7f56a419bc40 2024-03-30T04:04:33.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.170+0000 7f56bd5aab80 1 Processor -- start 2024-03-30T04:04:33.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f569b400700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 0x7f56880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 -- start start 2024-03-30T04:04:33.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f569b400700 1 -- 172.21.15.23:0/973724683 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f569c0ccb50 con 0x7f56a419bc40 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2000700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 0x7f56880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eb9bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eae5990 con 0x559c1eb8cef0 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eae3350 con 0x559c1eb9c200 2024-03-30T04:04:33.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eadec40 con 0x559c1eb8c5d0 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eb9bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eb9bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54738/0 (socket says 172.21.15.23:54738) 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50494/0 (socket says 172.21.15.23:50494) 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.171+0000 7f56a2000700 1 -- 172.21.15.23:0/2801378076 learned_addr learned my addr 172.21.15.23:0/2801378076 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a3400700 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 -- 172.21.15.23:0/2801378076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 msgr2=0x559c1eb9e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 -- 172.21.15.23:0/2801378076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 msgr2=0x559c1eb9bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eb9bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 -- 172.21.15.23:0/2801378076 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c1e9603b0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2000700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 0x7f56880a3750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5690005cd0 tx=0x7f5690005c20 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f56a2a00700 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56940114f0 tx=0x7f5694011800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c71a88b023d6a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f568fe00700 1 -- 172.21.15.23:0/2801378076 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694011310 con 0x559c1eb8c5d0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f568fe00700 1 -- 172.21.15.23:0/2801378076 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56940044f0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.172+0000 7f568fe00700 1 -- 172.21.15.23:0/2801378076 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569401a6e0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 -- 172.21.15.23:0/2801378076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 msgr2=0x559c1eb8c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56940114f0 tx=0x7f5694011800 comp rx=0 tx=0).stop 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 -- 172.21.15.23:0/2801378076 shutdown_connections 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eb8c9b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/2801378076 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eb9bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 -- 172.21.15.23:0/2801378076 >> 172.21.15.23:0/2801378076 conn(0x559c1eaec6a0 msgr2=0x559c1eb88a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 -- 172.21.15.23:0/2801378076 shutdown_connections 2024-03-30T04:04:33.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.173+0000 7f56bd5aab80 1 -- 172.21.15.23:0/2801378076 wait complete. 2024-03-30T04:04:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.174+0000 7f56bd5aab80 1 -- start start 2024-03-30T04:04:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.174+0000 7f56bd5aab80 1 Processor -- start 2024-03-30T04:04:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.174+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb9bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb995e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1ea7ad80 con 0x559c1eb8c5d0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1ea7aa20 con 0x559c1eb8cef0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1e9626a0 con 0x559c1eb9c200 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56bd5aab80 10 client.0 _ll_register_callbacks cb 0x559c1ea84470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb9bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb9bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54754/0 (socket says 172.21.15.23:54754) 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2a00700 1 -- 172.21.15.23:0/1371852096 learned_addr learned my addr 172.21.15.23:0/1371852096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58666/0 (socket says 172.21.15.23:58666) 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a3400700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 -- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 msgr2=0x559c1eb995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 -- 172.21.15.23:0/1371852096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 msgr2=0x559c1eb9bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb9bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.175+0000 7f56a2000700 1 -- 172.21.15.23:0/1371852096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5694009c90 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f56a2000700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f569000b690 tx=0x7f5690009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95d31224514e5147 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f568cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690004960 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c1ea9f380 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f5690004ac0 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5690004c80 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f56bd5aab80 10 client.26318 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.176+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559c1e90cb60 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559c1ea9f380 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f5690031070 con 0x559c1eb8cef0 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 10 client.26318 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 10 client.26318 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:04:33.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 20 client.26318 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ebabc60 0x559c1ebae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 4 client.26318 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c1ead7700 con 0x559c1ebabc60 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.177+0000 7f56a2a00700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ebabc60 0x559c1ebae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.178+0000 7f56a2a00700 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ebabc60 0x559c1ebae080 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a17f0b24b16f98e5 server_cookie=c72ea0566f89f94e in_seq=0 out_seq=0 2024-03-30T04:04:33.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.178+0000 7f568cc00700 10 client.26318 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.178+0000 7f568cc00700 1 -- 172.21.15.23:0/1371852096 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559c1ead7700 con 0x559c1ebabc60 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.178+0000 7f568cc00700 10 client.26318 handle_command_reply: tid=0 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.178+0000 7f56bd5aab80 1 client.26318 shutdown 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 msgr2=0x559c1eb98fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f569000b690 tx=0x7f5690009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 shutdown_connections 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ebabc60 0x559c1ebae080 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f568cc00700 0 client.26318 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb9bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/1371852096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eb98fb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 >> 172.21.15.23:0/1371852096 conn(0x559c1eaec6a0 msgr2=0x559c1eb8e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 shutdown_connections 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 1 -- 172.21.15.23:0/1371852096 wait complete. 2024-03-30T04:04:33.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.179+0000 7f56bd5aab80 20 client.26318 trim_cache size 0 max 16384 2024-03-30T04:04:33.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 149601 11.77 MB/sec warmup 70 sec latency 9.082 ms 2024-03-30T04:04:33.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.195+0000 7f56bd5aab80 1 Processor -- start 2024-03-30T04:04:33.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.195+0000 7f56bd5aab80 1 -- start start 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.195+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb98f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.195+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.195+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb815a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1ea9f380 con 0x559c1eb8c5d0 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eb99be0 con 0x559c1eb8cef0 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eb8df60 con 0x559c1eb9c200 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58668/0 (socket says 172.21.15.23:58668) 2024-03-30T04:04:33.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb98f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 -- 172.21.15.23:0/3201470803 learned_addr learned my addr 172.21.15.23:0/3201470803 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a3400700 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 -- 172.21.15.23:0/3201470803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 msgr2=0x559c1eb815a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb815a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 -- 172.21.15.23:0/3201470803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 msgr2=0x559c1eb98f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb98f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 -- 172.21.15.23:0/3201470803 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c1eaa4ea0 con 0x559c1eb8cef0 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a3400700 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb815a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:33.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.196+0000 7f56a2000700 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5690005450 tx=0x7f56900058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b81a712dbf920b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f5698c00700 1 -- 172.21.15.23:0/3201470803 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569000f5c0 con 0x559c1eb8cef0 2024-03-30T04:04:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f5698c00700 1 -- 172.21.15.23:0/3201470803 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f569000fd20 con 0x559c1eb8cef0 2024-03-30T04:04:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f5698c00700 1 -- 172.21.15.23:0/3201470803 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690029470 con 0x559c1eb8cef0 2024-03-30T04:04:33.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 -- 172.21.15.23:0/3201470803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 msgr2=0x559c1eba2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5690005450 tx=0x7f56900058c0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 -- 172.21.15.23:0/3201470803 shutdown_connections 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb9c200 0x559c1eb815a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8c5d0 0x559c1eb98f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/3201470803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb8cef0 0x559c1eba2e80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.198+0000 7f56bd5aab80 1 -- 172.21.15.23:0/3201470803 >> 172.21.15.23:0/3201470803 conn(0x559c1eaec6a0 msgr2=0x559c1eb98840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.199+0000 7f56bd5aab80 1 -- 172.21.15.23:0/3201470803 shutdown_connections 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.199+0000 7f56bd5aab80 1 -- 172.21.15.23:0/3201470803 wait complete. 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.199+0000 7f56bd5aab80 1 -- start start 2024-03-30T04:04:33.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.199+0000 7f56bd5aab80 1 Processor -- start 2024-03-30T04:04:33.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eba2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eb9add0 con 0x559c1eb8cef0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eb9af40 con 0x559c1eb9c200 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c1eb9b100 con 0x559c1eb8c5d0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56bd5aab80 10 client.0 _ll_register_callbacks cb 0x559c1ea84470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50520/0 (socket says 172.21.15.23:50520) 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eba2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58672/0 (socket says 172.21.15.23:58672) 2024-03-30T04:04:33.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 -- 172.21.15.23:0/568496724 learned_addr learned my addr 172.21.15.23:0/568496724 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 -- 172.21.15.23:0/568496724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 msgr2=0x559c1eb9a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 -- 172.21.15.23:0/568496724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 msgr2=0x559c1eba2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eba2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.200+0000 7f56a2a00700 1 -- 172.21.15.23:0/568496724 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559c1e90cb60 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f56a2a00700 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56940095c0 tx=0x7f5694005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f8d0aa0cb5a41b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f568cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569401e040 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56940044f0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f5694029140 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5694029300 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f56bd5aab80 10 client.26317 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x559c1e964cd0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.201+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f56940044f0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f56940253f0 con 0x559c1eb8c5d0 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 10 client.26317 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 10 client.26317 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 20 client.26317 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ec00a60 0x559c1ec02e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 4 client.26317 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559c1e92dcf0 con 0x559c1ec00a60 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56a2000700 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ec00a60 0x559c1ec02e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f568cc00700 10 client.26317 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.202+0000 7f56a2000700 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ec00a60 0x559c1ec02e80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e2ec4350fec1a351 server_cookie=f22cb16f5fbb2347 in_seq=0 out_seq=0 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f568cc00700 1 -- 172.21.15.23:0/568496724 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559c1e92dcf0 con 0x559c1ec00a60 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f568cc00700 10 client.26317 handle_command_reply: tid=0 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 client.26317 shutdown 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 msgr2=0x559c1eba27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56940095c0 tx=0x7f5694005d50 comp rx=0 tx=0).stop 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 shutdown_connections 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x559c1ec00a60 0x559c1ec02e80 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559c1eb9c200 0x559c1eb9a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f568cc00700 0 client.26317 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559c1eb8c5d0 0x559c1eba27e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 --2- 172.21.15.23:0/568496724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559c1eb8cef0 0x559c1eba2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 >> 172.21.15.23:0/568496724 conn(0x559c1eaec6a0 msgr2=0x559c1eb942a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 shutdown_connections 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 1 -- 172.21.15.23:0/568496724 wait complete. 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.203+0000 7f56bd5aab80 20 client.26317 trim_cache size 0 max 16384 2024-03-30T04:04:33.205 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 msgr2=0x7f56880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 0x7f56880a3750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5690005cd0 tx=0x7f5690005c20 comp rx=0 tx=0).stop 2024-03-30T04:04:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 msgr2=0x7f56a419c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f569c00ef30 tx=0x7f569c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 shutdown_connections 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f56a419bc40 0x7f56a419c0c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f56880a1290 0x7f56880a3750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56a4100f60 0x7f56a4197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 --2- 172.21.15.23:0/973724683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56a4106080 0x7f56a41978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 >> 172.21.15.23:0/973724683 conn(0x7f56a40fc7c0 msgr2=0x7f56a40fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 shutdown_connections 2024-03-30T04:04:33.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.204+0000 7f56a9a00700 1 -- 172.21.15.23:0/973724683 wait complete. 2024-03-30T04:04:33.221 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:33 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1624383628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:04:33.226 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 -- 172.21.15.23:0/175120590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff984067e30 msgr2=0x7ff9840682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 --2- 172.21.15.23:0/175120590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff984067e30 0x7ff9840682b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff9700099c0 tx=0x7ff970009cd0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 -- 172.21.15.23:0/175120590 shutdown_connections 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 --2- 172.21.15.23:0/175120590 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9840687f0 0x7ff98410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 --2- 172.21.15.23:0/175120590 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 0x7ff9841060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 --2- 172.21.15.23:0/175120590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff984067e30 0x7ff9840682b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 -- 172.21.15.23:0/175120590 >> 172.21.15.23:0/175120590 conn(0x7ff984076d40 msgr2=0x7ff984077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 -- 172.21.15.23:0/175120590 shutdown_connections 2024-03-30T04:04:33.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.465+0000 7ff98a600700 1 -- 172.21.15.23:0/175120590 wait complete. 2024-03-30T04:04:33.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.466+0000 7ff98a600700 1 Processor -- start 2024-03-30T04:04:33.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.466+0000 7ff98a600700 1 -- start start 2024-03-30T04:04:33.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff984067e30 0x7ff984197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 0x7ff98419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98419c140 con 0x7ff984067e30 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98419c2b0 con 0x7ff984105ce0 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff98a600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98419c470 con 0x7ff9840687f0 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff983400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff984067e30 0x7ff984197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50538/0 (socket says 172.21.15.23:50538) 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff982a00700 1 -- 172.21.15.23:0/2735638816 learned_addr learned my addr 172.21.15.23:0/2735638816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.467+0000 7ff983e00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 0x7ff98419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 -- 172.21.15.23:0/2735638816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 msgr2=0x7ff98419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 0x7ff98419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 -- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff984067e30 msgr2=0x7ff984197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff984067e30 0x7ff984197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 -- 172.21.15.23:0/2735638816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700095f0 con 0x7ff9840687f0 2024-03-30T04:04:33.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff982a00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff970006010 tx=0x7ff97000b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff97be00700 1 -- 172.21.15.23:0/2735638816 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97001d070 con 0x7ff9840687f0 2024-03-30T04:04:33.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff97be00700 1 -- 172.21.15.23:0/2735638816 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff97000be00 con 0x7ff9840687f0 2024-03-30T04:04:33.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98419c6f0 con 0x7ff9840687f0 2024-03-30T04:04:33.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.468+0000 7ff97be00700 1 -- 172.21.15.23:0/2735638816 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97000fa90 con 0x7ff9840687f0 2024-03-30T04:04:33.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.469+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9841a3f90 con 0x7ff9840687f0 2024-03-30T04:04:33.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.471+0000 7ff97be00700 1 -- 172.21.15.23:0/2735638816 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff97002b440 con 0x7ff9840687f0 2024-03-30T04:04:33.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff99e192b80 1 Processor -- start 2024-03-30T04:04:33.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff99e192b80 1 -- start start 2024-03-30T04:04:33.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff97be00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 0x7ff9680a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff97be00700 1 -- 172.21.15.23:0/2735638816 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff9700cdc20 con 0x7ff9840687f0 2024-03-30T04:04:33.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff983e00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 0x7ff9680a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.473+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff99e192b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f333990 con 0x556c3f3e5df0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff99e192b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f331350 con 0x556c3f3daef0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff99e192b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f32cc40 con 0x556c3f3da5d0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58704/0 (socket says 172.21.15.23:58704) 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 -- 172.21.15.23:0/1665690999 learned_addr learned my addr 172.21.15.23:0/1665690999 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff983e00700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff983400700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 -- 172.21.15.23:0/1665690999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 msgr2=0x556c3f3da9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3da9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 -- 172.21.15.23:0/1665690999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 msgr2=0x556c3f3e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 -- 172.21.15.23:0/1665690999 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c3f1ae3b0 con 0x556c3f3daef0 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff983400700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.474+0000 7ff982a00700 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a28c0 tx=0x7ff970009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=975ac2a931db684b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff978c00700 1 -- 172.21.15.23:0/1665690999 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97009e830 con 0x556c3f3daef0 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff983e00700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 0x7ff9680a37d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00f870 tx=0x7ff97c00fc30 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff978c00700 1 -- 172.21.15.23:0/1665690999 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9700a2df0 con 0x556c3f3daef0 2024-03-30T04:04:33.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff978c00700 1 -- 172.21.15.23:0/1665690999 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9700a2960 con 0x556c3f3daef0 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff99e192b80 1 -- 172.21.15.23:0/1665690999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 msgr2=0x556c3f3e58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a28c0 tx=0x7ff970009310 comp rx=0 tx=0).stop 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.475+0000 7ff99e192b80 1 -- 172.21.15.23:0/1665690999 shutdown_connections 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3da9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1665690999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3e58b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 -- 172.21.15.23:0/1665690999 >> 172.21.15.23:0/1665690999 conn(0x556c3f33a6a0 msgr2=0x556c3f3d6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 -- 172.21.15.23:0/1665690999 shutdown_connections 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 -- 172.21.15.23:0/1665690999 wait complete. 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 -- start start 2024-03-30T04:04:33.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.476+0000 7ff99e192b80 1 Processor -- start 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 0x556c3f3e2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3e5df0 0x556c3f3e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f2c8d80 con 0x556c3f3daef0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f2c8a20 con 0x556c3f3e5df0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f1b06a0 con 0x556c3f3da5d0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff99e192b80 10 client.0 _ll_register_callbacks cb 0x556c3f2d2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50552/0 (socket says 172.21.15.23:50552) 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff983400700 1 -- 172.21.15.23:0/1476447661 learned_addr learned my addr 172.21.15.23:0/1476447661 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.477+0000 7ff983e00700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3e5df0 0x556c3f3e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff982a00700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 0x556c3f3e2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 -- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3e5df0 msgr2=0x556c3f3e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3e5df0 0x556c3f3e3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 -- 172.21.15.23:0/1476447661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 msgr2=0x556c3f3e2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 0x556c3f3e2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 -- 172.21.15.23:0/1476447661 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9841a3f90 con 0x556c3f3da5d0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff982a00700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 0x556c3f3e2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff983400700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9740126d0 tx=0x7ff9740129e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d1954b57fad151e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff96d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974017070 con 0x556c3f3da5d0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff97400f480 con 0x556c3f3da5d0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.478+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556c3f2ed380 con 0x556c3f3da5d0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9740122e0 con 0x556c3f3da5d0 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 10 client.26329 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556c3f15ab60 con 0x556c3f3da5d0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556c3f2ed380 con 0x556c3f3da5d0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff974010790 con 0x556c3f3da5d0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 10 client.26329 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 10 client.26329 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 20 client.26329 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f3f9ad0 0x556c3f3fbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 4 client.26329 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.479+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c3f325700 con 0x556c3f3f9ad0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.480+0000 7ff982a00700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f3f9ad0 0x556c3f3fbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.480+0000 7ff96d600700 10 client.26329 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.480+0000 7ff982a00700 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f3f9ad0 0x556c3f3fbef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c2c382c658727780 server_cookie=f56eb9d71b1e8b09 in_seq=0 out_seq=0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff96d600700 1 -- 172.21.15.23:0/1476447661 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556c3f325700 con 0x556c3f3f9ad0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff96d600700 10 client.26329 handle_command_reply: tid=0 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 client.26329 shutdown 2024-03-30T04:04:33.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 msgr2=0x556c3f3e5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff9740126d0 tx=0x7ff9740129e0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 shutdown_connections 2024-03-30T04:04:33.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f3f9ad0 0x556c3f3fbef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3e5df0 0x556c3f3e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3e5810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 --2- 172.21.15.23:0/1476447661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3daef0 0x556c3f3e2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 >> 172.21.15.23:0/1476447661 conn(0x556c3f33a6a0 msgr2=0x556c3f3dc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff96d600700 0 client.26329 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:04:33.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 shutdown_connections 2024-03-30T04:04:33.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 1 -- 172.21.15.23:0/1476447661 wait complete. 2024-03-30T04:04:33.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.481+0000 7ff99e192b80 20 client.26329 trim_cache size 0 max 16384 2024-03-30T04:04:33.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:33 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1624383628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:04:33.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 Processor -- start 2024-03-30T04:04:33.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 -- start start 2024-03-30T04:04:33.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3dde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3e3af0 con 0x556c3f3e5df0 2024-03-30T04:04:33.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3f1130 con 0x556c3f3da5d0 2024-03-30T04:04:33.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff99e192b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3e1130 con 0x556c3f3daef0 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff983400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff982a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50568/0 (socket says 172.21.15.23:50568) 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff983e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3dde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.498+0000 7ff982a00700 1 -- 172.21.15.23:0/2753824918 learned_addr learned my addr 172.21.15.23:0/2753824918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 -- 172.21.15.23:0/2753824918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3da5d0 msgr2=0x556c3f3f0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 -- 172.21.15.23:0/2753824918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 msgr2=0x556c3f3dde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3dde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 -- 172.21.15.23:0/2753824918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c3f2f2ea0 con 0x556c3f3daef0 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff983e00700 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3dde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff982a00700 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff970096f00 tx=0x7ff97009a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15863f436b2e2bbe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff979600700 1 -- 172.21.15.23:0/2753824918 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9700b31c0 con 0x556c3f3daef0 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff979600700 1 -- 172.21.15.23:0/2753824918 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9700b3320 con 0x556c3f3daef0 2024-03-30T04:04:33.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.499+0000 7ff979600700 1 -- 172.21.15.23:0/2753824918 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9700a2960 con 0x556c3f3daef0 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 -- 172.21.15.23:0/2753824918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 msgr2=0x556c3f3e53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff970096f00 tx=0x7ff97009a490 comp rx=0 tx=0).stop 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 -- 172.21.15.23:0/2753824918 shutdown_connections 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3dde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 --2- 172.21.15.23:0/2753824918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3daef0 0x556c3f3e53a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 -- 172.21.15.23:0/2753824918 >> 172.21.15.23:0/2753824918 conn(0x556c3f33a6a0 msgr2=0x556c3f3cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 -- 172.21.15.23:0/2753824918 shutdown_connections 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.500+0000 7ff99e192b80 1 -- 172.21.15.23:0/2753824918 wait complete. 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.501+0000 7ff99e192b80 1 -- start start 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.501+0000 7ff99e192b80 1 Processor -- start 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3eea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3eeff0 con 0x556c3f3e5df0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3ef160 con 0x556c3f3daef0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c3f3ef320 con 0x556c3f3da5d0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff99e192b80 10 client.0 _ll_register_callbacks cb 0x556c3f2d2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff983e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58738/0 (socket says 172.21.15.23:58738) 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 -- 172.21.15.23:0/214793738 learned_addr learned my addr 172.21.15.23:0/214793738 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff983400700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 -- 172.21.15.23:0/214793738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 msgr2=0x556c3f3f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 -- 172.21.15.23:0/214793738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 msgr2=0x556c3f3eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.502+0000 7ff982a00700 1 -- 172.21.15.23:0/214793738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97c00fdb0 con 0x556c3f3daef0 2024-03-30T04:04:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff982a00700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a28c0 tx=0x7ff97000f460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=930fda079f293f0f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff96d600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9700b31c0 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9700a2df0 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff97009ae70 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff970099ef0 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff99e192b80 10 client.26342 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556c3f1ae3b0 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.503+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9700a2df0 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff970079b70 con 0x556c3f3daef0 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 10 client.26342 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 10 client.26342 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 20 client.26342 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f44e6f0 0x556c3f450b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 4 client.26342 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556c3f17bcf0 con 0x556c3f44e6f0 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff983400700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f44e6f0 0x556c3f450b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff983400700 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f44e6f0 0x556c3f450b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38f33f71efc86aab server_cookie=f5ce6d022f77987f in_seq=0 out_seq=0 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.504+0000 7ff96d600700 10 client.26342 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff96d600700 1 -- 172.21.15.23:0/214793738 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556c3f17bcf0 con 0x556c3f44e6f0 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff96d600700 10 client.26342 handle_command_reply: tid=0 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 client.26342 shutdown 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 msgr2=0x556c3f3f08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff9700a28c0 tx=0x7ff97000f460 comp rx=0 tx=0).stop 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 shutdown_connections 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x556c3f44e6f0 0x556c3f450b10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556c3f3e5df0 0x556c3f3eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556c3f3da5d0 0x556c3f3f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 --2- 172.21.15.23:0/214793738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556c3f3daef0 0x556c3f3f08b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff96d600700 0 client.26342 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 >> 172.21.15.23:0/214793738 conn(0x556c3f33a6a0 msgr2=0x556c3f3dde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 shutdown_connections 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 1 -- 172.21.15.23:0/214793738 wait complete. 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.505+0000 7ff99e192b80 20 client.26342 trim_cache size 0 max 16384 2024-03-30T04:04:33.507 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 msgr2=0x7ff9680a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 0x7ff9680a37d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00f870 tx=0x7ff97c00fc30 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 msgr2=0x7ff9841977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff970006010 tx=0x7ff97000b9c0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 shutdown_connections 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9680a1310 0x7ff9680a37d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff984105ce0 0x7ff98419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff984067e30 0x7ff984197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 --2- 172.21.15.23:0/2735638816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9840687f0 0x7ff9841977b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 >> 172.21.15.23:0/2735638816 conn(0x7ff984076d40 msgr2=0x7ff9840fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 shutdown_connections 2024-03-30T04:04:33.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.506+0000 7ff98a600700 1 -- 172.21.15.23:0/2735638816 wait complete. 2024-03-30T04:04:33.523 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:04:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:33 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1624383628' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 -- 172.21.15.23:0/3183592280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118105ce0 msgr2=0x7f11181060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 --2- 172.21.15.23:0/3183592280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118105ce0 0x7f11181060c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f11080099c0 tx=0x7f1108009cd0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 -- 172.21.15.23:0/3183592280 shutdown_connections 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 --2- 172.21.15.23:0/3183592280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11180687f0 0x7f111810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 --2- 172.21.15.23:0/3183592280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118105ce0 0x7f11181060c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 --2- 172.21.15.23:0/3183592280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1118067e30 0x7f11180682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 -- 172.21.15.23:0/3183592280 >> 172.21.15.23:0/3183592280 conn(0x7f1118076d40 msgr2=0x7f1118077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 -- 172.21.15.23:0/3183592280 shutdown_connections 2024-03-30T04:04:33.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.744+0000 7f111d400700 1 -- 172.21.15.23:0/3183592280 wait complete. 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.745+0000 7f111d400700 1 Processor -- start 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.745+0000 7f111d400700 1 -- start start 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11180687f0 0x7f1118199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1118105ce0 0x7f111819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111819e360 con 0x7f11180687f0 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111819e4d0 con 0x7f1118067e30 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f111d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111819e690 con 0x7f1118105ce0 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58762/0 (socket says 172.21.15.23:58762) 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 -- 172.21.15.23:0/1908137852 learned_addr learned my addr 172.21.15.23:0/1908137852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1115e00700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11180687f0 0x7f1118199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1117200700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1118105ce0 0x7f111819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 -- 172.21.15.23:0/1908137852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1118105ce0 msgr2=0x7f111819dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1118105ce0 0x7f111819dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 -- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11180687f0 msgr2=0x7f1118199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11180687f0 0x7f1118199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.746+0000 7f1116800700 1 -- 172.21.15.23:0/1908137852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11080095f0 con 0x7f1118067e30 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f1116800700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f110800b580 tx=0x7f110800bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f110f400700 1 -- 172.21.15.23:0/1908137852 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110801c070 con 0x7f1118067e30 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f110f400700 1 -- 172.21.15.23:0/1908137852 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110800fdf0 con 0x7f1118067e30 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f110f400700 1 -- 172.21.15.23:0/1908137852 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1108022c40 con 0x7f1118067e30 2024-03-30T04:04:33.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111819e910 con 0x7f1118067e30 2024-03-30T04:04:33.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.747+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11181a61b0 con 0x7f1118067e30 2024-03-30T04:04:33.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.751+0000 7f110f400700 1 -- 172.21.15.23:0/1908137852 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1108020070 con 0x7f1118067e30 2024-03-30T04:04:33.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.753+0000 7f1130f19b80 1 Processor -- start 2024-03-30T04:04:33.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f110f400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 0x7f10fc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 -- start start 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f110f400700 1 -- 172.21.15.23:0/1908137852 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f11080cdd30 con 0x7f1118067e30 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e904f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e90659f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8fa8990 con 0x55b3e9063600 2024-03-30T04:04:33.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8fa6350 con 0x55b3e904f5d0 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1115e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1115e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50600/0 (socket says 172.21.15.23:50600) 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.755+0000 7f1115e00700 1 -- 172.21.15.23:0/2537310538 learned_addr learned my addr 172.21.15.23:0/2537310538 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.755+0000 7f1116800700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e904f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.754+0000 7f1130f19b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8fa1c40 con 0x55b3e904fef0 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.755+0000 7f1117200700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e90659f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1117200700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 0x7f10fc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 -- 172.21.15.23:0/2537310538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 msgr2=0x55b3e90630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 -- 172.21.15.23:0/2537310538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 msgr2=0x55b3e90659f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e90659f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 -- 172.21.15.23:0/2537310538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3e8e233b0 con 0x55b3e904f5d0 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1117200700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e90659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1115e00700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:33.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1116800700 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e904f9b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f11080cb0c0 tx=0x7f110809a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5892003850525b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1103400700 1 -- 172.21.15.23:0/2537310538 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110809ab10 con 0x55b3e904f5d0 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1117200700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 0x7f10fc0a37d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1110009fd0 tx=0x7f111000b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1103400700 1 -- 172.21.15.23:0/2537310538 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110809e940 con 0x55b3e904f5d0 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.756+0000 7f1103400700 1 -- 172.21.15.23:0/2537310538 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110809a260 con 0x55b3e904f5d0 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 -- 172.21.15.23:0/2537310538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 msgr2=0x55b3e904f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e904f9b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f11080cb0c0 tx=0x7f110809a650 comp rx=0 tx=0).stop 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 -- 172.21.15.23:0/2537310538 shutdown_connections 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e90659f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e904f9b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2537310538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 -- 172.21.15.23:0/2537310538 >> 172.21.15.23:0/2537310538 conn(0x55b3e8faf6a0 msgr2=0x55b3e904ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 -- 172.21.15.23:0/2537310538 shutdown_connections 2024-03-30T04:04:33.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.757+0000 7f1130f19b80 1 -- 172.21.15.23:0/2537310538 wait complete. 2024-03-30T04:04:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.758+0000 7f1130f19b80 1 -- start start 2024-03-30T04:04:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.758+0000 7f1130f19b80 1 Processor -- start 2024-03-30T04:04:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 0x55b3e9062fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90602d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8f3dd80 con 0x55b3e904f5d0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8f3da20 con 0x55b3e9063600 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e8e256a0 con 0x55b3e904fef0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1130f19b80 10 client.0 _ll_register_callbacks cb 0x55b3e8f47470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 0x55b3e9062fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1117200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1115e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1117200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58784/0 (socket says 172.21.15.23:58784) 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1117200700 1 -- 172.21.15.23:0/2234910216 learned_addr learned my addr 172.21.15.23:0/2234910216 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 -- 172.21.15.23:0/2234910216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 msgr2=0x55b3e90602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 -- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 msgr2=0x55b3e9060890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1116800700 1 -- 172.21.15.23:0/2234910216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1104009980 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.759+0000 7f1117200700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f1116800700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 0x55b3e9062fb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1108098e60 tx=0x7f1108022650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8105acf7b328f5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f10f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110809ab10 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f11080a2f00 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3e8f62380 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f110809ad70 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f1130f19b80 10 client.16389 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f11181a61b0 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.760+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b3e8f62380 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f110809bb20 con 0x55b3e904f5d0 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 10 client.16389 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 10 client.16389 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 20 client.16389 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90729a0 0x55b3e9074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 4 client.16389 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3e8f9a700 con 0x55b3e90729a0 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1115e00700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90729a0 0x55b3e9074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f1115e00700 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90729a0 0x55b3e9074dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7bf3fd0daf3e39e server_cookie=927c473cf063306f in_seq=0 out_seq=0 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.761+0000 7f10f7e00700 10 client.16389 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f10f7e00700 1 -- 172.21.15.23:0/2234910216 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b3e8f9a700 con 0x55b3e90729a0 2024-03-30T04:04:33.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f10f7e00700 10 client.16389 handle_command_reply: tid=0 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 client.16389 shutdown 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 msgr2=0x55b3e9062fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 0x55b3e9062fb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f1108098e60 tx=0x7f1108022650 comp rx=0 tx=0).stop 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 shutdown_connections 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e9063600 0x55b3e9060890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904f5d0 0x55b3e9062fb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90729a0 0x55b3e9074dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 --2- 172.21.15.23:0/2234910216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904fef0 0x55b3e90602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 >> 172.21.15.23:0/2234910216 conn(0x55b3e8faf6a0 msgr2=0x55b3e9051510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.762+0000 7f10f7e00700 0 client.16389 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.763+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 shutdown_connections 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.763+0000 7f1130f19b80 1 -- 172.21.15.23:0/2234910216 wait complete. 2024-03-30T04:04:33.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.763+0000 7f1130f19b80 20 client.16389 trim_cache size 0 max 16384 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 Processor -- start 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 -- start start 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 0x55b3e9069ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e9063600 0x55b3e905b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9060e90 con 0x55b3e904fef0 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e906a530 con 0x55b3e904f5d0 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1130f19b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e905e8c0 con 0x55b3e9063600 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1115e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 0x55b3e9069ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1116800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1117200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e9063600 0x55b3e905b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1116800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58794/0 (socket says 172.21.15.23:58794) 2024-03-30T04:04:33.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1116800700 1 -- 172.21.15.23:0/1177710216 learned_addr learned my addr 172.21.15.23:0/1177710216 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.778+0000 7f1115e00700 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 0x55b3e9069ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54888/0 (socket says 172.21.15.23:54888) 2024-03-30T04:04:33.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 -- 172.21.15.23:0/1177710216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e9063600 msgr2=0x55b3e905b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e9063600 0x55b3e905b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 -- 172.21.15.23:0/1177710216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 msgr2=0x55b3e9069ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 0x55b3e9069ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 -- 172.21.15.23:0/1177710216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3e8dcfb60 con 0x55b3e904f5d0 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1116800700 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1108000c00 tx=0x7f110809a330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfdf08b497eff553 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1103e00700 1 -- 172.21.15.23:0/1177710216 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110809a650 con 0x55b3e904f5d0 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1103e00700 1 -- 172.21.15.23:0/1177710216 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11080a2f00 con 0x55b3e904f5d0 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.779+0000 7f1103e00700 1 -- 172.21.15.23:0/1177710216 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110809afd0 con 0x55b3e904f5d0 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- 172.21.15.23:0/1177710216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 msgr2=0x55b3e9060850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1108000c00 tx=0x7f110809a330 comp rx=0 tx=0).stop 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- 172.21.15.23:0/1177710216 shutdown_connections 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e9063600 0x55b3e905b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904f5d0 0x55b3e9060850 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 --2- 172.21.15.23:0/1177710216 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e904fef0 0x55b3e9069ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- 172.21.15.23:0/1177710216 >> 172.21.15.23:0/1177710216 conn(0x55b3e8faf6a0 msgr2=0x55b3e90445a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- 172.21.15.23:0/1177710216 shutdown_connections 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- 172.21.15.23:0/1177710216 wait complete. 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.780+0000 7f1130f19b80 1 -- start start 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 Processor -- start 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 0x55b3e9051dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e9069080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9069790 con 0x55b3e9063600 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9069900 con 0x55b3e904fef0 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3e9069ac0 con 0x55b3e904f5d0 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1130f19b80 10 client.0 _ll_register_callbacks cb 0x55b3e8f47470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1116800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1117200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e9069080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1116800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50632/0 (socket says 172.21.15.23:50632) 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1116800700 1 -- 172.21.15.23:0/4035083867 learned_addr learned my addr 172.21.15.23:0/4035083867 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.781+0000 7f1115e00700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 0x55b3e9051dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 -- 172.21.15.23:0/4035083867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 msgr2=0x55b3e9051890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 -- 172.21.15.23:0/4035083867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 msgr2=0x55b3e9069080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e9069080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 -- 172.21.15.23:0/4035083867 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11181a61b0 con 0x55b3e904fef0 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1116800700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f1115e00700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 0x55b3e9051dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55b3e9075660 tx=0x7f1104004590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d54c88add2178eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.782+0000 7f10f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:33.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1104017070 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1104021c00 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f110809e940 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1104021dc0 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 10 client.26360 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b3e8f67ea0 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f110809e940 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f11040124b0 con 0x55b3e904fef0 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 10 client.26360 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 10 client.26360 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 20 client.26360 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90c7a00 0x55b3e90c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 4 client.26360 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:33.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.783+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b3e8df0cf0 con 0x55b3e90c7a00 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f1116800700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90c7a00 0x55b3e90c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f1116800700 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90c7a00 0x55b3e90c9e20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6e056ad5933ddb50 server_cookie=8d6eb5ad08fa67a0 in_seq=0 out_seq=0 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f10f7e00700 10 client.26360 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f10f7e00700 1 -- 172.21.15.23:0/4035083867 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b3e8df0cf0 con 0x55b3e90c7a00 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f10f7e00700 10 client.26360 handle_command_reply: tid=0 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f1130f19b80 1 client.26360 shutdown 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.784+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 msgr2=0x55b3e9051dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 0x55b3e9051dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55b3e9075660 tx=0x7f1104004590 comp rx=0 tx=0).stop 2024-03-30T04:04:33.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 shutdown_connections 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3e9063600 0x55b3e9069080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55b3e90c7a00 0x55b3e90c9e20 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3e904f5d0 0x55b3e9051890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 --2- 172.21.15.23:0/4035083867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3e904fef0 0x55b3e9051dd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f10f7e00700 0 client.26360 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 >> 172.21.15.23:0/4035083867 conn(0x55b3e8faf6a0 msgr2=0x55b3e905b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 shutdown_connections 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 1 -- 172.21.15.23:0/4035083867 wait complete. 2024-03-30T04:04:33.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f1130f19b80 20 client.26360 trim_cache size 0 max 16384 2024-03-30T04:04:33.790 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:33.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 msgr2=0x7f10fc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 0x7f10fc0a37d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1110009fd0 tx=0x7f111000b040 comp rx=0 tx=0).stop 2024-03-30T04:04:33.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 msgr2=0x7f1118199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.785+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f110800b580 tx=0x7f110800bfd0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 shutdown_connections 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1118105ce0 0x7f111819dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f10fc0a1310 0x7f10fc0a37d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1118067e30 0x7f1118199440 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 --2- 172.21.15.23:0/1908137852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11180687f0 0x7f1118199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 >> 172.21.15.23:0/1908137852 conn(0x7f1118076d40 msgr2=0x7f111810c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 shutdown_connections 2024-03-30T04:04:33.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:33.786+0000 7f111d400700 1 -- 172.21.15.23:0/1908137852 wait complete. 2024-03-30T04:04:33.803 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:04:34.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.019+0000 7f8190400700 1 -- 172.21.15.23:0/2335814895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8188100f50 msgr2=0x7f8188101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 --2- 172.21.15.23:0/2335814895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8188100f50 0x7f8188101330 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f817c009990 tx=0x7f817c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 -- 172.21.15.23:0/2335814895 shutdown_connections 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 --2- 172.21.15.23:0/2335814895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188106070 0x7f818810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 --2- 172.21.15.23:0/2335814895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8188100f50 0x7f8188101330 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 --2- 172.21.15.23:0/2335814895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188101900 0x7f8188105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 -- 172.21.15.23:0/2335814895 >> 172.21.15.23:0/2335814895 conn(0x7f81880fc7f0 msgr2=0x7f81880fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 -- 172.21.15.23:0/2335814895 shutdown_connections 2024-03-30T04:04:34.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.020+0000 7f8190400700 1 -- 172.21.15.23:0/2335814895 wait complete. 2024-03-30T04:04:34.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 Processor -- start 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 -- start start 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188100f50 0x7f8188199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f818819dd50 0x7f818819e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819e7d0 con 0x7f818819dd50 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819e940 con 0x7f8188106070 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.021+0000 7f8190400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818819eb00 con 0x7f8188100f50 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188100f50 0x7f8188199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58830/0 (socket says 172.21.15.23:58830) 2024-03-30T04:04:34.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 -- 172.21.15.23:0/3749416690 learned_addr learned my addr 172.21.15.23:0/3749416690 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818e400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f818819dd50 0x7f818819e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 -- 172.21.15.23:0/3749416690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188100f50 msgr2=0x7f8188199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188100f50 0x7f8188199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 -- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f818819dd50 msgr2=0x7f818819e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f818819dd50 0x7f818819e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.022+0000 7f818d000700 1 -- 172.21.15.23:0/3749416690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817c0095f0 con 0x7f8188106070 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f818d000700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f817400e970 tx=0x7f817400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f8186000700 1 -- 172.21.15.23:0/3749416690 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8174009790 con 0x7f8188106070 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f8186000700 1 -- 172.21.15.23:0/3749416690 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f817400c7e0 con 0x7f8188106070 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81881a6060 con 0x7f8188106070 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f8186000700 1 -- 172.21.15.23:0/3749416690 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81740054e0 con 0x7f8188106070 2024-03-30T04:04:34.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.023+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81881a6540 con 0x7f8188106070 2024-03-30T04:04:34.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.028+0000 7f8186000700 1 -- 172.21.15.23:0/3749416690 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f817400c950 con 0x7f8188106070 2024-03-30T04:04:34.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 Processor -- start 2024-03-30T04:04:34.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 -- start start 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f8186000700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 0x7f81700a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 0x56488cacc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488ca17990 con 0x56488cac9df0 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488ca15350 con 0x56488cabeef0 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.029+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488ca10c40 con 0x56488cabe5d0 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f8186000700 1 -- 172.21.15.23:0/3749416690 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8174014070 con 0x7f8188106070 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50652/0 (socket says 172.21.15.23:50652) 2024-03-30T04:04:34.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 -- 172.21.15.23:0/1082207521 learned_addr learned my addr 172.21.15.23:0/1082207521 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818d000700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 -- 172.21.15.23:0/1082207521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 msgr2=0x56488cac98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 -- 172.21.15.23:0/1082207521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 msgr2=0x56488cacc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818e400700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 0x56488cacc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.030+0000 7f818da00700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 0x56488cacc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f818da00700 1 -- 172.21.15.23:0/1082207521 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56488c8923b0 con 0x56488cabe5d0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f818e400700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 0x56488cacc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f818e400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 0x7f81700a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f818da00700 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f817c005ec0 tx=0x7f817c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18e653798f4ba8d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f817aa00700 1 -- 172.21.15.23:0/1082207521 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c01d070 con 0x56488cabe5d0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f817aa00700 1 -- 172.21.15.23:0/1082207521 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f817c01e040 con 0x56488cabe5d0 2024-03-30T04:04:34.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.031+0000 7f817aa00700 1 -- 172.21.15.23:0/1082207521 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c0047d0 con 0x56488cabe5d0 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1082207521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 msgr2=0x56488cabe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f817c005ec0 tx=0x7f817c005ef0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f818e400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 0x7f81700a3750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8180009550 tx=0x7f818000b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1082207521 shutdown_connections 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cac9df0 0x56488cacc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cabe9b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1082207521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1082207521 >> 172.21.15.23:0/1082207521 conn(0x56488ca1e6a0 msgr2=0x56488cabaa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1082207521 shutdown_connections 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.032+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1082207521 wait complete. 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.033+0000 7f81a3f68b80 1 -- start start 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.033+0000 7f81a3f68b80 1 Processor -- start 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabe5d0 0x56488cac9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cac9df0 0x56488cac6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488c9acd80 con 0x56488cabe5d0 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488c9aca20 con 0x56488cabeef0 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488c8946a0 con 0x56488cac9df0 2024-03-30T04:04:34.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f81a3f68b80 10 client.0 _ll_register_callbacks cb 0x56488c9b6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f818d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f818d000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58854/0 (socket says 172.21.15.23:58854) 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f818d000700 1 -- 172.21.15.23:0/3788642202 learned_addr learned my addr 172.21.15.23:0/3788642202 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f818da00700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabe5d0 0x56488cac9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.034+0000 7f818e400700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cac9df0 0x56488cac6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 -- 172.21.15.23:0/3788642202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cac9df0 msgr2=0x56488cac6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cac9df0 0x56488cac6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 -- 172.21.15.23:0/3788642202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabe5d0 msgr2=0x56488cac9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabe5d0 0x56488cac9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 -- 172.21.15.23:0/3788642202 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f818810e460 con 0x56488cabeef0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f818d000700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81740ca6c0 tx=0x7f8174099a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c31075374ec05f79 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f816f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:34.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81740caae0 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f817409de50 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56488c9d1380 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.035+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81740a24a0 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 10 client.26372 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56488c83eb60 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81740b1070 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f81740b2bd0 con 0x56488cabeef0 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 10 client.26372 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 10 client.26372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 20 client.26372 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cad9790 0x56488cadbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 4 client.26372 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:34.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.036+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56488ca09700 con 0x56488cad9790 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.037+0000 7f818da00700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cad9790 0x56488cadbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.037+0000 7f818da00700 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cad9790 0x56488cadbbb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67f6b2494a8ee5e server_cookie=c68d643d3f2d8cc9 in_seq=0 out_seq=0 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.037+0000 7f816f400700 10 client.26372 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f816f400700 1 -- 172.21.15.23:0/3788642202 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56488ca09700 con 0x56488cad9790 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f816f400700 10 client.26372 handle_command_reply: tid=0 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 client.26372 shutdown 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 msgr2=0x56488cac6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f81740ca6c0 tx=0x7f8174099a90 comp rx=0 tx=0).stop 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 shutdown_connections 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cad9790 0x56488cadbbb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cac9df0 0x56488cac6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabe5d0 0x56488cac9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f816f400700 0 client.26372 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:34.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3788642202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cabeef0 0x56488cac6a50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 >> 172.21.15.23:0/3788642202 conn(0x56488ca1e6a0 msgr2=0x56488cac0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 shutdown_connections 2024-03-30T04:04:34.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.038+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3788642202 wait complete. 2024-03-30T04:04:34.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.039+0000 7f81a3f68b80 20 client.26372 trim_cache size 0 max 16384 2024-03-30T04:04:34.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 Processor -- start 2024-03-30T04:04:34.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 -- start start 2024-03-30T04:04:34.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cad0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cab3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cac8000 con 0x56488cabeef0 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cad95b0 con 0x56488cac9df0 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cad0f80 con 0x56488cabe5d0 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.055+0000 7f818d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cad0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50666/0 (socket says 172.21.15.23:50666) 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 -- 172.21.15.23:0/1353047671 learned_addr learned my addr 172.21.15.23:0/1353047671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:34.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818e400700 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cab3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 -- 172.21.15.23:0/1353047671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 msgr2=0x56488cab3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cab3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 -- 172.21.15.23:0/1353047671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 msgr2=0x56488cad0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cad0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 -- 172.21.15.23:0/1353047671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56488c9d6ea0 con 0x56488cabe5d0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f818da00700 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f817c009990 tx=0x7f817c01a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b2d8251e503350 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f817b400700 1 -- 172.21.15.23:0/1353047671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c032070 con 0x56488cabe5d0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.056+0000 7f817b400700 1 -- 172.21.15.23:0/1353047671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f817c005350 con 0x56488cabe5d0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f817b400700 1 -- 172.21.15.23:0/1353047671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c01d040 con 0x56488cabe5d0 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1353047671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 msgr2=0x56488cadbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f817c009990 tx=0x7f817c01a040 comp rx=0 tx=0).stop 2024-03-30T04:04:34.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1353047671 shutdown_connections 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cab3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cadbb80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/1353047671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cad0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1353047671 >> 172.21.15.23:0/1353047671 conn(0x56488ca1e6a0 msgr2=0x56488cab35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1353047671 shutdown_connections 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.057+0000 7f81a3f68b80 1 -- 172.21.15.23:0/1353047671 wait complete. 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 -- start start 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 Processor -- start 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cacefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cb26700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cb26c40 con 0x56488cabeef0 2024-03-30T04:04:34.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cb26db0 con 0x56488cac9df0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.058+0000 7f81a3f68b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56488cb26f20 con 0x56488cabe5d0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f81a3f68b80 10 client.0 _ll_register_callbacks cb 0x56488c9b6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818e400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cb26700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818d000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cacefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50676/0 (socket says 172.21.15.23:50676) 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 -- 172.21.15.23:0/3940935505 learned_addr learned my addr 172.21.15.23:0/3940935505 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818e400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cb26700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58866/0 (socket says 172.21.15.23:58866) 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 -- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 msgr2=0x56488cb26700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cb26700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 -- 172.21.15.23:0/3940935505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 msgr2=0x56488cacefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cacefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 -- 172.21.15.23:0/3940935505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56488c8923b0 con 0x56488cabe5d0 2024-03-30T04:04:34.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f818da00700 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f817c005f50 tx=0x7f817c005670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fabfaec091fab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:34.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.059+0000 7f816f400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f817c01d050 con 0x56488cabe5d0 2024-03-30T04:04:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f817c00bcc0 con 0x56488cabe5d0 2024-03-30T04:04:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56488c85fcf0 con 0x56488cabe5d0 2024-03-30T04:04:34.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f817c032040 con 0x56488cabe5d0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f81a3f68b80 10 client.26371 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56488c896cd0 con 0x56488cabe5d0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56488c85fcf0 con 0x56488cabe5d0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.060+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f817c045070 con 0x56488cabe5d0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 10 client.26371 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 10 client.26371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 20 client.26371 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cb2e240 0x56488cb30660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 4 client.26371 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56488c894ce0 con 0x56488cb2e240 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f818d000700 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cb2e240 0x56488cb30660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f818d000700 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cb2e240 0x56488cb30660 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b157f7db4745bb0d server_cookie=88e2538b622ee0c7 in_seq=0 out_seq=0 2024-03-30T04:04:34.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.061+0000 7f816f400700 10 client.26371 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f816f400700 1 -- 172.21.15.23:0/3940935505 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56488c894ce0 con 0x56488cb2e240 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f816f400700 10 client.26371 handle_command_reply: tid=0 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 client.26371 shutdown 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 msgr2=0x56488cacea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f817c005f50 tx=0x7f817c005670 comp rx=0 tx=0).stop 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 shutdown_connections 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56488cac9df0 0x56488cb26700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56488cb2e240 0x56488cb30660 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56488cabe5d0 0x56488cacea80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 --2- 172.21.15.23:0/3940935505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56488cabeef0 0x56488cacefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f816f400700 0 client.26371 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.062+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 >> 172.21.15.23:0/3940935505 conn(0x56488ca1e6a0 msgr2=0x56488cac1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 shutdown_connections 2024-03-30T04:04:34.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f81a3f68b80 1 -- 172.21.15.23:0/3940935505 wait complete. 2024-03-30T04:04:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f81a3f68b80 20 client.26371 trim_cache size 0 max 16384 2024-03-30T04:04:34.066 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:04:34.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 msgr2=0x7f81700a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 0x7f81700a3750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8180009550 tx=0x7f818000b040 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 msgr2=0x7f81881999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.063+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f817400e970 tx=0x7f817400ec80 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 shutdown_connections 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f818819dd50 0x7f818819e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8188100f50 0x7f8188199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f81700a1290 0x7f81700a3750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 --2- 172.21.15.23:0/3749416690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8188106070 0x7f81881999b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 >> 172.21.15.23:0/3749416690 conn(0x7f81880fc7f0 msgr2=0x7f81880fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 shutdown_connections 2024-03-30T04:04:34.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:34.064+0000 7f8190400700 1 -- 172.21.15.23:0/3749416690 wait complete. 2024-03-30T04:04:34.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 152393 11.81 MB/sec warmup 71 sec latency 14.203 ms 2024-03-30T04:04:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:34 smithi193 ceph-mon[79581]: pgmap v82: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:34 smithi023 ceph-mon[38254]: pgmap v82: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:34.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:34 smithi154 ceph-mon[79161]: pgmap v82: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:35.083 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:04:35.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:04:35.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 154188 11.78 MB/sec warmup 72 sec latency 8.490 ms 2024-03-30T04:04:35.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 -- 172.21.15.23:0/1062809294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 msgr2=0x7efdb410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1062809294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb410db50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efdac00b0d0 tx=0x7efdac00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 -- 172.21.15.23:0/1062809294 shutdown_connections 2024-03-30T04:04:35.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1062809294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb410db50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1062809294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb41060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1062809294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb40682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 -- 172.21.15.23:0/1062809294 >> 172.21.15.23:0/1062809294 conn(0x7efdb4076d40 msgr2=0x7efdb4077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 -- 172.21.15.23:0/1062809294 shutdown_connections 2024-03-30T04:04:35.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.325+0000 7efdb9c00700 1 -- 172.21.15.23:0/1062809294 wait complete. 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.326+0000 7efdb9c00700 1 Processor -- start 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.326+0000 7efdb9c00700 1 -- start start 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb4199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419e480 con 0x7efdb4067e30 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419e5f0 con 0x7efdb40687f0 2024-03-30T04:04:35.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb9c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb419e7b0 con 0x7efdb4105ce0 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb4199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58878/0 (socket says 172.21.15.23:58878) 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb4199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54992/0 (socket says 172.21.15.23:54992) 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 -- 172.21.15.23:0/1627641916 learned_addr learned my addr 172.21.15.23:0/1627641916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb3400700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 -- 172.21.15.23:0/1627641916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 msgr2=0x7efdb419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 -- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 msgr2=0x7efdb4199560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb4199560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb2000700 1 -- 172.21.15.23:0/1627641916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac009d00 con 0x7efdb40687f0 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.327+0000 7efdb3400700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:35.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdb2000700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9a0 tx=0x7efda000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdab400700 1 -- 172.21.15.23:0/1627641916 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0009790 con 0x7efdb40687f0 2024-03-30T04:04:35.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb419ea90 con 0x7efdb40687f0 2024-03-30T04:04:35.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdab400700 1 -- 172.21.15.23:0/1627641916 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efda0010430 con 0x7efdb40687f0 2024-03-30T04:04:35.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdab400700 1 -- 172.21.15.23:0/1627641916 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda0018680 con 0x7efdb40687f0 2024-03-30T04:04:35.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.328+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb41a62d0 con 0x7efdb40687f0 2024-03-30T04:04:35.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.331+0000 7efdab400700 1 -- 172.21.15.23:0/1627641916 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7efda00105a0 con 0x7efdb40687f0 2024-03-30T04:04:35.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 Processor -- start 2024-03-30T04:04:35.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 -- start start 2024-03-30T04:04:35.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdab400700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 0x7efd980a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x556022122260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602206da10 con 0x556022114f70 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602206b3d0 con 0x556022114650 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.333+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556022066cc0 con 0x55602211fe70 2024-03-30T04:04:35.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdab400700 1 -- 172.21.15.23:0/1627641916 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efda0014070 con 0x7efdb40687f0 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58892/0 (socket says 172.21.15.23:58892) 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 -- 172.21.15.23:0/2548995481 learned_addr learned my addr 172.21.15.23:0/2548995481 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb3400700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x556022122260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2000700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 -- 172.21.15.23:0/2548995481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 msgr2=0x556022122260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x556022122260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 -- 172.21.15.23:0/2548995481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 msgr2=0x55602211f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 -- 172.21.15.23:0/2548995481 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556021ee8430 con 0x556022114650 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb3400700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x556022122260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2000700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb3400700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 0x7efd980a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.334+0000 7efdb2a00700 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efda4009930 tx=0x7efda4009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5e2fc34faaf7234 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.335+0000 7efd9fe00700 1 -- 172.21.15.23:0/2548995481 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda401c070 con 0x556022114650 2024-03-30T04:04:35.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.335+0000 7efd9fe00700 1 -- 172.21.15.23:0/2548995481 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efda4009e80 con 0x556022114650 2024-03-30T04:04:35.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.335+0000 7efd9fe00700 1 -- 172.21.15.23:0/2548995481 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda40038e0 con 0x556022114650 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2548995481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 msgr2=0x556022114a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdb3400700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 0x7efd980a37d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdac009fd0 tx=0x7efdac00b3e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efda4009930 tx=0x7efda4009c40 comp rx=0 tx=0).stop 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2548995481 shutdown_connections 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x556022122260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022114a30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2548995481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2548995481 >> 172.21.15.23:0/2548995481 conn(0x556022074720 msgr2=0x556022110b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2548995481 shutdown_connections 2024-03-30T04:04:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2548995481 wait complete. 2024-03-30T04:04:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.336+0000 7efdcd7e7b80 1 -- start start 2024-03-30T04:04:35.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.337+0000 7efdcd7e7b80 1 Processor -- start 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.337+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602211f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x55602211d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556022002e00 con 0x55602211fe70 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556022002aa0 con 0x556022114650 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556021eea720 con 0x556022114f70 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdcd7e7b80 10 client.0 _ll_register_callbacks cb 0x55602200c4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:35.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50718/0 (socket says 172.21.15.23:50718) 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 -- 172.21.15.23:0/3710924480 learned_addr learned my addr 172.21.15.23:0/3710924480 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb3400700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x55602211d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2a00700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602211f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 -- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 msgr2=0x55602211f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602211f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 -- 172.21.15.23:0/3710924480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 msgr2=0x55602211d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x55602211d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2000700 1 -- 172.21.15.23:0/3710924480 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda4017d80 con 0x556022114f70 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.338+0000 7efdb2a00700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602211f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.339+0000 7efdb2000700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efda0096430 tx=0x7efda009a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddec4a1a1062a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.339+0000 7efd9cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:35.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.339+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda00a23b0 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.339+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efda0099420 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.339+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55602205f780 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efda00995e0 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 10 client.26383 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x556021e94be0 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55602205f780 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efda0095d20 con 0x556022114f70 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 10 client.26383 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 10 client.26383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 20 client.26383 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55602212f9c0 0x556022131de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 4 client.26383 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.340+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556021eb5d70 con 0x55602212f9c0 2024-03-30T04:04:35.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.341+0000 7efdb2a00700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55602212f9c0 0x556022131de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.341+0000 7efdb2a00700 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55602212f9c0 0x556022131de0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdf1eda9348f3f86 server_cookie=ff60ee486bf8d93 in_seq=0 out_seq=0 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.341+0000 7efd9cc00700 10 client.26383 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efd9cc00700 1 -- 172.21.15.23:0/3710924480 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556021eb5d70 con 0x55602212f9c0 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efd9cc00700 10 client.26383 handle_command_reply: tid=0 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 client.26383 shutdown 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 msgr2=0x55602211cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7efda0096430 tx=0x7efda009a060 comp rx=0 tx=0).stop 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 shutdown_connections 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x55602211d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602211f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55602212f9c0 0x556022131de0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/3710924480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602211cb40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 >> 172.21.15.23:0/3710924480 conn(0x556022074720 msgr2=0x556022116740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.342+0000 7efd9cc00700 0 client.26383 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.343+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 shutdown_connections 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.343+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/3710924480 wait complete. 2024-03-30T04:04:35.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.343+0000 7efdcd7e7b80 20 client.26383 trim_cache size 0 max 16384 2024-03-30T04:04:35.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 Processor -- start 2024-03-30T04:04:35.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 -- start start 2024-03-30T04:04:35.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022126b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602211e130 con 0x556022114f70 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602212c570 con 0x556022114650 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602212d070 con 0x55602211fe70 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdb3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdb2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdb3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50734/0 (socket says 172.21.15.23:50734) 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022126b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.363+0000 7efdb3400700 1 -- 172.21.15.23:0/2233764374 learned_addr learned my addr 172.21.15.23:0/2233764374 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 -- 172.21.15.23:0/2233764374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 msgr2=0x556022126b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022126b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 -- 172.21.15.23:0/2233764374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 msgr2=0x55602211b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 -- 172.21.15.23:0/2233764374 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556022136a10 con 0x55602211fe70 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efdb3400700 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efdac007c40 tx=0x7efdac01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6650e9aefb358ae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efda8c00700 1 -- 172.21.15.23:0/2233764374 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac00e040 con 0x55602211fe70 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efda8c00700 1 -- 172.21.15.23:0/2233764374 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdac003a00 con 0x55602211fe70 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.364+0000 7efda8c00700 1 -- 172.21.15.23:0/2233764374 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdac005610 con 0x55602211fe70 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2233764374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 msgr2=0x55602211b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efdac007c40 tx=0x7efdac01e000 comp rx=0 tx=0).stop 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2233764374 shutdown_connections 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55602211fe70 0x55602211b990 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x556022126b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/2233764374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556022114f70 0x55602211b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2233764374 >> 172.21.15.23:0/2233764374 conn(0x556022074720 msgr2=0x556022109620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2233764374 shutdown_connections 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.365+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/2233764374 wait complete. 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.366+0000 7efdcd7e7b80 1 -- start start 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.366+0000 7efdcd7e7b80 1 Processor -- start 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.366+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602217e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x556022125e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556022126530 con 0x55602211fe70 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560221266a0 con 0x556022114650 2024-03-30T04:04:35.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556022126860 con 0x556022114f70 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdcd7e7b80 10 client.0 _ll_register_callbacks cb 0x55602200c4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58922/0 (socket says 172.21.15.23:58922) 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 -- 172.21.15.23:0/552420277 learned_addr learned my addr 172.21.15.23:0/552420277 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2000700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602217e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb3400700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x556022125e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 -- 172.21.15.23:0/552420277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 msgr2=0x55602217e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602217e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 -- 172.21.15.23:0/552420277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 msgr2=0x556022125e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x556022125e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.367+0000 7efdb2a00700 1 -- 172.21.15.23:0/552420277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb410e550 con 0x556022114650 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efdb2a00700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efda4009340 tx=0x7efda400ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86abfa00eea71cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efd9cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda401c070 con 0x556022114650 2024-03-30T04:04:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556022027400 con 0x556022114650 2024-03-30T04:04:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7efda4005450 con 0x556022114650 2024-03-30T04:04:35.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efda400ea30 con 0x556022114650 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efdcd7e7b80 10 client.26396 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.368+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55602202cf20 con 0x556022114650 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556022027400 con 0x556022114650 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7efda4009960 con 0x556022114650 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 10 client.26396 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 20 client.26396 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556022185e00 0x556022188220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 4 client.26396 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556021eead60 con 0x556022185e00 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.369+0000 7efdb2000700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556022185e00 0x556022188220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.370+0000 7efd9cc00700 10 client.26396 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.370+0000 7efdb2000700 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556022185e00 0x556022188220 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4bdf51f535a6e22 server_cookie=2b56ee9e833cc512 in_seq=0 out_seq=0 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efd9cc00700 1 -- 172.21.15.23:0/552420277 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556021eead60 con 0x556022185e00 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efd9cc00700 10 client.26396 handle_command_reply: tid=0 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 client.26396 shutdown 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 msgr2=0x55602217e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efda4009340 tx=0x7efda400ef50 comp rx=0 tx=0).stop 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 shutdown_connections 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55602211fe70 0x556022125e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556022114650 0x55602217e010 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556022185e00 0x556022188220 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 --2- 172.21.15.23:0/552420277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556022114f70 0x55602217e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 >> 172.21.15.23:0/552420277 conn(0x556022074720 msgr2=0x55602211b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efd9cc00700 0 client.26396 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 shutdown_connections 2024-03-30T04:04:35.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.371+0000 7efdcd7e7b80 1 -- 172.21.15.23:0/552420277 wait complete. 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdcd7e7b80 20 client.26396 trim_cache size 0 max 16384 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "d7d8ec6b-e625-40ff-861c-181b41edae46", 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:04:35.374 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 msgr2=0x7efd980a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 0x7efd980a37d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdac009fd0 tx=0x7efdac00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 msgr2=0x7efdb4199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efda000e9a0 tx=0x7efda000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 shutdown_connections 2024-03-30T04:04:35.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efdb4105ce0 0x7efdb419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdb4067e30 0x7efdb4199560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efd980a1310 0x7efd980a37d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 --2- 172.21.15.23:0/1627641916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40687f0 0x7efdb4199aa0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 >> 172.21.15.23:0/1627641916 conn(0x7efdb4076d40 msgr2=0x7efdb4104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 shutdown_connections 2024-03-30T04:04:35.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.372+0000 7efdb9c00700 1 -- 172.21.15.23:0/1627641916 wait complete. 2024-03-30T04:04:35.389 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:04:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 -- 172.21.15.23:0/844469443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 msgr2=0x7f73ec0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 --2- 172.21.15.23:0/844469443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec0735c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009990 tx=0x7f73dc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 -- 172.21.15.23:0/844469443 shutdown_connections 2024-03-30T04:04:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 --2- 172.21.15.23:0/844469443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec10f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 --2- 172.21.15.23:0/844469443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec0735c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 --2- 172.21.15.23:0/844469443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 -- 172.21.15.23:0/844469443 >> 172.21.15.23:0/844469443 conn(0x7f73ec0fc7c0 msgr2=0x7f73ec0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 -- 172.21.15.23:0/844469443 shutdown_connections 2024-03-30T04:04:35.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.613+0000 7f73f1800700 1 -- 172.21.15.23:0/844469443 wait complete. 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.614+0000 7f73f1800700 1 Processor -- start 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 -- start start 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec19e310 con 0x7f73ec10d530 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec19e480 con 0x7f73ec0731e0 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73f1800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73ec19e640 con 0x7f73ec073b00 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73ea000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73ea000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50768/0 (socket says 172.21.15.23:50768) 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73ea000700 1 -- 172.21.15.23:0/3878935129 learned_addr learned my addr 172.21.15.23:0/3878935129 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.615+0000 7f73eb400700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 -- 172.21.15.23:0/3878935129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 msgr2=0x7f73ec1993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec1993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 -- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 msgr2=0x7f73ec19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 -- 172.21.15.23:0/3878935129 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8009520 con 0x7f73ec073b00 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73eb400700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73ea000700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73d800eb30 tx=0x7f73d800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73e3400700 1 -- 172.21.15.23:0/3878935129 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8009880 con 0x7f73ec073b00 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73dc0095f0 con 0x7f73ec073b00 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73e3400700 1 -- 172.21.15.23:0/3878935129 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73d80044e0 con 0x7f73ec073b00 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73e3400700 1 -- 172.21.15.23:0/3878935129 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d8003e90 con 0x7f73ec073b00 2024-03-30T04:04:35.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.616+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec19eb10 con 0x7f73ec073b00 2024-03-30T04:04:35.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.619+0000 7f73e3400700 1 -- 172.21.15.23:0/3878935129 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f73d8004000 con 0x7f73ec073b00 2024-03-30T04:04:35.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f73e3400700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 0x7f73d00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f73e3400700 1 -- 172.21.15.23:0/3878935129 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f73d8014070 con 0x7f73ec073b00 2024-03-30T04:04:35.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f74052bab80 1 Processor -- start 2024-03-30T04:04:35.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 0x7f73d00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f74052bab80 1 -- start start 2024-03-30T04:04:35.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f74052bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.621+0000 7f74052bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf12d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f74052bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f74052bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf1227990 con 0x55eaf12ce5d0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f74052bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf1225350 con 0x55eaf12ceef0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f74052bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf1220c40 con 0x55eaf12d9df0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50782/0 (socket says 172.21.15.23:50782) 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 -- 172.21.15.23:0/278723945 learned_addr learned my addr 172.21.15.23:0/278723945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eaa00700 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73ea000700 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf12d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 0x7f73d00a37d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73dc006010 tx=0x7f73dc00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 -- 172.21.15.23:0/278723945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 msgr2=0x55eaf12d98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf12d98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 -- 172.21.15.23:0/278723945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 msgr2=0x55eaf12ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.622+0000 7f73eb400700 1 -- 172.21.15.23:0/278723945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eaf10a23b0 con 0x55eaf12d9df0 2024-03-30T04:04:35.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.623+0000 7f73eb400700 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009440 tx=0x7f73e4009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=971562309d6f9a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.623+0000 7f73d7e00700 1 -- 172.21.15.23:0/278723945 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e400e070 con 0x55eaf12d9df0 2024-03-30T04:04:35.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.623+0000 7f73d7e00700 1 -- 172.21.15.23:0/278723945 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e400f040 con 0x55eaf12d9df0 2024-03-30T04:04:35.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.623+0000 7f73d7e00700 1 -- 172.21.15.23:0/278723945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e4015c90 con 0x55eaf12d9df0 2024-03-30T04:04:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- 172.21.15.23:0/278723945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 msgr2=0x55eaf12dc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e4009440 tx=0x7f73e4009920 comp rx=0 tx=0).stop 2024-03-30T04:04:35.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- 172.21.15.23:0/278723945 shutdown_connections 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf12dc1e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 --2- 172.21.15.23:0/278723945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf12d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- 172.21.15.23:0/278723945 >> 172.21.15.23:0/278723945 conn(0x55eaf122e6a0 msgr2=0x55eaf12caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- 172.21.15.23:0/278723945 shutdown_connections 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- 172.21.15.23:0/278723945 wait complete. 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 -- start start 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.624+0000 7f74052bab80 1 Processor -- start 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12d97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf11bcd80 con 0x55eaf12ce5d0 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf11bca20 con 0x55eaf12d9df0 2024-03-30T04:04:35.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf10a46a0 con 0x55eaf12ceef0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f74052bab80 10 client.0 _ll_register_callbacks cb 0x55eaf11c6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73eaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12d97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73ea000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73eb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73eaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12d97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55070/0 (socket says 172.21.15.23:55070) 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73ea000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50786/0 (socket says 172.21.15.23:50786) 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.625+0000 7f73eaa00700 1 -- 172.21.15.23:0/3622206223 learned_addr learned my addr 172.21.15.23:0/3622206223 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 -- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 msgr2=0x55eaf12d70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 -- 172.21.15.23:0/3622206223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 msgr2=0x55eaf12d97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12d97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 -- 172.21.15.23:0/3622206223 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40090f0 con 0x55eaf12ceef0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73eb400700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73ea000700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73d80ca840 tx=0x7f73d8099b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed28ecf47f229cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80b6070 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f73d800cc90 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eaf11e1380 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d809a200 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f74052bab80 10 client.26401 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.626+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55eaf104eb60 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eaf11e1380 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f73d809d050 con 0x55eaf12ceef0 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 10 client.26401 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 10 client.26401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 20 client.26401 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf12e9a60 0x55eaf12ebe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 4 client.26401 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eaf1219700 con 0x55eaf12e9a60 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.627+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf12e9a60 0x55eaf12ebe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf12e9a60 0x55eaf12ebe80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93fc51ade4be09a3 server_cookie=8857b5a94b52553f in_seq=0 out_seq=0 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f73d4c00700 10 client.26401 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f73d4c00700 1 -- 172.21.15.23:0/3622206223 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eaf1219700 con 0x55eaf12e9a60 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f73d4c00700 10 client.26401 handle_command_reply: tid=0 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f74052bab80 1 client.26401 shutdown 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 msgr2=0x55eaf12d6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.628+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73d80ca840 tx=0x7f73d8099b60 comp rx=0 tx=0).stop 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 shutdown_connections 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf12d97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf12e9a60 0x55eaf12ebe80 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 --2- 172.21.15.23:0/3622206223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ceef0 0x55eaf12d6ae0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 >> 172.21.15.23:0/3622206223 conn(0x55eaf122e6a0 msgr2=0x55eaf12d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f73d4c00700 0 client.26401 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 shutdown_connections 2024-03-30T04:04:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 1 -- 172.21.15.23:0/3622206223 wait complete. 2024-03-30T04:04:35.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.629+0000 7f74052bab80 20 client.26401 trim_cache size 0 max 16384 2024-03-30T04:04:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.657+0000 7f74052bab80 1 Processor -- start 2024-03-30T04:04:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.659+0000 7f74052bab80 1 -- start start 2024-03-30T04:04:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ce5d0 0x55eaf12e0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12e70b0 con 0x55eaf12ceef0 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12cc300 con 0x55eaf12d9df0 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.660+0000 7f74052bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12cb7d0 con 0x55eaf12ce5d0 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.661+0000 7f73ea000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.661+0000 7f73ea000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55074/0 (socket says 172.21.15.23:55074) 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.661+0000 7f73ea000700 1 -- 172.21.15.23:0/304785492 learned_addr learned my addr 172.21.15.23:0/304785492 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 -- 172.21.15.23:0/304785492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ce5d0 msgr2=0x55eaf12e0aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ce5d0 0x55eaf12e0aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 -- 172.21.15.23:0/304785492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 msgr2=0x55eaf12d53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 -- 172.21.15.23:0/304785492 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eaf11e6ea0 con 0x55eaf12ceef0 2024-03-30T04:04:35.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.662+0000 7f73ea000700 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73d800eb00 tx=0x7f73d8003930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b465f6d0d2c911a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.664+0000 7f73e0c00700 1 -- 172.21.15.23:0/304785492 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80b6070 con 0x55eaf12ceef0 2024-03-30T04:04:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.664+0000 7f73e0c00700 1 -- 172.21.15.23:0/304785492 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73d800cc90 con 0x55eaf12ceef0 2024-03-30T04:04:35.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.664+0000 7f73e0c00700 1 -- 172.21.15.23:0/304785492 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80a2680 con 0x55eaf12ceef0 2024-03-30T04:04:35.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.666+0000 7f74052bab80 1 -- 172.21.15.23:0/304785492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 msgr2=0x55eaf12d92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.666+0000 7f74052bab80 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73d800eb00 tx=0x7f73d8003930 comp rx=0 tx=0).stop 2024-03-30T04:04:35.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.667+0000 7f74052bab80 1 -- 172.21.15.23:0/304785492 shutdown_connections 2024-03-30T04:04:35.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.667+0000 7f74052bab80 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12d9df0 0x55eaf12d53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.667+0000 7f74052bab80 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12ce5d0 0x55eaf12e0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.667+0000 7f74052bab80 1 --2- 172.21.15.23:0/304785492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ceef0 0x55eaf12d92f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.667+0000 7f74052bab80 1 -- 172.21.15.23:0/304785492 >> 172.21.15.23:0/304785492 conn(0x55eaf122e6a0 msgr2=0x55eaf12d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.668+0000 7f74052bab80 1 -- 172.21.15.23:0/304785492 shutdown_connections 2024-03-30T04:04:35.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.668+0000 7f74052bab80 1 -- 172.21.15.23:0/304785492 wait complete. 2024-03-30T04:04:35.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.670+0000 7f74052bab80 1 -- start start 2024-03-30T04:04:35.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.671+0000 7f74052bab80 1 Processor -- start 2024-03-30T04:04:35.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf1338830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf13392b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12de660 con 0x55eaf12ce5d0 2024-03-30T04:04:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12de7d0 con 0x55eaf12ceef0 2024-03-30T04:04:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eaf12de990 con 0x55eaf12d9df0 2024-03-30T04:04:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.674+0000 7f74052bab80 10 client.0 _ll_register_callbacks cb 0x55eaf11c6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:04:35.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.675+0000 7f73ea000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.675+0000 7f73ea000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58966/0 (socket says 172.21.15.23:58966) 2024-03-30T04:04:35.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.675+0000 7f73ea000700 1 -- 172.21.15.23:0/3326348863 learned_addr learned my addr 172.21.15.23:0/3326348863 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:04:35.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.675+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf1338830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.675+0000 7f73eb400700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf13392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73ea000700 1 -- 172.21.15.23:0/3326348863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 msgr2=0x55eaf13392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73ea000700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf13392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73ea000700 1 -- 172.21.15.23:0/3326348863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 msgr2=0x55eaf1338830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:04:35.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73ea000700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf1338830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73ea000700 1 -- 172.21.15.23:0/3326348863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eaf104eb60 con 0x55eaf12ceef0 2024-03-30T04:04:35.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.677+0000 7f73eb400700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf13392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:04:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.681+0000 7f73ea000700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f73d80cd350 tx=0x7f73d8095ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0582f7fb9cf4f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:04:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.681+0000 7f73d4c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:04:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.682+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80b6070 con 0x55eaf12ceef0 2024-03-30T04:04:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.682+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f73d809e030 con 0x55eaf12ceef0 2024-03-30T04:04:35.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.682+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73e400f040 con 0x55eaf12ceef0 2024-03-30T04:04:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.683+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d809a200 con 0x55eaf12ceef0 2024-03-30T04:04:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.683+0000 7f74052bab80 10 client.26411 fetch_fsmap learned FSMap version 29 2024-03-30T04:04:35.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.683+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55eaf10a6cd0 con 0x55eaf12ceef0 2024-03-30T04:04:35.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.683+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73e400f040 con 0x55eaf12ceef0 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f73d809d070 con 0x55eaf12ceef0 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 10 client.26411 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 10 client.26411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 20 client.26411 populate_metadata read hostname 'smithi023' 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf13467c0 0x55eaf1348bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 4 client.26411 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.684+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55eaf106fcf0 con 0x55eaf13467c0 2024-03-30T04:04:35.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.685+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf13467c0 0x55eaf1348bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:04:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.687+0000 7f73eaa00700 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf13467c0 0x55eaf1348bb0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7899473b4f58d5 server_cookie=f5bdd2f5abd6ab51 in_seq=0 out_seq=0 2024-03-30T04:04:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.687+0000 7f73d4c00700 10 client.26411 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.688+0000 7f73d4c00700 1 -- 172.21.15.23:0/3326348863 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55eaf106fcf0 con 0x55eaf13467c0 2024-03-30T04:04:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.688+0000 7f73d4c00700 10 client.26411 handle_command_reply: tid=0 2024-03-30T04:04:35.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.688+0000 7f74052bab80 1 client.26411 shutdown 2024-03-30T04:04:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.688+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 msgr2=0x55eaf1338d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.688+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f73d80cd350 tx=0x7f73d8095ea0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 shutdown_connections 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55eaf13467c0 0x55eaf1348bb0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eaf12d9df0 0x55eaf13392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eaf12ce5d0 0x55eaf1338830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 --2- 172.21.15.23:0/3326348863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eaf12ceef0 0x55eaf1338d70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 >> 172.21.15.23:0/3326348863 conn(0x55eaf122e6a0 msgr2=0x55eaf12d5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f73d4c00700 0 client.26411 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 shutdown_connections 2024-03-30T04:04:35.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.689+0000 7f74052bab80 1 -- 172.21.15.23:0/3326348863 wait complete. 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.690+0000 7f74052bab80 20 client.26411 trim_cache size 0 max 16384 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "d7d8ec6b-e625-40ff-861c-181b41edae46": { 2024-03-30T04:04:35.689 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:04:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "d7d8ec6b-e625-40ff-861c-181b41edae46", 2024-03-30T04:04:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:04:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:04:35.690 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:04:35.690 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:04:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.691+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 msgr2=0x7f73d00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.691+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 0x7f73d00a37d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f73dc006010 tx=0x7f73dc00b260 comp rx=0 tx=0).stop 2024-03-30T04:04:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.691+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 msgr2=0x7f73ec199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:04:35.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.691+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f73d800eb30 tx=0x7f73d800ee40 comp rx=0 tx=0).stop 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 shutdown_connections 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73ec10d530 0x7f73ec19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73d00a1310 0x7f73d00a37d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73ec0731e0 0x7f73ec1993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 --2- 172.21.15.23:0/3878935129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f73ec073b00 0x7f73ec199930 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.692+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 >> 172.21.15.23:0/3878935129 conn(0x7f73ec0fc7c0 msgr2=0x7f73ec0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.693+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 shutdown_connections 2024-03-30T04:04:35.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:04:35.693+0000 7f73f1800700 1 -- 172.21.15.23:0/3878935129 wait complete. 2024-03-30T04:04:35.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:35 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:04:35.725 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:35 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:35.731 INFO:tasks.cephfs.filesystem:scrub status for tag:d7d8ec6b-e625-40ff-861c-181b41edae46 - {'path': '/', 'tag': 'd7d8ec6b-e625-40ff-861c-181b41edae46', 'options': 'recursive,force'} 2024-03-30T04:04:35.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:35 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:04:35.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:35 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:35.992 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:35 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:04:35.992 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:35 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:36.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 156408 11.81 MB/sec warmup 73 sec latency 85.989 ms 2024-03-30T04:04:37.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 158572 11.76 MB/sec warmup 74 sec latency 24.665 ms 2024-03-30T04:04:37.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:37 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:04:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:37 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:37 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:04:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:37 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:37 smithi023 ceph-mon[38254]: pgmap v83: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:37 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:04:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:37 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:37 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:04:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:37 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:37 smithi154 ceph-mon[79161]: pgmap v83: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:37 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:04:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:37 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:37 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:04:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:37 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:04:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:37 smithi193 ceph-mon[79581]: pgmap v83: 129 pgs: 129 active+clean; 88 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 210 op/s 2024-03-30T04:04:38.190 INFO:tasks.workunit.client.0.smithi023.stdout: 1 160553 11.76 MB/sec warmup 75 sec latency 9.598 ms 2024-03-30T04:04:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:38 smithi154 ceph-mon[79161]: pgmap v84: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:38 smithi193 ceph-mon[79581]: pgmap v84: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:38 smithi023 ceph-mon[38254]: pgmap v84: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:39.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 163428 11.85 MB/sec warmup 76 sec latency 15.596 ms 2024-03-30T04:04:40.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 165618 11.79 MB/sec warmup 77 sec latency 10.614 ms 2024-03-30T04:04:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:40 smithi154 ceph-mon[79161]: pgmap v85: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 204 op/s 2024-03-30T04:04:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:40 smithi193 ceph-mon[79581]: pgmap v85: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 204 op/s 2024-03-30T04:04:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:40 smithi023 ceph-mon[38254]: pgmap v85: 129 pgs: 129 active+clean; 93 MiB data, 817 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 204 op/s 2024-03-30T04:04:41.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 167397 11.79 MB/sec warmup 78 sec latency 52.725 ms 2024-03-30T04:04:42.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 169731 11.80 MB/sec warmup 79 sec latency 16.713 ms 2024-03-30T04:04:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:42 smithi154 ceph-mon[79161]: pgmap v86: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 308 op/s 2024-03-30T04:04:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:42 smithi193 ceph-mon[79581]: pgmap v86: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 308 op/s 2024-03-30T04:04:42.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:42 smithi023 ceph-mon[38254]: pgmap v86: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 308 op/s 2024-03-30T04:04:43.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 171553 11.76 MB/sec warmup 80 sec latency 11.963 ms 2024-03-30T04:04:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:43 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:04:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:43 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:04:43.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:43 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:04:44.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 174195 11.83 MB/sec warmup 81 sec latency 11.085 ms 2024-03-30T04:04:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:44 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:04:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:44 smithi154 ceph-mon[79161]: pgmap v87: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:44 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:04:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:44 smithi193 ceph-mon[79581]: pgmap v87: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:44.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:44 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:04:44.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:44 smithi023 ceph-mon[38254]: pgmap v87: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:45.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 176721 11.83 MB/sec warmup 82 sec latency 9.440 ms 2024-03-30T04:04:46.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 178178 11.78 MB/sec warmup 83 sec latency 43.980 ms 2024-03-30T04:04:47.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 181028 11.84 MB/sec warmup 84 sec latency 18.907 ms 2024-03-30T04:04:47.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:47 smithi023 ceph-mon[38254]: pgmap v88: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:47 smithi154 ceph-mon[79161]: pgmap v88: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:47 smithi193 ceph-mon[79581]: pgmap v88: 129 pgs: 129 active+clean; 98 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 204 op/s 2024-03-30T04:04:48.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 182926 11.79 MB/sec warmup 85 sec latency 9.238 ms 2024-03-30T04:04:48.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:48 smithi154 ceph-mon[79161]: pgmap v89: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 309 op/s 2024-03-30T04:04:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:48 smithi193 ceph-mon[79581]: pgmap v89: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 309 op/s 2024-03-30T04:04:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:48 smithi023 ceph-mon[38254]: pgmap v89: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 309 op/s 2024-03-30T04:04:49.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 185075 11.81 MB/sec warmup 86 sec latency 10.949 ms 2024-03-30T04:04:50.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 188002 11.87 MB/sec warmup 87 sec latency 7.255 ms 2024-03-30T04:04:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:50 smithi154 ceph-mon[79161]: pgmap v90: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:50 smithi193 ceph-mon[79581]: pgmap v90: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:50 smithi023 ceph-mon[38254]: pgmap v90: 129 pgs: 129 active+clean; 102 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:51.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 189237 11.78 MB/sec warmup 88 sec latency 66.748 ms 2024-03-30T04:04:52.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 191804 11.83 MB/sec warmup 89 sec latency 23.637 ms 2024-03-30T04:04:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:52 smithi154 ceph-mon[79161]: pgmap v91: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:52 smithi193 ceph-mon[79581]: pgmap v91: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:52 smithi023 ceph-mon[38254]: pgmap v91: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 312 op/s 2024-03-30T04:04:53.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 194118 11.80 MB/sec warmup 90 sec latency 10.623 ms 2024-03-30T04:04:54.191 INFO:tasks.workunit.client.0.smithi023.stdout: 1 196020 11.80 MB/sec warmup 91 sec latency 9.894 ms 2024-03-30T04:04:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:54 smithi193 ceph-mon[79581]: pgmap v92: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:54 smithi023 ceph-mon[38254]: pgmap v92: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:54 smithi154 ceph-mon[79161]: pgmap v92: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:55.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 198876 11.86 MB/sec warmup 92 sec latency 9.194 ms 2024-03-30T04:04:56.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 200615 11.80 MB/sec warmup 93 sec latency 44.037 ms 2024-03-30T04:04:57.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 202649 11.82 MB/sec warmup 94 sec latency 45.310 ms 2024-03-30T04:04:57.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:57 smithi154 ceph-mon[79161]: pgmap v93: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:57 smithi193 ceph-mon[79581]: pgmap v93: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:57 smithi023 ceph-mon[38254]: pgmap v93: 129 pgs: 129 active+clean; 107 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 208 op/s 2024-03-30T04:04:58.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 205375 11.83 MB/sec warmup 95 sec latency 16.861 ms 2024-03-30T04:04:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:04:58 smithi193 ceph-mon[79581]: pgmap v94: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-30T04:04:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:04:58 smithi023 ceph-mon[38254]: pgmap v94: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-30T04:04:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:04:58 smithi154 ceph-mon[79161]: pgmap v94: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.3 MiB/s wr, 308 op/s 2024-03-30T04:04:59.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 207222 11.82 MB/sec warmup 96 sec latency 10.269 ms 2024-03-30T04:05:00.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 209678 11.85 MB/sec warmup 97 sec latency 61.335 ms 2024-03-30T04:05:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:00 smithi193 ceph-mon[79581]: pgmap v95: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 203 op/s 2024-03-30T04:05:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:00 smithi023 ceph-mon[38254]: pgmap v95: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 203 op/s 2024-03-30T04:05:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:00 smithi154 ceph-mon[79161]: pgmap v95: 129 pgs: 129 active+clean; 114 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 203 op/s 2024-03-30T04:05:01.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 210310 11.74 MB/sec warmup 98 sec latency 60.398 ms 2024-03-30T04:05:02.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 212633 11.75 MB/sec warmup 99 sec latency 120.662 ms 2024-03-30T04:05:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:02 smithi193 ceph-mon[79581]: pgmap v96: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-30T04:05:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:02 smithi023 ceph-mon[38254]: pgmap v96: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-30T04:05:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:02 smithi154 ceph-mon[79161]: pgmap v96: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 322 op/s 2024-03-30T04:05:03.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 214205 11.73 MB/sec warmup 100 sec latency 20.241 ms 2024-03-30T04:05:04.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 216802 11.76 MB/sec warmup 101 sec latency 18.202 ms 2024-03-30T04:05:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:04 smithi193 ceph-mon[79581]: pgmap v97: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:04 smithi023 ceph-mon[38254]: pgmap v97: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:04 smithi154 ceph-mon[79161]: pgmap v97: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:05.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 219189 11.75 MB/sec warmup 102 sec latency 10.458 ms 2024-03-30T04:05:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:05 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:05:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:05 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:05:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:05 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:05:05.733 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:05:05.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 -- 172.21.15.23:0/2395653508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690100f60 msgr2=0x7fe690101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 --2- 172.21.15.23:0/2395653508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690100f60 0x7fe690101340 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe684009990 tx=0x7fe684009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 -- 172.21.15.23:0/2395653508 shutdown_connections 2024-03-30T04:05:05.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 --2- 172.21.15.23:0/2395653508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690106080 0x7fe69010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 --2- 172.21.15.23:0/2395653508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690100f60 0x7fe690101340 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 --2- 172.21.15.23:0/2395653508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690101910 0x7fe690105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.948+0000 7fe698800700 1 -- 172.21.15.23:0/2395653508 >> 172.21.15.23:0/2395653508 conn(0x7fe6900fc7c0 msgr2=0x7fe6900febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.949+0000 7fe698800700 1 -- 172.21.15.23:0/2395653508 shutdown_connections 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.949+0000 7fe698800700 1 -- 172.21.15.23:0/2395653508 wait complete. 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.950+0000 7fe698800700 1 Processor -- start 2024-03-30T04:05:05.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.950+0000 7fe698800700 1 -- start start 2024-03-30T04:05:05.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 0x7fe6900730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 0x7fe6900735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 0x7fe6901a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901a59c0 con 0x7fe690100f60 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901a5b30 con 0x7fe690073b30 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe698800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6901a5cf0 con 0x7fe690106080 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 0x7fe6900730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe696800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 0x7fe6901a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe695400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 0x7fe6900735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 0x7fe6900730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37584/0 (socket says 172.21.15.23:37584) 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe696800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 0x7fe6901a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52532/0 (socket says 172.21.15.23:52532) 2024-03-30T04:05:05.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.951+0000 7fe695e00700 1 -- 172.21.15.23:0/1102057785 learned_addr learned my addr 172.21.15.23:0/1102057785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 -- 172.21.15.23:0/1102057785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 msgr2=0x7fe6901a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 0x7fe6901a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 -- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 msgr2=0x7fe6900730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 0x7fe6900730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 -- 172.21.15.23:0/1102057785 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840095f0 con 0x7fe690106080 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe695400700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 0x7fe6900735f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e9c0 tx=0x7fe68000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe68aa00700 1 -- 172.21.15.23:0/1102057785 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68000c9c0 con 0x7fe690106080 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe68aa00700 1 -- 172.21.15.23:0/1102057785 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe680010430 con 0x7fe690106080 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6901a5fd0 con 0x7fe690106080 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe68aa00700 1 -- 172.21.15.23:0/1102057785 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680018680 con 0x7fe690106080 2024-03-30T04:05:05.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.952+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901a6450 con 0x7fe690106080 2024-03-30T04:05:05.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.956+0000 7fe68aa00700 1 -- 172.21.15.23:0/1102057785 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe68000cb20 con 0x7fe690106080 2024-03-30T04:05:05.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.957+0000 7fe68aa00700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 0x7fe6780a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.957+0000 7fe68aa00700 1 -- 172.21.15.23:0/1102057785 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe680014070 con 0x7fe690106080 2024-03-30T04:05:05.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.957+0000 7fe695e00700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 0x7fe6780a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 Processor -- start 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 -- start start 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe695e00700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 0x7fe6780a3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe684006010 tx=0x7fe68400b280 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccd19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccdf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecc2a990 con 0x561ceccd15d0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecc28350 con 0x561ceccd1ef0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.958+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecc23c40 con 0x561ceccdcdf0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccd19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52546/0 (socket says 172.21.15.23:52546) 2024-03-30T04:05:05.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 -- 172.21.15.23:0/3477654825 learned_addr learned my addr 172.21.15.23:0/3477654825 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe696800700 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccdf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 -- 172.21.15.23:0/3477654825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 msgr2=0x561ceccdf1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccdf1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 -- 172.21.15.23:0/3477654825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 msgr2=0x561ceccd19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccd19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 -- 172.21.15.23:0/3477654825 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cecaa53b0 con 0x561ceccd1ef0 2024-03-30T04:05:05.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.959+0000 7fe695400700 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a1e20 tx=0x7fe680099db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae6807bda614726 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.960+0000 7fe67f000700 1 -- 172.21.15.23:0/3477654825 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe680099ed0 con 0x561ceccd1ef0 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.960+0000 7fe67f000700 1 -- 172.21.15.23:0/3477654825 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe68009dd00 con 0x561ceccd1ef0 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.960+0000 7fe67f000700 1 -- 172.21.15.23:0/3477654825 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68009a130 con 0x561ceccd1ef0 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.960+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3477654825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 msgr2=0x561ceccdc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.960+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6800a1e20 tx=0x7fe680099db0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3477654825 shutdown_connections 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccdf1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccd19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3477654825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc8b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3477654825 >> 172.21.15.23:0/3477654825 conn(0x561cecc316a0 msgr2=0x561cecccda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3477654825 shutdown_connections 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3477654825 wait complete. 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 -- start start 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.961+0000 7fe6ac3cdb80 1 Processor -- start 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccdc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecbbfd80 con 0x561ceccd15d0 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecbbfa20 con 0x561ceccdcdf0 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecaa76a0 con 0x561ceccd1ef0 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe6ac3cdb80 10 client.0 _ll_register_callbacks cb 0x561cecbc9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccdc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe696800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe695400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe696800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52558/0 (socket says 172.21.15.23:52558) 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccdc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37602/0 (socket says 172.21.15.23:37602) 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe695400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35420/0 (socket says 172.21.15.23:35420) 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.962+0000 7fe696800700 1 -- 172.21.15.23:0/1084083258 learned_addr learned my addr 172.21.15.23:0/1084083258 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 -- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 msgr2=0x561ceccda2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 -- 172.21.15.23:0/1084083258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 msgr2=0x561ceccdc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccdc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 -- 172.21.15.23:0/1084083258 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6901a6450 con 0x561ceccd1ef0 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe696800700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe695400700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe68000c5b0 tx=0x7fe680009790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d18bde6928a538 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe673400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:05.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6800b82c0 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe680099ed0 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561cecbe4380 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6800105a0 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe6ac3cdb80 10 client.26416 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561ceca51b60 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.963+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561cecbe4380 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fe6800a2010 con 0x561ceccd1ef0 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 10 client.26416 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 10 client.26416 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 20 client.26416 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561ceccec5a0 0x561ceccee990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 4 client.26416 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:05.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561cecc1c700 con 0x561ceccec5a0 2024-03-30T04:05:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.964+0000 7fe695e00700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561ceccec5a0 0x561ceccee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe695e00700 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561ceccec5a0 0x561ceccee990 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48932230c508048f server_cookie=732050a5509dbead in_seq=0 out_seq=0 2024-03-30T04:05:05.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe673400700 10 client.26416 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe673400700 1 -- 172.21.15.23:0/1084083258 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561cecc1c700 con 0x561ceccec5a0 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe673400700 10 client.26416 handle_command_reply: tid=0 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe6ac3cdb80 1 client.26416 shutdown 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 msgr2=0x561ceccd9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.965+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe68000c5b0 tx=0x7fe680009790 comp rx=0 tx=0).stop 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 shutdown_connections 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561ceccda2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccdc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561ceccec5a0 0x561ceccee990 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/1084083258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561ceccd9b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 >> 172.21.15.23:0/1084083258 conn(0x561cecc316a0 msgr2=0x561ceccd36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe673400700 0 client.26416 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 shutdown_connections 2024-03-30T04:05:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/1084083258 wait complete. 2024-03-30T04:05:05.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.966+0000 7fe6ac3cdb80 20 client.26416 trim_cache size 0 max 16384 2024-03-30T04:05:05.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.984+0000 7fe6ac3cdb80 1 Processor -- start 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 -- start start 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccd6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ceccdb180 con 0x561ceccd15d0 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecccf160 con 0x561ceccd1ef0 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecccdfa0 con 0x561ceccdcdf0 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37610/0 (socket says 172.21.15.23:37610) 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695e00700 1 -- 172.21.15.23:0/4157787531 learned_addr learned my addr 172.21.15.23:0/4157787531 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:05.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe696800700 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccd6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695e00700 1 -- 172.21.15.23:0/4157787531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 msgr2=0x561ceccd6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.985+0000 7fe695e00700 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccd6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe695e00700 1 -- 172.21.15.23:0/4157787531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 msgr2=0x561ceccdc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe695e00700 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe695e00700 1 -- 172.21.15.23:0/4157787531 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cecbe9ea0 con 0x561ceccd15d0 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe695400700 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe695e00700 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe68400b6f0 tx=0x7fe684023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a124cadbe873a86e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe67fa00700 1 -- 172.21.15.23:0/4157787531 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68400be30 con 0x561ceccd15d0 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe67fa00700 1 -- 172.21.15.23:0/4157787531 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe68401e040 con 0x561ceccd15d0 2024-03-30T04:05:05.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.986+0000 7fe67fa00700 1 -- 172.21.15.23:0/4157787531 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684004cb0 con 0x561ceccd15d0 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/4157787531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 msgr2=0x561ceccc6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe68400b6f0 tx=0x7fe684023000 comp rx=0 tx=0).stop 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/4157787531 shutdown_connections 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccdcdf0 0x561ceccd6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561ceccc6f70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/4157787531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccd1ef0 0x561ceccdc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/4157787531 >> 172.21.15.23:0/4157787531 conn(0x561cecc316a0 msgr2=0x561ceccc65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/4157787531 shutdown_connections 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/4157787531 wait complete. 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.987+0000 7fe6ac3cdb80 1 -- start start 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 Processor -- start 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561cecce3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561cecce2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecce2c10 con 0x561ceccd15d0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecce2d80 con 0x561ceccdcdf0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cecd40340 con 0x561ceccd1ef0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe6ac3cdb80 10 client.0 _ll_register_callbacks cb 0x561cecbc9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe695400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe695400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35436/0 (socket says 172.21.15.23:35436) 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe695400700 1 -- 172.21.15.23:0/3535027238 learned_addr learned my addr 172.21.15.23:0/3535027238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe695e00700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561cecce3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.988+0000 7fe696800700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561cecce2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 -- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 msgr2=0x561cecce2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561cecce2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 -- 172.21.15.23:0/3535027238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 msgr2=0x561cecce3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561cecce3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 -- 172.21.15.23:0/3535027238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ceca51b60 con 0x561ceccd1ef0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe695400700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe680099bd0 tx=0x7fe680099db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=826943da9028bdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe673400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6800b82c0 con 0x561ceccd1ef0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe680099ed0 con 0x561ceccd1ef0 2024-03-30T04:05:05.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe68009dd00 con 0x561ceccd1ef0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe680095ed0 con 0x561ceccd1ef0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe6ac3cdb80 10 client.26422 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.989+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fe69010e470 con 0x561ceccd1ef0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe68009dd00 con 0x561ceccd1ef0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fe6800a1ec0 con 0x561ceccd1ef0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 10 client.26422 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 20 client.26422 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561cecd416b0 0x561cecd43ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 4 client.26422 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561ceca72cf0 con 0x561cecd416b0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.990+0000 7fe695e00700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561cecd416b0 0x561cecd43ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:05.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe695e00700 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561cecd416b0 0x561cecd43ad0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0067f1e74f0c030 server_cookie=cbaaae8a36aa734a in_seq=0 out_seq=0 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe673400700 10 client.26422 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe673400700 1 -- 172.21.15.23:0/3535027238 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561ceca72cf0 con 0x561cecd416b0 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe673400700 10 client.26422 handle_command_reply: tid=0 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 client.26422 shutdown 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 msgr2=0x561cecce1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe680099bd0 tx=0x7fe680099db0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 shutdown_connections 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561ceccdcdf0 0x561cecce2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561ceccd15d0 0x561cecce3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561cecd416b0 0x561cecd43ad0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 --2- 172.21.15.23:0/3535027238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561ceccd1ef0 0x561cecce1f60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 >> 172.21.15.23:0/3535027238 conn(0x561cecc316a0 msgr2=0x561ceccc6f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe673400700 0 client.26422 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:05.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 shutdown_connections 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 1 -- 172.21.15.23:0/3535027238 wait complete. 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.991+0000 7fe6ac3cdb80 20 client.26422 trim_cache size 0 max 16384 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:05:05.995 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 msgr2=0x7fe6780a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 0x7fe6780a3750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe684006010 tx=0x7fe68400b280 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 msgr2=0x7fe6900735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 0x7fe6900735f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe68000e9c0 tx=0x7fe68000ecd0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 shutdown_connections 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a1290 0x7fe6780a3750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe690073b30 0x7fe6901a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe690100f60 0x7fe6900730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 --2- 172.21.15.23:0/1102057785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe690106080 0x7fe6900735f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:05.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 >> 172.21.15.23:0/1102057785 conn(0x7fe6900fc7c0 msgr2=0x7fe6900fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:05.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 shutdown_connections 2024-03-30T04:05:05.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:05.992+0000 7fe698800700 1 -- 172.21.15.23:0/1102057785 wait complete. 2024-03-30T04:05:06.009 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:05:06.009 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:05:06.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 220788 11.72 MB/sec warmup 103 sec latency 10.765 ms 2024-03-30T04:05:06.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 -- 172.21.15.23:0/3053426176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 msgr2=0x7f6674103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 --2- 172.21.15.23:0/3053426176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674103470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6668009990 tx=0x7f6668009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 -- 172.21.15.23:0/3053426176 shutdown_connections 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 --2- 172.21.15.23:0/3053426176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 0x7f667410fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 --2- 172.21.15.23:0/3053426176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674103470 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 --2- 172.21.15.23:0/3053426176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6674103a40 0x7f6674107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.229+0000 7f667be00700 1 -- 172.21.15.23:0/3053426176 >> 172.21.15.23:0/3053426176 conn(0x7f66740fe8b0 msgr2=0x7f6674100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.230+0000 7f667be00700 1 -- 172.21.15.23:0/3053426176 shutdown_connections 2024-03-30T04:05:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.230+0000 7f667be00700 1 -- 172.21.15.23:0/3053426176 wait complete. 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 Processor -- start 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 -- start start 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 0x7f6674199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 0x7f667419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667419e750 con 0x7f66741081b0 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667419e8c0 con 0x7f667419dcd0 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f667be00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667419ea80 con 0x7f6674103090 2024-03-30T04:05:06.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f6679400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f6679e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 0x7f667419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f6679400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35456/0 (socket says 172.21.15.23:35456) 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.231+0000 7f6679400700 1 -- 172.21.15.23:0/2584646236 learned_addr learned my addr 172.21.15.23:0/2584646236 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6678a00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 0x7f6674199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 -- 172.21.15.23:0/2584646236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 msgr2=0x7f667419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 0x7f667419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 -- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 msgr2=0x7f6674199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 0x7f6674199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 -- 172.21.15.23:0/2584646236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66680095f0 con 0x7f6674103090 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679e00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 0x7f667419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6679400700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6668005910 tx=0x7f6668004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666801d070 con 0x7f6674103090 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6668022470 con 0x7f6674103090 2024-03-30T04:05:06.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.232+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6668017650 con 0x7f6674103090 2024-03-30T04:05:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.233+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66741a5f80 con 0x7f6674103090 2024-03-30T04:05:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.233+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741a6340 con 0x7f6674103090 2024-03-30T04:05:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.234+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667410b420 con 0x7f6674103090 2024-03-30T04:05:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.235+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f66680225e0 con 0x7f6674103090 2024-03-30T04:05:06.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.237+0000 7f6672000700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 0x7f665c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.237+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f66680ce310 con 0x7f6674103090 2024-03-30T04:05:06.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.237+0000 7f6678a00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 0x7f665c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.238+0000 7f6678a00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 0x7f665c0a37d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f667419a940 tx=0x7f6660009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.241+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6668097550 con 0x7f6674103090 2024-03-30T04:05:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:06 smithi023 ceph-mon[38254]: pgmap v98: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:06.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.583+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6674104560 con 0x7f6674103090 2024-03-30T04:05:06.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.584+0000 7f6672000700 1 -- 172.21.15.23:0/2584646236 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+10925 (secure 0 0 0) 0x7f666809ab70 con 0x7f6674103090 2024-03-30T04:05:06.582 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:05:06.582 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:03:23.374313+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 msgr2=0x7f665c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 0x7f665c0a37d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f667419a940 tx=0x7f6660009260 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 msgr2=0x7f6674199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6668005910 tx=0x7f6668004730 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 shutdown_connections 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f665c0a1310 0x7f665c0a37d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667419dcd0 0x7f667419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6674103090 0x7f6674199440 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 --2- 172.21.15.23:0/2584646236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66741081b0 0x7f6674199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 >> 172.21.15.23:0/2584646236 conn(0x7f66740fe8b0 msgr2=0x7f66740ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.586+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 shutdown_connections 2024-03-30T04:05:06.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.587+0000 7f667be00700 1 -- 172.21.15.23:0/2584646236 wait complete. 2024-03-30T04:05:06.586 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-30T04:05:06.604 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:06 smithi154 ceph-mon[79161]: pgmap v98: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:06.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:05:06.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:06 smithi193 ceph-mon[79581]: pgmap v98: 129 pgs: 129 active+clean; 126 MiB data, 1008 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 218 op/s 2024-03-30T04:05:06.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 -- 172.21.15.23:0/1239539948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c100f60 msgr2=0x7fcd4c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 --2- 172.21.15.23:0/1239539948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c100f60 0x7fcd4c101340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009990 tx=0x7fcd3c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 -- 172.21.15.23:0/1239539948 shutdown_connections 2024-03-30T04:05:06.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 --2- 172.21.15.23:0/1239539948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c106080 0x7fcd4c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 --2- 172.21.15.23:0/1239539948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c100f60 0x7fcd4c101340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 --2- 172.21.15.23:0/1239539948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c101910 0x7fcd4c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.823+0000 7fcd51800700 1 -- 172.21.15.23:0/1239539948 >> 172.21.15.23:0/1239539948 conn(0x7fcd4c0fc7c0 msgr2=0x7fcd4c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.824+0000 7fcd51800700 1 -- 172.21.15.23:0/1239539948 shutdown_connections 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.824+0000 7fcd51800700 1 -- 172.21.15.23:0/1239539948 wait complete. 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 Processor -- start 2024-03-30T04:05:06.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 -- start start 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 0x7fcd4c199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 0x7fcd4c19e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c19e7d0 con 0x7fcd4c100f60 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c19e940 con 0x7fcd4c106080 2024-03-30T04:05:06.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.825+0000 7fcd51800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c19eb00 con 0x7fcd4c19dd50 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 0x7fcd4c199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52614/0 (socket says 172.21.15.23:52614) 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 -- 172.21.15.23:0/533698656 learned_addr learned my addr 172.21.15.23:0/533698656 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4b400700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 0x7fcd4c19e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 -- 172.21.15.23:0/533698656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 msgr2=0x7fcd4c19e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 0x7fcd4c19e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 -- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 msgr2=0x7fcd4c199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 0x7fcd4c199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 -- 172.21.15.23:0/533698656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44009c40 con 0x7fcd4c106080 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4b400700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 0x7fcd4c19e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 0x7fcd4c199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.826+0000 7fcd4a000700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800e970 tx=0x7fcd3800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.827+0000 7fcd43400700 1 -- 172.21.15.23:0/533698656 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38009790 con 0x7fcd4c106080 2024-03-30T04:05:06.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.827+0000 7fcd43400700 1 -- 172.21.15.23:0/533698656 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd3800eec0 con 0x7fcd4c106080 2024-03-30T04:05:06.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.827+0000 7fcd43400700 1 -- 172.21.15.23:0/533698656 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38005590 con 0x7fcd4c106080 2024-03-30T04:05:06.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.827+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3c0095f0 con 0x7fcd4c106080 2024-03-30T04:05:06.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.827+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c1a63b0 con 0x7fcd4c106080 2024-03-30T04:05:06.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.830+0000 7fcd43400700 1 -- 172.21.15.23:0/533698656 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fcd3800cbd0 con 0x7fcd4c106080 2024-03-30T04:05:06.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.831+0000 7fcd653b9b80 1 Processor -- start 2024-03-30T04:05:06.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.831+0000 7fcd653b9b80 1 -- start start 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c719b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8bca990 con 0x5611f8c715d0 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8bc8350 con 0x5611f8c7cdf0 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8bc3c40 con 0x5611f8c71ef0 2024-03-30T04:05:06.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35484/0 (socket says 172.21.15.23:35484) 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2348056047 learned_addr learned my addr 172.21.15.23:0/2348056047 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4b400700 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c719b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4a000700 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd43400700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 0x7fcd300a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.832+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2348056047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 msgr2=0x5611f8c7f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2348056047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 msgr2=0x5611f8c719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c719b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2348056047 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611f8a453b0 con 0x5611f8c71ef0 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd43400700 1 -- 172.21.15.23:0/533698656 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcd38014070 con 0x7fcd4c106080 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4b400700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 0x7fcd300a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c015000 tx=0x7fcd3c00b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbf09b7ef322c446 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd37e00700 1 -- 172.21.15.23:0/2348056047 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c026070 con 0x5611f8c71ef0 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd37e00700 1 -- 172.21.15.23:0/2348056047 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd3c01e040 con 0x5611f8c71ef0 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.833+0000 7fcd37e00700 1 -- 172.21.15.23:0/2348056047 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c01d070 con 0x5611f8c71ef0 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2348056047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 msgr2=0x5611f8c7c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd4b400700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 0x7fcd300a37d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009550 tx=0x7fcd44009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c015000 tx=0x7fcd3c00b610 comp rx=0 tx=0).stop 2024-03-30T04:05:06.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2348056047 shutdown_connections 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c719b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2348056047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c7c8b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2348056047 >> 172.21.15.23:0/2348056047 conn(0x5611f8bd16a0 msgr2=0x5611f8c6da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2348056047 shutdown_connections 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.834+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2348056047 wait complete. 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.835+0000 7fcd653b9b80 1 -- start start 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.835+0000 7fcd653b9b80 1 Processor -- start 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c79b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8b5fd80 con 0x5611f8c71ef0 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8b5fa20 con 0x5611f8c715d0 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8a476a0 con 0x5611f8c7cdf0 2024-03-30T04:05:06.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd653b9b80 10 client.0 _ll_register_callbacks cb 0x5611f8b69470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52622/0 (socket says 172.21.15.23:52622) 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3513699499 learned_addr learned my addr 172.21.15.23:0/3513699499 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4a000700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c79b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4b400700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3513699499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 msgr2=0x5611f8c7a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3513699499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 msgr2=0x5611f8c79b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c79b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.836+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3513699499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c10e470 con 0x5611f8c715d0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c004c90 tx=0x7fcd3c00bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2ad7469a8b90b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd34c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c01d030 con 0x5611f8c715d0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcd3c0053e0 con 0x5611f8c715d0 2024-03-30T04:05:06.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611f8b84380 con 0x5611f8c715d0 2024-03-30T04:05:06.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd3c026040 con 0x5611f8c715d0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd653b9b80 10 client.26447 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5611f89f1b60 con 0x5611f8c715d0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.837+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5611f8b84380 con 0x5611f8c715d0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fcd3c02a400 con 0x5611f8c715d0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 10 client.26447 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 10 client.26447 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 20 client.26447 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8c8c420 0x5611f8c8e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 4 client.26447 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611f8bbc700 con 0x5611f8c8c420 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd4a000700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8c8c420 0x5611f8c8e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd4a000700 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8c8c420 0x5611f8c8e810 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=603c0ee20cfb9e49 server_cookie=f4e50ffa81679b3f in_seq=0 out_seq=0 2024-03-30T04:05:06.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.838+0000 7fcd34c00700 10 client.26447 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.839+0000 7fcd34c00700 1 -- 172.21.15.23:0/3513699499 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5611f8bbc700 con 0x5611f8c8c420 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.839+0000 7fcd34c00700 10 client.26447 handle_command_reply: tid=0 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.839+0000 7fcd653b9b80 1 client.26447 shutdown 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.839+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 msgr2=0x5611f8c7c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c004c90 tx=0x7fcd3c00bdb0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 shutdown_connections 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8c8c420 0x5611f8c8e810 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8c7a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c7c7d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd34c00700 0 client.26447 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3513699499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c79b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 >> 172.21.15.23:0/3513699499 conn(0x5611f8bd16a0 msgr2=0x5611f8c736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 shutdown_connections 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3513699499 wait complete. 2024-03-30T04:05:06.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.840+0000 7fcd653b9b80 20 client.26447 trim_cache size 0 max 16384 2024-03-30T04:05:06.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.854+0000 7fcd653b9b80 1 Processor -- start 2024-03-30T04:05:06.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.854+0000 7fcd653b9b80 1 -- start start 2024-03-30T04:05:06.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c83a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c665a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c66f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8c6e660 con 0x5611f8c715d0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8c83f50 con 0x5611f8c7cdf0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8c66ae0 con 0x5611f8c71ef0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c83a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd4a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd4b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c66f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd4a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c665a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35506/0 (socket says 172.21.15.23:35506) 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.855+0000 7fcd4a000700 1 -- 172.21.15.23:0/2085545849 learned_addr learned my addr 172.21.15.23:0/2085545849 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2085545849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 msgr2=0x5611f8c665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2085545849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 msgr2=0x5611f8c66f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c66f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 -- 172.21.15.23:0/2085545849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611f8b89ea0 con 0x5611f8c715d0 2024-03-30T04:05:06.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c83a10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c02a790 tx=0x7fcd3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60f6d1d854e6af7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd40c00700 1 -- 172.21.15.23:0/2085545849 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c01d050 con 0x5611f8c715d0 2024-03-30T04:05:06.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd40c00700 1 -- 172.21.15.23:0/2085545849 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd3c004db0 con 0x5611f8c715d0 2024-03-30T04:05:06.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.856+0000 7fcd40c00700 1 -- 172.21.15.23:0/2085545849 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c026040 con 0x5611f8c715d0 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2085545849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 msgr2=0x5611f8c83a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c83a10 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c02a790 tx=0x7fcd3c000bc0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2085545849 shutdown_connections 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c7cdf0 0x5611f8c66f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c715d0 0x5611f8c83a10 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/2085545849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c71ef0 0x5611f8c665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2085545849 >> 172.21.15.23:0/2085545849 conn(0x5611f8bd16a0 msgr2=0x5611f8c930b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2085545849 shutdown_connections 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- 172.21.15.23:0/2085545849 wait complete. 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.857+0000 7fcd653b9b80 1 -- start start 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 Processor -- start 2024-03-30T04:05:06.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c83100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8cdf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8c839e0 con 0x5611f8c71ef0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8cdf6a0 con 0x5611f8c715d0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611f8cdf7e0 con 0x5611f8c7cdf0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.858+0000 7fcd653b9b80 10 client.0 _ll_register_callbacks cb 0x5611f8b69470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8cdf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c83100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52634/0 (socket says 172.21.15.23:52634) 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3596924458 learned_addr learned my addr 172.21.15.23:0/3596924458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8cdf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35520/0 (socket says 172.21.15.23:35520) 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3596924458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 msgr2=0x5611f8cdf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8cdf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3596924458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 msgr2=0x5611f8c83100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c83100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 -- 172.21.15.23:0/3596924458 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611f89f1b60 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd4aa00700 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c001eb0 tx=0x7fcd3c02aaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfa1e8edbf01f08e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd34c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.859+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c01d070 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcd3c000e40 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611f8a12cf0 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd3c026040 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 10 client.26453 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5611f8a49cd0 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5611f8a12cf0 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fcd3c00f990 con 0x5611f8c715d0 2024-03-30T04:05:06.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 10 client.26453 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 10 client.26453 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 20 client.26453 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8ce0b00 0x5611f8ce2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 4 client.26453 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.860+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5611f8a47ce0 con 0x5611f8ce0b00 2024-03-30T04:05:06.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd4a000700 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8ce0b00 0x5611f8ce2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:06.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd34c00700 10 client.26453 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:06.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd4a000700 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8ce0b00 0x5611f8ce2ef0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6fde6bdd277effe0 server_cookie=e5f5c62a3d768428 in_seq=0 out_seq=0 2024-03-30T04:05:06.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd34c00700 1 -- 172.21.15.23:0/3596924458 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5611f8a47ce0 con 0x5611f8ce0b00 2024-03-30T04:05:06.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd34c00700 10 client.26453 handle_command_reply: tid=0 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 client.26453 shutdown 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 msgr2=0x5611f8c82bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c001eb0 tx=0x7fcd3c02aaa0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 shutdown_connections 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5611f8ce0b00 0x5611f8ce2ef0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5611f8c7cdf0 0x5611f8cdf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd34c00700 0 client.26453 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611f8c715d0 0x5611f8c82bc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 --2- 172.21.15.23:0/3596924458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611f8c71ef0 0x5611f8c83100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.861+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 >> 172.21.15.23:0/3596924458 conn(0x5611f8bd16a0 msgr2=0x5611f8c88160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 shutdown_connections 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd653b9b80 1 -- 172.21.15.23:0/3596924458 wait complete. 2024-03-30T04:05:06.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd653b9b80 20 client.26453 trim_cache size 0 max 16384 2024-03-30T04:05:06.865 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:06.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 msgr2=0x7fcd300a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 0x7fcd300a37d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009550 tx=0x7fcd44009040 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 msgr2=0x7fcd4c1999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd3800e970 tx=0x7fcd3800ec80 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.862+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 shutdown_connections 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd4c19dd50 0x7fcd4c19e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd4c100f60 0x7fcd4c199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd300a1310 0x7fcd300a37d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 --2- 172.21.15.23:0/533698656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd4c106080 0x7fcd4c1999b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 >> 172.21.15.23:0/533698656 conn(0x7fcd4c0fc7c0 msgr2=0x7fcd4c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 shutdown_connections 2024-03-30T04:05:06.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:06.863+0000 7fcd51800700 1 -- 172.21.15.23:0/533698656 wait complete. 2024-03-30T04:05:06.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 -- 172.21.15.23:0/731876133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984111d00 msgr2=0x7fd9841140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 --2- 172.21.15.23:0/731876133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984111d00 0x7fd9841140f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00b0a0 tx=0x7fd97c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 -- 172.21.15.23:0/731876133 shutdown_connections 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 --2- 172.21.15.23:0/731876133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984111d00 0x7fd9841140f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 --2- 172.21.15.23:0/731876133 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984073150 0x7fd984073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 --2- 172.21.15.23:0/731876133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073a70 0x7fd9841117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 -- 172.21.15.23:0/731876133 >> 172.21.15.23:0/731876133 conn(0x7fd9840fc7c0 msgr2=0x7fd9840febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 -- 172.21.15.23:0/731876133 shutdown_connections 2024-03-30T04:05:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.098+0000 7fd98a800700 1 -- 172.21.15.23:0/731876133 wait complete. 2024-03-30T04:05:07.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.100+0000 7fd98a800700 1 Processor -- start 2024-03-30T04:05:07.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.100+0000 7fd98a800700 1 -- start start 2024-03-30T04:05:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073150 0x7fd9841a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984111d00 0x7fd98419d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98419dea0 con 0x7fd984073150 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98419e010 con 0x7fd984073a70 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd98a800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98419e1d0 con 0x7fd984111d00 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52646/0 (socket says 172.21.15.23:52646) 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 -- 172.21.15.23:0/3182296089 learned_addr learned my addr 172.21.15.23:0/3182296089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd983e00700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984111d00 0x7fd98419d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd983400700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073150 0x7fd9841a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 -- 172.21.15.23:0/3182296089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984111d00 msgr2=0x7fd98419d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984111d00 0x7fd98419d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 -- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073150 msgr2=0x7fd9841a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073150 0x7fd9841a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.101+0000 7fd982a00700 1 -- 172.21.15.23:0/3182296089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c009d00 con 0x7fd984073a70 2024-03-30T04:05:07.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd982a00700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd97000e9a0 tx=0x7fd97000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd97be00700 1 -- 172.21.15.23:0/3182296089 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970009790 con 0x7fd984073a70 2024-03-30T04:05:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd97be00700 1 -- 172.21.15.23:0/3182296089 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd970010430 con 0x7fd984073a70 2024-03-30T04:05:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98419e4b0 con 0x7fd984073a70 2024-03-30T04:05:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd97be00700 1 -- 172.21.15.23:0/3182296089 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970018680 con 0x7fd984073a70 2024-03-30T04:05:07.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.102+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9841aa6f0 con 0x7fd984073a70 2024-03-30T04:05:07.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.105+0000 7fd97be00700 1 -- 172.21.15.23:0/3182296089 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd9700105a0 con 0x7fd984073a70 2024-03-30T04:05:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd97be00700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 0x7fd9680a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd97be00700 1 -- 172.21.15.23:0/3182296089 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd970014070 con 0x7fd984073a70 2024-03-30T04:05:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd99e2c0b80 1 Processor -- start 2024-03-30T04:05:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd983400700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 0x7fd9680a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd99e2c0b80 1 -- start start 2024-03-30T04:05:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.107+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x5578858fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557885908df0 0x55788590b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557885856990 con 0x5578858fd5d0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557885854350 con 0x5578858fdef0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55788584fc40 con 0x557885908df0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd983400700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 0x7fd9680a3750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd974005950 tx=0x7fd97400b410 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd983400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x5578858fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52648/0 (socket says 172.21.15.23:52648) 2024-03-30T04:05:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 -- 172.21.15.23:0/2866190671 learned_addr learned my addr 172.21.15.23:0/2866190671 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd983e00700 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557885908df0 0x55788590b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 -- 172.21.15.23:0/2866190671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557885908df0 msgr2=0x55788590b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557885908df0 0x55788590b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 -- 172.21.15.23:0/2866190671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 msgr2=0x5578858fd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x5578858fd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.108+0000 7fd982a00700 1 -- 172.21.15.23:0/2866190671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578856d13b0 con 0x5578858fdef0 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.109+0000 7fd982a00700 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9700969c0 tx=0x7fd9700999a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85dd770305ed44b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.109+0000 7fd978c00700 1 -- 172.21.15.23:0/2866190671 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97009dd80 con 0x5578858fdef0 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.109+0000 7fd978c00700 1 -- 172.21.15.23:0/2866190671 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9700a23d0 con 0x5578858fdef0 2024-03-30T04:05:07.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.109+0000 7fd978c00700 1 -- 172.21.15.23:0/2866190671 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9700996a0 con 0x5578858fdef0 2024-03-30T04:05:07.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2866190671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 msgr2=0x5578859088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9700969c0 tx=0x7fd9700999a0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2866190671 shutdown_connections 2024-03-30T04:05:07.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557885908df0 0x55788590b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x5578858fd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2866190671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x5578859088b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2866190671 >> 172.21.15.23:0/2866190671 conn(0x55788585d6a0 msgr2=0x5578858f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2866190671 shutdown_connections 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2866190671 wait complete. 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.110+0000 7fd99e2c0b80 1 -- start start 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 Processor -- start 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fdef0 0x557885905b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x557885906280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578857ebd80 con 0x5578858fdef0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578857eba20 con 0x557885908df0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578856d36a0 con 0x5578858fd5d0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd99e2c0b80 10 client.0 _ll_register_callbacks cb 0x5578857f5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35554/0 (socket says 172.21.15.23:35554) 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd983400700 1 -- 172.21.15.23:0/2360672187 learned_addr learned my addr 172.21.15.23:0/2360672187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.111+0000 7fd983e00700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x557885906280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd982a00700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fdef0 0x557885905b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 -- 172.21.15.23:0/2360672187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 msgr2=0x557885906280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x557885906280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 -- 172.21.15.23:0/2360672187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fdef0 msgr2=0x557885905b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fdef0 0x557885905b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 -- 172.21.15.23:0/2360672187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9841aa6f0 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd983400700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9740116f0 tx=0x7fd9740047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a631f25f999285b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd96d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974010ee0 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd974017070 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.112+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557885810380 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd974011e90 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 10 client.26458 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55788567db60 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557885810380 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd9740277e0 con 0x5578858fd5d0 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 10 client.26458 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 10 client.26458 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 20 client.26458 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x557885918ad0 0x55788591aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 4 client.26458 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557885848700 con 0x557885918ad0 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.113+0000 7fd982a00700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x557885918ad0 0x55788591aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.114+0000 7fd982a00700 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x557885918ad0 0x55788591aef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=49d7234aef40e705 server_cookie=80bd102e75f086e4 in_seq=0 out_seq=0 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.114+0000 7fd96d600700 10 client.26458 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd96d600700 1 -- 172.21.15.23:0/2360672187 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557885848700 con 0x557885918ad0 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd96d600700 10 client.26458 handle_command_reply: tid=0 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 client.26458 shutdown 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 msgr2=0x557885908850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9740116f0 tx=0x7fd9740047c0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 shutdown_connections 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x557885918ad0 0x55788591aef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x557885906280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885908850 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd96d600700 0 client.26458 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2360672187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fdef0 0x557885905b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 >> 172.21.15.23:0/2360672187 conn(0x55788585d6a0 msgr2=0x5578858ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 shutdown_connections 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.115+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2360672187 wait complete. 2024-03-30T04:05:07.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.116+0000 7fd99e2c0b80 20 client.26458 trim_cache size 0 max 16384 2024-03-30T04:05:07.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.129+0000 7fd99e2c0b80 1 Processor -- start 2024-03-30T04:05:07.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.129+0000 7fd99e2c0b80 1 -- start start 2024-03-30T04:05:07.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x557885900570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 0x55788590fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578858f9fa0 con 0x557885908df0 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55788590ffb0 con 0x5578858fdef0 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578859153f0 con 0x5578858fd5d0 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd982a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x557885900570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35556/0 (socket says 172.21.15.23:35556) 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 -- 172.21.15.23:0/378088906 learned_addr learned my addr 172.21.15.23:0/378088906 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983e00700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 0x55788590fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 -- 172.21.15.23:0/378088906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 msgr2=0x557885900570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x557885900570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 -- 172.21.15.23:0/378088906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 msgr2=0x55788590fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 0x55788590fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983400700 1 -- 172.21.15.23:0/378088906 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557885815ea0 con 0x5578858fd5d0 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd983e00700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 0x55788590fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.130+0000 7fd982a00700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x557885900570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd983400700 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd974000c00 tx=0x7fd9740046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=392f7ab85386a0ca server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd979600700 1 -- 172.21.15.23:0/378088906 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974038070 con 0x5578858fd5d0 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd979600700 1 -- 172.21.15.23:0/378088906 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd974015070 con 0x5578858fd5d0 2024-03-30T04:05:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd979600700 1 -- 172.21.15.23:0/378088906 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974004c50 con 0x5578858fd5d0 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/378088906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 msgr2=0x557885905b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.131+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd974000c00 tx=0x7fd9740046b0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/378088906 shutdown_connections 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557885908df0 0x55788590fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fd5d0 0x557885905b00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/378088906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5578858fdef0 0x557885900570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/378088906 >> 172.21.15.23:0/378088906 conn(0x55788585d6a0 msgr2=0x55788591f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/378088906 shutdown_connections 2024-03-30T04:05:07.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/378088906 wait complete. 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 -- start start 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.132+0000 7fd99e2c0b80 1 Processor -- start 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fdef0 0x55788590f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x55788590da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55788590e160 con 0x5578858fd5d0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55788590e2d0 con 0x557885908df0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55788590e490 con 0x5578858fdef0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd99e2c0b80 10 client.0 _ll_register_callbacks cb 0x5578857f5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37754/0 (socket says 172.21.15.23:37754) 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983400700 1 -- 172.21.15.23:0/2272087361 learned_addr learned my addr 172.21.15.23:0/2272087361 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x55788590da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd982a00700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fdef0 0x55788590f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 -- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fdef0 msgr2=0x55788590f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fdef0 0x55788590f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 -- 172.21.15.23:0/2272087361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 msgr2=0x55788590f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983e00700 1 -- 172.21.15.23:0/2272087361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9700b1040 con 0x557885908df0 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.133+0000 7fd983400700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd983e00700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x55788590da50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0062a0 tx=0x7fd97c00be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90199b1dc1c99007 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd96d600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00e040 con 0x557885908df0 2024-03-30T04:05:07.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd97c003e10 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9700a23d0 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd97c0074e0 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd99e2c0b80 10 client.26471 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.134+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7fd97402c040 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd97c02f020 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd97c00b0d0 con 0x557885908df0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 10 client.26471 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 10 client.26471 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 20 client.26471 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55788596db00 0x55788596ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 4 client.26471 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55788569ecf0 con 0x55788596db00 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.135+0000 7fd983400700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55788596db00 0x55788596ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd983400700 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55788596db00 0x55788596ff20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=92784ab7e50422f9 server_cookie=8aa58d2acb19e78b in_seq=0 out_seq=0 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd96d600700 10 client.26471 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd96d600700 1 -- 172.21.15.23:0/2272087361 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55788569ecf0 con 0x55788596db00 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd96d600700 10 client.26471 handle_command_reply: tid=0 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 client.26471 shutdown 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 msgr2=0x55788590da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x55788590da50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd97c0062a0 tx=0x7fd97c00be20 comp rx=0 tx=0).stop 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 shutdown_connections 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55788596db00 0x55788596ff20 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557885908df0 0x55788590da50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5578858fd5d0 0x55788590f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd96d600700 0 client.26471 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 --2- 172.21.15.23:0/2272087361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578858fdef0 0x55788590f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 >> 172.21.15.23:0/2272087361 conn(0x55788585d6a0 msgr2=0x5578858f25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 shutdown_connections 2024-03-30T04:05:07.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 1 -- 172.21.15.23:0/2272087361 wait complete. 2024-03-30T04:05:07.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.136+0000 7fd99e2c0b80 20 client.26471 trim_cache size 0 max 16384 2024-03-30T04:05:07.141 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 msgr2=0x7fd9680a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 0x7fd9680a3750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd974005950 tx=0x7fd97400b410 comp rx=0 tx=0).stop 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 msgr2=0x7fd9841a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd97000e9a0 tx=0x7fd97000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 shutdown_connections 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd984111d00 0x7fd98419d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd984073150 0x7fd9841a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd9680a1290 0x7fd9680a3750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 --2- 172.21.15.23:0/3182296089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd984073a70 0x7fd9841a3c40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 >> 172.21.15.23:0/3182296089 conn(0x7fd9840fc7c0 msgr2=0x7fd9840fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 shutdown_connections 2024-03-30T04:05:07.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.137+0000 7fd98a800700 1 -- 172.21.15.23:0/3182296089 wait complete. 2024-03-30T04:05:07.155 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:05:07.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 223327 11.74 MB/sec warmup 104 sec latency 120.415 ms 2024-03-30T04:05:07.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.404+0000 7ff977e00700 1 -- 172.21.15.23:0/921071430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 msgr2=0x7ff970101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.404+0000 7ff977e00700 1 --2- 172.21.15.23:0/921071430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff970101330 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff964009990 tx=0x7ff964009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 -- 172.21.15.23:0/921071430 shutdown_connections 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 --2- 172.21.15.23:0/921071430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff97010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 --2- 172.21.15.23:0/921071430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff970101330 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 --2- 172.21.15.23:0/921071430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff970101900 0x7ff970105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 -- 172.21.15.23:0/921071430 >> 172.21.15.23:0/921071430 conn(0x7ff9700fc7f0 msgr2=0x7ff9700fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.405+0000 7ff977e00700 1 -- 172.21.15.23:0/921071430 shutdown_connections 2024-03-30T04:05:07.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.406+0000 7ff977e00700 1 -- 172.21.15.23:0/921071430 wait complete. 2024-03-30T04:05:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.407+0000 7ff977e00700 1 Processor -- start 2024-03-30T04:05:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.407+0000 7ff977e00700 1 -- start start 2024-03-30T04:05:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.408+0000 7ff977e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff9701a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.408+0000 7ff977e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.408+0000 7ff977e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff97010cc50 0x7ff97010d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.408+0000 7ff975400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff9701a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff977e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010d750 con 0x7ff97010cc50 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff977e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010d8c0 con 0x7ff970106070 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff977e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97010da80 con 0x7ff970100f50 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52692/0 (socket says 172.21.15.23:52692) 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 -- 172.21.15.23:0/110199165 learned_addr learned my addr 172.21.15.23:0/110199165 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff975e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff97010cc50 0x7ff97010d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 -- 172.21.15.23:0/110199165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 msgr2=0x7ff9701a35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff9701a35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 -- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff97010cc50 msgr2=0x7ff97010d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff97010cc50 0x7ff97010d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.409+0000 7ff974a00700 1 -- 172.21.15.23:0/110199165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9640095f0 con 0x7ff970106070 2024-03-30T04:05:07.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff975400700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff9701a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff974a00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c8f0 tx=0x7ff96000cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff96e000700 1 -- 172.21.15.23:0/110199165 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9600078d0 con 0x7ff970106070 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff96e000700 1 -- 172.21.15.23:0/110199165 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96000f450 con 0x7ff970106070 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff96e000700 1 -- 172.21.15.23:0/110199165 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960018660 con 0x7ff970106070 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9701aa400 con 0x7ff970106070 2024-03-30T04:05:07.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.410+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9701aa8c0 con 0x7ff970106070 2024-03-30T04:05:07.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.416+0000 7ff96e000700 1 -- 172.21.15.23:0/110199165 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff9600187c0 con 0x7ff970106070 2024-03-30T04:05:07.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.418+0000 7ff98b92ab80 1 Processor -- start 2024-03-30T04:05:07.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.418+0000 7ff96e000700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 0x7ff9580a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.419+0000 7ff96e000700 1 -- 172.21.15.23:0/110199165 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff9600cce80 con 0x7ff970106070 2024-03-30T04:05:07.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.420+0000 7ff98b92ab80 1 -- start start 2024-03-30T04:05:07.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.420+0000 7ff975e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 0x7ff9580a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.420+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642a71990 con 0x557642b185d0 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642a6f350 con 0x557642b23df0 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642a6ac40 con 0x557642b18ef0 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.421+0000 7ff975400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff974a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37774/0 (socket says 172.21.15.23:37774) 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 -- 172.21.15.23:0/3565953222 learned_addr learned my addr 172.21.15.23:0/3565953222 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 -- 172.21.15.23:0/3565953222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 msgr2=0x557642b238b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b238b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 -- 172.21.15.23:0/3565953222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 msgr2=0x557642b261e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b261e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.422+0000 7ff975400700 1 -- 172.21.15.23:0/3565953222 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5576428ec3b0 con 0x557642b185d0 2024-03-30T04:05:07.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.423+0000 7ff975400700 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff964005bc0 tx=0x7ff964005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c660eb357063b9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.424+0000 7ff975e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 0x7ff9580a3850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff96800f540 tx=0x7ff96800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.424+0000 7ff95e600700 1 -- 172.21.15.23:0/3565953222 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96401d070 con 0x557642b185d0 2024-03-30T04:05:07.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.424+0000 7ff95e600700 1 -- 172.21.15.23:0/3565953222 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96401e040 con 0x557642b185d0 2024-03-30T04:05:07.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.424+0000 7ff95e600700 1 -- 172.21.15.23:0/3565953222 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964004d20 con 0x557642b185d0 2024-03-30T04:05:07.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 -- 172.21.15.23:0/3565953222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 msgr2=0x557642b189b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff964005bc0 tx=0x7ff964005bf0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 -- 172.21.15.23:0/3565953222 shutdown_connections 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b185d0 0x557642b189b0 secure :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff964005bc0 tx=0x7ff964005bf0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/3565953222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b238b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.425+0000 7ff98b92ab80 1 -- 172.21.15.23:0/3565953222 >> 172.21.15.23:0/3565953222 conn(0x557642a786a0 msgr2=0x557642b14a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.426+0000 7ff98b92ab80 1 -- 172.21.15.23:0/3565953222 shutdown_connections 2024-03-30T04:05:07.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.426+0000 7ff98b92ab80 1 -- 172.21.15.23:0/3565953222 wait complete. 2024-03-30T04:05:07.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.429+0000 7ff98b92ab80 1 -- start start 2024-03-30T04:05:07.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.430+0000 7ff98b92ab80 1 Processor -- start 2024-03-30T04:05:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.431+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 0x557642b20d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.431+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 0x557642b21250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.431+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642a06d80 con 0x557642b21790 2024-03-30T04:05:07.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642a06a20 con 0x557642b18ef0 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5576428ee6a0 con 0x557642b23df0 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff98b92ab80 10 client.0 _ll_register_callbacks cb 0x557642a10470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff975400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 0x557642b20d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff975400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 0x557642b20d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52708/0 (socket says 172.21.15.23:52708) 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff975400700 1 -- 172.21.15.23:0/2636384204 learned_addr learned my addr 172.21.15.23:0/2636384204 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff974a00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 0x557642b21250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff974a00700 1 -- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 msgr2=0x557642b20d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff974a00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 0x557642b20d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.432+0000 7ff974a00700 1 -- 172.21.15.23:0/2636384204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 msgr2=0x557642b2fac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.433+0000 7ff975e00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.433+0000 7ff974a00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.433+0000 7ff974a00700 1 -- 172.21.15.23:0/2636384204 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97010e460 con 0x557642b23df0 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.433+0000 7ff975e00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:07.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.434+0000 7ff974a00700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 0x557642b21250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff9600ca210 tx=0x7ff960099bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b725673a14a54dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.434+0000 7ff952a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:07.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.434+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96009abe0 con 0x557642b23df0 2024-03-30T04:05:07.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.434+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff96009ad40 con 0x557642b23df0 2024-03-30T04:05:07.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.434+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557642a2b380 con 0x557642b23df0 2024-03-30T04:05:07.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.435+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff960099e30 con 0x557642b23df0 2024-03-30T04:05:07.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.435+0000 7ff98b92ab80 10 client.26476 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.435+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557642898b60 con 0x557642b23df0 2024-03-30T04:05:07.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557642a2b380 con 0x557642b23df0 2024-03-30T04:05:07.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff9600a1bc0 con 0x557642b23df0 2024-03-30T04:05:07.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 10 client.26476 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 10 client.26476 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:05:07.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 20 client.26476 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b33de0 0x557642b36200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 4 client.26476 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:07.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.437+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557642a63700 con 0x557642b33de0 2024-03-30T04:05:07.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.438+0000 7ff975400700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b33de0 0x557642b36200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.438+0000 7ff975400700 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b33de0 0x557642b36200 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=956f097403f17c5b server_cookie=f7af5695d201998f in_seq=0 out_seq=0 2024-03-30T04:05:07.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.438+0000 7ff952a00700 10 client.26476 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:07.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.440+0000 7ff952a00700 1 -- 172.21.15.23:0/2636384204 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557642a63700 con 0x557642b33de0 2024-03-30T04:05:07.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.440+0000 7ff952a00700 10 client.26476 handle_command_reply: tid=0 2024-03-30T04:05:07.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.440+0000 7ff98b92ab80 1 client.26476 shutdown 2024-03-30T04:05:07.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.441+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 msgr2=0x557642b21250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.441+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 0x557642b21250 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff9600ca210 tx=0x7ff960099bd0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 shutdown_connections 2024-03-30T04:05:07.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b33de0 0x557642b36200 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b18ef0 0x557642b20d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2636384204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b23df0 0x557642b21250 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 >> 172.21.15.23:0/2636384204 conn(0x557642a786a0 msgr2=0x557642b1a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.442+0000 7ff952a00700 0 client.26476 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.443+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 shutdown_connections 2024-03-30T04:05:07.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.443+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2636384204 wait complete. 2024-03-30T04:05:07.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.444+0000 7ff98b92ab80 20 client.26476 trim_cache size 0 max 16384 2024-03-30T04:05:07.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.472+0000 7ff98b92ab80 1 Processor -- start 2024-03-30T04:05:07.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 -- start start 2024-03-30T04:05:07.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b1f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b21cd0 con 0x557642b21790 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b267b0 con 0x557642b23df0 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.473+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b18430 con 0x557642b18ef0 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b1f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35602/0 (socket says 172.21.15.23:35602) 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 -- 172.21.15.23:0/2968049099 learned_addr learned my addr 172.21.15.23:0/2968049099 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 -- 172.21.15.23:0/2968049099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 msgr2=0x557642b1f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b1f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 -- 172.21.15.23:0/2968049099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 msgr2=0x557642b2aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 -- 172.21.15.23:0/2968049099 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557642a30ea0 con 0x557642b18ef0 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.474+0000 7ff975400700 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9640048d0 tx=0x7ff96400b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8547307ff492294 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.475+0000 7ff95f000700 1 -- 172.21.15.23:0/2968049099 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964033070 con 0x557642b18ef0 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.475+0000 7ff95f000700 1 -- 172.21.15.23:0/2968049099 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96400be30 con 0x557642b18ef0 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.475+0000 7ff95f000700 1 -- 172.21.15.23:0/2968049099 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96401d040 con 0x557642b18ef0 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2968049099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 msgr2=0x557642b361d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff9640048d0 tx=0x7ff96400b260 comp rx=0 tx=0).stop 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2968049099 shutdown_connections 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b1f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b361d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/2968049099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b2aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.476+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2968049099 >> 172.21.15.23:0/2968049099 conn(0x557642a786a0 msgr2=0x557642b0d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.477+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2968049099 shutdown_connections 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.477+0000 7ff98b92ab80 1 -- 172.21.15.23:0/2968049099 wait complete. 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.477+0000 7ff98b92ab80 1 -- start start 2024-03-30T04:05:07.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.478+0000 7ff98b92ab80 1 Processor -- start 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b82bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b83130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b83940 con 0x557642b21790 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b28f30 con 0x557642b23df0 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557642b290f0 con 0x557642b18ef0 2024-03-30T04:05:07.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.480+0000 7ff98b92ab80 10 client.0 _ll_register_callbacks cb 0x557642a10470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.481+0000 7ff975e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.481+0000 7ff975e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52732/0 (socket says 172.21.15.23:52732) 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.481+0000 7ff975e00700 1 -- 172.21.15.23:0/1542219226 learned_addr learned my addr 172.21.15.23:0/1542219226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.482+0000 7ff974a00700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b83130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.482+0000 7ff975400700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b82bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 -- 172.21.15.23:0/1542219226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 msgr2=0x557642b82bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b82bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 -- 172.21.15.23:0/1542219226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 msgr2=0x557642b83130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b83130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 -- 172.21.15.23:0/1542219226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5576428ec3b0 con 0x557642b23df0 2024-03-30T04:05:07.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975400700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b82bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff975e00700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff968008a90 tx=0x7ff968004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=369c8ba4e4b68803 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.483+0000 7ff952a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96800e040 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff968017c60 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff96401e040 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff96800feb0 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff98b92ab80 10 client.26486 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.484+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5576428f0cd0 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff96401e040 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7ff968015410 con 0x557642b23df0 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 10 client.26486 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 10 client.26486 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:05:07.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 20 client.26486 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b99a60 0x557642b9be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 4 client.26486 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:07.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.485+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5576428b9cf0 con 0x557642b99a60 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.486+0000 7ff975400700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b99a60 0x557642b9be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.487+0000 7ff975400700 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b99a60 0x557642b9be80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5696718d83999ad1 server_cookie=b6204fe52e3c50d9 in_seq=0 out_seq=0 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.487+0000 7ff952a00700 10 client.26486 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.487+0000 7ff952a00700 1 -- 172.21.15.23:0/1542219226 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5576428b9cf0 con 0x557642b99a60 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.487+0000 7ff952a00700 10 client.26486 handle_command_reply: tid=0 2024-03-30T04:05:07.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.487+0000 7ff98b92ab80 1 client.26486 shutdown 2024-03-30T04:05:07.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 msgr2=0x557642b289f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff968008a90 tx=0x7ff968004ad0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 shutdown_connections 2024-03-30T04:05:07.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x557642b99a60 0x557642b9be80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557642b23df0 0x557642b289f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557642b18ef0 0x557642b82bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff952a00700 0 client.26486 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 --2- 172.21.15.23:0/1542219226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557642b21790 0x557642b83130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 >> 172.21.15.23:0/1542219226 conn(0x557642a786a0 msgr2=0x557642b1f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.488+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 shutdown_connections 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.489+0000 7ff98b92ab80 1 -- 172.21.15.23:0/1542219226 wait complete. 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.489+0000 7ff98b92ab80 20 client.26486 trim_cache size 0 max 16384 2024-03-30T04:05:07.488 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 msgr2=0x7ff9580a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 0x7ff9580a3850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff96800f540 tx=0x7ff96800b040 comp rx=0 tx=0).stop 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 msgr2=0x7ff9701a3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff96000c8f0 tx=0x7ff96000cc00 comp rx=0 tx=0).stop 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 shutdown_connections 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff97010cc50 0x7ff97010d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff970100f50 0x7ff9701a35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff9580a1390 0x7ff9580a3850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 --2- 172.21.15.23:0/110199165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff970106070 0x7ff9701a3ae0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 >> 172.21.15.23:0/110199165 conn(0x7ff9700fc7f0 msgr2=0x7ff9700fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 shutdown_connections 2024-03-30T04:05:07.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.490+0000 7ff977e00700 1 -- 172.21.15.23:0/110199165 wait complete. 2024-03-30T04:05:07.507 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:07 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2584646236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:07.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:05:07.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:07 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2584646236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:07.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:07 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2584646236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:07.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 -- 172.21.15.23:0/4252057858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 msgr2=0x7f736010fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 --2- 172.21.15.23:0/4252057858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f736010fe10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7358009f90 tx=0x7f735800b330 comp rx=0 tx=0).stop 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 -- 172.21.15.23:0/4252057858 shutdown_connections 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 --2- 172.21.15.23:0/4252057858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f736010fe10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 --2- 172.21.15.23:0/4252057858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7360103170 0x7f7360103550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 --2- 172.21.15.23:0/4252057858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103b20 0x7f7360107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 -- 172.21.15.23:0/4252057858 >> 172.21.15.23:0/4252057858 conn(0x7f73600fe990 msgr2=0x7f7360100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 -- 172.21.15.23:0/4252057858 shutdown_connections 2024-03-30T04:05:07.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.765+0000 7f7367a00700 1 -- 172.21.15.23:0/4252057858 wait complete. 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 Processor -- start 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 -- start start 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103170 0x7f7360199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f736019dc20 0x7f736019e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736019e6a0 con 0x7f7360103170 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736019e810 con 0x7f7360108290 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.766+0000 7f7367a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736019e9d0 con 0x7f736019dc20 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103170 0x7f7360199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52752/0 (socket says 172.21.15.23:52752) 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 -- 172.21.15.23:0/3214301102 learned_addr learned my addr 172.21.15.23:0/3214301102 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f7365a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f736019dc20 0x7f736019e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 -- 172.21.15.23:0/3214301102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f736019dc20 msgr2=0x7f736019e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f736019dc20 0x7f736019e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 -- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103170 msgr2=0x7f7360199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103170 0x7f7360199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 -- 172.21.15.23:0/3214301102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540095f0 con 0x7f7360108290 2024-03-30T04:05:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.767+0000 7f735fe00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e970 tx=0x7f734c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.768+0000 7f735d600700 1 -- 172.21.15.23:0/3214301102 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c009790 con 0x7f7360108290 2024-03-30T04:05:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.768+0000 7f735d600700 1 -- 172.21.15.23:0/3214301102 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f734c00c7e0 con 0x7f7360108290 2024-03-30T04:05:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.768+0000 7f735d600700 1 -- 172.21.15.23:0/3214301102 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0054e0 con 0x7f7360108290 2024-03-30T04:05:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.768+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7358009c40 con 0x7f7360108290 2024-03-30T04:05:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.768+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73601a6220 con 0x7f7360108290 2024-03-30T04:05:07.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.771+0000 7f735d600700 1 -- 172.21.15.23:0/3214301102 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f734c00c950 con 0x7f7360108290 2024-03-30T04:05:07.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.772+0000 7f737b56fb80 1 Processor -- start 2024-03-30T04:05:07.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.772+0000 7f735d600700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 0x7f73480a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 -- start start 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f735d600700 1 -- 172.21.15.23:0/3214301102 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f734c014070 con 0x7f7360108290 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f7365a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 0x7f73480a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c1b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417b66990 con 0x55e417c0d5d0 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417b64350 con 0x55e417c18df0 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f737b56fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417b5fc40 con 0x55e417c0def0 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37838/0 (socket says 172.21.15.23:37838) 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f7365000700 1 -- 172.21.15.23:0/3260250229 learned_addr learned my addr 172.21.15.23:0/3260250229 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.773+0000 7f735fe00700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365a00700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c1b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 -- 172.21.15.23:0/3260250229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 msgr2=0x55e417c188b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c188b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 -- 172.21.15.23:0/3260250229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 msgr2=0x55e417c1b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c1b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 -- 172.21.15.23:0/3260250229 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4179e13b0 con 0x55e417c0d5d0 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365a00700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c1b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 0x7f73480a37d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7358009790 tx=0x7f7358009720 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.774+0000 7f7365000700 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7354009fc0 tx=0x7f7354011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4631c38dc1753e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.775+0000 7f7352000700 1 -- 172.21.15.23:0/3260250229 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400e670 con 0x55e417c0d5d0 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.775+0000 7f7352000700 1 -- 172.21.15.23:0/3260250229 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f735400ec90 con 0x55e417c0d5d0 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.775+0000 7f7352000700 1 -- 172.21.15.23:0/3260250229 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354019d30 con 0x55e417c0d5d0 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.775+0000 7f737b56fb80 1 -- 172.21.15.23:0/3260250229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 msgr2=0x55e417c0d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.775+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7354009fc0 tx=0x7f7354011720 comp rx=0 tx=0).stop 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 -- 172.21.15.23:0/3260250229 shutdown_connections 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c1b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c0d9b0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3260250229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c188b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 -- 172.21.15.23:0/3260250229 >> 172.21.15.23:0/3260250229 conn(0x55e417b6d6a0 msgr2=0x55e417c09a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 -- 172.21.15.23:0/3260250229 shutdown_connections 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 -- 172.21.15.23:0/3260250229 wait complete. 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.776+0000 7f737b56fb80 1 -- start start 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.777+0000 7f737b56fb80 1 Processor -- start 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417afbd80 con 0x55e417c0d5d0 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417afba20 con 0x55e417c18df0 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4179e36a0 con 0x55e417c0def0 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f737b56fb80 10 client.0 _ll_register_callbacks cb 0x55e417b05470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52774/0 (socket says 172.21.15.23:52774) 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35646/0 (socket says 172.21.15.23:35646) 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365a00700 1 -- 172.21.15.23:0/3549171989 learned_addr learned my addr 172.21.15.23:0/3549171989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c187c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37840/0 (socket says 172.21.15.23:37840) 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 -- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 msgr2=0x55e417c16050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 -- 172.21.15.23:0/3549171989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 msgr2=0x55e417c187c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c187c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f735fe00700 1 -- 172.21.15.23:0/3549171989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354009c20 con 0x55e417c0def0 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.778+0000 7f7365a00700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f735fe00700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f734c0ca540 tx=0x7f734c099f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6618f446486e3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:07.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0a2380 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f734c09a160 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e417b20380 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f734c0b8250 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f737b56fb80 10 client.26494 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55e41798db60 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e417b20380 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.779+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f734c0a1ef0 con 0x55e417c0def0 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 10 client.26494 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 10 client.26494 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 20 client.26494 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c288c0 0x55e417c2ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 4 client.26494 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:07.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e417b58700 con 0x55e417c288c0 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.780+0000 7f7365000700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c288c0 0x55e417c2ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.781+0000 7f7365000700 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c288c0 0x55e417c2ace0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce24a02c5490764e server_cookie=6a3baec9bfb357f5 in_seq=0 out_seq=0 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.781+0000 7f7346a00700 10 client.26494 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f7346a00700 1 -- 172.21.15.23:0/3549171989 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e417b58700 con 0x55e417c288c0 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f7346a00700 10 client.26494 handle_command_reply: tid=0 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 client.26494 shutdown 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 msgr2=0x55e417c15a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f734c0ca540 tx=0x7f734c099f00 comp rx=0 tx=0).stop 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 shutdown_connections 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c16050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c187c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c288c0 0x55e417c2ace0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3549171989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c15a40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f7346a00700 0 client.26494 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 >> 172.21.15.23:0/3549171989 conn(0x55e417b6d6a0 msgr2=0x55e417c0f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 shutdown_connections 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 1 -- 172.21.15.23:0/3549171989 wait complete. 2024-03-30T04:05:07.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.782+0000 7f737b56fb80 20 client.26494 trim_cache size 0 max 16384 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 Processor -- start 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 -- start start 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c2acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c104d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c25510 con 0x55e417c0d5d0 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c0e670 con 0x55e417c18df0 2024-03-30T04:05:07.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.797+0000 7f737b56fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c0a850 con 0x55e417c0def0 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c104d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52778/0 (socket says 172.21.15.23:52778) 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c2acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365a00700 1 -- 172.21.15.23:0/3737265789 learned_addr learned my addr 172.21.15.23:0/3737265789 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 -- 172.21.15.23:0/3737265789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 msgr2=0x55e417c15a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 -- 172.21.15.23:0/3737265789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 msgr2=0x55e417c2acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c2acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 -- 172.21.15.23:0/3737265789 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e417b25ea0 con 0x55e417c0def0 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365a00700 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7365000700 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c2acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f735fe00700 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c104d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f734c096970 tx=0x7f734c00cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24e65ac30b08ae84 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7352a00700 1 -- 172.21.15.23:0/3737265789 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0996b0 con 0x55e417c0def0 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7352a00700 1 -- 172.21.15.23:0/3737265789 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f734c099f70 con 0x55e417c0def0 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.798+0000 7f7352a00700 1 -- 172.21.15.23:0/3737265789 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c09a160 con 0x55e417c0def0 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 -- 172.21.15.23:0/3737265789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 msgr2=0x55e417c104d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c104d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f734c096970 tx=0x7f734c00cf50 comp rx=0 tx=0).stop 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 -- 172.21.15.23:0/3737265789 shutdown_connections 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c18df0 0x55e417c15a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0d5d0 0x55e417c2acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3737265789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c0def0 0x55e417c104d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 -- 172.21.15.23:0/3737265789 >> 172.21.15.23:0/3737265789 conn(0x55e417b6d6a0 msgr2=0x55e417c152d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 -- 172.21.15.23:0/3737265789 shutdown_connections 2024-03-30T04:05:07.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.799+0000 7f737b56fb80 1 -- 172.21.15.23:0/3737265789 wait complete. 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.800+0000 7f737b56fb80 1 -- start start 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.800+0000 7f737b56fb80 1 Processor -- start 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 0x55e417c1df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c18df0 0x55e417c1e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c7b8c0 con 0x55e417c0def0 2024-03-30T04:05:07.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c7ba30 con 0x55e417c0d5d0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e417c7bbf0 con 0x55e417c18df0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f737b56fb80 10 client.0 _ll_register_callbacks cb 0x55e417b05470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f735fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 0x55e417c1df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:52782/0 (socket says 172.21.15.23:52782) 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 -- 172.21.15.23:0/3298824608 learned_addr learned my addr 172.21.15.23:0/3298824608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365a00700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c18df0 0x55e417c1e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 -- 172.21.15.23:0/3298824608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c18df0 msgr2=0x55e417c1e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c18df0 0x55e417c1e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 -- 172.21.15.23:0/3298824608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 msgr2=0x55e417c1df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 0x55e417c1df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f7365000700 1 -- 172.21.15.23:0/3298824608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4179e13b0 con 0x55e417c0d5d0 2024-03-30T04:05:07.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.801+0000 7f735fe00700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 0x55e417c1df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f7365000700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73540198e0 tx=0x7f7354019870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b92cf343879379e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f7346a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735402b070 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e4179aecf0 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f735402b3d0 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f735402b5b0 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f737b56fb80 10 client.26501 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.802+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f7360110680 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e4179aecf0 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f735400e7b0 con 0x55e417c0d5d0 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 10 client.26501 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 10 client.26501 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 20 client.26501 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:07.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c7d390 0x55e417c7f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 4 client.26501 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e4179e3ce0 con 0x55e417c7d390 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f735fe00700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c7d390 0x55e417c7f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f735fe00700 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c7d390 0x55e417c7f7b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d5ee2ed238d2ac47 server_cookie=744c0d3088bd610e in_seq=0 out_seq=0 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.803+0000 7f7346a00700 10 client.26501 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f7346a00700 1 -- 172.21.15.23:0/3298824608 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e4179e3ce0 con 0x55e417c7d390 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f7346a00700 10 client.26501 handle_command_reply: tid=0 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 client.26501 shutdown 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 msgr2=0x55e417c1da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f73540198e0 tx=0x7f7354019870 comp rx=0 tx=0).stop 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 shutdown_connections 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55e417c7d390 0x55e417c7f7b0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e417c18df0 0x55e417c1e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e417c0d5d0 0x55e417c1da50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f7346a00700 0 client.26501 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 --2- 172.21.15.23:0/3298824608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e417c0def0 0x55e417c1df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 >> 172.21.15.23:0/3298824608 conn(0x55e417b6d6a0 msgr2=0x55e417c142f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 shutdown_connections 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 1 -- 172.21.15.23:0/3298824608 wait complete. 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.804+0000 7f737b56fb80 20 client.26501 trim_cache size 0 max 16384 2024-03-30T04:05:07.807 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 msgr2=0x7f73480a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 0x7f73480a37d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7358009790 tx=0x7f7358009720 comp rx=0 tx=0).stop 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 msgr2=0x7f7360199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e970 tx=0x7f734c00ec80 comp rx=0 tx=0).stop 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 shutdown_connections 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f736019dc20 0x7f736019e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7360103170 0x7f7360199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f73480a1310 0x7f73480a37d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 --2- 172.21.15.23:0/3214301102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7360108290 0x7f7360199880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:07.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 >> 172.21.15.23:0/3214301102 conn(0x7f73600fe990 msgr2=0x7f7360100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:07.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 shutdown_connections 2024-03-30T04:05:07.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:07.805+0000 7f7367a00700 1 -- 172.21.15.23:0/3214301102 wait complete. 2024-03-30T04:05:07.822 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:05:08.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 -- 172.21.15.23:0/2030362301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 msgr2=0x7fb06010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 --2- 172.21.15.23:0/2030362301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb06010dc10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b0a0 tx=0x7fb05800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 -- 172.21.15.23:0/2030362301 shutdown_connections 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 --2- 172.21.15.23:0/2030362301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb06010dc10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 --2- 172.21.15.23:0/2030362301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 --2- 172.21.15.23:0/2030362301 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101920 0x7fb060105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.046+0000 7fb066800700 1 -- 172.21.15.23:0/2030362301 >> 172.21.15.23:0/2030362301 conn(0x7fb0600fc7f0 msgr2=0x7fb0600fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.047+0000 7fb066800700 1 -- 172.21.15.23:0/2030362301 shutdown_connections 2024-03-30T04:05:08.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.047+0000 7fb066800700 1 -- 172.21.15.23:0/2030362301 wait complete. 2024-03-30T04:05:08.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 Processor -- start 2024-03-30T04:05:08.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 -- start start 2024-03-30T04:05:08.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb0601997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06019db60 0x7fb06019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e5e0 con 0x7fb06019db60 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e750 con 0x7fb060106090 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb066800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019e910 con 0x7fb060100f70 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.048+0000 7fb05f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb0601997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55756/0 (socket says 172.21.15.23:55756) 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 -- 172.21.15.23:0/2905412377 learned_addr learned my addr 172.21.15.23:0/2905412377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05fe00700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06019db60 0x7fb06019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 -- 172.21.15.23:0/2905412377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 msgr2=0x7fb0601997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb0601997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 -- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06019db60 msgr2=0x7fb06019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06019db60 0x7fb06019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 -- 172.21.15.23:0/2905412377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009d00 con 0x7fb060100f70 2024-03-30T04:05:08.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05ea00700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb0601997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.049+0000 7fb05f400700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e9a0 tx=0x7fb05000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.050+0000 7fb057e00700 1 -- 172.21.15.23:0/2905412377 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050009790 con 0x7fb060100f70 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.050+0000 7fb057e00700 1 -- 172.21.15.23:0/2905412377 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb050010430 con 0x7fb060100f70 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.050+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0601a5e70 con 0x7fb060100f70 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.050+0000 7fb057e00700 1 -- 172.21.15.23:0/2905412377 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050018680 con 0x7fb060100f70 2024-03-30T04:05:08.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.050+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0601a6340 con 0x7fb060100f70 2024-03-30T04:05:08.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.053+0000 7fb057e00700 1 -- 172.21.15.23:0/2905412377 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb0500105a0 con 0x7fb060100f70 2024-03-30T04:05:08.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.054+0000 7fb057e00700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 0x7fb0440a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.054+0000 7fb057e00700 1 -- 172.21.15.23:0/2905412377 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb050014070 con 0x7fb060100f70 2024-03-30T04:05:08.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.054+0000 7fb07a2aeb80 1 Processor -- start 2024-03-30T04:05:08.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.054+0000 7fb05ea00700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 0x7fb0440a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 -- start start 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d3052069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d3052118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30515f990 con 0x55d3052065d0 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30515d350 con 0x55d305211df0 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d305158c40 con 0x55d305206ef0 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d3052069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41810/0 (socket says 172.21.15.23:41810) 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05fe00700 1 -- 172.21.15.23:0/3565870227 learned_addr learned my addr 172.21.15.23:0/3565870227 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:08.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d3052118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.055+0000 7fb05ea00700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 0x7fb0440a37d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb06019a7d0 tx=0x7fb04c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 -- 172.21.15.23:0/3565870227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 msgr2=0x55d3052118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d3052118b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 -- 172.21.15.23:0/3565870227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 msgr2=0x55d3052069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d3052069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 -- 172.21.15.23:0/3565870227 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d304fda3b0 con 0x55d305211df0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb05fe00700 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb0580094d0 tx=0x7fb058009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4035ad46073030b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb054c00700 1 -- 172.21.15.23:0/3565870227 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800e040 con 0x55d305211df0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb054c00700 1 -- 172.21.15.23:0/3565870227 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb058007e80 con 0x55d305211df0 2024-03-30T04:05:08.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.056+0000 7fb054c00700 1 -- 172.21.15.23:0/3565870227 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580078e0 con 0x55d305211df0 2024-03-30T04:05:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3565870227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 msgr2=0x55d3052141e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb0580094d0 tx=0x7fb058009500 comp rx=0 tx=0).stop 2024-03-30T04:05:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3565870227 shutdown_connections 2024-03-30T04:05:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d3052141e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d3052069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3565870227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d3052118b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3565870227 >> 172.21.15.23:0/3565870227 conn(0x55d3051666a0 msgr2=0x55d305202a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3565870227 shutdown_connections 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.057+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3565870227 wait complete. 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.058+0000 7fb07a2aeb80 1 -- start start 2024-03-30T04:05:08.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.058+0000 7fb07a2aeb80 1 Processor -- start 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d305206ef0 0x55d30520eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305211df0 0x55d30520f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3050f4d80 con 0x55d305206ef0 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3050f4a20 con 0x55d3052065d0 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d304fdc6a0 con 0x55d305211df0 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb07a2aeb80 10 client.0 _ll_register_callbacks cb 0x55d3050fe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:08.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb05f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d305206ef0 0x55d30520eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.059+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305211df0 0x55d30520f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41822/0 (socket says 172.21.15.23:41822) 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 -- 172.21.15.23:0/3100819164 learned_addr learned my addr 172.21.15.23:0/3100819164 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 -- 172.21.15.23:0/3100819164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305211df0 msgr2=0x55d30520f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305211df0 0x55d30520f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 -- 172.21.15.23:0/3100819164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d305206ef0 msgr2=0x55d30520eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d305206ef0 0x55d30520eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 -- 172.21.15.23:0/3100819164 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06010e480 con 0x55d3052065d0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb05f400700 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb0500969a0 tx=0x7fb05009a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af62854a9cb26820 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb049600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05009dd60 con 0x55d3052065d0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb0500a23b0 con 0x55d3052065d0 2024-03-30T04:05:08.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.060+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d305151700 con 0x55d3052065d0 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb050099420 con 0x55d3052065d0 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 10 client.26513 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d304f86b60 con 0x55d3052065d0 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d305151700 con 0x55d3052065d0 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb050095d20 con 0x55d3052065d0 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 10 client.26513 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 20 client.26513 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30522a150 0x55d30522c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 4 client.26513 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.061+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d304fa7cf0 con 0x55d30522a150 2024-03-30T04:05:08.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.062+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30522a150 0x55d30522c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.062+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30522a150 0x55d30522c570 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6564deff6d46e49b server_cookie=93134f2ae51a45d6 in_seq=0 out_seq=0 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.062+0000 7fb049600700 10 client.26513 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb049600700 1 -- 172.21.15.23:0/3100819164 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d304fa7cf0 con 0x55d30522a150 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb049600700 10 client.26513 handle_command_reply: tid=0 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 client.26513 shutdown 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 msgr2=0x55d3052117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb0500969a0 tx=0x7fb05009a780 comp rx=0 tx=0).stop 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 shutdown_connections 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30522a150 0x55d30522c570 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305211df0 0x55d30520f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d3052065d0 0x55d3052117e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3100819164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d305206ef0 0x55d30520eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 >> 172.21.15.23:0/3100819164 conn(0x55d3051666a0 msgr2=0x55d3052086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb049600700 0 client.26513 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 shutdown_connections 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.063+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3100819164 wait complete. 2024-03-30T04:05:08.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.064+0000 7fb07a2aeb80 20 client.26513 trim_cache size 0 max 16384 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 Processor -- start 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 -- start start 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305218a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d305211310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3052099a0 con 0x55d3052065d0 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d305204160 con 0x55d305211df0 2024-03-30T04:05:08.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.079+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d305229ef0 con 0x55d305206ef0 2024-03-30T04:05:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d305211310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305218a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55772/0 (socket says 172.21.15.23:55772) 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 -- 172.21.15.23:0/367423797 learned_addr learned my addr 172.21.15.23:0/367423797 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 -- 172.21.15.23:0/367423797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 msgr2=0x55d305211310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d305211310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 -- 172.21.15.23:0/367423797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 msgr2=0x55d305218a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305218a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 -- 172.21.15.23:0/367423797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d30511eea0 con 0x55d305206ef0 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.080+0000 7fb05ea00700 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55d305211dc0 tx=0x7fb04c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24f56c6045671cf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.081+0000 7fb055600700 1 -- 172.21.15.23:0/367423797 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c024030 con 0x55d305206ef0 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.081+0000 7fb055600700 1 -- 172.21.15.23:0/367423797 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb04c025040 con 0x55d305206ef0 2024-03-30T04:05:08.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.081+0000 7fb055600700 1 -- 172.21.15.23:0/367423797 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c028730 con 0x55d305206ef0 2024-03-30T04:05:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.081+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/367423797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 msgr2=0x55d30521d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.081+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55d305211dc0 tx=0x7fb04c014000 comp rx=0 tx=0).stop 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/367423797 shutdown_connections 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d305211310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305218a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/367423797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d30521d190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/367423797 >> 172.21.15.23:0/367423797 conn(0x55d3051666a0 msgr2=0x55d3051fb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/367423797 shutdown_connections 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/367423797 wait complete. 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 -- start start 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.082+0000 7fb07a2aeb80 1 Processor -- start 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305216c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d30527cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3052178b0 con 0x55d3052065d0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30527d4c0 con 0x55d305211df0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30527d680 con 0x55d305206ef0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb07a2aeb80 10 client.0 _ll_register_callbacks cb 0x55d3050fe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305216c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d30527cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55778/0 (socket says 172.21.15.23:55778) 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 -- 172.21.15.23:0/3632336089 learned_addr learned my addr 172.21.15.23:0/3632336089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d30527cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41842/0 (socket says 172.21.15.23:41842) 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05f400700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305216c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38640/0 (socket says 172.21.15.23:38640) 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 -- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 msgr2=0x55d30527cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d30527cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 -- 172.21.15.23:0/3632336089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 msgr2=0x55d305216c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:08.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305216c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.083+0000 7fb05ea00700 1 -- 172.21.15.23:0/3632336089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c017070 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb05ea00700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00efd0 tx=0x7fb04c00e640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c64f029bc12f6dc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb049600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c024070 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb04c0053c0 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d305119380 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb04c005580 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb07a2aeb80 10 client.26518 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.084+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55d304fdecd0 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d305119380 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fb04c004d20 con 0x55d305206ef0 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 10 client.26518 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:08.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 20 client.26518 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:08.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30527ed40 0x55d305281160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:08.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 4 client.26518 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:08.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d304fdcce0 con 0x55d30527ed40 2024-03-30T04:05:08.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.085+0000 7fb05f400700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30527ed40 0x55d305281160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:08.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb05f400700 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30527ed40 0x55d305281160 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60276fa5f989a136 server_cookie=1df06403aaab5585 in_seq=0 out_seq=0 2024-03-30T04:05:08.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb049600700 10 client.26518 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:08.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb049600700 1 -- 172.21.15.23:0/3632336089 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d304fdcce0 con 0x55d30527ed40 2024-03-30T04:05:08.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb049600700 10 client.26518 handle_command_reply: tid=0 2024-03-30T04:05:08.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 client.26518 shutdown 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 msgr2=0x55d305217150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb04c00efd0 tx=0x7fb04c00e640 comp rx=0 tx=0).stop 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 shutdown_connections 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d305211df0 0x55d30527cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d3052065d0 0x55d305216c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30527ed40 0x55d305281160 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 --2- 172.21.15.23:0/3632336089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d305206ef0 0x55d305217150 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb049600700 0 client.26518 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 >> 172.21.15.23:0/3632336089 conn(0x55d3051666a0 msgr2=0x55d30521d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 shutdown_connections 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 1 -- 172.21.15.23:0/3632336089 wait complete. 2024-03-30T04:05:08.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.086+0000 7fb07a2aeb80 20 client.26518 trim_cache size 0 max 16384 2024-03-30T04:05:08.090 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:08.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 msgr2=0x7fb0440a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 0x7fb0440a37d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb06019a7d0 tx=0x7fb04c009260 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 msgr2=0x7fb060199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e9a0 tx=0x7fb05000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 shutdown_connections 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb06019db60 0x7fb06019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb060100f70 0x7fb060199280 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb0440a1310 0x7fb0440a37d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 --2- 172.21.15.23:0/2905412377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb060106090 0x7fb0601997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 >> 172.21.15.23:0/2905412377 conn(0x7fb0600fc7f0 msgr2=0x7fb06010c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 shutdown_connections 2024-03-30T04:05:08.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:08.087+0000 7fb066800700 1 -- 172.21.15.23:0/2905412377 wait complete. 2024-03-30T04:05:08.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 224700 11.69 MB/sec warmup 105 sec latency 15.897 ms 2024-03-30T04:05:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:08 smithi023 ceph-mon[38254]: pgmap v99: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 318 op/s 2024-03-30T04:05:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:08 smithi154 ceph-mon[79161]: pgmap v99: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 318 op/s 2024-03-30T04:05:08.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:08 smithi193 ceph-mon[79581]: pgmap v99: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 318 op/s 2024-03-30T04:05:09.105 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:05:09.106 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:05:09.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 227387 11.74 MB/sec warmup 106 sec latency 9.540 ms 2024-03-30T04:05:09.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.331+0000 7f6397a00700 1 -- 172.21.15.23:0/1293676371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390101900 msgr2=0x7f6390105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.331+0000 7f6397a00700 1 --2- 172.21.15.23:0/1293676371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390101900 0x7f6390105940 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f637c009990 tx=0x7f637c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 -- 172.21.15.23:0/1293676371 shutdown_connections 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 --2- 172.21.15.23:0/1293676371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390106070 0x7f639010dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 --2- 172.21.15.23:0/1293676371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390100f50 0x7f6390101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 --2- 172.21.15.23:0/1293676371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390101900 0x7f6390105940 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 -- 172.21.15.23:0/1293676371 >> 172.21.15.23:0/1293676371 conn(0x7f63900fc7f0 msgr2=0x7f63900fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 -- 172.21.15.23:0/1293676371 shutdown_connections 2024-03-30T04:05:09.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.332+0000 7f6397a00700 1 -- 172.21.15.23:0/1293676371 wait complete. 2024-03-30T04:05:09.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.334+0000 7f6397a00700 1 Processor -- start 2024-03-30T04:05:09.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.334+0000 7f6397a00700 1 -- start start 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390106070 0x7f639010c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f639010c820 0x7f63901a9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639010df70 con 0x7f6390106070 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63901aa010 con 0x7f639010c820 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6397a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63901aa180 con 0x7f6390100f50 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6395000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6395000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55810/0 (socket says 172.21.15.23:55810) 2024-03-30T04:05:09.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f6395000700 1 -- 172.21.15.23:0/1699682860 learned_addr learned my addr 172.21.15.23:0/1699682860 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.335+0000 7f638fe00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390106070 0x7f639010c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.336+0000 7f6395000700 1 -- 172.21.15.23:0/1699682860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f639010c820 msgr2=0x7f63901a9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.336+0000 7f6395000700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f639010c820 0x7f63901a9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.336+0000 7f6395000700 1 -- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390106070 msgr2=0x7f639010c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.336+0000 7f6395000700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390106070 0x7f639010c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.336+0000 7f6395000700 1 -- 172.21.15.23:0/1699682860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c0095f0 con 0x7f6390100f50 2024-03-30T04:05:09.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.337+0000 7f6395000700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f638400d950 tx=0x7f638400dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.338+0000 7f638d600700 1 -- 172.21.15.23:0/1699682860 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384009b40 con 0x7f6390100f50 2024-03-30T04:05:09.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.338+0000 7f638d600700 1 -- 172.21.15.23:0/1699682860 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6384009ca0 con 0x7f6390100f50 2024-03-30T04:05:09.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.338+0000 7f638d600700 1 -- 172.21.15.23:0/1699682860 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638400f620 con 0x7f6390100f50 2024-03-30T04:05:09.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.339+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63901aa460 con 0x7f6390100f50 2024-03-30T04:05:09.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.339+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63901aa920 con 0x7f6390100f50 2024-03-30T04:05:09.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.345+0000 7f638d600700 1 -- 172.21.15.23:0/1699682860 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f638400f780 con 0x7f6390100f50 2024-03-30T04:05:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.349+0000 7f638d600700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 0x7f63780a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.349+0000 7f63ab5c8b80 1 Processor -- start 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.349+0000 7f638d600700 1 -- 172.21.15.23:0/1699682860 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f63840ccf60 con 0x7f6390100f50 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 -- start start 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f638fe00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 0x7f63780a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635afea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b09930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b0c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635a57a10 con 0x560635afe650 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635a553d0 con 0x560635afef70 2024-03-30T04:05:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.350+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635a50cc0 con 0x560635b09e70 2024-03-30T04:05:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f6395000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635afea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b09930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f6395000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635afea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38662/0 (socket says 172.21.15.23:38662) 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f6395000700 1 -- 172.21.15.23:0/1418622596 learned_addr learned my addr 172.21.15.23:0/1418622596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f6395a00700 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b0c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 0x7f63780a38a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f639010d4d0 tx=0x7f637c00b560 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 -- 172.21.15.23:0/1418622596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 msgr2=0x560635b0c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b0c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 -- 172.21.15.23:0/1418622596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 msgr2=0x560635afea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635afea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f638fe00700 1 -- 172.21.15.23:0/1418622596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606358d2430 con 0x560635afef70 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.352+0000 7f6395a00700 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b0c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.353+0000 7f638fe00700 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b09930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c01a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32bd0590647d449d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.353+0000 7f6382000700 1 -- 172.21.15.23:0/1418622596 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c0052c0 con 0x560635afef70 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.353+0000 7f6382000700 1 -- 172.21.15.23:0/1418622596 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f637c01e040 con 0x560635afef70 2024-03-30T04:05:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.353+0000 7f6382000700 1 -- 172.21.15.23:0/1418622596 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c0107f0 con 0x560635afef70 2024-03-30T04:05:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.355+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/1418622596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 msgr2=0x560635b09930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.355+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b09930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c01a070 comp rx=0 tx=0).stop 2024-03-30T04:05:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/1418622596 shutdown_connections 2024-03-30T04:05:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b0c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635afea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/1418622596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b09930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/1418622596 >> 172.21.15.23:0/1418622596 conn(0x560635a5e720 msgr2=0x560635afab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/1418622596 shutdown_connections 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.356+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/1418622596 wait complete. 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.357+0000 7f63ab5c8b80 1 -- start start 2024-03-30T04:05:09.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.357+0000 7f63ab5c8b80 1 Processor -- start 2024-03-30T04:05:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.358+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635b098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.358+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b06b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b07280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606359ece00 con 0x560635afe650 2024-03-30T04:05:09.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606359ecaa0 con 0x560635afef70 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606358d4720 con 0x560635b09e70 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f63ab5c8b80 10 client.0 _ll_register_callbacks cb 0x5606359f64f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f6395a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b07280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.359+0000 7f6395a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b07280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55832/0 (socket says 172.21.15.23:55832) 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f6395a00700 1 -- 172.21.15.23:0/2459427013 learned_addr learned my addr 172.21.15.23:0/2459427013 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b06b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 -- 172.21.15.23:0/2459427013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 msgr2=0x560635b07280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b07280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 -- 172.21.15.23:0/2459427013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 msgr2=0x560635b098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635b098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.360+0000 7f638fe00700 1 -- 172.21.15.23:0/2459427013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63880097e0 con 0x560635afef70 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.361+0000 7f638fe00700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b06b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f637c00bc60 tx=0x7f637c005660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb56edb522c86aea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.361+0000 7f6376a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:09.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.362+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c003d90 con 0x560635afef70 2024-03-30T04:05:09.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.362+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f637c02e400 con 0x560635afef70 2024-03-30T04:05:09.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.362+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560635a11400 con 0x560635afef70 2024-03-30T04:05:09.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.363+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f637c02e5c0 con 0x560635afef70 2024-03-30T04:05:09.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.363+0000 7f63ab5c8b80 10 client.26528 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:09.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.363+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f6390073360 con 0x560635afef70 2024-03-30T04:05:09.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.364+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560635a11400 con 0x560635afef70 2024-03-30T04:05:09.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.364+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f637c010930 con 0x560635afef70 2024-03-30T04:05:09.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.364+0000 7f63ab5c8b80 10 client.26528 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:09.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.364+0000 7f63ab5c8b80 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:09.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.364+0000 7f63ab5c8b80 20 client.26528 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:09.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.365+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b19590 0x560635b1b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.365+0000 7f63ab5c8b80 4 client.26528 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:09.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.365+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560635a49780 con 0x560635b19590 2024-03-30T04:05:09.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.366+0000 7f6395000700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b19590 0x560635b1b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.367+0000 7f6395000700 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b19590 0x560635b1b980 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77134c014b7c8c9 server_cookie=69ce8eedc8da8ded in_seq=0 out_seq=0 2024-03-30T04:05:09.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.367+0000 7f6376a00700 10 client.26528 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f6376a00700 1 -- 172.21.15.23:0/2459427013 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560635a49780 con 0x560635b19590 2024-03-30T04:05:09.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f6376a00700 10 client.26528 handle_command_reply: tid=0 2024-03-30T04:05:09.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 client.26528 shutdown 2024-03-30T04:05:09.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 msgr2=0x560635b06b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b06b40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f637c00bc60 tx=0x7f637c005660 comp rx=0 tx=0).stop 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 shutdown_connections 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b19590 0x560635b1b980 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635b09e70 0x560635b07280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635afe650 0x560635b098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/2459427013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b06b40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 >> 172.21.15.23:0/2459427013 conn(0x560635a5e720 msgr2=0x560635b00740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.368+0000 7f6376a00700 0 client.26528 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.369+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 shutdown_connections 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.369+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/2459427013 wait complete. 2024-03-30T04:05:09.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.370+0000 7f63ab5c8b80 20 client.26528 trim_cache size 0 max 16384 2024-03-30T04:05:09.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.392+0000 7f63ab5c8b80 1 Processor -- start 2024-03-30T04:05:09.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.392+0000 7f63ab5c8b80 1 -- start start 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b10af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635b07880 con 0x560635b09e70 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635b081b0 con 0x560635afef70 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635afc3b0 con 0x560635afe650 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f6395a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41884/0 (socket says 172.21.15.23:41884) 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 -- 172.21.15.23:0/3162027048 learned_addr learned my addr 172.21.15.23:0/3162027048 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f6395000700 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b10af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 -- 172.21.15.23:0/3162027048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 msgr2=0x560635b10af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b10af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 -- 172.21.15.23:0/3162027048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 msgr2=0x560635b093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.393+0000 7f638fe00700 1 -- 172.21.15.23:0/3162027048 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56063587ebe0 con 0x560635afef70 2024-03-30T04:05:09.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.394+0000 7f638fe00700 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c00bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93829764773a9367 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.394+0000 7f6382a00700 1 -- 172.21.15.23:0/3162027048 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c003d90 con 0x560635afef70 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.394+0000 7f6382a00700 1 -- 172.21.15.23:0/3162027048 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f637c010930 con 0x560635afef70 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.394+0000 7f6382a00700 1 -- 172.21.15.23:0/3162027048 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c022980 con 0x560635afef70 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3162027048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 msgr2=0x560635b1b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f637c0098e0 tx=0x7f637c00bfd0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3162027048 shutdown_connections 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b10af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3162027048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b1b980 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3162027048 >> 172.21.15.23:0/3162027048 conn(0x560635a5e720 msgr2=0x560635af3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3162027048 shutdown_connections 2024-03-30T04:05:09.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3162027048 wait complete. 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.395+0000 7f63ab5c8b80 1 -- start start 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 Processor -- start 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b0f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b0fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635b6d280 con 0x560635b09e70 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635b6d3c0 con 0x560635afef70 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560635b6d550 con 0x560635afe650 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f63ab5c8b80 10 client.0 _ll_register_callbacks cb 0x5606359f64f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f638fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f638fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41890/0 (socket says 172.21.15.23:41890) 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f638fe00700 1 -- 172.21.15.23:0/3746942434 learned_addr learned my addr 172.21.15.23:0/3746942434 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f6395000700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b0f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.396+0000 7f6395a00700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b0fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 -- 172.21.15.23:0/3746942434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 msgr2=0x560635b0f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b0f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 -- 172.21.15.23:0/3746942434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 msgr2=0x560635b0fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b0fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 -- 172.21.15.23:0/3746942434 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6390073360 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f6395000700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b0f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f638fe00700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f637c009990 tx=0x7f637c010750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffb150972582aab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f6376a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c010d70 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f637c027070 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.397+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f637c01e040 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f637c022850 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 10 client.26534 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x560635a16f20 con 0x560635afef70 2024-03-30T04:05:09.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f637c01e040 con 0x560635afef70 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f637c053930 con 0x560635afef70 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 10 client.26534 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 10 client.26534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 20 client.26534 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b6e880 0x560635b70c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 4 client.26534 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56063589fd70 con 0x560635b6e880 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.398+0000 7f6395000700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b6e880 0x560635b70c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.399+0000 7f6395000700 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b6e880 0x560635b70c70 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3814ceeadc1a178 server_cookie=f5a328ce7f0f454a in_seq=0 out_seq=0 2024-03-30T04:05:09.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.399+0000 7f6376a00700 10 client.26534 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f6376a00700 1 -- 172.21.15.23:0/3746942434 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56063589fd70 con 0x560635b6e880 2024-03-30T04:05:09.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f6376a00700 10 client.26534 handle_command_reply: tid=0 2024-03-30T04:05:09.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 client.26534 shutdown 2024-03-30T04:05:09.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 msgr2=0x560635b0f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f637c009990 tx=0x7f637c010750 comp rx=0 tx=0).stop 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 shutdown_connections 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560635b09e70 0x560635b0fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560635b6e880 0x560635b70c70 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f6376a00700 0 client.26534 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560635afe650 0x560635b0f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 --2- 172.21.15.23:0/3746942434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560635afef70 0x560635b0f740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.400+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 >> 172.21.15.23:0/3746942434 conn(0x560635a5e720 msgr2=0x560635b1b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.401+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 shutdown_connections 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.401+0000 7f63ab5c8b80 1 -- 172.21.15.23:0/3746942434 wait complete. 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.401+0000 7f63ab5c8b80 20 client.26534 trim_cache size 0 max 16384 2024-03-30T04:05:09.403 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:09.404 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:05:09.404 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "90f29a0b-34c7-4294-908f-e8fe415b4fed", 2024-03-30T04:05:09.404 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:05:09.404 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:09.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 msgr2=0x7f63780a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 0x7f63780a38a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f639010d4d0 tx=0x7f637c00b560 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 msgr2=0x7f639010bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f638400d950 tx=0x7f638400dd10 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 shutdown_connections 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a13e0 0x7f63780a38a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f639010c820 0x7f63901a9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390100f50 0x7f639010bda0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 --2- 172.21.15.23:0/1699682860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390106070 0x7f639010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 >> 172.21.15.23:0/1699682860 conn(0x7f63900fc7f0 msgr2=0x7f63900fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 shutdown_connections 2024-03-30T04:05:09.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.402+0000 7f6397a00700 1 -- 172.21.15.23:0/1699682860 wait complete. 2024-03-30T04:05:09.420 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:05:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.660+0000 7f063b800700 1 -- 172.21.15.23:0/2001810253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0634069c10 msgr2=0x7f0634105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.660+0000 7f063b800700 1 --2- 172.21.15.23:0/2001810253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0634069c10 0x7f0634105980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f06200099c0 tx=0x7f0620009cd0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 -- 172.21.15.23:0/2001810253 shutdown_connections 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 --2- 172.21.15.23:0/2001810253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06341060b0 0x7f063410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 --2- 172.21.15.23:0/2001810253 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f0634069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 --2- 172.21.15.23:0/2001810253 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0634069c10 0x7f0634105980 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 -- 172.21.15.23:0/2001810253 >> 172.21.15.23:0/2001810253 conn(0x7f0634076d80 msgr2=0x7f0634077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 -- 172.21.15.23:0/2001810253 shutdown_connections 2024-03-30T04:05:09.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.661+0000 7f063b800700 1 -- 172.21.15.23:0/2001810253 wait complete. 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 Processor -- start 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 -- start start 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f06341993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341060b0 0x7f0634199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063419e750 con 0x7f0634069260 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063419e8c0 con 0x7f063419dcd0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.662+0000 7f063b800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063419ea80 con 0x7f06341060b0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0638e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f06341993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41918/0 (socket says 172.21.15.23:41918) 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 -- 172.21.15.23:0/1841050945 learned_addr learned my addr 172.21.15.23:0/1841050945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0633e00700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341060b0 0x7f0634199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 -- 172.21.15.23:0/1841050945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341060b0 msgr2=0x7f0634199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341060b0 0x7f0634199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 -- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 msgr2=0x7f06341993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f06341993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 -- 172.21.15.23:0/1841050945 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0628009520 con 0x7f063419dcd0 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0638e00700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f06341993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.663+0000 7f0639800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f062c00ef30 tx=0x7f062c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.664+0000 7f0631600700 1 -- 172.21.15.23:0/1841050945 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c018070 con 0x7f063419dcd0 2024-03-30T04:05:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.664+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06200095f0 con 0x7f063419dcd0 2024-03-30T04:05:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.664+0000 7f0631600700 1 -- 172.21.15.23:0/1841050945 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f062c00f040 con 0x7f063419dcd0 2024-03-30T04:05:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.664+0000 7f0631600700 1 -- 172.21.15.23:0/1841050945 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c00cdb0 con 0x7f063419dcd0 2024-03-30T04:05:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.664+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06341a6280 con 0x7f063419dcd0 2024-03-30T04:05:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.668+0000 7f0631600700 1 -- 172.21.15.23:0/1841050945 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f062c003a00 con 0x7f063419dcd0 2024-03-30T04:05:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.668+0000 7f064f37cb80 1 Processor -- start 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 -- start start 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd2f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 0x5594fd2fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd2fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd249990 con 0x5594fd2fbdf0 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd247350 con 0x5594fd2f0ef0 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f064f37cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd242c40 con 0x5594fd2f05d0 2024-03-30T04:05:09.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.669+0000 7f0633e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd2f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0633e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd2f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55880/0 (socket says 172.21.15.23:55880) 2024-03-30T04:05:09.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0633e00700 1 -- 172.21.15.23:0/1493327603 learned_addr learned my addr 172.21.15.23:0/1493327603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 0x5594fd2fb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0633e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd2fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 -- 172.21.15.23:0/1493327603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 msgr2=0x5594fd2f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd2f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 -- 172.21.15.23:0/1493327603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 msgr2=0x5594fd2fe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd2fe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0638e00700 1 -- 172.21.15.23:0/1493327603 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594fd0c43b0 con 0x5594fd2f0ef0 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.670+0000 7f0633e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd2fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0631600700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 0x7f061c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0638e00700 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 0x5594fd2fb8b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e50 tx=0x7f06280116c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c176b7022bfa4383 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0631600700 1 -- 172.21.15.23:0/1841050945 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f062c0ccb90 con 0x7f063419dcd0 2024-03-30T04:05:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0633e00700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 0x7f061c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0626000700 1 -- 172.21.15.23:0/1493327603 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628011e20 con 0x5594fd2f0ef0 2024-03-30T04:05:09.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.671+0000 7f0626000700 1 -- 172.21.15.23:0/1493327603 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0628012910 con 0x5594fd2f0ef0 2024-03-30T04:05:09.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.672+0000 7f0626000700 1 -- 172.21.15.23:0/1493327603 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062800f8c0 con 0x5594fd2f0ef0 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.672+0000 7f064f37cb80 1 -- 172.21.15.23:0/1493327603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 msgr2=0x5594fd2fb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.672+0000 7f064f37cb80 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 0x5594fd2fb8b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0628009e50 tx=0x7f06280116c0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.672+0000 7f0633e00700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 0x7f061c0a3750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0620009990 tx=0x7f0620005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 -- 172.21.15.23:0/1493327603 shutdown_connections 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd2fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd2f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 --2- 172.21.15.23:0/1493327603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f0ef0 0x5594fd2fb8b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 -- 172.21.15.23:0/1493327603 >> 172.21.15.23:0/1493327603 conn(0x5594fd2506a0 msgr2=0x5594fd2eca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 -- 172.21.15.23:0/1493327603 shutdown_connections 2024-03-30T04:05:09.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 -- 172.21.15.23:0/1493327603 wait complete. 2024-03-30T04:05:09.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.673+0000 7f064f37cb80 1 -- start start 2024-03-30T04:05:09.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 Processor -- start 2024-03-30T04:05:09.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd1ded80 con 0x5594fd2f0ef0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd1dea20 con 0x5594fd2f05d0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.674+0000 7f064f37cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd0c66a0 con 0x5594fd2fbdf0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f064f37cb80 10 client.0 _ll_register_callbacks cb 0x5594fd1e8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41944/0 (socket says 172.21.15.23:41944) 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 -- 172.21.15.23:0/206890774 learned_addr learned my addr 172.21.15.23:0/206890774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0639800700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0633e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 -- 172.21.15.23:0/206890774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2fbdf0 msgr2=0x5594fd2f9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 -- 172.21.15.23:0/206890774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 msgr2=0x5594fd2f8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0638e00700 1 -- 172.21.15.23:0/206890774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0620020040 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.675+0000 7f0633e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f0638e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f062800a010 tx=0x7f0628012cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f52147b763556f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f061aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0628005040 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f06280051a0 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594fd203380 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0628005360 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.676+0000 7f064f37cb80 10 client.26546 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f0628009a80 con 0x5594fd2f05d0 2024-03-30T04:05:09.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5594fd203380 con 0x5594fd2f05d0 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f0628022040 con 0x5594fd2f05d0 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 10 client.26546 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 10 client.26546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 20 client.26546 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd313fd0 0x5594fd3163f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 4 client.26546 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.677+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594fd23b700 con 0x5594fd313fd0 2024-03-30T04:05:09.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.678+0000 7f0633e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd313fd0 0x5594fd3163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.678+0000 7f0633e00700 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd313fd0 0x5594fd3163f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faf44de60e7f0a09 server_cookie=27c5df7dc09143e in_seq=0 out_seq=0 2024-03-30T04:05:09.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.679+0000 7f061aa00700 10 client.26546 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.679+0000 7f061aa00700 1 -- 172.21.15.23:0/206890774 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5594fd23b700 con 0x5594fd313fd0 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.679+0000 7f061aa00700 10 client.26546 handle_command_reply: tid=0 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.679+0000 7f064f37cb80 1 client.26546 shutdown 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 msgr2=0x5594fd2fb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f062800a010 tx=0x7f0628012cb0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 shutdown_connections 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd313fd0 0x5594fd3163f0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd2fb7d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 --2- 172.21.15.23:0/206890774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 >> 172.21.15.23:0/206890774 conn(0x5594fd2506a0 msgr2=0x5594fd2f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f061aa00700 0 client.26546 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 shutdown_connections 2024-03-30T04:05:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 1 -- 172.21.15.23:0/206890774 wait complete. 2024-03-30T04:05:09.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.680+0000 7f064f37cb80 20 client.26546 trim_cache size 0 max 16384 2024-03-30T04:05:09.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.702+0000 7f064f37cb80 1 Processor -- start 2024-03-30T04:05:09.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 -- start start 2024-03-30T04:05:09.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd310cc0 con 0x5594fd2f0ef0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd2f1640 con 0x5594fd2fbdf0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f064f37cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd2fa0c0 con 0x5594fd2f05d0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f0638e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f0639800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f0638e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:55894/0 (socket says 172.21.15.23:55894) 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.703+0000 7f0638e00700 1 -- 172.21.15.23:0/3084300118 learned_addr learned my addr 172.21.15.23:0/3084300118 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0633e00700 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 -- 172.21.15.23:0/3084300118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 msgr2=0x5594fd302a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 -- 172.21.15.23:0/3084300118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 msgr2=0x5594fd2f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 -- 172.21.15.23:0/3084300118 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594fd070b60 con 0x5594fd2fbdf0 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0638e00700 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0639800700 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f4290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f062c0c9f60 tx=0x7f062c009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee2be5664f178992 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0626a00700 1 -- 172.21.15.23:0/3084300118 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c004ad0 con 0x5594fd2fbdf0 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0626a00700 1 -- 172.21.15.23:0/3084300118 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f062c01b320 con 0x5594fd2fbdf0 2024-03-30T04:05:09.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.704+0000 7f0626a00700 1 -- 172.21.15.23:0/3084300118 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062c099c10 con 0x5594fd2fbdf0 2024-03-30T04:05:09.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 -- 172.21.15.23:0/3084300118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 msgr2=0x5594fd2f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f4290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f062c0c9f60 tx=0x7f062c009370 comp rx=0 tx=0).stop 2024-03-30T04:05:09.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 -- 172.21.15.23:0/3084300118 shutdown_connections 2024-03-30T04:05:09.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2fbdf0 0x5594fd2f4290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f05d0 0x5594fd302a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 --2- 172.21.15.23:0/3084300118 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2f0ef0 0x5594fd2f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 -- 172.21.15.23:0/3084300118 >> 172.21.15.23:0/3084300118 conn(0x5594fd2506a0 msgr2=0x5594fd2e5f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 -- 172.21.15.23:0/3084300118 shutdown_connections 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.705+0000 7f064f37cb80 1 -- 172.21.15.23:0/3084300118 wait complete. 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.706+0000 7f064f37cb80 1 -- start start 2024-03-30T04:05:09.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.706+0000 7f064f37cb80 1 Processor -- start 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f0ef0 0x5594fd300d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd301290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd301810 con 0x5594fd2fbdf0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd301950 con 0x5594fd2f05d0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594fd301b10 con 0x5594fd2f0ef0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f064f37cb80 10 client.0 _ll_register_callbacks cb 0x5594fd1e8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:41960/0 (socket says 172.21.15.23:41960) 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 -- 172.21.15.23:0/2912949752 learned_addr learned my addr 172.21.15.23:0/2912949752 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0639800700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd301290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0633e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f0ef0 0x5594fd300d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 -- 172.21.15.23:0/2912949752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f0ef0 msgr2=0x5594fd300d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f0ef0 0x5594fd300d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 -- 172.21.15.23:0/2912949752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 msgr2=0x5594fd301290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd301290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.707+0000 7f0638e00700 1 -- 172.21.15.23:0/2912949752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0628009a80 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f0638e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0628000be0 tx=0x7f0628009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2995272b585c4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f062800fc50 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0628012910 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0628000e40 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0628011a10 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 10 client.26552 fetch_fsmap learned FSMap version 29 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7f06341a6280 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0628012910 con 0x5594fd2f05d0 2024-03-30T04:05:09.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 29) v1 ==== 7474+0+0 (secure 0 0 0) 0x7f0628011730 con 0x5594fd2f05d0 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 10 client.26552 fetch_fsmap finished waiting for FSMap version 29 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 10 client.26552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.708+0000 7f064f37cb80 20 client.26552 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd369040 0x5594fd36b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f064f37cb80 4 client.26552 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5594fd091cf0 con 0x5594fd369040 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f0633e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd369040 0x5594fd36b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f0633e00700 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd369040 0x5594fd36b460 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65af4c6898c74041 server_cookie=58b71ed8d9b95753 in_seq=0 out_seq=0 2024-03-30T04:05:09.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f061aa00700 10 client.26552 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.709+0000 7f061aa00700 1 -- 172.21.15.23:0/2912949752 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5594fd091cf0 con 0x5594fd369040 2024-03-30T04:05:09.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f061aa00700 10 client.26552 handle_command_reply: tid=0 2024-03-30T04:05:09.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 client.26552 shutdown 2024-03-30T04:05:09.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 msgr2=0x5594fd302920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0628000be0 tx=0x7f0628009750 comp rx=0 tx=0).stop 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 shutdown_connections 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5594fd2fbdf0 0x5594fd301290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5594fd2f05d0 0x5594fd302920 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5594fd369040 0x5594fd36b460 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 --2- 172.21.15.23:0/2912949752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594fd2f0ef0 0x5594fd300d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 >> 172.21.15.23:0/2912949752 conn(0x5594fd2506a0 msgr2=0x5594fd2f3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f061aa00700 0 client.26552 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 shutdown_connections 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.710+0000 7f064f37cb80 1 -- 172.21.15.23:0/2912949752 wait complete. 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f064f37cb80 20 client.26552 trim_cache size 0 max 16384 2024-03-30T04:05:09.714 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "90f29a0b-34c7-4294-908f-e8fe415b4fed": { 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "90f29a0b-34c7-4294-908f-e8fe415b4fed", 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:05:09.715 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 msgr2=0x7f061c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 0x7f061c0a3750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0620009990 tx=0x7f0620005fd0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 msgr2=0x7f063419e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f062c00ef30 tx=0x7f062c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 shutdown_connections 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f061c0a1290 0x7f061c0a3750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f063419dcd0 0x7f063419e150 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0634069260 0x7f06341993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 --2- 172.21.15.23:0/1841050945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06341060b0 0x7f0634199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 >> 172.21.15.23:0/1841050945 conn(0x7f0634076d80 msgr2=0x7f0634107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.711+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 shutdown_connections 2024-03-30T04:05:09.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:09.712+0000 7f063b800700 1 -- 172.21.15.23:0/1841050945 wait complete. 2024-03-30T04:05:09.731 INFO:tasks.cephfs.filesystem:scrub status for tag:90f29a0b-34c7-4294-908f-e8fe415b4fed - {'path': '/', 'tag': '90f29a0b-34c7-4294-908f-e8fe415b4fed', 'options': 'recursive,force'} 2024-03-30T04:05:09.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:09 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:05:09.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:09 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:09.930 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:09 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:05:09.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:09 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:09.992 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:09 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:05:09.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:09 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:10.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 229646 11.71 MB/sec warmup 107 sec latency 16.941 ms 2024-03-30T04:05:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:10 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:05:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:10 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:10 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:05:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:10 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:10.931 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:10 smithi154 ceph-mon[79161]: pgmap v100: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 218 op/s 2024-03-30T04:05:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:10 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:05:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:10 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:10 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:05:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:10 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:10.993 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:10 smithi193 ceph-mon[79581]: pgmap v100: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 218 op/s 2024-03-30T04:05:11.021 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:10 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:05:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:10 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:10 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:05:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:10 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:11.022 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:10 smithi023 ceph-mon[38254]: pgmap v100: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 218 op/s 2024-03-30T04:05:11.192 INFO:tasks.workunit.client.0.smithi023.stdout: 1 231518 11.71 MB/sec warmup 108 sec latency 11.064 ms 2024-03-30T04:05:12.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 234161 11.76 MB/sec warmup 109 sec latency 82.162 ms 2024-03-30T04:05:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:12 smithi193 ceph-mon[79581]: pgmap v101: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-30T04:05:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:12 smithi023 ceph-mon[38254]: pgmap v101: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-30T04:05:12.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:12 smithi154 ceph-mon[79161]: pgmap v101: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 319 op/s 2024-03-30T04:05:13.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 235677 11.71 MB/sec warmup 110 sec latency 17.858 ms 2024-03-30T04:05:14.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 238158 11.73 MB/sec warmup 111 sec latency 11.011 ms 2024-03-30T04:05:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:14 smithi193 ceph-mon[79581]: pgmap v102: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:14 smithi023 ceph-mon[38254]: pgmap v102: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:14 smithi154 ceph-mon[79161]: pgmap v102: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:15.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 240705 11.74 MB/sec warmup 112 sec latency 12.227 ms 2024-03-30T04:05:16.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 242393 11.71 MB/sec warmup 113 sec latency 14.268 ms 2024-03-30T04:05:17.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 244898 11.75 MB/sec warmup 114 sec latency 84.375 ms 2024-03-30T04:05:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:17 smithi154 ceph-mon[79161]: pgmap v103: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:17.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:17 smithi193 ceph-mon[79581]: pgmap v103: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:17 smithi023 ceph-mon[38254]: pgmap v103: 129 pgs: 129 active+clean; 137 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 200 op/s 2024-03-30T04:05:18.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 246379 11.70 MB/sec warmup 115 sec latency 61.991 ms 2024-03-30T04:05:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:18 smithi193 ceph-mon[79581]: pgmap v104: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-30T04:05:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:18 smithi023 ceph-mon[38254]: pgmap v104: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-30T04:05:18.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:18 smithi154 ceph-mon[79161]: pgmap v104: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.5 MiB/s wr, 306 op/s 2024-03-30T04:05:19.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 248801 11.72 MB/sec warmup 116 sec latency 26.575 ms 2024-03-30T04:05:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:19 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:05:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:19 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:05:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:19 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:05:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:19 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:05:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:19 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:05:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:19 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:05:20.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 251502 11.73 MB/sec warmup 117 sec latency 11.727 ms 2024-03-30T04:05:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:20 smithi193 ceph-mon[79581]: pgmap v105: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:05:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:20 smithi023 ceph-mon[38254]: pgmap v105: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:05:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:20 smithi154 ceph-mon[79161]: pgmap v105: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:05:21.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 253298 11.73 MB/sec warmup 118 sec latency 9.132 ms 2024-03-30T04:05:22.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 255716 11.74 MB/sec warmup 119 sec latency 64.012 ms 2024-03-30T04:05:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:22 smithi193 ceph-mon[79581]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 311 op/s 2024-03-30T04:05:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:22 smithi023 ceph-mon[38254]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 311 op/s 2024-03-30T04:05:22.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:22 smithi154 ceph-mon[79161]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 311 op/s 2024-03-30T04:05:24.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 259759 12.37 MB/sec execute 1 sec latency 14.531 ms 2024-03-30T04:05:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:24 smithi193 ceph-mon[79581]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:24 smithi023 ceph-mon[38254]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:24 smithi154 ceph-mon[79161]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:25.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 262611 14.83 MB/sec execute 2 sec latency 12.306 ms 2024-03-30T04:05:26.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 264161 12.11 MB/sec execute 3 sec latency 10.729 ms 2024-03-30T04:05:27.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 266455 12.55 MB/sec execute 4 sec latency 84.129 ms 2024-03-30T04:05:27.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:27 smithi154 ceph-mon[79161]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:27.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:27 smithi193 ceph-mon[79581]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:27.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:27 smithi023 ceph-mon[38254]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.2 MiB/s wr, 209 op/s 2024-03-30T04:05:28.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 268818 12.24 MB/sec execute 5 sec latency 9.521 ms 2024-03-30T04:05:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:28 smithi193 ceph-mon[79581]: pgmap v109: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-03-30T04:05:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:28 smithi023 ceph-mon[38254]: pgmap v109: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-03-30T04:05:28.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:28 smithi154 ceph-mon[79161]: pgmap v109: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 313 op/s 2024-03-30T04:05:29.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 270675 11.88 MB/sec execute 6 sec latency 8.395 ms 2024-03-30T04:05:30.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 273459 12.59 MB/sec execute 7 sec latency 14.496 ms 2024-03-30T04:05:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:30 smithi193 ceph-mon[79581]: pgmap v110: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-30T04:05:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:30 smithi023 ceph-mon[38254]: pgmap v110: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-30T04:05:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:30 smithi154 ceph-mon[79161]: pgmap v110: 129 pgs: 129 active+clean; 159 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 208 op/s 2024-03-30T04:05:31.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 275910 12.43 MB/sec execute 8 sec latency 8.473 ms 2024-03-30T04:05:32.193 INFO:tasks.workunit.client.0.smithi023.stdout: 1 277483 12.00 MB/sec execute 9 sec latency 29.460 ms 2024-03-30T04:05:32.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:32 smithi193 ceph-mon[79581]: pgmap v111: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-03-30T04:05:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:32 smithi023 ceph-mon[38254]: pgmap v111: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-03-30T04:05:32.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:32 smithi154 ceph-mon[79161]: pgmap v111: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 306 op/s 2024-03-30T04:05:33.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 280250 12.50 MB/sec execute 10 sec latency 14.874 ms 2024-03-30T04:05:34.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 282136 12.01 MB/sec execute 11 sec latency 11.383 ms 2024-03-30T04:05:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:34 smithi193 ceph-mon[79581]: pgmap v112: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:34 smithi023 ceph-mon[38254]: pgmap v112: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:34.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:34 smithi154 ceph-mon[79161]: pgmap v112: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:35.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 284342 12.15 MB/sec execute 12 sec latency 61.181 ms 2024-03-30T04:05:36.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 287090 12.32 MB/sec execute 13 sec latency 13.908 ms 2024-03-30T04:05:37.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 288373 11.92 MB/sec execute 14 sec latency 45.276 ms 2024-03-30T04:05:37.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:37 smithi154 ceph-mon[79161]: pgmap v113: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:37.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:37 smithi193 ceph-mon[79581]: pgmap v113: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:37.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:37 smithi023 ceph-mon[38254]: pgmap v113: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 202 op/s 2024-03-30T04:05:38.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 291035 12.18 MB/sec execute 15 sec latency 19.339 ms 2024-03-30T04:05:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:38 smithi193 ceph-mon[79581]: pgmap v114: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-03-30T04:05:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:38 smithi023 ceph-mon[38254]: pgmap v114: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-03-30T04:05:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:38 smithi154 ceph-mon[79161]: pgmap v114: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 294 op/s 2024-03-30T04:05:39.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 293272 11.94 MB/sec execute 16 sec latency 7.525 ms 2024-03-30T04:05:39.732 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 -- 172.21.15.23:0/1863043294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 msgr2=0x7f3e101077d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1863043294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e101077d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc009990 tx=0x7f3dfc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 -- 172.21.15.23:0/1863043294 shutdown_connections 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1863043294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e10107d10 0x7f3e101141b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1863043294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e1010d410 0x7f3e1010d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1863043294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e101077d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 -- 172.21.15.23:0/1863043294 >> 172.21.15.23:0/1863043294 conn(0x7f3e10076ba0 msgr2=0x7f3e10105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 -- 172.21.15.23:0/1863043294 shutdown_connections 2024-03-30T04:05:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.939+0000 7f3e14a00700 1 -- 172.21.15.23:0/1863043294 wait complete. 2024-03-30T04:05:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.941+0000 7f3e14a00700 1 Processor -- start 2024-03-30T04:05:39.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.941+0000 7f3e14a00700 1 -- start start 2024-03-30T04:05:39.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.941+0000 7f3e14a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e1019d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e14a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 0x7f3e1019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e14a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e1010d410 0x7f3e101a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e14a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e101a2600 con 0x7f3e1010d410 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e14a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e101a2770 con 0x7f3e10107350 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e14a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e101a2930 con 0x7f3e10107d10 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e1019d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 0x7f3e1019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e1019d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42378/0 (socket says 172.21.15.23:42378) 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0de00700 1 -- 172.21.15.23:0/1868553006 learned_addr learned my addr 172.21.15.23:0/1868553006 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:39.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0e800700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e1010d410 0x7f3e101a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 -- 172.21.15.23:0/1868553006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 msgr2=0x7f3e1019d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e1019d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 -- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e1010d410 msgr2=0x7f3e101a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e1010d410 0x7f3e101a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.942+0000 7f3e0d400700 1 -- 172.21.15.23:0/1868553006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc0095f0 con 0x7f3e10107d10 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e0d400700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 0x7f3e1019dc20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0059d0 tx=0x7f3dfc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e06a00700 1 -- 172.21.15.23:0/1868553006 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc01d070 con 0x7f3e10107d10 2024-03-30T04:05:39.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e101a2bb0 con 0x7f3e10107d10 2024-03-30T04:05:39.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e06a00700 1 -- 172.21.15.23:0/1868553006 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dfc022470 con 0x7f3e10107d10 2024-03-30T04:05:39.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e06a00700 1 -- 172.21.15.23:0/1868553006 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc017650 con 0x7f3e10107d10 2024-03-30T04:05:39.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.943+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101aa450 con 0x7f3e10107d10 2024-03-30T04:05:39.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.948+0000 7f3e06a00700 1 -- 172.21.15.23:0/1868553006 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3dfc0225e0 con 0x7f3e10107d10 2024-03-30T04:05:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.949+0000 7f3e28614b80 1 Processor -- start 2024-03-30T04:05:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.949+0000 7f3e06a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 0x7f3df40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.949+0000 7f3e28614b80 1 -- start start 2024-03-30T04:05:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.949+0000 7f3e06a00700 1 -- 172.21.15.23:0/1868553006 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3dfc0ce110 con 0x7f3e10107d10 2024-03-30T04:05:39.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.949+0000 7f3e0de00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 0x7f3df40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a129b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a201e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c3696b990 con 0x562c36a1ddf0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36969350 con 0x562c36a125d0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e28614b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36964c40 con 0x562c36a12ef0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50426/0 (socket says 172.21.15.23:50426) 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a129b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42382/0 (socket says 172.21.15.23:42382) 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 -- 172.21.15.23:0/987148571 learned_addr learned my addr 172.21.15.23:0/987148571 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0e800700 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a201e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 -- 172.21.15.23:0/987148571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 msgr2=0x562c36a129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 -- 172.21.15.23:0/987148571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 msgr2=0x562c36a201e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a201e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0d400700 1 -- 172.21.15.23:0/987148571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c367e63b0 con 0x562c36a12ef0 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.950+0000 7f3e0e800700 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.951+0000 7f3e0de00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 0x7f3df40a3750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e00011630 tx=0x7f3e000119f0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.951+0000 7f3e0d400700 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0979a0 tx=0x7f3dfc09ad50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e9b4688cdfcecd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.951+0000 7f3dfb000700 1 -- 172.21.15.23:0/987148571 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc09b630 con 0x562c36a12ef0 2024-03-30T04:05:39.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.951+0000 7f3dfb000700 1 -- 172.21.15.23:0/987148571 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dfc09af30 con 0x562c36a12ef0 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.951+0000 7f3dfb000700 1 -- 172.21.15.23:0/987148571 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc09b3d0 con 0x562c36a12ef0 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 -- 172.21.15.23:0/987148571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 msgr2=0x562c36a1d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0979a0 tx=0x7f3dfc09ad50 comp rx=0 tx=0).stop 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 -- 172.21.15.23:0/987148571 shutdown_connections 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a201e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 --2- 172.21.15.23:0/987148571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1d8b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 -- 172.21.15.23:0/987148571 >> 172.21.15.23:0/987148571 conn(0x562c369726a0 msgr2=0x562c36a0ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:39.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 -- 172.21.15.23:0/987148571 shutdown_connections 2024-03-30T04:05:39.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.952+0000 7f3e28614b80 1 -- 172.21.15.23:0/987148571 wait complete. 2024-03-30T04:05:39.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.953+0000 7f3e28614b80 1 -- start start 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.953+0000 7f3e28614b80 1 Processor -- start 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a29250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36900d80 con 0x562c36a1ddf0 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36900a20 con 0x562c36a125d0 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c367e86a0 con 0x562c36a12ef0 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e28614b80 10 client.0 _ll_register_callbacks cb 0x562c3690a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:39.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a29250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42396/0 (socket says 172.21.15.23:42396) 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 -- 172.21.15.23:0/3245255181 learned_addr learned my addr 172.21.15.23:0/3245255181 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 -- 172.21.15.23:0/3245255181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 msgr2=0x562c36a1e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 -- 172.21.15.23:0/3245255181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 msgr2=0x562c36a29250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a29250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.954+0000 7f3e0de00700 1 -- 172.21.15.23:0/3245255181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101aa450 con 0x562c36a125d0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3e0de00700 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009ce0 tx=0x7f3e000047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a2fa335d271165 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3def400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00029670 con 0x562c36a125d0 2024-03-30T04:05:39.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c36925380 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3e00009de0 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e000297d0 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3e28614b80 10 client.26564 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.955+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562c36792b60 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562c36925380 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3e00020bd0 con 0x562c36a125d0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 10 client.26564 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 20 client.26564 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a2d5b0 0x562c36a2f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 4 client.26564 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c3695d700 con 0x562c36a2d5b0 2024-03-30T04:05:39.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e0d400700 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a2d5b0 0x562c36a2f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3e0d400700 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a2d5b0 0x562c36a2f9d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb68388c21135d9a server_cookie=2d89f0613fe1d10c in_seq=0 out_seq=0 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.956+0000 7f3def400700 10 client.26564 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3def400700 1 -- 172.21.15.23:0/3245255181 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562c3695d700 con 0x562c36a2d5b0 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3def400700 10 client.26564 handle_command_reply: tid=0 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 client.26564 shutdown 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 msgr2=0x562c36a1e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009ce0 tx=0x7f3e000047c0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 shutdown_connections 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a29250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a1e1d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a2d5b0 0x562c36a2f9d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 --2- 172.21.15.23:0/3245255181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a1e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 >> 172.21.15.23:0/3245255181 conn(0x562c369726a0 msgr2=0x562c36a146c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3def400700 0 client.26564 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.957+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 shutdown_connections 2024-03-30T04:05:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.958+0000 7f3e28614b80 1 -- 172.21.15.23:0/3245255181 wait complete. 2024-03-30T04:05:39.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.958+0000 7f3e28614b80 20 client.26564 trim_cache size 0 max 16384 2024-03-30T04:05:39.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 Processor -- start 2024-03-30T04:05:39.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 -- start start 2024-03-30T04:05:39.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a18400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a18940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a1b630 con 0x562c36a1ddf0 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a29a80 con 0x562c36a125d0 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.977+0000 7f3e28614b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a0f660 con 0x562c36a12ef0 2024-03-30T04:05:39.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a18400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42408/0 (socket says 172.21.15.23:42408) 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 -- 172.21.15.23:0/1239832531 learned_addr learned my addr 172.21.15.23:0/1239832531 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0e800700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a18940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 -- 172.21.15.23:0/1239832531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 msgr2=0x562c36a18400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a18400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 -- 172.21.15.23:0/1239832531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 msgr2=0x562c36a18940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a18940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 -- 172.21.15.23:0/1239832531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c3692aea0 con 0x562c36a125d0 2024-03-30T04:05:39.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0d400700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a18400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:39.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0e800700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a18940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:39.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3e0de00700 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e00004770 tx=0x7f3e000092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a3bed645fcdbb2c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3dfba00700 1 -- 172.21.15.23:0/1239832531 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00009e60 con 0x562c36a125d0 2024-03-30T04:05:39.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3dfba00700 1 -- 172.21.15.23:0/1239832531 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e0000e4a0 con 0x562c36a125d0 2024-03-30T04:05:39.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.978+0000 7f3dfba00700 1 -- 172.21.15.23:0/1239832531 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000edf0 con 0x562c36a125d0 2024-03-30T04:05:39.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 -- 172.21.15.23:0/1239832531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 msgr2=0x562c36a22f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:39.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3e00004770 tx=0x7f3e000092d0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 -- 172.21.15.23:0/1239832531 shutdown_connections 2024-03-30T04:05:39.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a18940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a125d0 0x562c36a22f20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 --2- 172.21.15.23:0/1239832531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a12ef0 0x562c36a18400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 -- 172.21.15.23:0/1239832531 >> 172.21.15.23:0/1239832531 conn(0x562c369726a0 msgr2=0x562c36a1e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 -- 172.21.15.23:0/1239832531 shutdown_connections 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.979+0000 7f3e28614b80 1 -- 172.21.15.23:0/1239832531 wait complete. 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.980+0000 7f3e28614b80 1 -- start start 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.980+0000 7f3e28614b80 1 Processor -- start 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 0x562c36a22ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a12ef0 0x562c36a21500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a21a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a21f80 con 0x562c36a1ddf0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a220f0 con 0x562c36a12ef0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c36a222b0 con 0x562c36a125d0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e28614b80 10 client.0 _ll_register_callbacks cb 0x562c3690a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a21a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a12ef0 0x562c36a21500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 0x562c36a22ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a21a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49562/0 (socket says 172.21.15.23:49562) 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0e800700 1 -- 172.21.15.23:0/496647112 learned_addr learned my addr 172.21.15.23:0/496647112 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 -- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a12ef0 msgr2=0x562c36a21500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a12ef0 0x562c36a21500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 -- 172.21.15.23:0/496647112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 msgr2=0x562c36a21a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a21a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.981+0000 7f3e0de00700 1 -- 172.21.15.23:0/496647112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c367e63b0 con 0x562c36a125d0 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3e0de00700 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 0x562c36a22ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009410 tx=0x7f3e0000fd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a9a4537e2046999 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3def400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e00020620 con 0x562c36a125d0 2024-03-30T04:05:39.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3e00020780 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3dfc09af30 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e0000ee80 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3e28614b80 10 client.26572 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.982+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3e0002e040 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dfc09af30 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3e00036b20 con 0x562c36a125d0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 10 client.26572 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 10 client.26572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 20 client.26572 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a82410 0x562c36a84800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 4 client.26572 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562c367b3cf0 con 0x562c36a82410 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e0d400700 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a82410 0x562c36a84800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3e0d400700 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a82410 0x562c36a84800 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5078e5e9209a097e server_cookie=dde71788e5db5cb5 in_seq=0 out_seq=0 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.983+0000 7f3def400700 10 client.26572 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3def400700 1 -- 172.21.15.23:0/496647112 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562c367b3cf0 con 0x562c36a82410 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3def400700 10 client.26572 handle_command_reply: tid=0 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 client.26572 shutdown 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 msgr2=0x562c36a22ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 0x562c36a22ef0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e00009410 tx=0x7f3e0000fd10 comp rx=0 tx=0).stop 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 shutdown_connections 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562c36a1ddf0 0x562c36a21a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562c36a82410 0x562c36a84800 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562c36a125d0 0x562c36a22ef0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 --2- 172.21.15.23:0/496647112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562c36a12ef0 0x562c36a21500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3def400700 0 client.26572 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 >> 172.21.15.23:0/496647112 conn(0x562c369726a0 msgr2=0x562c36a18400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 shutdown_connections 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 1 -- 172.21.15.23:0/496647112 wait complete. 2024-03-30T04:05:40.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.984+0000 7f3e28614b80 20 client.26572 trim_cache size 0 max 16384 2024-03-30T04:05:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:05:40.003 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:05:40.003 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 msgr2=0x7f3df40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 0x7f3df40a3750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e00011630 tx=0x7f3e000119f0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 msgr2=0x7f3e1019dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 0x7f3e1019dc20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0059d0 tx=0x7f3dfc004730 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 shutdown_connections 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e1010d410 0x7f3e101a1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3df40a1290 0x7f3df40a3750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e10107350 0x7f3e1019d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 --2- 172.21.15.23:0/1868553006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e10107d10 0x7f3e1019dc20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 >> 172.21.15.23:0/1868553006 conn(0x7f3e10076ba0 msgr2=0x7f3e10077f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 shutdown_connections 2024-03-30T04:05:40.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:39.985+0000 7f3e14a00700 1 -- 172.21.15.23:0/1868553006 wait complete. 2024-03-30T04:05:40.006 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:05:40.006 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:05:40.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 295035 11.95 MB/sec execute 17 sec latency 34.155 ms 2024-03-30T04:05:40.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 -- 172.21.15.23:0/3185308549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501026e0 msgr2=0x7f4a5010abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 --2- 172.21.15.23:0/3185308549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501026e0 0x7f4a5010abc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009990 tx=0x7f4a3c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 -- 172.21.15.23:0/3185308549 shutdown_connections 2024-03-30T04:05:40.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 --2- 172.21.15.23:0/3185308549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a5010b230 0x7f4a5010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 --2- 172.21.15.23:0/3185308549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a50101dc0 0x7f4a501021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 --2- 172.21.15.23:0/3185308549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a501026e0 0x7f4a5010abc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 -- 172.21.15.23:0/3185308549 >> 172.21.15.23:0/3185308549 conn(0x7f4a500fb620 msgr2=0x7f4a500fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 -- 172.21.15.23:0/3185308549 shutdown_connections 2024-03-30T04:05:40.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.217+0000 7f4a55e00700 1 -- 172.21.15.23:0/3185308549 wait complete. 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.218+0000 7f4a55e00700 1 Processor -- start 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.218+0000 7f4a55e00700 1 -- start start 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a50101dc0 0x7f4a500ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a501026e0 0x7f4a500fff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a501a54f0 con 0x7f4a501026e0 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a501a5660 con 0x7f4a5010b230 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a55e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a501a5820 con 0x7f4a50101dc0 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a501026e0 0x7f4a500fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42434/0 (socket says 172.21.15.23:42434) 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 -- 172.21.15.23:0/934837215 learned_addr learned my addr 172.21.15.23:0/934837215 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4ea00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a50101dc0 0x7f4a500ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 -- 172.21.15.23:0/934837215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a50101dc0 msgr2=0x7f4a500ffa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a50101dc0 0x7f4a500ffa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 -- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a501026e0 msgr2=0x7f4a500fff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a501026e0 0x7f4a500fff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.219+0000 7f4a4f400700 1 -- 172.21.15.23:0/934837215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0095f0 con 0x7f4a5010b230 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a4f400700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800ef30 tx=0x7f4a4800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a48018070 con 0x7f4a5010b230 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a4800f040 con 0x7f4a5010b230 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a501a5aa0 con 0x7f4a5010b230 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4800cd00 con 0x7f4a5010b230 2024-03-30T04:05:40.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.220+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a501a6000 con 0x7f4a5010b230 2024-03-30T04:05:40.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.221+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a50106310 con 0x7f4a5010b230 2024-03-30T04:05:40.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.223+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f4a48003a00 con 0x7f4a5010b230 2024-03-30T04:05:40.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.224+0000 7f4a54a00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 0x7f4a340a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.225+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f4a480ccbd0 con 0x7f4a5010b230 2024-03-30T04:05:40.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.225+0000 7f4a4ea00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 0x7f4a340a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.225+0000 7f4a4ea00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 0x7f4a340a37d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009c60 tx=0x7f4a40009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.228+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a48095d30 con 0x7f4a5010b230 2024-03-30T04:05:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:40 smithi193 ceph-mon[79581]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:05:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:40 smithi193 ceph-mon[79581]: pgmap v115: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-30T04:05:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:40 smithi023 ceph-mon[38254]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:05:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:40 smithi023 ceph-mon[38254]: pgmap v115: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-30T04:05:40.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.550+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a50100f90 con 0x7f4a5010b230 2024-03-30T04:05:40.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.551+0000 7f4a54a00700 1 -- 172.21.15.23:0/934837215 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f4a48099350 con 0x7f4a5010b230 2024-03-30T04:05:40.551 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:05:40.551 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 msgr2=0x7f4a340a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 0x7f4a340a37d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009c60 tx=0x7f4a40009260 comp rx=0 tx=0).stop 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 msgr2=0x7f4a50100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4a4800ef30 tx=0x7f4a4800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 shutdown_connections 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a340a1310 0x7f4a340a37d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a5010b230 0x7f4a50100490 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a50101dc0 0x7f4a500ffa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 --2- 172.21.15.23:0/934837215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a501026e0 0x7f4a500fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 >> 172.21.15.23:0/934837215 conn(0x7f4a500fb620 msgr2=0x7f4a500fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 shutdown_connections 2024-03-30T04:05:40.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.554+0000 7f4a55e00700 1 -- 172.21.15.23:0/934837215 wait complete. 2024-03-30T04:05:40.555 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:05:40.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:05:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:40 smithi154 ceph-mon[79161]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:05:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:40 smithi154 ceph-mon[79161]: pgmap v115: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 189 op/s 2024-03-30T04:05:40.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 -- 172.21.15.23:0/1777601101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 msgr2=0x7f1cd810dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 --2- 172.21.15.23:0/1777601101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd810dc00 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000b0a0 tx=0x7f1cd000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 -- 172.21.15.23:0/1777601101 shutdown_connections 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 --2- 172.21.15.23:0/1777601101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd810dc00 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 --2- 172.21.15.23:0/1777601101 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd8100f60 0x7f1cd8101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 --2- 172.21.15.23:0/1777601101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8101910 0x7f1cd8105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 -- 172.21.15.23:0/1777601101 >> 172.21.15.23:0/1777601101 conn(0x7f1cd80fc7c0 msgr2=0x7f1cd80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 -- 172.21.15.23:0/1777601101 shutdown_connections 2024-03-30T04:05:40.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.787+0000 7f1cdd400700 1 -- 172.21.15.23:0/1777601101 wait complete. 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.788+0000 7f1cdd400700 1 Processor -- start 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.788+0000 7f1cdd400700 1 -- start start 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.788+0000 7f1cdd400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.788+0000 7f1cdd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd81998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.788+0000 7f1cdd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd819dc90 0x7f1cd819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cdd400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd819e710 con 0x7f1cd819dc90 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cdd400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd819e880 con 0x7f1cd8106080 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cdd400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd819ea40 con 0x7f1cd8100f60 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50460/0 (socket says 172.21.15.23:50460) 2024-03-30T04:05:40.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 -- 172.21.15.23:0/2933706515 learned_addr learned my addr 172.21.15.23:0/2933706515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd81998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd7200700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd819dc90 0x7f1cd819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 -- 172.21.15.23:0/2933706515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 msgr2=0x7f1cd81998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd81998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 -- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd819dc90 msgr2=0x7f1cd819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd819dc90 0x7f1cd819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 -- 172.21.15.23:0/2933706515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd0009d00 con 0x7f1cd8100f60 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd81998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.789+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800e9a0 tx=0x7f1cc800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.790+0000 7f1ccf400700 1 -- 172.21.15.23:0/2933706515 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8009790 con 0x7f1cd8100f60 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.790+0000 7f1ccf400700 1 -- 172.21.15.23:0/2933706515 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc8010430 con 0x7f1cd8100f60 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.790+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd81a5fa0 con 0x7f1cd8100f60 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.790+0000 7f1ccf400700 1 -- 172.21.15.23:0/2933706515 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8018680 con 0x7f1cd8100f60 2024-03-30T04:05:40.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.790+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd81a6420 con 0x7f1cd8100f60 2024-03-30T04:05:40.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.795+0000 7f1ccf400700 1 -- 172.21.15.23:0/2933706515 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1cc80105a0 con 0x7f1cd8100f60 2024-03-30T04:05:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.795+0000 7f1cf0fdbb80 1 Processor -- start 2024-03-30T04:05:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 -- start start 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d8762128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d8762151e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876160990 con 0x55d8762075d0 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d87615e350 con 0x55d876212df0 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876159c40 con 0x55d876207ef0 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1ccf400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 0x7f1cbc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cd7200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d8762128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cd5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d8762151e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d8762128b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50474/0 (socket says 172.21.15.23:50474) 2024-03-30T04:05:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.796+0000 7f1cd6800700 1 -- 172.21.15.23:0/3896802055 learned_addr learned my addr 172.21.15.23:0/3896802055 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1ccf400700 1 -- 172.21.15.23:0/2933706515 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1cc8014070 con 0x7f1cd8100f60 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 -- 172.21.15.23:0/3896802055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 msgr2=0x55d8762128b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d8762128b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 -- 172.21.15.23:0/3896802055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 msgr2=0x55d8762079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 -- 172.21.15.23:0/3896802055 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d875fdb3b0 con 0x55d876212df0 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd7200700 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd7200700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 0x7f1cbc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d8762151e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1cd806a380 tx=0x7f1cc4011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3491c79c4b4981f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cc3400700 1 -- 172.21.15.23:0/3896802055 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4004230 con 0x55d876212df0 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cc3400700 1 -- 172.21.15.23:0/3896802055 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc4004cf0 con 0x55d876212df0 2024-03-30T04:05:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.797+0000 7f1cc3400700 1 -- 172.21.15.23:0/3896802055 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4003680 con 0x55d876212df0 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/3896802055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 msgr2=0x55d8762151e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cd7200700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 0x7f1cbc0a3750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009610 tx=0x7f1cd00099e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d8762151e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1cd806a380 tx=0x7f1cc4011720 comp rx=0 tx=0).stop 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/3896802055 shutdown_connections 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d8762151e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/3896802055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d8762128b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/3896802055 >> 172.21.15.23:0/3896802055 conn(0x55d8761676a0 msgr2=0x55d876203a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/3896802055 shutdown_connections 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.798+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/3896802055 wait complete. 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.799+0000 7f1cf0fdbb80 1 -- start start 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.799+0000 7f1cf0fdbb80 1 Processor -- start 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d87620fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 0x55d8762102a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8760f5d80 con 0x55d8762075d0 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8760f5a20 con 0x55d876207ef0 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d875fdd6a0 con 0x55d876212df0 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cf0fdbb80 10 client.0 _ll_register_callbacks cb 0x55d8760ff470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49620/0 (socket says 172.21.15.23:49620) 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd6800700 1 -- 172.21.15.23:0/1642693849 learned_addr learned my addr 172.21.15.23:0/1642693849 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 0x55d8762102a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d87620fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 -- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 msgr2=0x55d87620fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d87620fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 -- 172.21.15.23:0/1642693849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 msgr2=0x55d876212820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd7200700 1 -- 172.21.15.23:0/1642693849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc4009c90 con 0x55d876212df0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.800+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 0x55d8762102a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009fd0 tx=0x7f1cd0006230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b9017e4b4191ae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cb7e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000e040 con 0x55d876212df0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d87611a380 con 0x55d876212df0 2024-03-30T04:05:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1cd00076e0 con 0x55d876212df0 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cd00078a0 con 0x55d876212df0 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cf0fdbb80 10 client.26590 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d875f87b60 con 0x55d876212df0 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.801+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d87611a380 con 0x55d876212df0 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1cd001c410 con 0x55d876212df0 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 10 client.26590 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 10 client.26590 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 20 client.26590 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876222900 0x55d876224cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 4 client.26590 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d876152700 con 0x55d876222900 2024-03-30T04:05:40.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.802+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876222900 0x55d876224cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.803+0000 7f1cb7e00700 10 client.26590 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.803+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876222900 0x55d876224cf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=320945c93564bdb9 server_cookie=f833b8ca3686dac4 in_seq=0 out_seq=0 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.803+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1642693849 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d876152700 con 0x55d876222900 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.803+0000 7f1cb7e00700 10 client.26590 handle_command_reply: tid=0 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.803+0000 7f1cf0fdbb80 1 client.26590 shutdown 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 msgr2=0x55d8762102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 0x55d8762102a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009fd0 tx=0x7f1cd0006230 comp rx=0 tx=0).stop 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 shutdown_connections 2024-03-30T04:05:40.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876222900 0x55d876224cf0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876212df0 0x55d8762102a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d876212820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1642693849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d87620fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 >> 172.21.15.23:0/1642693849 conn(0x55d8761676a0 msgr2=0x55d876209530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cb7e00700 0 client.26590 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 shutdown_connections 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1642693849 wait complete. 2024-03-30T04:05:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.804+0000 7f1cf0fdbb80 20 client.26590 trim_cache size 0 max 16384 2024-03-30T04:05:40.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.820+0000 7f1cf0fdbb80 1 Processor -- start 2024-03-30T04:05:40.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.820+0000 7f1cf0fdbb80 1 -- start start 2024-03-30T04:05:40.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d876202f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 0x55d876219aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876219fe0 con 0x55d876212df0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d87621ee40 con 0x55d876207ef0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876204660 con 0x55d8762075d0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50500/0 (socket says 172.21.15.23:50500) 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 -- 172.21.15.23:0/2616870042 learned_addr learned my addr 172.21.15.23:0/2616870042 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd7200700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 0x55d876219aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd5e00700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d876202f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 -- 172.21.15.23:0/2616870042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 msgr2=0x55d876202f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d876202f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 -- 172.21.15.23:0/2616870042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 msgr2=0x55d876219aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.821+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 0x55d876219aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cd6800700 1 -- 172.21.15.23:0/2616870042 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d876207440 con 0x55d8762075d0 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cd7200700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 0x55d876219aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cd6800700 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8099f40 tx=0x7f1cc80cf4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c25d3f03663617b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cc3e00700 1 -- 172.21.15.23:0/2616870042 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc809dde0 con 0x55d8762075d0 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cc3e00700 1 -- 172.21.15.23:0/2616870042 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc80a2430 con 0x55d8762075d0 2024-03-30T04:05:40.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.822+0000 7f1cc3e00700 1 -- 172.21.15.23:0/2616870042 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc80994a0 con 0x55d8762075d0 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/2616870042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 msgr2=0x55d8762079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8099f40 tx=0x7f1cc80cf4b0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/2616870042 shutdown_connections 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d876212df0 0x55d876219aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8762075d0 0x55d8762079b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/2616870042 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876207ef0 0x55d876202f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/2616870042 >> 172.21.15.23:0/2616870042 conn(0x55d8761676a0 msgr2=0x55d8761fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/2616870042 shutdown_connections 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/2616870042 wait complete. 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 -- start start 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.823+0000 7f1cf0fdbb80 1 Processor -- start 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d876219a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876218360 con 0x55d8762075d0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8762184d0 con 0x55d876212df0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d876218690 con 0x55d876207ef0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cf0fdbb80 10 client.0 _ll_register_callbacks cb 0x55d8760ff470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cd6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cd5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d876219a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.824+0000 7f1cd7200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d876219a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50502/0 (socket says 172.21.15.23:50502) 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42484/0 (socket says 172.21.15.23:42484) 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd5e00700 1 -- 172.21.15.23:0/1539164858 learned_addr learned my addr 172.21.15.23:0/1539164858 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 -- 172.21.15.23:0/1539164858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 msgr2=0x55d876219a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d876219a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 -- 172.21.15.23:0/1539164858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 msgr2=0x55d8762194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 -- 172.21.15.23:0/1539164858 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d875f87b60 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cd7200700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001c840 tx=0x7f1cd0003cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5fbc99212f155c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cb7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd00076a0 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1cd0007800 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.825+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1cc4004cf0 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cd00079c0 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 10 client.26594 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f1cd81a6420 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc4004cf0 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1cd0030580 con 0x55d876212df0 2024-03-30T04:05:40.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 10 client.26594 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 10 client.26594 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:05:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 20 client.26594 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876277660 0x55d876279a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 4 client.26594 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.826+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d875fa8cf0 con 0x55d876277660 2024-03-30T04:05:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876277660 0x55d876279a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cd6800700 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876277660 0x55d876279a80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f5d64241b3dabb62 server_cookie=a76c6b6d60528d6e in_seq=0 out_seq=0 2024-03-30T04:05:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cb7e00700 10 client.26594 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cb7e00700 1 -- 172.21.15.23:0/1539164858 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d875fa8cf0 con 0x55d876277660 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cb7e00700 10 client.26594 handle_command_reply: tid=0 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.827+0000 7f1cf0fdbb80 1 client.26594 shutdown 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 msgr2=0x55d876217c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1cd001c840 tx=0x7f1cd0003cf0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 shutdown_connections 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d876212df0 0x55d876217c80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d8762075d0 0x55d8762194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d876277660 0x55d876279a80 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 --2- 172.21.15.23:0/1539164858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d876207ef0 0x55d876219a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cb7e00700 0 client.26594 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 >> 172.21.15.23:0/1539164858 conn(0x55d8761676a0 msgr2=0x55d87620aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 shutdown_connections 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 1 -- 172.21.15.23:0/1539164858 wait complete. 2024-03-30T04:05:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cf0fdbb80 20 client.26594 trim_cache size 0 max 16384 2024-03-30T04:05:40.832 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:40.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 msgr2=0x7f1cbc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 0x7f1cbc0a3750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1cd0009610 tx=0x7f1cd00099e0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 msgr2=0x7f1cd81993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.828+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1cc800e9a0 tx=0x7f1cc800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 shutdown_connections 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1cd819dc90 0x7f1cd819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1cd8100f60 0x7f1cd81993b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cbc0a1290 0x7f1cbc0a3750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 --2- 172.21.15.23:0/2933706515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cd8106080 0x7f1cd81998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 >> 172.21.15.23:0/2933706515 conn(0x7f1cd80fc7c0 msgr2=0x7f1cd80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 shutdown_connections 2024-03-30T04:05:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:40.829+0000 7f1cdd400700 1 -- 172.21.15.23:0/2933706515 wait complete. 2024-03-30T04:05:40.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:05:41.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 -- 172.21.15.23:0/2407301282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c0073160 msgr2=0x7f27c0073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2407301282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c0073160 0x7f27c0073540 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f27b0009990 tx=0x7f27b0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 -- 172.21.15.23:0/2407301282 shutdown_connections 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2407301282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c010d5a0 0x7f27c010f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2407301282 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c0073160 0x7f27c0073540 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2407301282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073a80 0x7f27c010d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 -- 172.21.15.23:0/2407301282 >> 172.21.15.23:0/2407301282 conn(0x7f27c00fc7f0 msgr2=0x7f27c00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.066+0000 7f27c6a00700 1 -- 172.21.15.23:0/2407301282 shutdown_connections 2024-03-30T04:05:41.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.067+0000 7f27c6a00700 1 -- 172.21.15.23:0/2407301282 wait complete. 2024-03-30T04:05:41.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.067+0000 7f27c6a00700 1 Processor -- start 2024-03-30T04:05:41.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 -- start start 2024-03-30T04:05:41.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c0073a80 0x7f27c0199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 0x7f27c019dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c019e2e0 con 0x7f27c010d5a0 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c019e450 con 0x7f27c0073a80 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c6a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c019e610 con 0x7f27c0073160 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50530/0 (socket says 172.21.15.23:50530) 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27bfe00700 1 -- 172.21.15.23:0/2780810716 learned_addr learned my addr 172.21.15.23:0/2780810716 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27c4a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 0x7f27c019dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.068+0000 7f27bf400700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c0073a80 0x7f27c0199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 -- 172.21.15.23:0/2780810716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c0073a80 msgr2=0x7f27c0199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c0073a80 0x7f27c0199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 -- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 msgr2=0x7f27c019dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 0x7f27c019dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 -- 172.21.15.23:0/2780810716 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b00095f0 con 0x7f27c0073160 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27c4a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 0x7f27c019dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bfe00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f27b0005910 tx=0x7f27b0009570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bcc00700 1 -- 172.21.15.23:0/2780810716 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b001d070 con 0x7f27c0073160 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c019e890 con 0x7f27c0073160 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bcc00700 1 -- 172.21.15.23:0/2780810716 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27b0022470 con 0x7f27c0073160 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27bcc00700 1 -- 172.21.15.23:0/2780810716 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b0017650 con 0x7f27c0073160 2024-03-30T04:05:41.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.069+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c01a6130 con 0x7f27c0073160 2024-03-30T04:05:41.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.073+0000 7f27bcc00700 1 -- 172.21.15.23:0/2780810716 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f27b00225e0 con 0x7f27c0073160 2024-03-30T04:05:41.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.074+0000 7f27da59fb80 1 Processor -- start 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.074+0000 7f27bcc00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 0x7f27a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.074+0000 7f27da59fb80 1 -- start start 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.074+0000 7f27bcc00700 1 -- 172.21.15.23:0/2780810716 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f27b00ce610 con 0x7f27c0073160 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27c4a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 0x7f27a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e292222990 con 0x55e2922c9ef0 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e292220350 con 0x55e2922c95d0 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27da59fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e29221bc40 con 0x55e2922d4df0 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42498/0 (socket says 172.21.15.23:42498) 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27bfe00700 1 -- 172.21.15.23:0/3431951337 learned_addr learned my addr 172.21.15.23:0/3431951337 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27bf400700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 -- 172.21.15.23:0/3431951337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 msgr2=0x55e2922d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.075+0000 7f27c4a00700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922d71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 -- 172.21.15.23:0/3431951337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 msgr2=0x55e2922d48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 -- 172.21.15.23:0/3431951337 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e29209d3b0 con 0x55e2922c95d0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bf400700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27c4a00700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27c4a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 0x7f27a40a3750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27b8009d20 tx=0x7f27b8009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27b00a33d0 tx=0x7f27b0022c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e108cf728eb74790 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27b5200700 1 -- 172.21.15.23:0/3431951337 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b009f170 con 0x55e2922c95d0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27b5200700 1 -- 172.21.15.23:0/3431951337 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27b009bc10 con 0x55e2922c95d0 2024-03-30T04:05:41.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.076+0000 7f27b5200700 1 -- 172.21.15.23:0/3431951337 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00ce430 con 0x55e2922c95d0 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 -- 172.21.15.23:0/3431951337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 msgr2=0x55e2922c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27b00a33d0 tx=0x7f27b0022c90 comp rx=0 tx=0).stop 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 -- 172.21.15.23:0/3431951337 shutdown_connections 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922c99b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3431951337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 -- 172.21.15.23:0/3431951337 >> 172.21.15.23:0/3431951337 conn(0x55e2922296a0 msgr2=0x55e2922c5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 -- 172.21.15.23:0/3431951337 shutdown_connections 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.077+0000 7f27da59fb80 1 -- 172.21.15.23:0/3431951337 wait complete. 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.078+0000 7f27da59fb80 1 -- start start 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.078+0000 7f27da59fb80 1 Processor -- start 2024-03-30T04:05:41.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 0x55e2922d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 0x55e2922d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2921b7d80 con 0x55e2922d4df0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2921b7a20 con 0x55e2922c9ef0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e29209f6a0 con 0x55e2922c95d0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27da59fb80 10 client.0 _ll_register_callbacks cb 0x55e2921c1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 0x55e2922d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50544/0 (socket says 172.21.15.23:50544) 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 0x55e2922d1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42506/0 (socket says 172.21.15.23:42506) 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 -- 172.21.15.23:0/944681311 learned_addr learned my addr 172.21.15.23:0/944681311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27c4a00700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 0x55e2922d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 -- 172.21.15.23:0/944681311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 msgr2=0x55e2922d1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 0x55e2922d1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 -- 172.21.15.23:0/944681311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 msgr2=0x55e2922d1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 0x55e2922d1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.079+0000 7f27bfe00700 1 -- 172.21.15.23:0/944681311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c01a6130 con 0x55e2922c95d0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27c4a00700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 0x55e2922d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27bfe00700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27b0005f50 tx=0x7f27b0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda440295e529836 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27a9600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00b9050 con 0x55e2922c95d0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f27b00ce030 con 0x55e2922c95d0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2921dc380 con 0x55e2922c95d0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27b009b510 con 0x55e2922c95d0 2024-03-30T04:05:41.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27da59fb80 10 client.26605 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e292049b60 con 0x55e2922c95d0 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.080+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e2921dc380 con 0x55e2922c95d0 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f27b009ab50 con 0x55e2922c95d0 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 10 client.26605 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 10 client.26605 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 20 client.26605 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e2922e4040 0x55e2922e6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 4 client.26605 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e292214700 con 0x55e2922e4040 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27bf400700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e2922e4040 0x55e2922e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27bf400700 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e2922e4040 0x55e2922e6430 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6098192140d4d1dd server_cookie=2c6a6a9e4b525c87 in_seq=0 out_seq=0 2024-03-30T04:05:41.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.081+0000 7f27a9600700 10 client.26605 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27a9600700 1 -- 172.21.15.23:0/944681311 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e292214700 con 0x55e2922e4040 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27a9600700 10 client.26605 handle_command_reply: tid=0 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 client.26605 shutdown 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 msgr2=0x55e2922d4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f27b0005f50 tx=0x7f27b0009310 comp rx=0 tx=0).stop 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 shutdown_connections 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e2922e4040 0x55e2922e6430 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922d4df0 0x55e2922d1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c95d0 0x55e2922d4710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27a9600700 0 client.26605 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:41.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 --2- 172.21.15.23:0/944681311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c9ef0 0x55e2922d1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.082+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 >> 172.21.15.23:0/944681311 conn(0x55e2922296a0 msgr2=0x55e2922cb6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.083+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 shutdown_connections 2024-03-30T04:05:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.083+0000 7f27da59fb80 1 -- 172.21.15.23:0/944681311 wait complete. 2024-03-30T04:05:41.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.083+0000 7f27da59fb80 20 client.26605 trim_cache size 0 max 16384 2024-03-30T04:05:41.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.097+0000 7f27da59fb80 1 Processor -- start 2024-03-30T04:05:41.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.097+0000 7f27da59fb80 1 -- start start 2024-03-30T04:05:41.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922dff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2922d25e0 con 0x55e2922c9ef0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2922ca570 con 0x55e2922c95d0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27da59fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2922c7240 con 0x55e2922d4df0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bf400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42516/0 (socket says 172.21.15.23:42516) 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27c4a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922dff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 -- 172.21.15.23:0/3991250279 learned_addr learned my addr 172.21.15.23:0/3991250279 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 -- 172.21.15.23:0/3991250279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 msgr2=0x55e2922dff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922dff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 -- 172.21.15.23:0/3991250279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 msgr2=0x55e2922d4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.098+0000 7f27bfe00700 1 -- 172.21.15.23:0/3991250279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2921e1ea0 con 0x55e2922c95d0 2024-03-30T04:05:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f27b009b4a0 tx=0x7f27b0022c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1ebaaf666c80ec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27b5c00700 1 -- 172.21.15.23:0/3991250279 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00a3470 con 0x55e2922c95d0 2024-03-30T04:05:41.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27b5c00700 1 -- 172.21.15.23:0/3991250279 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27b009b050 con 0x55e2922c95d0 2024-03-30T04:05:41.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27b5c00700 1 -- 172.21.15.23:0/3991250279 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b00998b0 con 0x55e2922c95d0 2024-03-30T04:05:41.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 -- 172.21.15.23:0/3991250279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 msgr2=0x55e2922db7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f27b009b4a0 tx=0x7f27b0022c90 comp rx=0 tx=0).stop 2024-03-30T04:05:41.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 -- 172.21.15.23:0/3991250279 shutdown_connections 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922d4df0 0x55e2922dff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922c95d0 0x55e2922db7e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3991250279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c9ef0 0x55e2922d4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.099+0000 7f27da59fb80 1 -- 172.21.15.23:0/3991250279 >> 172.21.15.23:0/3991250279 conn(0x55e2922296a0 msgr2=0x55e2922bef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.100+0000 7f27da59fb80 1 -- 172.21.15.23:0/3991250279 shutdown_connections 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.100+0000 7f27da59fb80 1 -- 172.21.15.23:0/3991250279 wait complete. 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.100+0000 7f27da59fb80 1 -- start start 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.100+0000 7f27da59fb80 1 Processor -- start 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 0x55e2922d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922d4df0 0x55e292336d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2922da210 con 0x55e2922c95d0 2024-03-30T04:05:41.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2923372b0 con 0x55e2922d4df0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e292337470 con 0x55e2922c9ef0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27da59fb80 10 client.0 _ll_register_callbacks cb 0x55e2921c1470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49712/0 (socket says 172.21.15.23:49712) 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bfe00700 1 -- 172.21.15.23:0/3775266148 learned_addr learned my addr 172.21.15.23:0/3775266148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 0x55e2922d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27c4a00700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922d4df0 0x55e292336d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 -- 172.21.15.23:0/3775266148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922d4df0 msgr2=0x55e292336d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922d4df0 0x55e292336d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 -- 172.21.15.23:0/3775266148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 msgr2=0x55e2922d9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bf400700 1 -- 172.21.15.23:0/3775266148 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e292049b60 con 0x55e2922c9ef0 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.101+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27bf400700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 0x55e2922d9ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e2922e6c10 tx=0x7f27ac01c590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76257af9d4c53bb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ac017040 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f27ac004500 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27b009bc10 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27ac003680 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27da59fb80 10 client.26611 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e2920a1cd0 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27b009bc10 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.102+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f27ac0119e0 con 0x55e2922c9ef0 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 10 client.26611 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 10 client.26611 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 20 client.26611 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e292338c60 0x55e29233b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 4 client.26611 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e29206acf0 con 0x55e292338c60 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e292338c60 0x55e29233b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27bfe00700 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e292338c60 0x55e29233b080 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=90a984a38fe261b1 server_cookie=1f0fc0dc0a3ef6c1 in_seq=0 out_seq=0 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27a9600700 10 client.26611 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27a9600700 1 -- 172.21.15.23:0/3775266148 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e29206acf0 con 0x55e292338c60 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27a9600700 10 client.26611 handle_command_reply: tid=0 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.103+0000 7f27da59fb80 1 client.26611 shutdown 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 msgr2=0x55e2922d9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 0x55e2922d9ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x55e2922e6c10 tx=0x7f27ac01c590 comp rx=0 tx=0).stop 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 shutdown_connections 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e292338c60 0x55e29233b080 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e2922d4df0 0x55e292336d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27a9600700 0 client.26611 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e2922c95d0 0x55e2922d9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 --2- 172.21.15.23:0/3775266148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2922c9ef0 0x55e2922d9ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 >> 172.21.15.23:0/3775266148 conn(0x55e2922296a0 msgr2=0x55e2922dff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 shutdown_connections 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 1 -- 172.21.15.23:0/3775266148 wait complete. 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.104+0000 7f27da59fb80 20 client.26611 trim_cache size 0 max 16384 2024-03-30T04:05:41.109 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 msgr2=0x7f27a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 0x7f27a40a3750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f27b8009d20 tx=0x7f27b8009040 comp rx=0 tx=0).stop 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 msgr2=0x7f27c01993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f27b0005910 tx=0x7f27b0009570 comp rx=0 tx=0).stop 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 shutdown_connections 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f27c010d5a0 0x7f27c019dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f27c0073160 0x7f27c01993c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f27a40a1290 0x7f27a40a3750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 --2- 172.21.15.23:0/2780810716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27c0073a80 0x7f27c0199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 >> 172.21.15.23:0/2780810716 conn(0x7f27c00fc7f0 msgr2=0x7f27c00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 shutdown_connections 2024-03-30T04:05:41.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.105+0000 7f27c6a00700 1 -- 172.21.15.23:0/2780810716 wait complete. 2024-03-30T04:05:41.122 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:05:41.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 297927 12.25 MB/sec execute 18 sec latency 10.795 ms 2024-03-30T04:05:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 -- 172.21.15.23:0/2515091205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614101910 msgr2=0x7ff614105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 --2- 172.21.15.23:0/2515091205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614101910 0x7ff614105950 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff600009990 tx=0x7ff600009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 -- 172.21.15.23:0/2515091205 shutdown_connections 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 --2- 172.21.15.23:0/2515091205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff614106080 0x7ff61410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 --2- 172.21.15.23:0/2515091205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614100f60 0x7ff614101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 --2- 172.21.15.23:0/2515091205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614101910 0x7ff614105950 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 -- 172.21.15.23:0/2515091205 >> 172.21.15.23:0/2515091205 conn(0x7ff6140fc7c0 msgr2=0x7ff6140febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 -- 172.21.15.23:0/2515091205 shutdown_connections 2024-03-30T04:05:41.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.338+0000 7ff61a400700 1 -- 172.21.15.23:0/2515091205 wait complete. 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.339+0000 7ff61a400700 1 Processor -- start 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.339+0000 7ff61a400700 1 -- start start 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61419e700 con 0x7ff61419dc80 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61419e870 con 0x7ff614100f60 2024-03-30T04:05:41.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff61a400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61419ea30 con 0x7ff614106080 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff613e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff612a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff612a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50578/0 (socket says 172.21.15.23:50578) 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff612a00700 1 -- 172.21.15.23:0/1717603376 learned_addr learned my addr 172.21.15.23:0/1717603376 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42554/0 (socket says 172.21.15.23:42554) 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.340+0000 7ff613e00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49722/0 (socket says 172.21.15.23:49722) 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 -- 172.21.15.23:0/1717603376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 msgr2=0x7ff6141993a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 -- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 msgr2=0x7ff61419e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 -- 172.21.15.23:0/1717603376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000095f0 con 0x7ff614106080 2024-03-30T04:05:41.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff613400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff613e00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff612a00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff600005770 tx=0x7ff6000057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff60be00700 1 -- 172.21.15.23:0/1717603376 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60001d070 con 0x7ff614106080 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff60be00700 1 -- 172.21.15.23:0/1717603376 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff600021410 con 0x7ff614106080 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6141a5f30 con 0x7ff614106080 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff60be00700 1 -- 172.21.15.23:0/1717603376 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600017650 con 0x7ff614106080 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.341+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6141a63b0 con 0x7ff614106080 2024-03-30T04:05:41.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.344+0000 7ff60be00700 1 -- 172.21.15.23:0/1717603376 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff60000fb00 con 0x7ff614106080 2024-03-30T04:05:41.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.346+0000 7ff60be00700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 0x7ff5f80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.346+0000 7ff60be00700 1 -- 172.21.15.23:0/1717603376 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff6000ce0d0 con 0x7ff614106080 2024-03-30T04:05:41.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.346+0000 7ff613400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 0x7ff5f80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.346+0000 7ff62ded3b80 1 Processor -- start 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.346+0000 7ff62ded3b80 1 -- start start 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff613400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 0x7ff5f80a3750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff604011490 tx=0x7ff604009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447d5d0 0x56035447d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603543d6990 con 0x560354488df0 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603543d4350 con 0x56035447def0 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603543cfc40 con 0x56035447d5d0 2024-03-30T04:05:41.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff613400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447d5d0 0x56035447d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42564/0 (socket says 172.21.15.23:42564) 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 -- 172.21.15.23:0/2390611021 learned_addr learned my addr 172.21.15.23:0/2390611021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff613e00700 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 -- 172.21.15.23:0/2390611021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447d5d0 msgr2=0x56035447d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447d5d0 0x56035447d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 -- 172.21.15.23:0/2390611021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 msgr2=0x56035448b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.347+0000 7ff612a00700 1 -- 172.21.15.23:0/2390611021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603542513b0 con 0x56035447def0 2024-03-30T04:05:41.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff612a00700 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff600097920 tx=0x7ff6000972c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9a25e037f0e673d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff608c00700 1 -- 172.21.15.23:0/2390611021 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60009b5b0 con 0x56035447def0 2024-03-30T04:05:41.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff608c00700 1 -- 172.21.15.23:0/2390611021 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff60009aeb0 con 0x56035447def0 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff608c00700 1 -- 172.21.15.23:0/2390611021 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600021810 con 0x56035447def0 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff62ded3b80 1 -- 172.21.15.23:0/2390611021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 msgr2=0x5603544888b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.348+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff600097920 tx=0x7ff6000972c0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 -- 172.21.15.23:0/2390611021 shutdown_connections 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447d5d0 0x56035447d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/2390611021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x5603544888b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 -- 172.21.15.23:0/2390611021 >> 172.21.15.23:0/2390611021 conn(0x5603543dd6a0 msgr2=0x560354479a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 -- 172.21.15.23:0/2390611021 shutdown_connections 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 -- 172.21.15.23:0/2390611021 wait complete. 2024-03-30T04:05:41.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.349+0000 7ff62ded3b80 1 -- start start 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 Processor -- start 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447def0 0x560354485a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x560354486020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56035436bd80 con 0x560354488df0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56035436ba20 con 0x56035447d5d0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.350+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603542536a0 con 0x56035447def0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff62ded3b80 10 client.0 _ll_register_callbacks cb 0x560354375470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42570/0 (socket says 172.21.15.23:42570) 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 -- 172.21.15.23:0/450218068 learned_addr learned my addr 172.21.15.23:0/450218068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff612a00700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447def0 0x560354485a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613e00700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x560354486020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 -- 172.21.15.23:0/450218068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447def0 msgr2=0x560354485a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447def0 0x560354485a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 -- 172.21.15.23:0/450218068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 msgr2=0x560354486020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x560354486020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 -- 172.21.15.23:0/450218068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6141a63b0 con 0x56035447d5d0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff613400700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff604003ad0 tx=0x7ff6040047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2acdd1b7b5c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.351+0000 7ff5fd600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60401fca0 con 0x56035447d5d0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff60401fe00 con 0x56035447d5d0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560354390380 con 0x56035447d5d0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff604003e90 con 0x56035447d5d0 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff62ded3b80 10 client.26624 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5603541fdb60 con 0x56035447d5d0 2024-03-30T04:05:41.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560354390380 con 0x56035447d5d0 2024-03-30T04:05:41.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.352+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff604027640 con 0x56035447d5d0 2024-03-30T04:05:41.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 10 client.26624 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 10 client.26624 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:05:41.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 20 client.26624 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544986e0 0x56035449ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 4 client.26624 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603543c8700 con 0x5603544986e0 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff612a00700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544986e0 0x56035449ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff612a00700 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544986e0 0x56035449ab00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1cc6dfce1603bc5d server_cookie=109cb2ff7ce4cf0c in_seq=0 out_seq=0 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.353+0000 7ff5fd600700 10 client.26624 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff5fd600700 1 -- 172.21.15.23:0/450218068 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5603543c8700 con 0x5603544986e0 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff5fd600700 10 client.26624 handle_command_reply: tid=0 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 client.26624 shutdown 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 msgr2=0x560354488720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff604003ad0 tx=0x7ff6040047c0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 shutdown_connections 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544986e0 0x56035449ab00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x560354486020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447d5d0 0x560354488720 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff5fd600700 0 client.26624 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/450218068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56035447def0 0x560354485a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 >> 172.21.15.23:0/450218068 conn(0x5603543dd6a0 msgr2=0x56035447f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.355+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 shutdown_connections 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.356+0000 7ff62ded3b80 1 -- 172.21.15.23:0/450218068 wait complete. 2024-03-30T04:05:41.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.356+0000 7ff62ded3b80 20 client.26624 trim_cache size 0 max 16384 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 Processor -- start 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 -- start start 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x560354481210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56035447e640 con 0x560354488df0 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560354486b50 con 0x56035447def0 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560354481780 con 0x560354480db0 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff613400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42572/0 (socket says 172.21.15.23:42572) 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff613400700 1 -- 172.21.15.23:0/3855878723 learned_addr learned my addr 172.21.15.23:0/3855878723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613e00700 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x560354481210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.370+0000 7ff612a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 -- 172.21.15.23:0/3855878723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 msgr2=0x560354481210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x560354481210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 -- 172.21.15.23:0/3855878723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 msgr2=0x56035448f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 -- 172.21.15.23:0/3855878723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560354395ea0 con 0x56035447def0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff613400700 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff6040279f0 tx=0x7ff604011d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a808e15a9de52d36 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff609600700 1 -- 172.21.15.23:0/3855878723 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604027e10 con 0x56035447def0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff609600700 1 -- 172.21.15.23:0/3855878723 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff60401f6e0 con 0x56035447def0 2024-03-30T04:05:41.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.371+0000 7ff609600700 1 -- 172.21.15.23:0/3855878723 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604028730 con 0x56035447def0 2024-03-30T04:05:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 -- 172.21.15.23:0/3855878723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 msgr2=0x560354485fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff6040279f0 tx=0x7ff604011d10 comp rx=0 tx=0).stop 2024-03-30T04:05:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 -- 172.21.15.23:0/3855878723 shutdown_connections 2024-03-30T04:05:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x560354481210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56035447def0 0x560354485fc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/3855878723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560354488df0 0x56035448f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 -- 172.21.15.23:0/3855878723 >> 172.21.15.23:0/3855878723 conn(0x5603543dd6a0 msgr2=0x56035449f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 -- 172.21.15.23:0/3855878723 shutdown_connections 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.372+0000 7ff62ded3b80 1 -- 172.21.15.23:0/3855878723 wait complete. 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.373+0000 7ff62ded3b80 1 -- start start 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.373+0000 7ff62ded3b80 1 Processor -- start 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56035447def0 0x56035448f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56035448e7f0 con 0x56035447def0 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56035448e960 con 0x560354488df0 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603544ec4e0 con 0x560354480db0 2024-03-30T04:05:41.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff62ded3b80 10 client.0 _ll_register_callbacks cb 0x560354375470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff613e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff613400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56035447def0 0x56035448f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50610/0 (socket says 172.21.15.23:50610) 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff613e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42578/0 (socket says 172.21.15.23:42578) 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 -- 172.21.15.23:0/405086870 learned_addr learned my addr 172.21.15.23:0/405086870 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 -- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 msgr2=0x56035448e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 -- 172.21.15.23:0/405086870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56035447def0 msgr2=0x56035448f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56035447def0 0x56035448f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff612a00700 1 -- 172.21.15.23:0/405086870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603541fdb60 con 0x560354480db0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.374+0000 7ff613e00700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff612a00700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6000a1140 tx=0x7ff600009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff1c2b27a83dc961 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:41.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6000a32a0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff60009b5b0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff60009aeb0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60009ece0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff62ded3b80 10 client.26629 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x560354255cd0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff60009aeb0 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.375+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff60009b110 con 0x560354480db0 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 10 client.26629 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 10 client.26629 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 20 client.26629 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544ed840 0x5603544efc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 4 client.26629 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:41.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56035421ecf0 con 0x5603544ed840 2024-03-30T04:05:41.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff613400700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544ed840 0x5603544efc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff613400700 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544ed840 0x5603544efc60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19dc7bc04d0fa353 server_cookie=1ea517b5d6db5db2 in_seq=0 out_seq=0 2024-03-30T04:05:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.376+0000 7ff5fd600700 10 client.26629 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff5fd600700 1 -- 172.21.15.23:0/405086870 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56035421ecf0 con 0x5603544ed840 2024-03-30T04:05:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff5fd600700 10 client.26629 handle_command_reply: tid=0 2024-03-30T04:05:41.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 client.26629 shutdown 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 msgr2=0x56035448dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6000a1140 tx=0x7ff600009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 shutdown_connections 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5603544ed840 0x5603544efc60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff5fd600700 0 client.26629 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560354488df0 0x56035448e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56035447def0 0x56035448f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 --2- 172.21.15.23:0/405086870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560354480db0 0x56035448dbf0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 >> 172.21.15.23:0/405086870 conn(0x5603543dd6a0 msgr2=0x5603544842c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 shutdown_connections 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.377+0000 7ff62ded3b80 1 -- 172.21.15.23:0/405086870 wait complete. 2024-03-30T04:05:41.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff62ded3b80 20 client.26629 trim_cache size 0 max 16384 2024-03-30T04:05:41.382 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:41.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 msgr2=0x7ff5f80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 0x7ff5f80a3750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff604011490 tx=0x7ff604009260 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 msgr2=0x7ff6141998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff600005770 tx=0x7ff6000057a0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 shutdown_connections 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff61419dc80 0x7ff61419e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5f80a1290 0x7ff5f80a3750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff614100f60 0x7ff6141993a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 --2- 172.21.15.23:0/1717603376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff614106080 0x7ff6141998e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 >> 172.21.15.23:0/1717603376 conn(0x7ff6140fc7c0 msgr2=0x7ff6140feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 shutdown_connections 2024-03-30T04:05:41.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.378+0000 7ff61a400700 1 -- 172.21.15.23:0/1717603376 wait complete. 2024-03-30T04:05:41.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:41 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/934837215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:41.395 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:05:41.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:41 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/934837215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:41.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 -- 172.21.15.23:0/3645755492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b380752e0 msgr2=0x7f2b380756c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 --2- 172.21.15.23:0/3645755492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b380752e0 0x7f2b380756c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009990 tx=0x7f2b28009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 -- 172.21.15.23:0/3645755492 shutdown_connections 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 --2- 172.21.15.23:0/3645755492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38111bb0 0x7f2b38113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 --2- 172.21.15.23:0/3645755492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b380752e0 0x7f2b380756c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 --2- 172.21.15.23:0/3645755492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b38075c00 0x7f2b38111670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 -- 172.21.15.23:0/3645755492 >> 172.21.15.23:0/3645755492 conn(0x7f2b380fe9c0 msgr2=0x7f2b38100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 -- 172.21.15.23:0/3645755492 shutdown_connections 2024-03-30T04:05:41.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.605+0000 7f2b3e600700 1 -- 172.21.15.23:0/3645755492 wait complete. 2024-03-30T04:05:41.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 Processor -- start 2024-03-30T04:05:41.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 -- start start 2024-03-30T04:05:41.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38075c00 0x7f2b381a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b38111bb0 0x7f2b3819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3819de80 con 0x7f2b38111bb0 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3819dff0 con 0x7f2b38075c00 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b3e600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3819e1b0 con 0x7f2b380752e0 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.606+0000 7f2b37400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50634/0 (socket says 172.21.15.23:50634) 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 -- 172.21.15.23:0/4053879259 learned_addr learned my addr 172.21.15.23:0/4053879259 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b36a00700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38075c00 0x7f2b381a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37e00700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b38111bb0 0x7f2b3819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 -- 172.21.15.23:0/4053879259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38075c00 msgr2=0x7f2b381a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38075c00 0x7f2b381a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 -- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b38111bb0 msgr2=0x7f2b3819d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b38111bb0 0x7f2b3819d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 -- 172.21.15.23:0/4053879259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b280095f0 con 0x7f2b380752e0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b37400700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2b28000c00 tx=0x7f2b28005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b2fe00700 1 -- 172.21.15.23:0/4053879259 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2801d070 con 0x7f2b380752e0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3819e430 con 0x7f2b380752e0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.607+0000 7f2b2fe00700 1 -- 172.21.15.23:0/4053879259 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b28022470 con 0x7f2b380752e0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.608+0000 7f2b2fe00700 1 -- 172.21.15.23:0/4053879259 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b28017650 con 0x7f2b380752e0 2024-03-30T04:05:41.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.608+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b381aa540 con 0x7f2b380752e0 2024-03-30T04:05:41.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.611+0000 7f2b2fe00700 1 -- 172.21.15.23:0/4053879259 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2b280225e0 con 0x7f2b380752e0 2024-03-30T04:05:41.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.612+0000 7f2b2fe00700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 0x7f2b1c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.612+0000 7f2b2fe00700 1 -- 172.21.15.23:0/4053879259 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2b280ce490 con 0x7f2b380752e0 2024-03-30T04:05:41.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.612+0000 7f2b521ffb80 1 Processor -- start 2024-03-30T04:05:41.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.612+0000 7f2b36a00700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 0x7f2b1c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.612+0000 7f2b521ffb80 1 -- start start 2024-03-30T04:05:41.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 0x55c8e77898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e778c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e76d7990 con 0x55c8e777e5d0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e76d5350 con 0x55c8e777eef0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e76d0c40 con 0x55c8e7789df0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b37400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b37400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49780/0 (socket says 172.21.15.23:49780) 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b37400700 1 -- 172.21.15.23:0/1818270221 learned_addr learned my addr 172.21.15.23:0/1818270221 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 0x55c8e77898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b37e00700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e778c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 0x7f2b1c0a3750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b3819ee40 tx=0x7f2b24009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 -- 172.21.15.23:0/1818270221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 msgr2=0x55c8e778c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e778c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 -- 172.21.15.23:0/1818270221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 msgr2=0x55c8e777e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b36a00700 1 -- 172.21.15.23:0/1818270221 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8e75523b0 con 0x55c8e777eef0 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.613+0000 7f2b37400700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.614+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 0x55c8e77898b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011c20 tx=0x7f2b24014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbd5f1af468a09a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.614+0000 7f2b2cc00700 1 -- 172.21.15.23:0/1818270221 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24004230 con 0x55c8e777eef0 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.614+0000 7f2b2cc00700 1 -- 172.21.15.23:0/1818270221 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b24021070 con 0x55c8e777eef0 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.614+0000 7f2b2cc00700 1 -- 172.21.15.23:0/1818270221 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24003b40 con 0x55c8e777eef0 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1818270221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 msgr2=0x55c8e77898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 0x55c8e77898b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011c20 tx=0x7f2b24014000 comp rx=0 tx=0).stop 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1818270221 shutdown_connections 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e778c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e777e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1818270221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777eef0 0x55c8e77898b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1818270221 >> 172.21.15.23:0/1818270221 conn(0x55c8e76de6a0 msgr2=0x55c8e777aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1818270221 shutdown_connections 2024-03-30T04:05:41.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1818270221 wait complete. 2024-03-30T04:05:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.615+0000 7f2b521ffb80 1 -- start start 2024-03-30T04:05:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 Processor -- start 2024-03-30T04:05:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e7786b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e7789df0 0x55c8e7787140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e766cd80 con 0x55c8e7789df0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e766ca20 con 0x55c8e777e5d0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e75546a0 con 0x55c8e777eef0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b521ffb80 10 client.0 _ll_register_callbacks cb 0x55c8e7676470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b37e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e7789df0 0x55c8e7787140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42612/0 (socket says 172.21.15.23:42612) 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b36a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e7786b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.616+0000 7f2b37400700 1 -- 172.21.15.23:0/1397034298 learned_addr learned my addr 172.21.15.23:0/1397034298 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 -- 172.21.15.23:0/1397034298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 msgr2=0x55c8e7786b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e7786b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 -- 172.21.15.23:0/1397034298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e7789df0 msgr2=0x55c8e7787140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e7789df0 0x55c8e7787140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 -- 172.21.15.23:0/1397034298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b381aa540 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b37400700 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009350 tx=0x7f2b28097740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4353707528abac7a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b21600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2809aed0 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2b2809ba90 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8e7691380 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b2809acf0 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b521ffb80 10 client.26642 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.617+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f2b24017070 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8e7691380 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b2809a880 con 0x55c8e777e5d0 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 10 client.26642 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 10 client.26642 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:05:41.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 20 client.26642 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e7799630 0x55c8e779ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 4 client.26642 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8e76c9700 con 0x55c8e7799630 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.618+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e7799630 0x55c8e779ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.619+0000 7f2b36a00700 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e7799630 0x55c8e779ba50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f64642d3248518be server_cookie=88f50a62fb2801a5 in_seq=0 out_seq=0 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.619+0000 7f2b21600700 10 client.26642 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b21600700 1 -- 172.21.15.23:0/1397034298 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c8e76c9700 con 0x55c8e7799630 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b21600700 10 client.26642 handle_command_reply: tid=0 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 client.26642 shutdown 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 msgr2=0x55c8e77897f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b28009350 tx=0x7f2b28097740 comp rx=0 tx=0).stop 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 shutdown_connections 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e7789df0 0x55c8e7787140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e77897f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e7799630 0x55c8e779ba50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/1397034298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e7786b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b21600700 0 client.26642 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 >> 172.21.15.23:0/1397034298 conn(0x55c8e76de6a0 msgr2=0x55c8e77806c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 shutdown_connections 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 1 -- 172.21.15.23:0/1397034298 wait complete. 2024-03-30T04:05:41.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.620+0000 7f2b521ffb80 20 client.26642 trim_cache size 0 max 16384 2024-03-30T04:05:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 Processor -- start 2024-03-30T04:05:41.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 -- start start 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e77735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e777f850 con 0x55c8e777eef0 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e779bde0 con 0x55c8e777e5d0 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e777c160 con 0x55c8e7789df0 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.634+0000 7f2b36a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49804/0 (socket says 172.21.15.23:49804) 2024-03-30T04:05:41.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 -- 172.21.15.23:0/2795484365 learned_addr learned my addr 172.21.15.23:0/2795484365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b37400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42626/0 (socket says 172.21.15.23:42626) 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b37e00700 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e77735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 -- 172.21.15.23:0/2795484365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 msgr2=0x55c8e77735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e77735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 -- 172.21.15.23:0/2795484365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 msgr2=0x55c8e7779f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 -- 172.21.15.23:0/2795484365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8e74feb60 con 0x55c8e777eef0 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b37400700 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b36a00700 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011c20 tx=0x7f2b24005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d024fa1f1960d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.635+0000 7f2b2d600700 1 -- 172.21.15.23:0/2795484365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2402a040 con 0x55c8e777eef0 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b2d600700 1 -- 172.21.15.23:0/2795484365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b2400ee40 con 0x55c8e777eef0 2024-03-30T04:05:41.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b2d600700 1 -- 172.21.15.23:0/2795484365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24005040 con 0x55c8e777eef0 2024-03-30T04:05:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 -- 172.21.15.23:0/2795484365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 msgr2=0x55c8e7790a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2b24011c20 tx=0x7f2b24005ce0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 -- 172.21.15.23:0/2795484365 shutdown_connections 2024-03-30T04:05:41.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e7789df0 0x55c8e77735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e777e5d0 0x55c8e7779f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/2795484365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777eef0 0x55c8e7790a70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 -- 172.21.15.23:0/2795484365 >> 172.21.15.23:0/2795484365 conn(0x55c8e76de6a0 msgr2=0x55c8e77863c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 -- 172.21.15.23:0/2795484365 shutdown_connections 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.636+0000 7f2b521ffb80 1 -- 172.21.15.23:0/2795484365 wait complete. 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.637+0000 7f2b521ffb80 1 -- start start 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.637+0000 7f2b521ffb80 1 Processor -- start 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.637+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e7790990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.637+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e778ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.637+0000 7f2b521ffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e778fb50 con 0x55c8e777e5d0 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e778fcc0 con 0x55c8e7789df0 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b521ffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e77ed4a0 con 0x55c8e777eef0 2024-03-30T04:05:41.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b521ffb80 10 client.0 _ll_register_callbacks cb 0x55c8e7676470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e7790990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b36a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e778ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42634/0 (socket says 172.21.15.23:42634) 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 -- 172.21.15.23:0/4236819964 learned_addr learned my addr 172.21.15.23:0/4236819964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 -- 172.21.15.23:0/4236819964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 msgr2=0x55c8e778ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e778ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 -- 172.21.15.23:0/4236819964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 msgr2=0x55c8e7790990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e7790990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 -- 172.21.15.23:0/4236819964 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b24017070 con 0x55c8e7789df0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b37e00700 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007ee0 tx=0x7f2b30007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44fb695c1797d5f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.638+0000 7f2b21600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:41.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3000e040 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2b30018ce0 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b24021070 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b30003a00 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b521ffb80 10 client.26648 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c8e75523b0 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b24021070 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.639+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2b30004760 con 0x55c8e7789df0 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 10 client.26648 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 10 client.26648 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 20 client.26648 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e77eed50 0x55c8e77f1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 4 client.26648 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:41.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c8e751fcf0 con 0x55c8e77eed50 2024-03-30T04:05:41.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b37400700 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e77eed50 0x55c8e77f1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b37400700 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e77eed50 0x55c8e77f1170 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf5d4249a9e42af0 server_cookie=81747df208218e49 in_seq=0 out_seq=0 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.640+0000 7f2b21600700 10 client.26648 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b21600700 1 -- 172.21.15.23:0/4236819964 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c8e751fcf0 con 0x55c8e77eed50 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b21600700 10 client.26648 handle_command_reply: tid=0 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 client.26648 shutdown 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 msgr2=0x55c8e778f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007ee0 tx=0x7f2b30007f10 comp rx=0 tx=0).stop 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 shutdown_connections 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8e7789df0 0x55c8e778f470 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8e777e5d0 0x55c8e7790990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c8e77eed50 0x55c8e77f1170 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 --2- 172.21.15.23:0/4236819964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8e777eef0 0x55c8e778ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 >> 172.21.15.23:0/4236819964 conn(0x55c8e76de6a0 msgr2=0x55c8e777a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b21600700 0 client.26648 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 shutdown_connections 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 1 -- 172.21.15.23:0/4236819964 wait complete. 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.641+0000 7f2b521ffb80 20 client.26648 trim_cache size 0 max 16384 2024-03-30T04:05:41.645 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 msgr2=0x7f2b1c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 0x7f2b1c0a3750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2b3819ee40 tx=0x7f2b24009260 comp rx=0 tx=0).stop 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 msgr2=0x7f2b381a35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2b28000c00 tx=0x7f2b28005740 comp rx=0 tx=0).stop 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 shutdown_connections 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2b38111bb0 0x7f2b3819d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b380752e0 0x7f2b381a35c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2b1c0a1290 0x7f2b1c0a3750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 --2- 172.21.15.23:0/4053879259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2b38075c00 0x7f2b381a3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 >> 172.21.15.23:0/4053879259 conn(0x7f2b380fe9c0 msgr2=0x7f2b38100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 shutdown_connections 2024-03-30T04:05:41.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.642+0000 7f2b3e600700 1 -- 172.21.15.23:0/4053879259 wait complete. 2024-03-30T04:05:41.660 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:05:41.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:41 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/934837215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 -- 172.21.15.23:0/4054019584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d30101910 msgr2=0x7f3d30105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 --2- 172.21.15.23:0/4054019584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d30101910 0x7f3d30105950 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009990 tx=0x7f3d1c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 -- 172.21.15.23:0/4054019584 shutdown_connections 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 --2- 172.21.15.23:0/4054019584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 0x7f3d3010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 --2- 172.21.15.23:0/4054019584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d30101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 --2- 172.21.15.23:0/4054019584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d30101910 0x7f3d30105950 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 -- 172.21.15.23:0/4054019584 >> 172.21.15.23:0/4054019584 conn(0x7f3d300fc7c0 msgr2=0x7f3d300febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 -- 172.21.15.23:0/4054019584 shutdown_connections 2024-03-30T04:05:41.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.881+0000 7f3d36800700 1 -- 172.21.15.23:0/4054019584 wait complete. 2024-03-30T04:05:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.882+0000 7f3d36800700 1 Processor -- start 2024-03-30T04:05:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.882+0000 7f3d36800700 1 -- start start 2024-03-30T04:05:41.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 0x7f3d301999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 0x7f3d3019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3019e7b0 con 0x7f3d3019dd30 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3019e920 con 0x7f3d30106080 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d36800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3019eae0 con 0x7f3d30100f60 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50702/0 (socket says 172.21.15.23:50702) 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 -- 172.21.15.23:0/4146518808 learned_addr learned my addr 172.21.15.23:0/4146518808 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 0x7f3d301999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 0x7f3d3019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 -- 172.21.15.23:0/4146518808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 msgr2=0x7f3d301999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 0x7f3d301999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 -- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 msgr2=0x7f3d3019e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 0x7f3d3019e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2f400700 1 -- 172.21.15.23:0/4146518808 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0095f0 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.883+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 0x7f3d3019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d2f400700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e9a0 tx=0x7f3d2000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d27e00700 1 -- 172.21.15.23:0/4146518808 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20009790 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d27e00700 1 -- 172.21.15.23:0/4146518808 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d2000cbd0 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d301a5fe0 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d27e00700 1 -- 172.21.15.23:0/4146518808 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20018680 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.884+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301a64b0 con 0x7f3d30100f60 2024-03-30T04:05:41.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.887+0000 7f3d27e00700 1 -- 172.21.15.23:0/4146518808 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3d2000cd40 con 0x7f3d30100f60 2024-03-30T04:05:41.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.889+0000 7f3d4a323b80 1 Processor -- start 2024-03-30T04:05:41.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.889+0000 7f3d4a323b80 1 -- start start 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d27e00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 0x7f3d140a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55d79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a5530990 con 0x5575a55d75d0 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a552e350 con 0x5575a55e2df0 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a5529c40 con 0x5575a55d7ef0 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d27e00700 1 -- 172.21.15.23:0/4146518808 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3d20014070 con 0x7f3d30100f60 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50718/0 (socket says 172.21.15.23:50718) 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2ea00700 1 -- 172.21.15.23:0/211613680 learned_addr learned my addr 172.21.15.23:0/211613680 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 0x7f3d140a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2f400700 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55d79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.890+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 -- 172.21.15.23:0/211613680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 msgr2=0x5575a55e51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55e51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 -- 172.21.15.23:0/211613680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 msgr2=0x5575a55d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55d79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 -- 172.21.15.23:0/211613680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575a53ab3b0 con 0x5575a55d7ef0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009990 tx=0x7f3d1c005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834a1a58dee9394b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 0x7f3d140a3750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d28015b80 tx=0x7f3d28009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d24c00700 1 -- 172.21.15.23:0/211613680 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c01d070 con 0x5575a55d7ef0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d24c00700 1 -- 172.21.15.23:0/211613680 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d1c01e040 con 0x5575a55d7ef0 2024-03-30T04:05:41.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.891+0000 7f3d24c00700 1 -- 172.21.15.23:0/211613680 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0047f0 con 0x5575a55d7ef0 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 -- 172.21.15.23:0/211613680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 msgr2=0x5575a55e28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009990 tx=0x7f3d1c005ec0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 -- 172.21.15.23:0/211613680 shutdown_connections 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55e51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55d79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/211613680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55e28b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 -- 172.21.15.23:0/211613680 >> 172.21.15.23:0/211613680 conn(0x5575a55376a0 msgr2=0x5575a55d3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 -- 172.21.15.23:0/211613680 shutdown_connections 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.892+0000 7f3d4a323b80 1 -- 172.21.15.23:0/211613680 wait complete. 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.893+0000 7f3d4a323b80 1 -- start start 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.893+0000 7f3d4a323b80 1 Processor -- start 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55dff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a54c5d80 con 0x5575a55d75d0 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a54c5a20 con 0x5575a55e2df0 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a53ad6a0 con 0x5575a55d7ef0 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d4a323b80 10 client.0 _ll_register_callbacks cb 0x5575a54cf470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50728/0 (socket says 172.21.15.23:50728) 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d2ea00700 1 -- 172.21.15.23:0/122038845 learned_addr learned my addr 172.21.15.23:0/122038845 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.894+0000 7f3d2f400700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55dff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 -- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 msgr2=0x5575a55dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55dff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 -- 172.21.15.23:0/122038845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 msgr2=0x5575a55e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 -- 172.21.15.23:0/122038845 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3010e470 con 0x5575a55d7ef0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00be80 tx=0x7f3d1c00b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f51f214fc932809 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.895+0000 7f3d19600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c01d030 con 0x5575a55d7ef0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3d1c0058e0 con 0x5575a55d7ef0 2024-03-30T04:05:41.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575a54ea380 con 0x5575a55d7ef0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d1c005bc0 con 0x5575a55d7ef0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 10 client.26659 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5575a5357b60 con 0x5575a55d7ef0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5575a54ea380 con 0x5575a55d7ef0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d1c00f920 con 0x5575a55d7ef0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 10 client.26659 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 10 client.26659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 20 client.26659 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a55f23e0 0x5575a55f47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 4 client.26659 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.896+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575a5522700 con 0x5575a55f23e0 2024-03-30T04:05:41.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.897+0000 7f3d2f400700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a55f23e0 0x5575a55f47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.897+0000 7f3d19600700 10 client.26659 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.897+0000 7f3d2f400700 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a55f23e0 0x5575a55f47d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89d191f71447ef4f server_cookie=dae41d0d80d1bf6b in_seq=0 out_seq=0 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d19600700 1 -- 172.21.15.23:0/122038845 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5575a5522700 con 0x5575a55f23e0 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d19600700 10 client.26659 handle_command_reply: tid=0 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 client.26659 shutdown 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 msgr2=0x5575a55dfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00be80 tx=0x7f3d1c00b900 comp rx=0 tx=0).stop 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 shutdown_connections 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55e2df0 0x5575a55dff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a55f23e0 0x5575a55f47d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/122038845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d7ef0 0x5575a55dfa40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 >> 172.21.15.23:0/122038845 conn(0x5575a55376a0 msgr2=0x5575a55d96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d19600700 0 client.26659 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 shutdown_connections 2024-03-30T04:05:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.898+0000 7f3d4a323b80 1 -- 172.21.15.23:0/122038845 wait complete. 2024-03-30T04:05:41.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.899+0000 7f3d4a323b80 20 client.26659 trim_cache size 0 max 16384 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.912+0000 7f3d4a323b80 1 Processor -- start 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.912+0000 7f3d4a323b80 1 -- start start 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.912+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.912+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55de310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 0x5575a55de850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a55d9140 con 0x5575a55d75d0 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a55d8630 con 0x5575a55d7ef0 2024-03-30T04:05:41.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a55f4b90 con 0x5575a55e2df0 2024-03-30T04:05:41.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55de310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49870/0 (socket says 172.21.15.23:49870) 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 -- 172.21.15.23:0/1822957206 learned_addr learned my addr 172.21.15.23:0/1822957206 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 0x5575a55de850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 -- 172.21.15.23:0/1822957206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 msgr2=0x5575a55de850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 0x5575a55de850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 -- 172.21.15.23:0/1822957206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 msgr2=0x5575a55de310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55de310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2f400700 1 -- 172.21.15.23:0/1822957206 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575a54efea0 con 0x5575a55d75d0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2fe00700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 0x5575a55de850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.913+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55de310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.914+0000 7f3d2f400700 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005bf0 tx=0x7f3d20086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b48ecfeb467283 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.914+0000 7f3d25600700 1 -- 172.21.15.23:0/1822957206 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2009a050 con 0x5575a55d75d0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.914+0000 7f3d25600700 1 -- 172.21.15.23:0/1822957206 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d2009de80 con 0x5575a55d75d0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.914+0000 7f3d25600700 1 -- 172.21.15.23:0/1822957206 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2009a510 con 0x5575a55d75d0 2024-03-30T04:05:41.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- 172.21.15.23:0/1822957206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 msgr2=0x5575a55e9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005bf0 tx=0x7f3d20086040 comp rx=0 tx=0).stop 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- 172.21.15.23:0/1822957206 shutdown_connections 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55e2df0 0x5575a55de850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55d75d0 0x5575a55e9a70 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/1822957206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55de310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- 172.21.15.23:0/1822957206 >> 172.21.15.23:0/1822957206 conn(0x5575a55376a0 msgr2=0x5575a55ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- 172.21.15.23:0/1822957206 shutdown_connections 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- 172.21.15.23:0/1822957206 wait complete. 2024-03-30T04:05:41.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 -- start start 2024-03-30T04:05:41.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.915+0000 7f3d4a323b80 1 Processor -- start 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55e2df0 0x5575a55e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a55e88f0 con 0x5575a55e2df0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a55e8a60 con 0x5575a55d7ef0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575a564e960 con 0x5575a55d75d0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d4a323b80 10 client.0 _ll_register_callbacks cb 0x5575a54cf470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55e2df0 0x5575a55e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42696/0 (socket says 172.21.15.23:42696) 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50746/0 (socket says 172.21.15.23:50746) 2024-03-30T04:05:41.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2ea00700 1 -- 172.21.15.23:0/3121916435 learned_addr learned my addr 172.21.15.23:0/3121916435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.916+0000 7f3d2f400700 1 -- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 msgr2=0x5575a55e7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2f400700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2f400700 1 -- 172.21.15.23:0/3121916435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55e2df0 msgr2=0x5575a55e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2f400700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55e2df0 0x5575a55e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2f400700 1 -- 172.21.15.23:0/3121916435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575a5357b60 con 0x5575a55d75d0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d2f400700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3d20098640 tx=0x7f3d2009a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7755ecc47700c0f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d19600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d20099540 con 0x5575a55d75d0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3d200996a0 con 0x5575a55d75d0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d1c01e040 con 0x5575a55d75d0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d2009a2b0 con 0x5575a55d75d0 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d4a323b80 10 client.26665 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:41.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3d280090d0 con 0x5575a55d75d0 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.917+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d1c01e040 con 0x5575a55d75d0 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3d200b8120 con 0x5575a55d75d0 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 10 client.26665 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 10 client.26665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 20 client.26665 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a564fc80 0x5575a5652070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 4 client.26665 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5575a5378cf0 con 0x5575a564fc80 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a564fc80 0x5575a5652070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d2ea00700 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a564fc80 0x5575a5652070 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2601af03d94a5c6 server_cookie=6c83c231ca772e5c in_seq=0 out_seq=0 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.918+0000 7f3d19600700 10 client.26665 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d19600700 1 -- 172.21.15.23:0/3121916435 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5575a5378cf0 con 0x5575a564fc80 2024-03-30T04:05:41.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d19600700 10 client.26665 handle_command_reply: tid=0 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 client.26665 shutdown 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 msgr2=0x5575a55e9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3d20098640 tx=0x7f3d2009a050 comp rx=0 tx=0).stop 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 shutdown_connections 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5575a55e2df0 0x5575a55e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5575a564fc80 0x5575a5652070 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5575a55d75d0 0x5575a55e9860 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 --2- 172.21.15.23:0/3121916435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5575a55d7ef0 0x5575a55e7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d19600700 0 client.26665 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 >> 172.21.15.23:0/3121916435 conn(0x5575a55376a0 msgr2=0x5575a55dad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 shutdown_connections 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 1 -- 172.21.15.23:0/3121916435 wait complete. 2024-03-30T04:05:41.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.919+0000 7f3d4a323b80 20 client.26665 trim_cache size 0 max 16384 2024-03-30T04:05:41.924 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:05:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 msgr2=0x7f3d140a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 0x7f3d140a3750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3d28015b80 tx=0x7f3d28009920 comp rx=0 tx=0).stop 2024-03-30T04:05:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 msgr2=0x7f3d301994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:41.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000e9a0 tx=0x7f3d2000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 shutdown_connections 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3d3019dd30 0x7f3d3019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30100f60 0x7f3d301994a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3d140a1290 0x7f3d140a3750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 --2- 172.21.15.23:0/4146518808 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d30106080 0x7f3d301999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 >> 172.21.15.23:0/4146518808 conn(0x7f3d300fc7c0 msgr2=0x7f3d300feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 shutdown_connections 2024-03-30T04:05:41.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:41.920+0000 7f3d36800700 1 -- 172.21.15.23:0/4146518808 wait complete. 2024-03-30T04:05:42.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 299611 11.97 MB/sec execute 19 sec latency 7.619 ms 2024-03-30T04:05:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:42 smithi193 ceph-mon[79581]: pgmap v116: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-30T04:05:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:42 smithi023 ceph-mon[38254]: pgmap v116: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-30T04:05:42.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:42 smithi154 ceph-mon[79161]: pgmap v116: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 283 op/s 2024-03-30T04:05:42.940 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:05:42.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:05:43.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 -- 172.21.15.23:0/3353793447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c101910 msgr2=0x7f1d0c105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 --2- 172.21.15.23:0/3353793447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c101910 0x7f1d0c105950 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1cf8009990 tx=0x7f1cf8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 -- 172.21.15.23:0/3353793447 shutdown_connections 2024-03-30T04:05:43.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 --2- 172.21.15.23:0/3353793447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 --2- 172.21.15.23:0/3353793447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c100f60 0x7f1d0c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 --2- 172.21.15.23:0/3353793447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c101910 0x7f1d0c105950 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 -- 172.21.15.23:0/3353793447 >> 172.21.15.23:0/3353793447 conn(0x7f1d0c0fc7c0 msgr2=0x7f1d0c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 -- 172.21.15.23:0/3353793447 shutdown_connections 2024-03-30T04:05:43.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.157+0000 7f1d13600700 1 -- 172.21.15.23:0/3353793447 wait complete. 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.158+0000 7f1d13600700 1 Processor -- start 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 -- start start 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 0x7f1d0c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 0x7f1d0c1a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c1a5b70 con 0x7f1d0c073b30 2024-03-30T04:05:43.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c1a5ce0 con 0x7f1d0c106080 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d13600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c1a5ea0 con 0x7f1d0c100f60 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d10c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 0x7f1d0c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d10c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 0x7f1d0c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50754/0 (socket says 172.21.15.23:50754) 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42704/0 (socket says 172.21.15.23:42704) 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d10c00700 1 -- 172.21.15.23:0/3928198840 learned_addr learned my addr 172.21.15.23:0/3928198840 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.159+0000 7f1d11600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 0x7f1d0c1a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 -- 172.21.15.23:0/3928198840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 msgr2=0x7f1d0c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 0x7f1d0c0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 -- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 msgr2=0x7f1d0c1a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 0x7f1d0c1a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 -- 172.21.15.23:0/3928198840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf80095f0 con 0x7f1d0c106080 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d11600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 0x7f1d0c1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:05:43.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098e0 tx=0x7f1cf8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d09600700 1 -- 172.21.15.23:0/3928198840 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf801d070 con 0x7f1d0c106080 2024-03-30T04:05:43.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d09600700 1 -- 172.21.15.23:0/3928198840 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cf8022470 con 0x7f1d0c106080 2024-03-30T04:05:43.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c1a6120 con 0x7f1d0c106080 2024-03-30T04:05:43.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d09600700 1 -- 172.21.15.23:0/3928198840 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8017650 con 0x7f1d0c106080 2024-03-30T04:05:43.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.160+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c1a65a0 con 0x7f1d0c106080 2024-03-30T04:05:43.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.164+0000 7f1d09600700 1 -- 172.21.15.23:0/3928198840 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1cf80225e0 con 0x7f1d0c106080 2024-03-30T04:05:43.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 Processor -- start 2024-03-30T04:05:43.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 -- start start 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a416a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a421930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a36fa10 con 0x55f37a416650 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a36d3d0 con 0x55f37a421e70 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d27062b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a368cc0 con 0x55f37a416f70 2024-03-30T04:05:43.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.165+0000 7f1d09600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 0x7f1cf40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d11600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a416a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42716/0 (socket says 172.21.15.23:42716) 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 -- 172.21.15.23:0/3259847739 learned_addr learned my addr 172.21.15.23:0/3259847739 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d10c00700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a421930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d09600700 1 -- 172.21.15.23:0/3928198840 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1cf80ce310 con 0x7f1d0c106080 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 -- 172.21.15.23:0/3259847739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 msgr2=0x55f37a421930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a421930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 -- 172.21.15.23:0/3259847739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 msgr2=0x55f37a416a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a416a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 -- 172.21.15.23:0/3259847739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f37a1ea430 con 0x55f37a421e70 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d10c00700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a421930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:43.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d11600700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a416a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d11600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 0x7f1cf40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80cb700 tx=0x7f1cf8009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e48ab0ef08c8d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1cfdc00700 1 -- 172.21.15.23:0/3259847739 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf809ad50 con 0x55f37a421e70 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1cfdc00700 1 -- 172.21.15.23:0/3259847739 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cf809ab70 con 0x55f37a421e70 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.166+0000 7f1cfdc00700 1 -- 172.21.15.23:0/3259847739 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf80228c0 con 0x55f37a421e70 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 -- 172.21.15.23:0/3259847739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 msgr2=0x55f37a424260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d11600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 0x7f1cf40a37d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009480 tx=0x7f1d04004ad0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80cb700 tx=0x7f1cf8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 -- 172.21.15.23:0/3259847739 shutdown_connections 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a424260 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a416a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3259847739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a421930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 -- 172.21.15.23:0/3259847739 >> 172.21.15.23:0/3259847739 conn(0x55f37a376720 msgr2=0x55f37a412b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 -- 172.21.15.23:0/3259847739 shutdown_connections 2024-03-30T04:05:43.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.167+0000 7f1d27062b80 1 -- 172.21.15.23:0/3259847739 wait complete. 2024-03-30T04:05:43.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.168+0000 7f1d27062b80 1 -- start start 2024-03-30T04:05:43.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.168+0000 7f1d27062b80 1 Processor -- start 2024-03-30T04:05:43.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a421770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a41f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a304e00 con 0x55f37a416650 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a304aa0 con 0x55f37a421e70 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a1ec720 con 0x55f37a416f70 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d27062b80 10 client.0 _ll_register_callbacks cb 0x55f37a30e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d10c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a421770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50770/0 (socket says 172.21.15.23:50770) 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d0be00700 1 -- 172.21.15.23:0/3155948474 learned_addr learned my addr 172.21.15.23:0/3155948474 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.169+0000 7f1d11600700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a41f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 -- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 msgr2=0x55f37a41f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a41f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 -- 172.21.15.23:0/3155948474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 msgr2=0x55f37a421770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a421770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 -- 172.21.15.23:0/3155948474 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c1a65a0 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d0be00700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80976c0 tx=0x7f1cf80a3690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c41df023028cacc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1cf2000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf809c220 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1cf809f040 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f37a329400 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cf809b910 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d27062b80 10 client.26674 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.170+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f37a196be0 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f37a329400 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1cf809b590 con 0x55f37a416f70 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 10 client.26674 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:43.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 10 client.26674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 20 client.26674 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a4310b0 0x55f37a4334d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 4 client.26674 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f37a361780 con 0x55f37a4310b0 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.171+0000 7f1d10c00700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a4310b0 0x55f37a4334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.172+0000 7f1cf2000700 10 client.26674 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.172+0000 7f1d10c00700 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a4310b0 0x55f37a4334d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d106a929d480cb0a server_cookie=e755169ce92e9fcf in_seq=0 out_seq=0 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.172+0000 7f1cf2000700 1 -- 172.21.15.23:0/3155948474 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f37a361780 con 0x55f37a4310b0 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.172+0000 7f1cf2000700 10 client.26674 handle_command_reply: tid=0 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 client.26674 shutdown 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 msgr2=0x55f37a41eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80976c0 tx=0x7f1cf80a3690 comp rx=0 tx=0).stop 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 shutdown_connections 2024-03-30T04:05:43.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a41f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416650 0x55f37a421770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a4310b0 0x55f37a4334d0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 --2- 172.21.15.23:0/3155948474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416f70 0x55f37a41eb60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1cf2000700 0 client.26674 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 >> 172.21.15.23:0/3155948474 conn(0x55f37a376720 msgr2=0x55f37a418740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 shutdown_connections 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 1 -- 172.21.15.23:0/3155948474 wait complete. 2024-03-30T04:05:43.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.173+0000 7f1d27062b80 20 client.26674 trim_cache size 0 max 16384 2024-03-30T04:05:43.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.192+0000 7f1d27062b80 1 Processor -- start 2024-03-30T04:05:43.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.192+0000 7f1d27062b80 1 -- start start 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a428860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 0x55f37a41d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a41de90 con 0x55f37a421e70 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a40c670 con 0x55f37a416f70 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d27062b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a41f410 con 0x55f37a416650 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 0x55f37a41d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42734/0 (socket says 172.21.15.23:42734) 2024-03-30T04:05:43.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d0be00700 1 -- 172.21.15.23:0/2038432138 learned_addr learned my addr 172.21.15.23:0/2038432138 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d10c00700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a428860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 -- 172.21.15.23:0/2038432138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 msgr2=0x55f37a428860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a428860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 -- 172.21.15.23:0/2038432138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 msgr2=0x55f37a41d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d11600700 1 -- 172.21.15.23:0/2038432138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f37a40b8f0 con 0x55f37a421e70 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d10c00700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a428860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.193+0000 7f1d0be00700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.194+0000 7f1d11600700 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 0x55f37a41d950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009c10 tx=0x7f1d040094b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7277d954eadd0f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.194+0000 7f1cfe600700 1 -- 172.21.15.23:0/2038432138 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04027070 con 0x55f37a421e70 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.194+0000 7f1cfe600700 1 -- 172.21.15.23:0/2038432138 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d0401f040 con 0x55f37a421e70 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.194+0000 7f1cfe600700 1 -- 172.21.15.23:0/2038432138 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0401e050 con 0x55f37a421e70 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- 172.21.15.23:0/2038432138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 msgr2=0x55f37a41d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 0x55f37a41d950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009c10 tx=0x7f1d040094b0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- 172.21.15.23:0/2038432138 shutdown_connections 2024-03-30T04:05:43.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a421e70 0x55f37a41d950 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a428860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 --2- 172.21.15.23:0/2038432138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a416f70 0x55f37a41d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- 172.21.15.23:0/2038432138 >> 172.21.15.23:0/2038432138 conn(0x55f37a376720 msgr2=0x55f37a419e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 301783 12.05 MB/sec execute 20 sec latency 72.848 ms 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- 172.21.15.23:0/2038432138 shutdown_connections 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- 172.21.15.23:0/2038432138 wait complete. 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 -- start start 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.195+0000 7f1d27062b80 1 Processor -- start 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416f70 0x55f37a47b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a47b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a485f90 con 0x55f37a416f70 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a4860d0 con 0x55f37a421e70 2024-03-30T04:05:43.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f37a486290 con 0x55f37a416650 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d27062b80 10 client.0 _ll_register_callbacks cb 0x55f37a30e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50790/0 (socket says 172.21.15.23:50790) 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 -- 172.21.15.23:0/1921975953 learned_addr learned my addr 172.21.15.23:0/1921975953 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d0be00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416f70 0x55f37a47b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d11600700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a47b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 -- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 msgr2=0x55f37a47b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a47b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 -- 172.21.15.23:0/1921975953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416f70 msgr2=0x55f37a47b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416f70 0x55f37a47b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.196+0000 7f1d10c00700 1 -- 172.21.15.23:0/1921975953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f37a196be0 con 0x55f37a416650 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1d10c00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000c00 tx=0x7f1d0001dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a768fc4d9d774fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1cf2000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00018040 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f1d00004520 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1cf809ab70 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d00003680 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1d27062b80 10 client.26680 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.197+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f1d04019480 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cf809ab70 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1d00012a50 con 0x55f37a416650 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 10 client.26680 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 10 client.26680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 20 client.26680 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a487a30 0x55f37a489e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 4 client.26680 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:05:43.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f37a1b7d70 con 0x55f37a487a30 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.198+0000 7f1d0be00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a487a30 0x55f37a489e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.199+0000 7f1d0be00700 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a487a30 0x55f37a489e20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39092acdad8e875a server_cookie=f4aa2c21cdca36a in_seq=0 out_seq=0 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.199+0000 7f1cf2000700 10 client.26680 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1cf2000700 1 -- 172.21.15.23:0/1921975953 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f37a1b7d70 con 0x55f37a487a30 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1cf2000700 10 client.26680 handle_command_reply: tid=0 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 client.26680 shutdown 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 msgr2=0x55f37a47ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000c00 tx=0x7f1d0001dbe0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 shutdown_connections 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f37a421e70 0x55f37a47b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f37a487a30 0x55f37a489e20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f37a416650 0x55f37a47ae80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 --2- 172.21.15.23:0/1921975953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f37a416f70 0x55f37a47b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 >> 172.21.15.23:0/1921975953 conn(0x55f37a376720 msgr2=0x55f37a419e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1cf2000700 0 client.26680 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 shutdown_connections 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.200+0000 7f1d27062b80 1 -- 172.21.15.23:0/1921975953 wait complete. 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d27062b80 20 client.26680 trim_cache size 0 max 16384 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "41d25f78-4457-4341-9969-64c8922d0b2c", 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:05:43.204 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 msgr2=0x7f1cf40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 0x7f1cf40a37d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009480 tx=0x7f1d04004ad0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 msgr2=0x7f1d0c0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1cf80098e0 tx=0x7f1cf8005740 comp rx=0 tx=0).stop 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 shutdown_connections 2024-03-30T04:05:43.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d0c073b30 0x7f1d0c1a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d0c100f60 0x7f1d0c0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1cf40a1310 0x7f1cf40a37d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 --2- 172.21.15.23:0/3928198840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d0c106080 0x7f1d0c0735f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 >> 172.21.15.23:0/3928198840 conn(0x7f1d0c0fc7c0 msgr2=0x7f1d0c0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 shutdown_connections 2024-03-30T04:05:43.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.201+0000 7f1d13600700 1 -- 172.21.15.23:0/3928198840 wait complete. 2024-03-30T04:05:43.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 -- 172.21.15.23:0/2215999629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 msgr2=0x7f8da410dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 --2- 172.21.15.23:0/2215999629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da410dc30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00b0a0 tx=0x7f8d9c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 -- 172.21.15.23:0/2215999629 shutdown_connections 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 --2- 172.21.15.23:0/2215999629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da410dc30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 --2- 172.21.15.23:0/2215999629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 0x7f8da4069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 --2- 172.21.15.23:0/2215999629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da4069c10 0x7f8da4105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 -- 172.21.15.23:0/2215999629 >> 172.21.15.23:0/2215999629 conn(0x7f8da4076d80 msgr2=0x7f8da4077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.433+0000 7f8da9600700 1 -- 172.21.15.23:0/2215999629 shutdown_connections 2024-03-30T04:05:43.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.434+0000 7f8da9600700 1 -- 172.21.15.23:0/2215999629 wait complete. 2024-03-30T04:05:43.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.435+0000 7f8da9600700 1 Processor -- start 2024-03-30T04:05:43.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.435+0000 7f8da9600700 1 -- start start 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 0x7f8da41972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da419c640 con 0x7f8da419bbc0 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da419c7b0 con 0x7f8da41060b0 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da9600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da419c970 con 0x7f8da4069260 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49928/0 (socket says 172.21.15.23:49928) 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42776/0 (socket says 172.21.15.23:42776) 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 -- 172.21.15.23:0/2455450968 learned_addr learned my addr 172.21.15.23:0/2455450968 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da2a00700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 0x7f8da41972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 -- 172.21.15.23:0/2455450968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 msgr2=0x7f8da41972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 0x7f8da41972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 -- 172.21.15.23:0/2455450968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 msgr2=0x7f8da4197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da3400700 1 -- 172.21.15.23:0/2455450968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90009520 con 0x7f8da419bbc0 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.436+0000 7f8da2000700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8da3400700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00b070 tx=0x7f8d9c00bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8d9b400700 1 -- 172.21.15.23:0/2455450968 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c00e040 con 0x7f8da419bbc0 2024-03-30T04:05:43.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c009d00 con 0x7f8da419bbc0 2024-03-30T04:05:43.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8d9b400700 1 -- 172.21.15.23:0/2455450968 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d9c01d470 con 0x7f8da419bbc0 2024-03-30T04:05:43.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8d9b400700 1 -- 172.21.15.23:0/2455450968 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0052a0 con 0x7f8da419bbc0 2024-03-30T04:05:43.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.437+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41a4160 con 0x7f8da419bbc0 2024-03-30T04:05:43.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.440+0000 7f8d9b400700 1 -- 172.21.15.23:0/2455450968 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8d9c01d850 con 0x7f8da419bbc0 2024-03-30T04:05:43.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.441+0000 7f8d9b400700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 0x7f8d880a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8d9b400700 1 -- 172.21.15.23:0/2455450968 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8d9c0ce320 con 0x7f8da419bbc0 2024-03-30T04:05:43.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8da2a00700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 0x7f8d880a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8dbd15bb80 1 Processor -- start 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8dbd15bb80 1 -- start start 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e90fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8da2a00700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 0x7f8d880a37d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009c60 tx=0x7f8d94009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.442+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90feef0 0x55a8e91098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 0x55a8e910c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9057990 con 0x55a8e90fe5d0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9055350 con 0x55a8e9109df0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9050c40 con 0x55a8e90feef0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e90fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 0x55a8e910c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e90fe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49944/0 (socket says 172.21.15.23:49944) 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da2a00700 1 -- 172.21.15.23:0/2894814315 learned_addr learned my addr 172.21.15.23:0/2894814315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da2000700 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90feef0 0x55a8e91098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 -- 172.21.15.23:0/2894814315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90feef0 msgr2=0x55a8e91098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90feef0 0x55a8e91098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 -- 172.21.15.23:0/2894814315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 msgr2=0x55a8e90fe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e90fe9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.443+0000 7f8da3400700 1 -- 172.21.15.23:0/2894814315 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8e8ed23b0 con 0x55a8e9109df0 2024-03-30T04:05:43.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8da3400700 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 0x55a8e910c1e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0cb6b0 tx=0x7f8d9c00b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9743ea4a7a883fbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8d8f400700 1 -- 172.21.15.23:0/2894814315 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c09b800 con 0x55a8e9109df0 2024-03-30T04:05:43.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8d8f400700 1 -- 172.21.15.23:0/2894814315 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d9c0ce120 con 0x55a8e9109df0 2024-03-30T04:05:43.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8d8f400700 1 -- 172.21.15.23:0/2894814315 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0995c0 con 0x55a8e9109df0 2024-03-30T04:05:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/2894814315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 msgr2=0x55a8e910c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 0x55a8e910c1e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0cb6b0 tx=0x7f8d9c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/2894814315 shutdown_connections 2024-03-30T04:05:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e9109df0 0x55a8e910c1e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e90fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.444+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/2894814315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90feef0 0x55a8e91098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.445+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/2894814315 >> 172.21.15.23:0/2894814315 conn(0x55a8e905e6a0 msgr2=0x55a8e90faa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.445+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/2894814315 shutdown_connections 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.445+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/2894814315 wait complete. 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.445+0000 7f8dbd15bb80 1 -- start start 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.445+0000 7f8dbd15bb80 1 Processor -- start 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e9109770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e91070a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e8fecd80 con 0x55a8e90fe5d0 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e8feca20 con 0x55a8e90feef0 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e8ed46a0 con 0x55a8e9109df0 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8dbd15bb80 10 client.0 _ll_register_callbacks cb 0x55a8e8ff6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:43.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e9109770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42792/0 (socket says 172.21.15.23:42792) 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e9109770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49952/0 (socket says 172.21.15.23:49952) 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da2000700 1 -- 172.21.15.23:0/194742961 learned_addr learned my addr 172.21.15.23:0/194742961 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.446+0000 7f8da3400700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e91070a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 -- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 msgr2=0x55a8e9106b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 -- 172.21.15.23:0/194742961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 msgr2=0x55a8e9109770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e9109770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 -- 172.21.15.23:0/194742961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41a4160 con 0x55a8e9109df0 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da2000700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8da3400700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e91070a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c09f0f0 tx=0x7f8d9c0099e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7871a9617a899e79 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8d83e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0a34f0 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8d9c09ef30 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8e9011380 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d9c0b92a0 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8dbd15bb80 10 client.26692 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a8e8e7eb60 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.447+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a8e9011380 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8d9c09a5f0 con 0x55a8e9109df0 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 10 client.26692 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 10 client.26692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 20 client.26692 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e9119180 0x55a8e911b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 4 client.26692 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:05:43.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a8e9049700 con 0x55a8e9119180 2024-03-30T04:05:43.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8da2a00700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e9119180 0x55a8e911b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8da2a00700 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e9119180 0x55a8e911b570 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53345efcd9c04814 server_cookie=3036104a19f74824 in_seq=0 out_seq=0 2024-03-30T04:05:43.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.448+0000 7f8d83e00700 10 client.26692 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.449+0000 7f8d83e00700 1 -- 172.21.15.23:0/194742961 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a8e9049700 con 0x55a8e9119180 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.449+0000 7f8d83e00700 10 client.26692 handle_command_reply: tid=0 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.449+0000 7f8dbd15bb80 1 client.26692 shutdown 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 msgr2=0x55a8e91070a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e91070a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c09f0f0 tx=0x7f8d9c0099e0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 shutdown_connections 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e9119180 0x55a8e911b570 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e91070a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e9109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8d83e00700 0 client.26692 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/194742961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e9106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 >> 172.21.15.23:0/194742961 conn(0x55a8e905e6a0 msgr2=0x55a8e9100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 shutdown_connections 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/194742961 wait complete. 2024-03-30T04:05:43.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.450+0000 7f8dbd15bb80 20 client.26692 trim_cache size 0 max 16384 2024-03-30T04:05:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.467+0000 7f8dbd15bb80 1 Processor -- start 2024-03-30T04:05:43.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 -- start start 2024-03-30T04:05:43.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 0x55a8e911b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e9109df0 0x55a8e90f35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e90ff640 con 0x55a8e9109df0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9108060 con 0x55a8e90feef0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e90fc2b0 con 0x55a8e90fe5d0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 0x55a8e911b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42798/0 (socket says 172.21.15.23:42798) 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8da2000700 1 -- 172.21.15.23:0/3178192577 learned_addr learned my addr 172.21.15.23:0/3178192577 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.468+0000 7f8da3400700 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e9109df0 0x55a8e90f35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 -- 172.21.15.23:0/3178192577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 msgr2=0x55a8e91107e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 -- 172.21.15.23:0/3178192577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e9109df0 msgr2=0x55a8e90f35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e9109df0 0x55a8e90f35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 -- 172.21.15.23:0/3178192577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8e9016ea0 con 0x55a8e90fe5d0 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2000700 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8da2a00700 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 0x55a8e911b570 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c01d750 tx=0x7f8d94014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0146ab0b883a58 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8d8fe00700 1 -- 172.21.15.23:0/3178192577 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94022030 con 0x55a8e90fe5d0 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8d8fe00700 1 -- 172.21.15.23:0/3178192577 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d94023040 con 0x55a8e90fe5d0 2024-03-30T04:05:43.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.469+0000 7f8d8fe00700 1 -- 172.21.15.23:0/3178192577 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9401d410 con 0x55a8e90fe5d0 2024-03-30T04:05:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/3178192577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 msgr2=0x55a8e911b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 0x55a8e911b570 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c01d750 tx=0x7f8d94014000 comp rx=0 tx=0).stop 2024-03-30T04:05:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/3178192577 shutdown_connections 2024-03-30T04:05:43.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e9109df0 0x55a8e90f35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e90fe5d0 0x55a8e911b570 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/3178192577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e91107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/3178192577 >> 172.21.15.23:0/3178192577 conn(0x55a8e905e6a0 msgr2=0x55a8e911fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/3178192577 shutdown_connections 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/3178192577 wait complete. 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 -- start start 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.470+0000 7f8dbd15bb80 1 Processor -- start 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e910e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9166450 con 0x55a8e90fe5d0 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e91665c0 con 0x55a8e90feef0 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8e9166780 con 0x55a8e9109df0 2024-03-30T04:05:43.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8dbd15bb80 10 client.0 _ll_register_callbacks cb 0x55a8e8ff6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e910e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42806/0 (socket says 172.21.15.23:42806) 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 -- 172.21.15.23:0/1318235408 learned_addr learned my addr 172.21.15.23:0/1318235408 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da3400700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:50828/0 (socket says 172.21.15.23:50828) 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 -- 172.21.15.23:0/1318235408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 msgr2=0x55a8e9165f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 -- 172.21.15.23:0/1318235408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 msgr2=0x55a8e910e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.471+0000 7f8da2000700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e910e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8da2000700 1 -- 172.21.15.23:0/1318235408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d90004170 con 0x55a8e90feef0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8da3400700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8da2000700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d900044c0 tx=0x7f8d90003960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4acff8b40c98ab server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8d83e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:05:43.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90016040 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8d9001d890 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d9c0ce120 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d900045b0 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8dbd15bb80 10 client.26702 fetch_fsmap learned FSMap version 30 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.472+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f8d94017070 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d9c0ce120 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8d9001eb10 con 0x55a8e90feef0 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 10 client.26702 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 10 client.26702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 20 client.26702 populate_metadata read hostname 'smithi023' 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e916e120 0x55a8e9170540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 4 client.26702 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:05:43.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a8e8e9fcf0 con 0x55a8e916e120 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.473+0000 7f8da2a00700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e916e120 0x55a8e9170540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8da2a00700 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e916e120 0x55a8e9170540 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53631197cca53a12 server_cookie=524558e1448abf9 in_seq=0 out_seq=0 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8d83e00700 10 client.26702 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8d83e00700 1 -- 172.21.15.23:0/1318235408 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a8e8e9fcf0 con 0x55a8e916e120 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8d83e00700 10 client.26702 handle_command_reply: tid=0 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 client.26702 shutdown 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 msgr2=0x55a8e910ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8d900044c0 tx=0x7f8d90003960 comp rx=0 tx=0).stop 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 shutdown_connections 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8e916e120 0x55a8e9170540 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8e9109df0 0x55a8e9165f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8e90fe5d0 0x55a8e910e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8d83e00700 0 client.26702 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:05:43.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 --2- 172.21.15.23:0/1318235408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8e90feef0 0x55a8e910ef10 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 >> 172.21.15.23:0/1318235408 conn(0x55a8e905e6a0 msgr2=0x55a8e90f35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 shutdown_connections 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 1 -- 172.21.15.23:0/1318235408 wait complete. 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.474+0000 7f8dbd15bb80 20 client.26702 trim_cache size 0 max 16384 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout: "41d25f78-4457-4341-9969-64c8922d0b2c": { 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:05:43.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "41d25f78-4457-4341-9969-64c8922d0b2c", 2024-03-30T04:05:43.480 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:05:43.480 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:05:43.480 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:05:43.480 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:05:43.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 msgr2=0x7f8d880a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 0x7f8d880a37d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009c60 tx=0x7f8d94009260 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 msgr2=0x7f8da419c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00b070 tx=0x7f8d9c00bdb0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 shutdown_connections 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8da419bbc0 0x7f8da419c040 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8da4069260 0x7f8da41972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8d880a1310 0x7f8d880a37d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 --2- 172.21.15.23:0/2455450968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8da41060b0 0x7f8da4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 >> 172.21.15.23:0/2455450968 conn(0x7f8da4076d80 msgr2=0x7f8da41071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 shutdown_connections 2024-03-30T04:05:43.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:05:43.475+0000 7f8da9600700 1 -- 172.21.15.23:0/2455450968 wait complete. 2024-03-30T04:05:43.493 INFO:tasks.cephfs.filesystem:scrub status for tag:41d25f78-4457-4341-9969-64c8922d0b2c - {'path': '/', 'tag': '41d25f78-4457-4341-9969-64c8922d0b2c', 'options': 'recursive,force'} 2024-03-30T04:05:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:43 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:05:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:43 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:43 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:05:43.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:43 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:43.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:43 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:05:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:43 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:43 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:05:43.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:43 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:43.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:43 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:05:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:43 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:43 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:05:43.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:43 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:44.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 304117 11.88 MB/sec execute 21 sec latency 7.110 ms 2024-03-30T04:05:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:44 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:05:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:44 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:44 smithi154 ceph-mon[79161]: pgmap v117: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:44 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:05:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:44 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:44.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:44 smithi193 ceph-mon[79581]: pgmap v117: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:44 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:05:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:44 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:05:44.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:44 smithi023 ceph-mon[38254]: pgmap v117: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:45.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 305952 11.87 MB/sec execute 22 sec latency 24.135 ms 2024-03-30T04:05:46.194 INFO:tasks.workunit.client.0.smithi023.stdout: 1 308881 12.14 MB/sec execute 23 sec latency 8.056 ms 2024-03-30T04:05:47.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 311172 11.98 MB/sec execute 24 sec latency 10.370 ms 2024-03-30T04:05:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:47 smithi154 ceph-mon[79161]: pgmap v118: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:47 smithi193 ceph-mon[79581]: pgmap v118: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:47.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:47 smithi023 ceph-mon[38254]: pgmap v118: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.2 MiB/s wr, 190 op/s 2024-03-30T04:05:48.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 312747 11.95 MB/sec execute 25 sec latency 79.813 ms 2024-03-30T04:05:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:48 smithi193 ceph-mon[79581]: pgmap v119: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 294 op/s 2024-03-30T04:05:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:48 smithi023 ceph-mon[38254]: pgmap v119: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 294 op/s 2024-03-30T04:05:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:48 smithi154 ceph-mon[79161]: pgmap v119: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 294 op/s 2024-03-30T04:05:49.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 315455 12.04 MB/sec execute 26 sec latency 16.535 ms 2024-03-30T04:05:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:49 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:05:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:49 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:05:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:49 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:05:49.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:49 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:05:49.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:49 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:05:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:49 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:05:50.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 317114 11.96 MB/sec execute 27 sec latency 11.265 ms 2024-03-30T04:05:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:50 smithi193 ceph-mon[79581]: pgmap v120: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-03-30T04:05:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:50 smithi023 ceph-mon[38254]: pgmap v120: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-03-30T04:05:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:50 smithi154 ceph-mon[79161]: pgmap v120: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 202 op/s 2024-03-30T04:05:51.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 319632 12.04 MB/sec execute 28 sec latency 12.990 ms 2024-03-30T04:05:52.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 322147 12.03 MB/sec execute 29 sec latency 11.104 ms 2024-03-30T04:05:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:52 smithi193 ceph-mon[79581]: pgmap v121: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 259 op/s 2024-03-30T04:05:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:52 smithi023 ceph-mon[38254]: pgmap v121: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 259 op/s 2024-03-30T04:05:52.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:52 smithi154 ceph-mon[79161]: pgmap v121: 129 pgs: 129 active+clean; 174 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 259 op/s 2024-03-30T04:05:53.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 323597 11.89 MB/sec execute 30 sec latency 64.441 ms 2024-03-30T04:05:54.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 326409 12.06 MB/sec execute 31 sec latency 14.072 ms 2024-03-30T04:05:54.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:54 smithi193 ceph-mon[79581]: pgmap v122: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-03-30T04:05:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:54 smithi023 ceph-mon[38254]: pgmap v122: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-03-30T04:05:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:54 smithi154 ceph-mon[79161]: pgmap v122: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 209 op/s 2024-03-30T04:05:55.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 328317 11.90 MB/sec execute 32 sec latency 16.939 ms 2024-03-30T04:05:56.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 330505 11.95 MB/sec execute 33 sec latency 11.929 ms 2024-03-30T04:05:57.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 333247 12.02 MB/sec execute 34 sec latency 9.114 ms 2024-03-30T04:05:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:57 smithi154 ceph-mon[79161]: pgmap v123: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-30T04:05:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:57 smithi193 ceph-mon[79581]: pgmap v123: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-30T04:05:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:57 smithi023 ceph-mon[38254]: pgmap v123: 129 pgs: 129 active+clean; 178 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 204 op/s 2024-03-30T04:05:58.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 334466 11.87 MB/sec execute 35 sec latency 47.184 ms 2024-03-30T04:05:58.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:05:58 smithi193 ceph-mon[79581]: pgmap v124: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 306 op/s 2024-03-30T04:05:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:05:58 smithi023 ceph-mon[38254]: pgmap v124: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 306 op/s 2024-03-30T04:05:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:05:58 smithi154 ceph-mon[79161]: pgmap v124: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 306 op/s 2024-03-30T04:05:59.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 337189 12.01 MB/sec execute 36 sec latency 21.827 ms 2024-03-30T04:06:00.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 339242 11.85 MB/sec execute 37 sec latency 21.375 ms 2024-03-30T04:06:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:00 smithi193 ceph-mon[79581]: pgmap v125: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-03-30T04:06:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:00 smithi023 ceph-mon[38254]: pgmap v125: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-03-30T04:06:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:00 smithi154 ceph-mon[79161]: pgmap v125: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 202 op/s 2024-03-30T04:06:01.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 339754 11.67 MB/sec execute 38 sec latency 73.163 ms 2024-03-30T04:06:02.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 341439 11.60 MB/sec execute 39 sec latency 18.740 ms 2024-03-30T04:06:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:02 smithi193 ceph-mon[79581]: pgmap v126: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-30T04:06:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:02 smithi023 ceph-mon[38254]: pgmap v126: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-30T04:06:02.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:02 smithi154 ceph-mon[79161]: pgmap v126: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 302 op/s 2024-03-30T04:06:03.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 344096 11.73 MB/sec execute 40 sec latency 86.348 ms 2024-03-30T04:06:04.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 345576 11.61 MB/sec execute 41 sec latency 18.373 ms 2024-03-30T04:06:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:04 smithi193 ceph-mon[79581]: pgmap v127: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 268 op/s 2024-03-30T04:06:04.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:04 smithi023 ceph-mon[38254]: pgmap v127: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 268 op/s 2024-03-30T04:06:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:04 smithi154 ceph-mon[79161]: pgmap v127: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 268 op/s 2024-03-30T04:06:05.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 348059 11.67 MB/sec execute 42 sec latency 9.674 ms 2024-03-30T04:06:06.195 INFO:tasks.workunit.client.0.smithi023.stdout: 1 350589 11.69 MB/sec execute 43 sec latency 11.742 ms 2024-03-30T04:06:07.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 352126 11.59 MB/sec execute 44 sec latency 9.075 ms 2024-03-30T04:06:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:07 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:06:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:07 smithi154 ceph-mon[79161]: pgmap v128: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 224 op/s 2024-03-30T04:06:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:07 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:06:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:07 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:06:07.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:07 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:06:07.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:07 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:06:07.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:07 smithi193 ceph-mon[79581]: pgmap v128: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 224 op/s 2024-03-30T04:06:07.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:07 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:06:07.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:07 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:06:07.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:07 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:06:07.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:07 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:06:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:07 smithi023 ceph-mon[38254]: pgmap v128: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 224 op/s 2024-03-30T04:06:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:07 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:06:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:07 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:06:07.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:07 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:06:08.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 354785 11.71 MB/sec execute 45 sec latency 90.181 ms 2024-03-30T04:06:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:08 smithi193 ceph-mon[79581]: pgmap v129: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-03-30T04:06:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:08 smithi023 ceph-mon[38254]: pgmap v129: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-03-30T04:06:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:08 smithi154 ceph-mon[79161]: pgmap v129: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 326 op/s 2024-03-30T04:06:09.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 356322 11.60 MB/sec execute 46 sec latency 16.953 ms 2024-03-30T04:06:10.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 358726 11.64 MB/sec execute 47 sec latency 7.244 ms 2024-03-30T04:06:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:10 smithi193 ceph-mon[79581]: pgmap v130: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-30T04:06:10.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:10 smithi023 ceph-mon[38254]: pgmap v130: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-30T04:06:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:10 smithi154 ceph-mon[79161]: pgmap v130: 129 pgs: 129 active+clean; 194 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-03-30T04:06:11.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 361458 11.68 MB/sec execute 48 sec latency 13.549 ms 2024-03-30T04:06:12.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 363260 11.66 MB/sec execute 49 sec latency 8.557 ms 2024-03-30T04:06:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:12 smithi193 ceph-mon[79581]: pgmap v131: 129 pgs: 129 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 322 op/s 2024-03-30T04:06:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:12 smithi023 ceph-mon[38254]: pgmap v131: 129 pgs: 129 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 322 op/s 2024-03-30T04:06:12.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:12 smithi154 ceph-mon[79161]: pgmap v131: 129 pgs: 129 active+clean; 201 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 322 op/s 2024-03-30T04:06:13.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 365680 11.71 MB/sec execute 50 sec latency 52.764 ms 2024-03-30T04:06:13.494 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 -- 172.21.15.23:0/3458431431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1060b0 msgr2=0x7fcccc10dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 --2- 172.21.15.23:0/3458431431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1060b0 0x7fcccc10dc30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fccc4009fc0 tx=0x7fccc400b330 comp rx=0 tx=0).stop 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 -- 172.21.15.23:0/3458431431 shutdown_connections 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 --2- 172.21.15.23:0/3458431431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc1060b0 0x7fcccc10dc30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 --2- 172.21.15.23:0/3458431431 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc069260 0x7fcccc069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 --2- 172.21.15.23:0/3458431431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069c10 0x7fcccc105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 -- 172.21.15.23:0/3458431431 >> 172.21.15.23:0/3458431431 conn(0x7fcccc076d80 msgr2=0x7fcccc077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 -- 172.21.15.23:0/3458431431 shutdown_connections 2024-03-30T04:06:13.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.781+0000 7fccd3a00700 1 -- 172.21.15.23:0/3458431431 wait complete. 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.782+0000 7fccd3a00700 1 Processor -- start 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.782+0000 7fccd3a00700 1 -- start start 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 0x7fcccc199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc19de10 0x7fcccc19e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc19e890 con 0x7fcccc069260 2024-03-30T04:06:13.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc19ea00 con 0x7fcccc1060b0 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd3a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc19ebc0 con 0x7fcccc19de10 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 0x7fcccc199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46168/0 (socket says 172.21.15.23:46168) 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fcccbe00700 1 -- 172.21.15.23:0/952127375 learned_addr learned my addr 172.21.15.23:0/952127375 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.783+0000 7fccd1a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc19de10 0x7fcccc19e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 -- 172.21.15.23:0/952127375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc19de10 msgr2=0x7fcccc19e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc19de10 0x7fcccc19e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 -- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 msgr2=0x7fcccc199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 0x7fcccc199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 -- 172.21.15.23:0/952127375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc4009c40 con 0x7fcccc1060b0 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccd1000700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 0x7fcccc199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:13.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fcccbe00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fccb800e970 tx=0x7fccb800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccc9600700 1 -- 172.21.15.23:0/952127375 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8009790 con 0x7fcccc1060b0 2024-03-30T04:06:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc1a6120 con 0x7fcccc1060b0 2024-03-30T04:06:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccc9600700 1 -- 172.21.15.23:0/952127375 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccb800c7c0 con 0x7fcccc1060b0 2024-03-30T04:06:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccc9600700 1 -- 172.21.15.23:0/952127375 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8005460 con 0x7fcccc1060b0 2024-03-30T04:06:13.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.784+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc1a6650 con 0x7fcccc1060b0 2024-03-30T04:06:13.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.787+0000 7fccc9600700 1 -- 172.21.15.23:0/952127375 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fccb8005620 con 0x7fcccc1060b0 2024-03-30T04:06:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.789+0000 7fcce75d9b80 1 Processor -- start 2024-03-30T04:06:13.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.789+0000 7fccc9600700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 0x7fccb40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.789+0000 7fccc9600700 1 -- 172.21.15.23:0/952127375 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fccb8014070 con 0x7fcccc1060b0 2024-03-30T04:06:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.789+0000 7fcce75d9b80 1 -- start start 2024-03-30T04:06:13.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.789+0000 7fccd1a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 0x7fccb40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fcef0 0x5599d32078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d320a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d3155990 con 0x5599d31fc5d0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d3153350 con 0x5599d3207df0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d314ec40 con 0x5599d31fcef0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33342/0 (socket says 172.21.15.23:33342) 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 -- 172.21.15.23:0/1256439885 learned_addr learned my addr 172.21.15.23:0/1256439885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1a00700 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d320a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fcccbe00700 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fcef0 0x5599d32078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 0x7fccb40a37d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fccc400b6c0 tx=0x7fccc4009f10 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 -- 172.21.15.23:0/1256439885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fcef0 msgr2=0x5599d32078b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fcef0 0x5599d32078b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 -- 172.21.15.23:0/1256439885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 msgr2=0x5599d320a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d320a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.790+0000 7fccd1000700 1 -- 172.21.15.23:0/1256439885 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599d2fd03b0 con 0x5599d31fc5d0 2024-03-30T04:06:13.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.791+0000 7fccd1000700 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fccc00115a0 tx=0x7fccc00118b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b302b710228c5cc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.791+0000 7fccbe000700 1 -- 172.21.15.23:0/1256439885 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0004230 con 0x5599d31fc5d0 2024-03-30T04:06:13.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.791+0000 7fccbe000700 1 -- 172.21.15.23:0/1256439885 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccc0004390 con 0x5599d31fc5d0 2024-03-30T04:06:13.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.791+0000 7fccbe000700 1 -- 172.21.15.23:0/1256439885 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0019960 con 0x5599d31fc5d0 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1256439885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 msgr2=0x5599d31fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fccc00115a0 tx=0x7fccc00118b0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1256439885 shutdown_connections 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d320a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fc5d0 0x5599d31fc9b0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1256439885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fcef0 0x5599d32078b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1256439885 >> 172.21.15.23:0/1256439885 conn(0x5599d315c6a0 msgr2=0x5599d31f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1256439885 shutdown_connections 2024-03-30T04:06:13.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.792+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1256439885 wait complete. 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 -- start start 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 Processor -- start 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fc5d0 0x5599d3207860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d3207df0 0x5599d32051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d30ead80 con 0x5599d31fcef0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d30eaa20 con 0x5599d31fc5d0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d2fd26a0 con 0x5599d3207df0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.793+0000 7fcce75d9b80 10 client.0 _ll_register_callbacks cb 0x5599d30f4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fccd1000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fc5d0 0x5599d3207860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33344/0 (socket says 172.21.15.23:33344) 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 -- 172.21.15.23:0/3892221676 learned_addr learned my addr 172.21.15.23:0/3892221676 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:13.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fccd1a00700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d3207df0 0x5599d32051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 -- 172.21.15.23:0/3892221676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d3207df0 msgr2=0x5599d32051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d3207df0 0x5599d32051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 -- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fc5d0 msgr2=0x5599d3207860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fc5d0 0x5599d3207860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 -- 172.21.15.23:0/3892221676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc0009d90 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fcccbe00700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fccb8099e20 tx=0x7fccb809dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abaec3128e599824 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.794+0000 7fccb2a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb809aea0 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fccb80982f0 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599d310f380 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb80b8250 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 10 client.16734 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5599d2f7cb60 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5599d310f380 con 0x5599d31fcef0 2024-03-30T04:06:13.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccb80a1e80 con 0x5599d31fcef0 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 10 client.16734 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.795+0000 7fcce75d9b80 20 client.16734 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d32179b0 0x5599d3219dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fcce75d9b80 4 client.16734 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599d3147700 con 0x5599d32179b0 2024-03-30T04:06:13.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fccd1000700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d32179b0 0x5599d3219dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fccd1000700 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d32179b0 0x5599d3219dd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d2213105b9c5a66 server_cookie=220864d8220854a5 in_seq=0 out_seq=0 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.796+0000 7fccb2a00700 10 client.16734 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fccb2a00700 1 -- 172.21.15.23:0/3892221676 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5599d3147700 con 0x5599d32179b0 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fccb2a00700 10 client.16734 handle_command_reply: tid=0 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 client.16734 shutdown 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 msgr2=0x5599d3204ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fccb8099e20 tx=0x7fccb809dd70 comp rx=0 tx=0).stop 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 shutdown_connections 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d32179b0 0x5599d3219dd0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d3207df0 0x5599d32051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fc5d0 0x5599d3207860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fccb2a00700 0 client.16734 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3892221676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d3204ae0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 >> 172.21.15.23:0/3892221676 conn(0x5599d315c6a0 msgr2=0x5599d31fe660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.797+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 shutdown_connections 2024-03-30T04:06:13.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.798+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3892221676 wait complete. 2024-03-30T04:06:13.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.798+0000 7fcce75d9b80 20 client.16734 trim_cache size 0 max 16384 2024-03-30T04:06:13.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 Processor -- start 2024-03-30T04:06:13.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 -- start start 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d3204aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d320ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d31f15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d320efb0 con 0x5599d31fcef0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d31f1ae0 con 0x5599d3207df0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d31f8f20 con 0x5599d31fc5d0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d320ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d3204aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d320ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33352/0 (socket says 172.21.15.23:33352) 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fcccbe00700 1 -- 172.21.15.23:0/3180870735 learned_addr learned my addr 172.21.15.23:0/3180870735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1a00700 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d31f15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 -- 172.21.15.23:0/3180870735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 msgr2=0x5599d31f15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d31f15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 -- 172.21.15.23:0/3180870735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 msgr2=0x5599d320ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d320ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.816+0000 7fccd1000700 1 -- 172.21.15.23:0/3180870735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599d3114ea0 con 0x5599d31fc5d0 2024-03-30T04:06:13.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fccd1000700 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d3204aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccc0011570 tx=0x7fccc00118b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3b01861cd649bf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fccbea00700 1 -- 172.21.15.23:0/3180870735 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0004130 con 0x5599d31fc5d0 2024-03-30T04:06:13.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fccbea00700 1 -- 172.21.15.23:0/3180870735 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccc0003b40 con 0x5599d31fc5d0 2024-03-30T04:06:13.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fccbea00700 1 -- 172.21.15.23:0/3180870735 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0005240 con 0x5599d31fc5d0 2024-03-30T04:06:13.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3180870735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 msgr2=0x5599d3204aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d3204aa0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccc0011570 tx=0x7fccc00118b0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3180870735 shutdown_connections 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d3207df0 0x5599d31f15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d3204aa0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/3180870735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d31fcef0 0x5599d320ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.817+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3180870735 >> 172.21.15.23:0/3180870735 conn(0x5599d315c6a0 msgr2=0x5599d3204370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.818+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3180870735 shutdown_connections 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.818+0000 7fcce75d9b80 1 -- 172.21.15.23:0/3180870735 wait complete. 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.818+0000 7fcce75d9b80 1 -- start start 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.818+0000 7fcce75d9b80 1 Processor -- start 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d320e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d3207df0 0x5599d320d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d320d6d0 con 0x5599d3207df0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d320d840 con 0x5599d31fcef0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599d320da00 con 0x5599d31fc5d0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcce75d9b80 10 client.0 _ll_register_callbacks cb 0x5599d30f4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fccd1000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d320e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcccbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46212/0 (socket says 172.21.15.23:46212) 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fccd1a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d3207df0 0x5599d320d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.819+0000 7fcccbe00700 1 -- 172.21.15.23:0/1198308482 learned_addr learned my addr 172.21.15.23:0/1198308482 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 -- 172.21.15.23:0/1198308482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 msgr2=0x5599d320e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d320e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 -- 172.21.15.23:0/1198308482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d3207df0 msgr2=0x5599d320d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:13.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d3207df0 0x5599d320d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 -- 172.21.15.23:0/1198308482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599d2fd03b0 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcccbe00700 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fccb8099e20 tx=0x7fccb8005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=368948a083d9db22 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fccb2a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb809aea0 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fccb80982f0 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccc0004390 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb8004a80 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcce75d9b80 10 client.26720 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.820+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5599d2fd4cd0 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccc0004390 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccb809a0f0 con 0x5599d31fcef0 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 10 client.26720 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 10 client.26720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:13.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 20 client.26720 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:13.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d326ca00 0x5599d326edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:13.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 4 client.26720 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:06:13.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5599d2f9dcf0 con 0x5599d326ca00 2024-03-30T04:06:13.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.821+0000 7fccd1000700 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d326ca00 0x5599d326edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fccd1000700 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d326ca00 0x5599d326edf0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1ea1e003b1ced0 server_cookie=4db569864b71aab0 in_seq=0 out_seq=0 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fccb2a00700 10 client.26720 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fccb2a00700 1 -- 172.21.15.23:0/1198308482 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5599d2f9dcf0 con 0x5599d326ca00 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fccb2a00700 10 client.26720 handle_command_reply: tid=0 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 client.26720 shutdown 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 msgr2=0x5599d320cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fccb8099e20 tx=0x7fccb8005bb0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 shutdown_connections 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5599d3207df0 0x5599d320d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5599d326ca00 0x5599d326edf0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5599d31fc5d0 0x5599d320e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 --2- 172.21.15.23:0/1198308482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5599d31fcef0 0x5599d320cb40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 >> 172.21.15.23:0/1198308482 conn(0x5599d315c6a0 msgr2=0x5599d32033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fccb2a00700 0 client.26720 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 shutdown_connections 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.822+0000 7fcce75d9b80 1 -- 172.21.15.23:0/1198308482 wait complete. 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fcce75d9b80 20 client.26720 trim_cache size 0 max 16384 2024-03-30T04:06:13.826 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:13.827 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:06:13.827 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:06:13.827 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:13.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 msgr2=0x7fccb40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 0x7fccb40a37d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fccc400b6c0 tx=0x7fccc4009f10 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 msgr2=0x7fcccc199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fccb800e970 tx=0x7fccb800ec80 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 shutdown_connections 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcccc19de10 0x7fcccc19e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcccc069260 0x7fcccc199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fccb40a1310 0x7fccb40a37d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 --2- 172.21.15.23:0/952127375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcccc1060b0 0x7fcccc199a70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 >> 172.21.15.23:0/952127375 conn(0x7fcccc076d80 msgr2=0x7fcccc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 shutdown_connections 2024-03-30T04:06:13.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:13.823+0000 7fccd3a00700 1 -- 172.21.15.23:0/952127375 wait complete. 2024-03-30T04:06:13.841 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:06:13.841 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:06:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 -- 172.21.15.23:0/2659990824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9894101810 msgr2=0x7f9894105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 --2- 172.21.15.23:0/2659990824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9894101810 0x7f9894105850 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9880009990 tx=0x7f9880009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 -- 172.21.15.23:0/2659990824 shutdown_connections 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 --2- 172.21.15.23:0/2659990824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f989410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 --2- 172.21.15.23:0/2659990824 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f9894101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 --2- 172.21.15.23:0/2659990824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9894101810 0x7f9894105850 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.054+0000 7f9899e00700 1 -- 172.21.15.23:0/2659990824 >> 172.21.15.23:0/2659990824 conn(0x7f98940fc700 msgr2=0x7f98940feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.055+0000 7f9899e00700 1 -- 172.21.15.23:0/2659990824 shutdown_connections 2024-03-30T04:06:14.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.055+0000 7f9899e00700 1 -- 172.21.15.23:0/2659990824 wait complete. 2024-03-30T04:06:14.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 Processor -- start 2024-03-30T04:06:14.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 -- start start 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f9894199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 0x7f989419e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989419e630 con 0x7f9894100e60 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989419e7a0 con 0x7f989419dbb0 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.056+0000 7f9899e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989419e960 con 0x7f9894105f80 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9893400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 0x7f989419e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33382/0 (socket says 172.21.15.23:33382) 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f9894199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 -- 172.21.15.23:0/2299353682 learned_addr learned my addr 172.21.15.23:0/2299353682 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 -- 172.21.15.23:0/2299353682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 msgr2=0x7f9894199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f9894199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 -- 172.21.15.23:0/2299353682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 msgr2=0x7f989419e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 0x7f989419e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 -- 172.21.15.23:0/2299353682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98800095f0 con 0x7f9894100e60 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892000700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f9894199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9893400700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 0x7f989419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.057+0000 7f9892a00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f988400e9a0 tx=0x7f988400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.058+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884009790 con 0x7f9894100e60 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.058+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9884010430 con 0x7f9894100e60 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.058+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98941a5ec0 con 0x7f9894100e60 2024-03-30T04:06:14.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.058+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884018680 con 0x7f9894100e60 2024-03-30T04:06:14.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.058+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98941a6340 con 0x7f9894100e60 2024-03-30T04:06:14.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.059+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9894102330 con 0x7f9894100e60 2024-03-30T04:06:14.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.061+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f98840105a0 con 0x7f9894100e60 2024-03-30T04:06:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.062+0000 7f9898a00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 0x7f98780a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.062+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f9884014070 con 0x7f9894100e60 2024-03-30T04:06:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.062+0000 7f9892000700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 0x7f98780a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.063+0000 7f9892000700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 0x7f98780a3750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f989419a820 tx=0x7f988000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.065+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9884096250 con 0x7f9894100e60 2024-03-30T04:06:14.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 367818 11.63 MB/sec execute 51 sec latency 15.762 ms 2024-03-30T04:06:14.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.363+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9894067a10 con 0x7f9894100e60 2024-03-30T04:06:14.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.364+0000 7f9898a00700 1 -- 172.21.15.23:0/2299353682 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f9884099870 con 0x7f9894100e60 2024-03-30T04:06:14.363 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:06:14.364 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:06:14.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 msgr2=0x7f98780a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 0x7f98780a3750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f989419a820 tx=0x7f988000b260 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 msgr2=0x7f98941992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f988400e9a0 tx=0x7f988400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 shutdown_connections 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f98780a1290 0x7f98780a3750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f989419dbb0 0x7f989419e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9894100e60 0x7f98941992d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 --2- 172.21.15.23:0/2299353682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9894105f80 0x7f9894199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 >> 172.21.15.23:0/2299353682 conn(0x7f98940fc700 msgr2=0x7f98940fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 shutdown_connections 2024-03-30T04:06:14.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.367+0000 7f9899e00700 1 -- 172.21.15.23:0/2299353682 wait complete. 2024-03-30T04:06:14.368 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:06:14.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:06:14.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:14 smithi023 ceph-mon[38254]: pgmap v132: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:06:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:14 smithi193 ceph-mon[79581]: pgmap v132: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 -- 172.21.15.23:0/373065204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e780731e0 msgr2=0x7f3e780735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/373065204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e780731e0 0x7f3e780735c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009990 tx=0x7f3e68009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 -- 172.21.15.23:0/373065204 shutdown_connections 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/373065204 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e7810d540 0x7f3e7810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/373065204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e780731e0 0x7f3e780735c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/373065204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e7810d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 -- 172.21.15.23:0/373065204 >> 172.21.15.23:0/373065204 conn(0x7f3e780fc7f0 msgr2=0x7f3e780fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 -- 172.21.15.23:0/373065204 shutdown_connections 2024-03-30T04:06:14.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.607+0000 7f3e7dc00700 1 -- 172.21.15.23:0/373065204 wait complete. 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.608+0000 7f3e7dc00700 1 Processor -- start 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.608+0000 7f3e7dc00700 1 -- start start 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e780731e0 0x7f3e78199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e78199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 0x7f3e7819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819e320 con 0x7f3e780731e0 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819e490 con 0x7f3e7810d540 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e7dc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7819e650 con 0x7f3e78073b00 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e77400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 0x7f3e7819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e78199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e78199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35598/0 (socket says 172.21.15.23:35598) 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e76000700 1 -- 172.21.15.23:0/192632055 learned_addr learned my addr 172.21.15.23:0/192632055 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.609+0000 7f3e76a00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e780731e0 0x7f3e78199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 -- 172.21.15.23:0/192632055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 msgr2=0x7f3e78199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e78199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 -- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e780731e0 msgr2=0x7f3e78199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e780731e0 0x7f3e78199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 -- 172.21.15.23:0/192632055 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e70009c40 con 0x7f3e7810d540 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e77400700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 0x7f3e7819dce0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000efb0 tx=0x7f3e7000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e6f400700 1 -- 172.21.15.23:0/192632055 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70003a00 con 0x7f3e7810d540 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e6f400700 1 -- 172.21.15.23:0/192632055 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e7000f040 con 0x7f3e7810d540 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e680095f0 con 0x7f3e7810d540 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e6f400700 1 -- 172.21.15.23:0/192632055 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70004540 con 0x7f3e7810d540 2024-03-30T04:06:14.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.610+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7819eb20 con 0x7f3e7810d540 2024-03-30T04:06:14.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.613+0000 7f3e6f400700 1 -- 172.21.15.23:0/192632055 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3e70003bf0 con 0x7f3e7810d540 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.614+0000 7f3e91715b80 1 Processor -- start 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 -- start start 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e6f400700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 0x7f3e5c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e6a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e6f400700 1 -- 172.21.15.23:0/192632055 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3e700ccbc0 con 0x7f3e7810d540 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 0x55bce5e6d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5db8990 con 0x55bce5e5fef0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5db6350 con 0x55bce5e5f5d0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e91715b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5db1c40 con 0x55bce5e6adf0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46256/0 (socket says 172.21.15.23:46256) 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e76000700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 0x7f3e5c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.615+0000 7f3e76a00700 1 -- 172.21.15.23:0/2687433210 learned_addr learned my addr 172.21.15.23:0/2687433210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76000700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e6a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e77400700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 0x55bce5e6d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 -- 172.21.15.23:0/2687433210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 msgr2=0x55bce5e6d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 0x55bce5e6d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 -- 172.21.15.23:0/2687433210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 msgr2=0x55bce5e6a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e6a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 -- 172.21.15.23:0/2687433210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bce5c333b0 con 0x55bce5e5f5d0 2024-03-30T04:06:14.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76000700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e6a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e77400700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 0x55bce5e6d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76a00700 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3e68005ec0 tx=0x7f3e68005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb90b4b7a451e74f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e76000700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 0x7f3e5c0a37d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009880 tx=0x7f3e64009a90 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e63e00700 1 -- 172.21.15.23:0/2687433210 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6801d070 con 0x55bce5e5f5d0 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e63e00700 1 -- 172.21.15.23:0/2687433210 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e6801e040 con 0x55bce5e5f5d0 2024-03-30T04:06:14.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.616+0000 7f3e63e00700 1 -- 172.21.15.23:0/2687433210 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e680049c0 con 0x55bce5e5f5d0 2024-03-30T04:06:14.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 -- 172.21.15.23:0/2687433210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 msgr2=0x55bce5e5f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3e68005ec0 tx=0x7f3e68005ef0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 -- 172.21.15.23:0/2687433210 shutdown_connections 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e6adf0 0x55bce5e6d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e5f5d0 0x55bce5e5f9b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 --2- 172.21.15.23:0/2687433210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e6a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 -- 172.21.15.23:0/2687433210 >> 172.21.15.23:0/2687433210 conn(0x55bce5dbf6a0 msgr2=0x55bce5e5ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 -- 172.21.15.23:0/2687433210 shutdown_connections 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.617+0000 7f3e91715b80 1 -- 172.21.15.23:0/2687433210 wait complete. 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 -- start start 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 Processor -- start 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5d4dd80 con 0x55bce5e5fef0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5d4da20 con 0x55bce5e6adf0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.618+0000 7f3e91715b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5c356a0 con 0x55bce5e5f5d0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e91715b80 10 client.0 _ll_register_callbacks cb 0x55bce5d57470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33422/0 (socket says 172.21.15.23:33422) 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76000700 1 -- 172.21.15.23:0/3355822669 learned_addr learned my addr 172.21.15.23:0/3355822669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 -- 172.21.15.23:0/3355822669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 msgr2=0x55bce5e6a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 -- 172.21.15.23:0/3355822669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 msgr2=0x55bce5e67ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e77400700 1 -- 172.21.15.23:0/3355822669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e78110670 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76000700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.619+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e77400700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e680f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3e700c9f50 tx=0x7f3e70099380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa1fc18b6d3b1085 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70099e00 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3e700048b0 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bce5d72380 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e7009d7d0 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e91715b80 10 client.26738 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55bce5bdfb60 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bce5d72380 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.620+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3e70004bd0 con 0x55bce5e6adf0 2024-03-30T04:06:14.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 10 client.26738 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 10 client.26738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 20 client.26738 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5e7a910 0x55bce5e7cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 4 client.26738 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bce5daa700 con 0x55bce5e7a910 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5e7a910 0x55bce5e7cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5e7a910 0x55bce5e7cd30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ef86a97925d068c server_cookie=af5f8e7097857f19 in_seq=0 out_seq=0 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.621+0000 7f3e60c00700 10 client.26738 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e60c00700 1 -- 172.21.15.23:0/3355822669 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bce5daa700 con 0x55bce5e7a910 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e60c00700 10 client.26738 handle_command_reply: tid=0 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 client.26738 shutdown 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 msgr2=0x55bce5e680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e680f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3e700c9f50 tx=0x7f3e70099380 comp rx=0 tx=0).stop 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 shutdown_connections 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e680f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5e7a910 0x55bce5e7cd30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3355822669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5fef0 0x55bce5e67ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e60c00700 0 client.26738 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 >> 172.21.15.23:0/3355822669 conn(0x55bce5dbf6a0 msgr2=0x55bce5e616c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.622+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 shutdown_connections 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.623+0000 7f3e91715b80 1 -- 172.21.15.23:0/3355822669 wait complete. 2024-03-30T04:06:14.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.623+0000 7f3e91715b80 20 client.26738 trim_cache size 0 max 16384 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 Processor -- start 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 -- start start 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e62df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e63330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e663d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5e62930 con 0x55bce5e5f5d0 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5e5bfc0 con 0x55bce5e6adf0 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e91715b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5e5c7d0 con 0x55bce5e5fef0 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e63330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e63330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35630/0 (socket says 172.21.15.23:35630) 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e76000700 1 -- 172.21.15.23:0/1514108066 learned_addr learned my addr 172.21.15.23:0/1514108066 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e77400700 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.639+0000 7f3e76a00700 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e62df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 -- 172.21.15.23:0/1514108066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 msgr2=0x55bce5e63330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e63330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 -- 172.21.15.23:0/1514108066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 msgr2=0x55bce5e62df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e62df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 -- 172.21.15.23:0/1514108066 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bce5d77ea0 con 0x55bce5e6adf0 2024-03-30T04:06:14.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e77400700 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e663d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000f80 tx=0x7f3e700094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772b58032092bc59 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e6cc00700 1 -- 172.21.15.23:0/1514108066 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700048b0 con 0x55bce5e6adf0 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e6cc00700 1 -- 172.21.15.23:0/1514108066 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e7009d7d0 con 0x55bce5e6adf0 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.640+0000 7f3e6cc00700 1 -- 172.21.15.23:0/1514108066 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70099c00 con 0x55bce5e6adf0 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 -- 172.21.15.23:0/1514108066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 msgr2=0x55bce5e663d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e663d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000f80 tx=0x7f3e700094e0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 -- 172.21.15.23:0/1514108066 shutdown_connections 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e663d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e62df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 --2- 172.21.15.23:0/1514108066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e63330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 -- 172.21.15.23:0/1514108066 >> 172.21.15.23:0/1514108066 conn(0x55bce5dbf6a0 msgr2=0x55bce5e545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 -- 172.21.15.23:0/1514108066 shutdown_connections 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.641+0000 7f3e91715b80 1 -- 172.21.15.23:0/1514108066 wait complete. 2024-03-30T04:06:14.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.642+0000 7f3e91715b80 1 -- start start 2024-03-30T04:06:14.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.642+0000 7f3e91715b80 1 Processor -- start 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5ecd870 con 0x55bce5e5f5d0 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5ecd9e0 con 0x55bce5e6adf0 2024-03-30T04:06:14.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bce5ecdba0 con 0x55bce5e5fef0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e91715b80 10 client.0 _ll_register_callbacks cb 0x55bce5d57470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e76a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46276/0 (socket says 172.21.15.23:46276) 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35646/0 (socket says 172.21.15.23:35646) 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 -- 172.21.15.23:0/3717496021 learned_addr learned my addr 172.21.15.23:0/3717496021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 -- 172.21.15.23:0/3717496021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 msgr2=0x55bce5e6ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 -- 172.21.15.23:0/3717496021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 msgr2=0x55bce5e6f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e77400700 1 -- 172.21.15.23:0/3717496021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bce5bdfb60 con 0x55bce5e6adf0 2024-03-30T04:06:14.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.643+0000 7f3e76000700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e77400700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000c6f0 tx=0x7f3e70099380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d28f01f80e14e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70099e00 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f3e700048b0 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e6801e040 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e700a1d90 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e91715b80 10 client.26744 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55bce5c37cd0 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e700a5020 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.644+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3e70098900 con 0x55bce5e6adf0 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 10 client.26744 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 10 client.26744 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 20 client.26744 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:14.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5ecee50 0x55bce5ed1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 4 client.26744 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bce5c00cf0 con 0x55bce5ecee50 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5ecee50 0x55bce5ed1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e60c00700 10 client.26744 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.645+0000 7f3e76a00700 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5ecee50 0x55bce5ed1240 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8c9fc9556a394f01 server_cookie=f117005175d3957e in_seq=0 out_seq=0 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e60c00700 1 -- 172.21.15.23:0/3717496021 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bce5c00cf0 con 0x55bce5ecee50 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e60c00700 10 client.26744 handle_command_reply: tid=0 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 client.26744 shutdown 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 msgr2=0x55bce5e70470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000c6f0 tx=0x7f3e70099380 comp rx=0 tx=0).stop 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 shutdown_connections 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bce5e6adf0 0x55bce5e70470 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bce5e5f5d0 0x55bce5e6f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55bce5ecee50 0x55bce5ed1240 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 --2- 172.21.15.23:0/3717496021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bce5e5fef0 0x55bce5e6ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 >> 172.21.15.23:0/3717496021 conn(0x55bce5dbf6a0 msgr2=0x55bce5e62df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e60c00700 0 client.26744 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 shutdown_connections 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 1 -- 172.21.15.23:0/3717496021 wait complete. 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.646+0000 7f3e91715b80 20 client.26744 trim_cache size 0 max 16384 2024-03-30T04:06:14.650 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 msgr2=0x7f3e5c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 0x7f3e5c0a37d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009880 tx=0x7f3e64009a90 comp rx=0 tx=0).stop 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 msgr2=0x7f3e7819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 0x7f3e7819dce0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3e7000efb0 tx=0x7f3e7000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 shutdown_connections 2024-03-30T04:06:14.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3e5c0a1310 0x7f3e5c0a37d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7810d540 0x7f3e7819dce0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e780731e0 0x7f3e78199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 --2- 172.21.15.23:0/192632055 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e78073b00 0x7f3e78199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 >> 172.21.15.23:0/192632055 conn(0x7f3e780fc7f0 msgr2=0x7f3e780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 shutdown_connections 2024-03-30T04:06:14.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.647+0000 7f3e7dc00700 1 -- 172.21.15.23:0/192632055 wait complete. 2024-03-30T04:06:14.666 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:06:14.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:14 smithi154 ceph-mon[79161]: pgmap v132: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 -- 172.21.15.23:0/4190207583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0100f60 msgr2=0x7fb2d0101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/4190207583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0100f60 0x7fb2d0101340 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0009990 tx=0x7fb2c0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 -- 172.21.15.23:0/4190207583 shutdown_connections 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/4190207583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d0106080 0x7fb2d010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/4190207583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0100f60 0x7fb2d0101340 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/4190207583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0101910 0x7fb2d0105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 -- 172.21.15.23:0/4190207583 >> 172.21.15.23:0/4190207583 conn(0x7fb2d00fc7c0 msgr2=0x7fb2d00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 -- 172.21.15.23:0/4190207583 shutdown_connections 2024-03-30T04:06:14.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.892+0000 7fb2d5e00700 1 -- 172.21.15.23:0/4190207583 wait complete. 2024-03-30T04:06:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.893+0000 7fb2d5e00700 1 Processor -- start 2024-03-30T04:06:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 -- start start 2024-03-30T04:06:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0106080 0x7fb2d0199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d019dce0 0x7fb2d019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d019e760 con 0x7fb2d0106080 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d019e8d0 con 0x7fb2d0100f60 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2d5e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d019ea90 con 0x7fb2d019dce0 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46282/0 (socket says 172.21.15.23:46282) 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2cea00700 1 -- 172.21.15.23:0/3708921763 learned_addr learned my addr 172.21.15.23:0/3708921763 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2ce000700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0106080 0x7fb2d0199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.894+0000 7fb2cf400700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d019dce0 0x7fb2d019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 -- 172.21.15.23:0/3708921763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d019dce0 msgr2=0x7fb2d019e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d019dce0 0x7fb2d019e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 -- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0106080 msgr2=0x7fb2d0199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0106080 0x7fb2d0199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 -- 172.21.15.23:0/3708921763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c00095f0 con 0x7fb2d0100f60 2024-03-30T04:06:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2cea00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0004760 tx=0x7fb2c0004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2d4a00700 1 -- 172.21.15.23:0/3708921763 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c001d070 con 0x7fb2d0100f60 2024-03-30T04:06:14.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2d4a00700 1 -- 172.21.15.23:0/3708921763 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2c0022470 con 0x7fb2d0100f60 2024-03-30T04:06:14.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2d4a00700 1 -- 172.21.15.23:0/3708921763 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c0017650 con 0x7fb2d0100f60 2024-03-30T04:06:14.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.895+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d01a5f90 con 0x7fb2d0100f60 2024-03-30T04:06:14.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.896+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d01a6300 con 0x7fb2d0100f60 2024-03-30T04:06:14.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.899+0000 7fb2d4a00700 1 -- 172.21.15.23:0/3708921763 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb2c00225e0 con 0x7fb2d0100f60 2024-03-30T04:06:14.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.900+0000 7fb2d4a00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 0x7fb2b40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.900+0000 7fb2d4a00700 1 -- 172.21.15.23:0/3708921763 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb2c00ce120 con 0x7fb2d0100f60 2024-03-30T04:06:14.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2ce000700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 0x7fb2b40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 Processor -- start 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 -- start start 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2ce000700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 0x7fb2b40a37d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2d019a950 tx=0x7fb2bc009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c2185d0 0x55a62c2189b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c171990 con 0x55a62c2185d0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c16f350 con 0x55a62c218ef0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.901+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c16ac40 con 0x55a62c223df0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46298/0 (socket says 172.21.15.23:46298) 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 -- 172.21.15.23:0/1297369899 learned_addr learned my addr 172.21.15.23:0/1297369899 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2cf400700 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2cea00700 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c2185d0 0x55a62c2189b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 -- 172.21.15.23:0/1297369899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 msgr2=0x55a62c2261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 -- 172.21.15.23:0/1297369899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c2185d0 msgr2=0x55a62c2189b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c2185d0 0x55a62c2189b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 -- 172.21.15.23:0/1297369899 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a62bfec3b0 con 0x55a62c218ef0 2024-03-30T04:06:14.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.902+0000 7fb2ce000700 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc011d20 tx=0x7fb2bc014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf187ada24315d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.903+0000 7fb2bbe00700 1 -- 172.21.15.23:0/1297369899 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00eb20 con 0x55a62c218ef0 2024-03-30T04:06:14.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.903+0000 7fb2bbe00700 1 -- 172.21.15.23:0/1297369899 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2bc021070 con 0x55a62c218ef0 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.903+0000 7fb2bbe00700 1 -- 172.21.15.23:0/1297369899 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc01d670 con 0x55a62c218ef0 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.903+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1297369899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 msgr2=0x55a62c2238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.903+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc011d20 tx=0x7fb2bc014000 comp rx=0 tx=0).stop 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1297369899 shutdown_connections 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c2185d0 0x55a62c2189b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1297369899 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c218ef0 0x55a62c2238b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1297369899 >> 172.21.15.23:0/1297369899 conn(0x55a62c1786a0 msgr2=0x55a62c214a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1297369899 shutdown_connections 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1297369899 wait complete. 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 -- start start 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.904+0000 7fb2e99aeb80 1 Processor -- start 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c220b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c106d80 con 0x55a62c218ef0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c106a20 con 0x55a62c2185d0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62bfee6a0 con 0x55a62c223df0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2e99aeb80 10 client.0 _ll_register_callbacks cb 0x55a62c110470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2ce000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c220b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2cf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46304/0 (socket says 172.21.15.23:46304) 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.905+0000 7fb2cea00700 1 -- 172.21.15.23:0/4212414181 learned_addr learned my addr 172.21.15.23:0/4212414181 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 -- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 msgr2=0x55a62c2212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 -- 172.21.15.23:0/4212414181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 msgr2=0x55a62c220b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c220b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 -- 172.21.15.23:0/4212414181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2bc017070 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2ce000700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c220b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2cea00700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0005cb0 tx=0x7fb2c009aac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42aa34bcde30fef7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2b8c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c009a5f0 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb2c009b600 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a62c12b380 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2c009af00 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2e99aeb80 10 client.26756 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.906+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a62bf98b60 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a62c12b380 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb2c0096cf0 con 0x55a62c2185d0 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 10 client.26756 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 10 client.26756 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:06:14.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 20 client.26756 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:14.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c2335e0 0x55a62c2359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 4 client.26756 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:14.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a62c163700 con 0x55a62c2335e0 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.907+0000 7fb2ce000700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c2335e0 0x55a62c2359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.908+0000 7fb2ce000700 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c2335e0 0x55a62c2359d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9456c275c698fbd0 server_cookie=bf625a6ca6b6bc03 in_seq=0 out_seq=0 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.908+0000 7fb2b8c00700 10 client.26756 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2b8c00700 1 -- 172.21.15.23:0/4212414181 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a62c163700 con 0x55a62c2335e0 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2b8c00700 10 client.26756 handle_command_reply: tid=0 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 client.26756 shutdown 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 msgr2=0x55a62c223880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0005cb0 tx=0x7fb2c009aac0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 shutdown_connections 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c2335e0 0x55a62c2359d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c223df0 0x55a62c2212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c223880 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2b8c00700 0 client.26756 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/4212414181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c220b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 >> 172.21.15.23:0/4212414181 conn(0x55a62c1786a0 msgr2=0x55a62c21a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 shutdown_connections 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/4212414181 wait complete. 2024-03-30T04:06:14.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.909+0000 7fb2e99aeb80 20 client.26756 trim_cache size 0 max 16384 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.924+0000 7fb2e99aeb80 1 Processor -- start 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.924+0000 7fb2e99aeb80 1 -- start start 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c218ef0 0x55a62c213f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c223df0 0x55a62c22aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c219890 con 0x55a62c223df0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c22afb0 con 0x55a62c2185d0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c21f0c0 con 0x55a62c218ef0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46314/0 (socket says 172.21.15.23:46314) 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 -- 172.21.15.23:0/2976773435 learned_addr learned my addr 172.21.15.23:0/2976773435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cf400700 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c223df0 0x55a62c22aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2ce000700 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c218ef0 0x55a62c213f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 -- 172.21.15.23:0/2976773435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c218ef0 msgr2=0x55a62c213f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c218ef0 0x55a62c213f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 -- 172.21.15.23:0/2976773435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c223df0 msgr2=0x55a62c22aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.925+0000 7fb2cea00700 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c223df0 0x55a62c22aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.926+0000 7fb2cea00700 1 -- 172.21.15.23:0/2976773435 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a62c130ea0 con 0x55a62c2185d0 2024-03-30T04:06:14.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.926+0000 7fb2cea00700 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0004730 tx=0x7fb2c0022cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f33b4539a62962 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.926+0000 7fb2c5200700 1 -- 172.21.15.23:0/2976773435 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c009b600 con 0x55a62c2185d0 2024-03-30T04:06:14.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.926+0000 7fb2c5200700 1 -- 172.21.15.23:0/2976773435 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2c009af00 con 0x55a62c2185d0 2024-03-30T04:06:14.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.926+0000 7fb2c5200700 1 -- 172.21.15.23:0/2976773435 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c009a3f0 con 0x55a62c2185d0 2024-03-30T04:06:14.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/2976773435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 msgr2=0x55a62c2212b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0004730 tx=0x7fb2c0022cb0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/2976773435 shutdown_connections 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c223df0 0x55a62c22aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c2185d0 0x55a62c2212b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/2976773435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c218ef0 0x55a62c213f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/2976773435 >> 172.21.15.23:0/2976773435 conn(0x55a62c1786a0 msgr2=0x55a62c20d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/2976773435 shutdown_connections 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/2976773435 wait complete. 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.927+0000 7fb2e99aeb80 1 -- start start 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 Processor -- start 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c228f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c223df0 0x55a62c2294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c2864b0 con 0x55a62c218ef0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c2865f0 con 0x55a62c223df0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a62c2867b0 con 0x55a62c2185d0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2e99aeb80 10 client.0 _ll_register_callbacks cb 0x55a62c110470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.928+0000 7fb2cea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35682/0 (socket says 172.21.15.23:35682) 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 -- 172.21.15.23:0/1978173608 learned_addr learned my addr 172.21.15.23:0/1978173608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cf400700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c223df0 0x55a62c2294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 -- 172.21.15.23:0/1978173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c223df0 msgr2=0x55a62c2294d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c223df0 0x55a62c2294d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 -- 172.21.15.23:0/1978173608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 msgr2=0x55a62c228f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c228f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 -- 172.21.15.23:0/1978173608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a62bf98b60 con 0x55a62c2185d0 2024-03-30T04:06:14.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2cea00700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00970e0 tx=0x7fb2c0096f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f20d8b6ce646deee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2b8c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c009ed30 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fb2c009b600 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.929+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2bc021070 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2c00a32f0 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 10 client.26758 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a62bff0cd0 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb2bc021070 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb2c000fb20 con 0x55a62c2185d0 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 10 client.26758 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 10 client.26758 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 20 client.26758 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c287a80 0x55a62c289e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:14.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 4 client.26758 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:14.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a62bfb9cf0 con 0x55a62c287a80 2024-03-30T04:06:14.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.930+0000 7fb2ce000700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c287a80 0x55a62c289e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2ce000700 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c287a80 0x55a62c289e70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=feb7520beffdb2e1 server_cookie=b8304a70f41021c9 in_seq=0 out_seq=0 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2b8c00700 10 client.26758 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2b8c00700 1 -- 172.21.15.23:0/1978173608 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a62bfb9cf0 con 0x55a62c287a80 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2b8c00700 10 client.26758 handle_command_reply: tid=0 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 client.26758 shutdown 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 msgr2=0x55a62c228a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00970e0 tx=0x7fb2c0096f20 comp rx=0 tx=0).stop 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 shutdown_connections 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55a62c287a80 0x55a62c289e70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a62c223df0 0x55a62c2294d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a62c2185d0 0x55a62c228a50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 --2- 172.21.15.23:0/1978173608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a62c218ef0 0x55a62c228f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2b8c00700 0 client.26758 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 >> 172.21.15.23:0/1978173608 conn(0x55a62c1786a0 msgr2=0x55a62c21be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 shutdown_connections 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.931+0000 7fb2e99aeb80 1 -- 172.21.15.23:0/1978173608 wait complete. 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2e99aeb80 20 client.26758 trim_cache size 0 max 16384 2024-03-30T04:06:14.935 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 msgr2=0x7fb2b40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 0x7fb2b40a37d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb2d019a950 tx=0x7fb2bc009260 comp rx=0 tx=0).stop 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 msgr2=0x7fb2d0199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2c0004760 tx=0x7fb2c0004790 comp rx=0 tx=0).stop 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 shutdown_connections 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d019dce0 0x7fb2d019e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb2b40a1310 0x7fb2b40a37d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2d0100f60 0x7fb2d0199400 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 --2- 172.21.15.23:0/3708921763 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2d0106080 0x7fb2d0199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:14.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 >> 172.21.15.23:0/3708921763 conn(0x7fb2d00fc7c0 msgr2=0x7fb2d00fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:14.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 shutdown_connections 2024-03-30T04:06:14.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:14.932+0000 7fb2d5e00700 1 -- 172.21.15.23:0/3708921763 wait complete. 2024-03-30T04:06:14.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:06:15.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 -- 172.21.15.23:0/2530542389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101900 msgr2=0x7f334c105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 --2- 172.21.15.23:0/2530542389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101900 0x7f334c105940 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f33380098f0 tx=0x7f3338009c00 comp rx=0 tx=0).stop 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 -- 172.21.15.23:0/2530542389 shutdown_connections 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 --2- 172.21.15.23:0/2530542389 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 0x7f334c10dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 --2- 172.21.15.23:0/2530542389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c100f50 0x7f334c101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 --2- 172.21.15.23:0/2530542389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101900 0x7f334c105940 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 -- 172.21.15.23:0/2530542389 >> 172.21.15.23:0/2530542389 conn(0x7f334c0fc7f0 msgr2=0x7f334c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.186+0000 7f3350c00700 1 -- 172.21.15.23:0/2530542389 shutdown_connections 2024-03-30T04:06:15.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.187+0000 7f3350c00700 1 -- 172.21.15.23:0/2530542389 wait complete. 2024-03-30T04:06:15.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.188+0000 7f3350c00700 1 Processor -- start 2024-03-30T04:06:15.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.188+0000 7f3350c00700 1 -- start start 2024-03-30T04:06:15.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.188+0000 7f3350c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3350c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 0x7f334c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3350c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c19de50 0x7f334c19e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3350c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c19e8d0 con 0x7f334c19de50 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3350c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c19ea40 con 0x7f334c106070 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3350c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c19ec00 con 0x7f334c100f50 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35700/0 (socket says 172.21.15.23:35700) 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 -- 172.21.15.23:0/3462776346 learned_addr learned my addr 172.21.15.23:0/3462776346 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f334a800700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c19de50 0x7f334c19e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349400700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 0x7f334c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 -- 172.21.15.23:0/3462776346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 msgr2=0x7f334c199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 0x7f334c199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 -- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c19de50 msgr2=0x7f334c19e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c19de50 0x7f334c19e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.189+0000 7f3349e00700 1 -- 172.21.15.23:0/3462776346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3338009520 con 0x7f334c100f50 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3349e00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f333c00e9c0 tx=0x7f333c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3342a00700 1 -- 172.21.15.23:0/3462776346 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c004a30 con 0x7f334c100f50 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3342a00700 1 -- 172.21.15.23:0/3462776346 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f333c00cc80 con 0x7f334c100f50 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c1a6160 con 0x7f334c100f50 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3342a00700 1 -- 172.21.15.23:0/3462776346 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0054e0 con 0x7f334c100f50 2024-03-30T04:06:15.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.190+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c1a6690 con 0x7f334c100f50 2024-03-30T04:06:15.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.194+0000 7f3342a00700 1 -- 172.21.15.23:0/3462776346 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f333c004b90 con 0x7f334c100f50 2024-03-30T04:06:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.195+0000 7f3364821b80 1 Processor -- start 2024-03-30T04:06:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.195+0000 7f3342a00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 0x7f33300a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3342a00700 1 -- 172.21.15.23:0/3462776346 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f333c014070 con 0x7f334c100f50 2024-03-30T04:06:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 -- start start 2024-03-30T04:06:15.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f334a800700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 0x7f33300a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482eef0 0x555cb48398b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb4787990 con 0x555cb482eef0 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb4785350 con 0x555cb482e5d0 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3364821b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb4780c40 con 0x555cb4839df0 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.196+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46346/0 (socket says 172.21.15.23:46346) 2024-03-30T04:06:15.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 -- 172.21.15.23:0/4179217438 learned_addr learned my addr 172.21.15.23:0/4179217438 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 369691 11.64 MB/sec execute 52 sec latency 12.530 ms 2024-03-30T04:06:15.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 -- 172.21.15.23:0/4179217438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 msgr2=0x555cb483c1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:06:15.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483c1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 -- 172.21.15.23:0/4179217438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482eef0 msgr2=0x555cb48398b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482eef0 0x555cb48398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 -- 172.21.15.23:0/4179217438 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cb46023b0 con 0x555cb482e5d0 2024-03-30T04:06:15.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f3349e00700 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f333c096370 tx=0x7f333c0998c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda89e39fbb930ae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.197+0000 7f334a800700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 0x7f33300a37d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f334400f590 tx=0x7f334400b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.200+0000 7f3337000700 1 -- 172.21.15.23:0/4179217438 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0b1070 con 0x555cb482e5d0 2024-03-30T04:06:15.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.200+0000 7f3337000700 1 -- 172.21.15.23:0/4179217438 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f333c09dca0 con 0x555cb482e5d0 2024-03-30T04:06:15.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.200+0000 7f3337000700 1 -- 172.21.15.23:0/4179217438 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c09a0d0 con 0x555cb482e5d0 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 -- 172.21.15.23:0/4179217438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 msgr2=0x555cb482e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f333c096370 tx=0x7f333c0998c0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 -- 172.21.15.23:0/4179217438 shutdown_connections 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb482e9b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 --2- 172.21.15.23:0/4179217438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482eef0 0x555cb48398b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 -- 172.21.15.23:0/4179217438 >> 172.21.15.23:0/4179217438 conn(0x555cb478e6a0 msgr2=0x555cb482aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 -- 172.21.15.23:0/4179217438 shutdown_connections 2024-03-30T04:06:15.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.201+0000 7f3364821b80 1 -- 172.21.15.23:0/4179217438 wait complete. 2024-03-30T04:06:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.202+0000 7f3364821b80 1 -- start start 2024-03-30T04:06:15.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 Processor -- start 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4836b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4837250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb471cd80 con 0x555cb4839df0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb471ca20 con 0x555cb482e5d0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb46046a0 con 0x555cb482eef0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3364821b80 10 client.0 _ll_register_callbacks cb 0x555cb4726470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.203+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46348/0 (socket says 172.21.15.23:46348) 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 -- 172.21.15.23:0/3593612346 learned_addr learned my addr 172.21.15.23:0/3593612346 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f334a800700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4837250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349400700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4836b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 -- 172.21.15.23:0/3593612346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 msgr2=0x555cb4836b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4836b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 -- 172.21.15.23:0/3593612346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 msgr2=0x555cb4837250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4837250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 -- 172.21.15.23:0/3593612346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c1a6690 con 0x555cb482e5d0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3349e00700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f333c0caa30 tx=0x7f333c0a2360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e04515362dfaa44 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f332b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0b1030 con 0x555cb482e5d0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f333c0b2070 con 0x555cb482e5d0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.204+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cb4741380 con 0x555cb482e5d0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f333c0b8250 con 0x555cb482e5d0 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 10 client.26774 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555cb45aeb60 con 0x555cb482e5d0 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555cb4741380 con 0x555cb482e5d0 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f333c098dd0 con 0x555cb482e5d0 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 10 client.26774 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 10 client.26774 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 20 client.26774 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb4849660 0x555cb484ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 4 client.26774 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cb4779700 con 0x555cb4849660 2024-03-30T04:06:15.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.205+0000 7f3349400700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb4849660 0x555cb484ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.206+0000 7f332b400700 10 client.26774 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.206+0000 7f3349400700 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb4849660 0x555cb484ba50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f13525f5ebcda4aa server_cookie=e80b9be10422f232 in_seq=0 out_seq=0 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.206+0000 7f332b400700 1 -- 172.21.15.23:0/3593612346 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555cb4779700 con 0x555cb4849660 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.206+0000 7f332b400700 10 client.26774 handle_command_reply: tid=0 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 client.26774 shutdown 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 msgr2=0x555cb4839870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f333c0caa30 tx=0x7f333c0a2360 comp rx=0 tx=0).stop 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 shutdown_connections 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb4849660 0x555cb484ba50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4837250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4839870 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 --2- 172.21.15.23:0/3593612346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4836b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 >> 172.21.15.23:0/3593612346 conn(0x555cb478e6a0 msgr2=0x555cb4830600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f332b400700 0 client.26774 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:15.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 shutdown_connections 2024-03-30T04:06:15.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 1 -- 172.21.15.23:0/3593612346 wait complete. 2024-03-30T04:06:15.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.208+0000 7f3364821b80 20 client.26774 trim_cache size 0 max 16384 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.224+0000 7f3364821b80 1 Processor -- start 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.224+0000 7f3364821b80 1 -- start start 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4831e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4832390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb482b660 con 0x555cb4839df0 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb482c160 con 0x555cb482e5d0 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3364821b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb482f850 con 0x555cb482eef0 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46350/0 (socket says 172.21.15.23:46350) 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 -- 172.21.15.23:0/449467037 learned_addr learned my addr 172.21.15.23:0/449467037 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349400700 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4831e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f334a800700 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4832390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 -- 172.21.15.23:0/449467037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 msgr2=0x555cb4831e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4831e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 -- 172.21.15.23:0/449467037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 msgr2=0x555cb4832390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4832390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.225+0000 7f3349e00700 1 -- 172.21.15.23:0/449467037 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cb4746ea0 con 0x555cb482e5d0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.226+0000 7f3349e00700 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f333c00cfd0 tx=0x7f333c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f470a0a04adbf47 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.226+0000 7f3337a00700 1 -- 172.21.15.23:0/449467037 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c0b8250 con 0x555cb482e5d0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.226+0000 7f3337a00700 1 -- 172.21.15.23:0/449467037 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f333c0961b0 con 0x555cb482e5d0 2024-03-30T04:06:15.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.226+0000 7f3337a00700 1 -- 172.21.15.23:0/449467037 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c097df0 con 0x555cb482e5d0 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- 172.21.15.23:0/449467037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 msgr2=0x555cb4840a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f333c00cfd0 tx=0x7f333c000bc0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- 172.21.15.23:0/449467037 shutdown_connections 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb4839df0 0x555cb4832390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482e5d0 0x555cb4840a70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 --2- 172.21.15.23:0/449467037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb482eef0 0x555cb4831e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- 172.21.15.23:0/449467037 >> 172.21.15.23:0/449467037 conn(0x555cb478e6a0 msgr2=0x555cb4823f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- 172.21.15.23:0/449467037 shutdown_connections 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- 172.21.15.23:0/449467037 wait complete. 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.227+0000 7f3364821b80 1 -- start start 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 Processor -- start 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482e5d0 0x555cb483ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb489c420 con 0x555cb482e5d0 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb489c560 con 0x555cb482eef0 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cb489c6a0 con 0x555cb4839df0 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3364821b80 10 client.0 _ll_register_callbacks cb 0x555cb4726470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3349400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3349400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46352/0 (socket says 172.21.15.23:46352) 2024-03-30T04:06:15.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3349400700 1 -- 172.21.15.23:0/2092405177 learned_addr learned my addr 172.21.15.23:0/2092405177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.228+0000 7f3349e00700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482e5d0 0x555cb483ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f334a800700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 -- 172.21.15.23:0/2092405177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 msgr2=0x555cb483f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 -- 172.21.15.23:0/2092405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482e5d0 msgr2=0x555cb483ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482e5d0 0x555cb483ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 -- 172.21.15.23:0/2092405177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cb45aeb60 con 0x555cb482eef0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f334a800700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3349400700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f333800c010 tx=0x7f3338024550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b9ce8e6468ec52 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f332b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800f720 con 0x555cb482eef0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f333c09dca0 con 0x555cb482eef0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.229+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f333800f880 con 0x555cb482eef0 2024-03-30T04:06:15.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3338024990 con 0x555cb482eef0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 10 client.26780 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555cb4606cd0 con 0x555cb482eef0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f333c09dca0 con 0x555cb482eef0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3338009920 con 0x555cb482eef0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 10 client.26780 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 10 client.26780 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 20 client.26780 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb489de20 0x555cb48a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 4 client.26780 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555cb45cfcf0 con 0x555cb489de20 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.230+0000 7f3349e00700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb489de20 0x555cb48a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.231+0000 7f3349e00700 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb489de20 0x555cb48a0240 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=20b6fe7d19202bfd server_cookie=cfb1333c3673e8df in_seq=0 out_seq=0 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.231+0000 7f332b400700 10 client.26780 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:15.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.231+0000 7f332b400700 1 -- 172.21.15.23:0/2092405177 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555cb45cfcf0 con 0x555cb489de20 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.231+0000 7f332b400700 10 client.26780 handle_command_reply: tid=0 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 client.26780 shutdown 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 msgr2=0x555cb483ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f333800c010 tx=0x7f3338024550 comp rx=0 tx=0).stop 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 shutdown_connections 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x555cb489de20 0x555cb48a0240 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555cb4839df0 0x555cb483f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f332b400700 0 client.26780 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555cb482e5d0 0x555cb483ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 --2- 172.21.15.23:0/2092405177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555cb482eef0 0x555cb483ef90 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 >> 172.21.15.23:0/2092405177 conn(0x555cb478e6a0 msgr2=0x555cb4831e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 shutdown_connections 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 1 -- 172.21.15.23:0/2092405177 wait complete. 2024-03-30T04:06:15.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.232+0000 7f3364821b80 20 client.26780 trim_cache size 0 max 16384 2024-03-30T04:06:15.237 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 msgr2=0x7f33300a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 0x7f33300a37d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f334400f590 tx=0x7f334400b040 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 msgr2=0x7f334c199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f333c00e9c0 tx=0x7f333c00ecd0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 shutdown_connections 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f334c19de50 0x7f334c19e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c100f50 0x7f334c199570 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33300a1310 0x7f33300a37d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 --2- 172.21.15.23:0/3462776346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f334c106070 0x7f334c199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 >> 172.21.15.23:0/3462776346 conn(0x7f334c0fc7f0 msgr2=0x7f334c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 shutdown_connections 2024-03-30T04:06:15.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.233+0000 7f3350c00700 1 -- 172.21.15.23:0/3462776346 wait complete. 2024-03-30T04:06:15.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:06:15.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:15 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2299353682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:15.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 -- 172.21.15.23:0/1542638664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24106080 msgr2=0x7f0a2410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/1542638664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24106080 0x7f0a2410dc00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009f90 tx=0x7f0a1c00b330 comp rx=0 tx=0).stop 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 -- 172.21.15.23:0/1542638664 shutdown_connections 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/1542638664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24106080 0x7f0a2410dc00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/1542638664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24100f60 0x7f0a24101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/1542638664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24101910 0x7f0a24105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 -- 172.21.15.23:0/1542638664 >> 172.21.15.23:0/1542638664 conn(0x7f0a240fc7c0 msgr2=0x7f0a240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 -- 172.21.15.23:0/1542638664 shutdown_connections 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.479+0000 7f0a2ba00700 1 -- 172.21.15.23:0/1542638664 wait complete. 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 Processor -- start 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 -- start start 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 0x7f0a240730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 0x7f0a241a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a241a5b40 con 0x7f0a24100f60 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a241a5c80 con 0x7f0a24106080 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.480+0000 7f0a2ba00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a241a5e40 con 0x7f0a24073b30 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a29000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 0x7f0a240730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46366/0 (socket says 172.21.15.23:46366) 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 -- 172.21.15.23:0/972806561 learned_addr learned my addr 172.21.15.23:0/972806561 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a29a00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 0x7f0a241a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 -- 172.21.15.23:0/972806561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 msgr2=0x7f0a241a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 0x7f0a241a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 -- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 msgr2=0x7f0a240730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 0x7f0a240730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 -- 172.21.15.23:0/972806561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a180095f0 con 0x7f0a24106080 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a29a00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 0x7f0a241a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a29000700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 0x7f0a240730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.481+0000 7f0a23e00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000e970 tx=0x7f0a1000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.482+0000 7f0a21600700 1 -- 172.21.15.23:0/972806561 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10009790 con 0x7f0a24106080 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.482+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c009c40 con 0x7f0a24106080 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.482+0000 7f0a21600700 1 -- 172.21.15.23:0/972806561 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a1000eec0 con 0x7f0a24106080 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.482+0000 7f0a21600700 1 -- 172.21.15.23:0/972806561 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10005590 con 0x7f0a24106080 2024-03-30T04:06:15.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.482+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a241a64a0 con 0x7f0a24106080 2024-03-30T04:06:15.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.485+0000 7f0a21600700 1 -- 172.21.15.23:0/972806561 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0a1000cbd0 con 0x7f0a24106080 2024-03-30T04:06:15.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.486+0000 7f0a3f456b80 1 Processor -- start 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 -- start start 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a21600700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 0x7f0a0c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a21600700 1 -- 172.21.15.23:0/972806561 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a10014070 con 0x7f0a24106080 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f978df0 0x561d0f97b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f8c6990 con 0x561d0f96d5d0 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f8c4350 con 0x561d0f978df0 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f8bfc40 con 0x561d0f96def0 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a29a00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 0x7f0a0c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35770/0 (socket says 172.21.15.23:35770) 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a29000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a23e00700 1 -- 172.21.15.23:0/3594485468 learned_addr learned my addr 172.21.15.23:0/3594485468 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.487+0000 7f0a29a00700 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f978df0 0x561d0f97b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 -- 172.21.15.23:0/3594485468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f978df0 msgr2=0x561d0f97b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f978df0 0x561d0f97b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 -- 172.21.15.23:0/3594485468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 msgr2=0x561d0f96d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 -- 172.21.15.23:0/3594485468 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d0f7413b0 con 0x561d0f96def0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a29000700 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0a100a1f40 tx=0x7f0a1009a770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5e65efe37371cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a29a00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 0x7f0a0c0a3750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009530 tx=0x7f0a1c0093a0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a16000700 1 -- 172.21.15.23:0/3594485468 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a100a2470 con 0x561d0f96def0 2024-03-30T04:06:15.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a16000700 1 -- 172.21.15.23:0/3594485468 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a10096700 con 0x561d0f96def0 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.488+0000 7f0a16000700 1 -- 172.21.15.23:0/3594485468 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1009a490 con 0x561d0f96def0 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3594485468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 msgr2=0x561d0f9788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0a100a1f40 tx=0x7f0a1009a770 comp rx=0 tx=0).stop 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3594485468 shutdown_connections 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f978df0 0x561d0f97b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3594485468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96def0 0x561d0f9788b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3594485468 >> 172.21.15.23:0/3594485468 conn(0x561d0f8cd6a0 msgr2=0x561d0f969a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3594485468 shutdown_connections 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.489+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3594485468 wait complete. 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.490+0000 7f0a3f456b80 1 -- start start 2024-03-30T04:06:15.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.490+0000 7f0a3f456b80 1 Processor -- start 2024-03-30T04:06:15.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:15 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2299353682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.490+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f975b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f976130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f85bd80 con 0x561d0f978df0 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f85ba20 con 0x561d0f96def0 2024-03-30T04:06:15.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f7436a0 con 0x561d0f96d5d0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a3f456b80 10 client.0 _ll_register_callbacks cb 0x561d0f865470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35786/0 (socket says 172.21.15.23:35786) 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 -- 172.21.15.23:0/3998284397 learned_addr learned my addr 172.21.15.23:0/3998284397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f975b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 -- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 msgr2=0x561d0f975b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f975b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 -- 172.21.15.23:0/3998284397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 msgr2=0x561d0f976130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f976130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.491+0000 7f0a29000700 1 -- 172.21.15.23:0/3998284397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a241a64a0 con 0x561d0f96d5d0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a29000700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0a180046e0 tx=0x7f0a18009ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e54b8c25f09b2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:15.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18017070 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a1800f9a0 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d0f880380 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a1800fb60 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a3f456b80 10 client.26785 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.492+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561d0f6edb60 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561d0f880380 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a18005120 con 0x561d0f96d5d0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 10 client.26785 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 10 client.26785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 20 client.26785 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9888b0 0x561d0f98acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 4 client.26785 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d0f8b8700 con 0x561d0f9888b0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.493+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9888b0 0x561d0f98acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.494+0000 7f0a23e00700 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9888b0 0x561d0f98acd0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93ce0ee7ab332892 server_cookie=9a282041181d411b in_seq=0 out_seq=0 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.494+0000 7f0a0aa00700 10 client.26785 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.494+0000 7f0a0aa00700 1 -- 172.21.15.23:0/3998284397 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561d0f8b8700 con 0x561d0f9888b0 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.494+0000 7f0a0aa00700 10 client.26785 handle_command_reply: tid=0 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.494+0000 7f0a3f456b80 1 client.26785 shutdown 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 msgr2=0x561d0f9787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0a180046e0 tx=0x7f0a18009ec0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 shutdown_connections 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f976130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9888b0 0x561d0f98acd0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f9787e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/3998284397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f975b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 >> 172.21.15.23:0/3998284397 conn(0x561d0f8cd6a0 msgr2=0x561d0f96f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a0aa00700 0 client.26785 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:15.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 shutdown_connections 2024-03-30T04:06:15.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 1 -- 172.21.15.23:0/3998284397 wait complete. 2024-03-30T04:06:15.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.495+0000 7f0a3f456b80 20 client.26785 trim_cache size 0 max 16384 2024-03-30T04:06:15.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 Processor -- start 2024-03-30T04:06:15.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 -- start start 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96def0 0x561d0f97f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f978df0 0x561d0f978310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f9770d0 con 0x561d0f96def0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f96f4d0 con 0x561d0f96d5d0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f970570 con 0x561d0f978df0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a29000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.513+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96def0 0x561d0f97f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46410/0 (socket says 172.21.15.23:46410) 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 -- 172.21.15.23:0/2003331648 learned_addr learned my addr 172.21.15.23:0/2003331648 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29a00700 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f978df0 0x561d0f978310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 -- 172.21.15.23:0/2003331648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f978df0 msgr2=0x561d0f978310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f978df0 0x561d0f978310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 -- 172.21.15.23:0/2003331648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96def0 msgr2=0x561d0f97f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96def0 0x561d0f97f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 -- 172.21.15.23:0/2003331648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d0f885ea0 con 0x561d0f96d5d0 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a29000700 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a18000c00 tx=0x7f0a18010430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bc1b2a4246a3c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a16a00700 1 -- 172.21.15.23:0/2003331648 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18017070 con 0x561d0f96d5d0 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a16a00700 1 -- 172.21.15.23:0/2003331648 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0a1800f6a0 con 0x561d0f96d5d0 2024-03-30T04:06:15.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.514+0000 7f0a16a00700 1 -- 172.21.15.23:0/2003331648 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18010bd0 con 0x561d0f96d5d0 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 -- 172.21.15.23:0/2003331648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 msgr2=0x561d0f96d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0a18000c00 tx=0x7f0a18010430 comp rx=0 tx=0).stop 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 -- 172.21.15.23:0/2003331648 shutdown_connections 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f978df0 0x561d0f978310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96d5d0 0x561d0f96d9b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/2003331648 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f96def0 0x561d0f97f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 -- 172.21.15.23:0/2003331648 >> 172.21.15.23:0/2003331648 conn(0x561d0f8cd6a0 msgr2=0x561d0f962f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 -- 172.21.15.23:0/2003331648 shutdown_connections 2024-03-30T04:06:15.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.515+0000 7f0a3f456b80 1 -- 172.21.15.23:0/2003331648 wait complete. 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 -- start start 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 Processor -- start 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f97fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f97cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f97d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f97dde0 con 0x561d0f978df0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f9db6e0 con 0x561d0f96def0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.516+0000 7f0a3f456b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d0f9db820 con 0x561d0f96d5d0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a3f456b80 10 client.0 _ll_register_callbacks cb 0x561d0f865470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f97cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a29000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f97fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a29000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f97fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35806/0 (socket says 172.21.15.23:35806) 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a29000700 1 -- 172.21.15.23:0/1658012816 learned_addr learned my addr 172.21.15.23:0/1658012816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a29a00700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f97d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 -- 172.21.15.23:0/1658012816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 msgr2=0x561d0f97fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f97fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 -- 172.21.15.23:0/1658012816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 msgr2=0x561d0f97d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f97d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a23e00700 1 -- 172.21.15.23:0/1658012816 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a18021040 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.517+0000 7f0a29a00700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f97d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a23e00700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f97cfb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0a10099cf0 tx=0x7f0a100cabb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43234e48794505ec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1005fea0 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0a100a2470 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d0f70ecf0 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a10016330 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a3f456b80 10 client.26798 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561d0f745cd0 con 0x561d0f96def0 2024-03-30T04:06:15.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561d0f70ecf0 con 0x561d0f96def0 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.518+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0a1009a250 con 0x561d0f96def0 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 10 client.26798 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 10 client.26798 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 20 client.26798 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9dcbc0 0x561d0f9defe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 4 client.26798 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:15.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d0f743ce0 con 0x561d0f9dcbc0 2024-03-30T04:06:15.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a29000700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9dcbc0 0x561d0f9defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a29000700 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9dcbc0 0x561d0f9defe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=437c0369779944d1 server_cookie=47800438ad2b71c5 in_seq=0 out_seq=0 2024-03-30T04:06:15.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a0aa00700 10 client.26798 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:15.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a0aa00700 1 -- 172.21.15.23:0/1658012816 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d0f743ce0 con 0x561d0f9dcbc0 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a0aa00700 10 client.26798 handle_command_reply: tid=0 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 client.26798 shutdown 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 msgr2=0x561d0f97cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f97cfb0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0a10099cf0 tx=0x7f0a100cabb0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 shutdown_connections 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561d0f978df0 0x561d0f97d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.520+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561d0f9dcbc0 0x561d0f9defe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561d0f96d5d0 0x561d0f97fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 1 --2- 172.21.15.23:0/1658012816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561d0f96def0 0x561d0f97cfb0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a0aa00700 0 client.26798 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 >> 172.21.15.23:0/1658012816 conn(0x561d0f8cd6a0 msgr2=0x561d0f974330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 shutdown_connections 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 1 -- 172.21.15.23:0/1658012816 wait complete. 2024-03-30T04:06:15.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.521+0000 7f0a3f456b80 20 client.26798 trim_cache size 0 max 16384 2024-03-30T04:06:15.524 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:15.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 msgr2=0x7f0a0c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 0x7f0a0c0a3750 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c009530 tx=0x7f0a1c0093a0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 msgr2=0x7f0a240735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0a1000e970 tx=0x7f0a1000ec80 comp rx=0 tx=0).stop 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 shutdown_connections 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a24073b30 0x7f0a241a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a24100f60 0x7f0a240730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0a0c0a1290 0x7f0a0c0a3750 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 --2- 172.21.15.23:0/972806561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a24106080 0x7f0a240735f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 >> 172.21.15.23:0/972806561 conn(0x7f0a240fc7c0 msgr2=0x7f0a240fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 shutdown_connections 2024-03-30T04:06:15.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.522+0000 7f0a2ba00700 1 -- 172.21.15.23:0/972806561 wait complete. 2024-03-30T04:06:15.542 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:06:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:15 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2299353682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 -- 172.21.15.23:0/481078070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c101510 msgr2=0x7ff53c10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 --2- 172.21.15.23:0/481078070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c101510 0x7ff53c10d9d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff53400b0d0 tx=0x7ff53400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 -- 172.21.15.23:0/481078070 shutdown_connections 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 --2- 172.21.15.23:0/481078070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c101510 0x7ff53c10d9d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 --2- 172.21.15.23:0/481078070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c106c10 0x7ff53c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 --2- 172.21.15.23:0/481078070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 -- 172.21.15.23:0/481078070 >> 172.21.15.23:0/481078070 conn(0x7ff53c0fc620 msgr2=0x7ff53c0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 -- 172.21.15.23:0/481078070 shutdown_connections 2024-03-30T04:06:15.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.843+0000 7ff53b200700 1 -- 172.21.15.23:0/481078070 wait complete. 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.844+0000 7ff53b200700 1 Processor -- start 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.844+0000 7ff53b200700 1 -- start start 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c101510 0x7ff53c199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c106c10 0x7ff53c19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c19e260 con 0x7ff53c106c10 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c19e3d0 con 0x7ff53c100b50 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53b200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c19e590 con 0x7ff53c101510 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff539e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff539e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46440/0 (socket says 172.21.15.23:46440) 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff539e00700 1 -- 172.21.15.23:0/3376067869 learned_addr learned my addr 172.21.15.23:0/3376067869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff53a800700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c106c10 0x7ff53c19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.845+0000 7ff539400700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c101510 0x7ff53c199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 -- 172.21.15.23:0/3376067869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c101510 msgr2=0x7ff53c199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c101510 0x7ff53c199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 -- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c106c10 msgr2=0x7ff53c19dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c106c10 0x7ff53c19dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 -- 172.21.15.23:0/3376067869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534009d00 con 0x7ff53c100b50 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff539e00700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00e9a0 tx=0x7ff52c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff532a00700 1 -- 172.21.15.23:0/3376067869 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c009790 con 0x7ff53c100b50 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c19e870 con 0x7ff53c100b50 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff532a00700 1 -- 172.21.15.23:0/3376067869 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff52c010430 con 0x7ff53c100b50 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff532a00700 1 -- 172.21.15.23:0/3376067869 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52c018680 con 0x7ff53c100b50 2024-03-30T04:06:15.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.846+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c1a60b0 con 0x7ff53c100b50 2024-03-30T04:06:15.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.851+0000 7ff532a00700 1 -- 172.21.15.23:0/3376067869 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff52c0105a0 con 0x7ff53c100b50 2024-03-30T04:06:15.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.852+0000 7ff554556b80 1 Processor -- start 2024-03-30T04:06:15.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.852+0000 7ff532a00700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 0x7ff5200a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.852+0000 7ff554556b80 1 -- start start 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ecbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821ed93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff532a00700 1 -- 172.21.15.23:0/3376067869 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff52c014070 con 0x7ff53c100b50 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821e24990 con 0x55b821ed6fc0 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821e22350 con 0x55b821ecc0c0 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff554556b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821e1dc40 con 0x55b821ecb7a0 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff53a800700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 0x7ff5200a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46456/0 (socket says 172.21.15.23:46456) 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 -- 172.21.15.23:0/3506392620 learned_addr learned my addr 172.21.15.23:0/3506392620 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539e00700 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ecbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 -- 172.21.15.23:0/3506392620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 msgr2=0x55b821ecbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ecbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 -- 172.21.15.23:0/3506392620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 msgr2=0x55b821ed93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821ed93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.853+0000 7ff539400700 1 -- 172.21.15.23:0/3506392620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b821c9f3b0 con 0x55b821ecc0c0 2024-03-30T04:06:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.854+0000 7ff539400700 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff528009d20 tx=0x7ff528011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dae6d59b661bea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.854+0000 7ff527000700 1 -- 172.21.15.23:0/3506392620 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528019630 con 0x55b821ecc0c0 2024-03-30T04:06:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.854+0000 7ff53a800700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 0x7ff5200a3750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff534006260 tx=0x7ff5340061f0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.854+0000 7ff527000700 1 -- 172.21.15.23:0/3506392620 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528019c50 con 0x55b821ecc0c0 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.854+0000 7ff527000700 1 -- 172.21.15.23:0/3506392620 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52800ec40 con 0x55b821ecc0c0 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- 172.21.15.23:0/3506392620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 msgr2=0x55b821ed6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff528009d20 tx=0x7ff528011720 comp rx=0 tx=0).stop 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- 172.21.15.23:0/3506392620 shutdown_connections 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821ed93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ecbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 --2- 172.21.15.23:0/3506392620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed6a80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- 172.21.15.23:0/3506392620 >> 172.21.15.23:0/3506392620 conn(0x55b821e2b6a0 msgr2=0x55b821ec7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- 172.21.15.23:0/3506392620 shutdown_connections 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- 172.21.15.23:0/3506392620 wait complete. 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.855+0000 7ff554556b80 1 -- start start 2024-03-30T04:06:15.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 Processor -- start 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ed6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ecc0c0 0x55b821ed3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821db9d80 con 0x55b821ecc0c0 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821db9a20 con 0x55b821ed6fc0 2024-03-30T04:06:15.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.856+0000 7ff554556b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821ca16a0 con 0x55b821ecb7a0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff554556b80 10 client.0 _ll_register_callbacks cb 0x55b821dc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46462/0 (socket says 172.21.15.23:46462) 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 -- 172.21.15.23:0/3673157806 learned_addr learned my addr 172.21.15.23:0/3673157806 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff539e00700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ed6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff539400700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ecc0c0 0x55b821ed3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 -- 172.21.15.23:0/3673157806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 msgr2=0x55b821ed6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ed6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 -- 172.21.15.23:0/3673157806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ecc0c0 msgr2=0x55b821ed3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ecc0c0 0x55b821ed3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 -- 172.21.15.23:0/3673157806 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff528009980 con 0x55b821ed6fc0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff539e00700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ed6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff53a800700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff53400bcd0 tx=0x7ff53400b3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2883aa3f165f5ca server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.857+0000 7ff51b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff534023040 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff5340052e0 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b821dde380 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff53400e040 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff554556b80 10 client.26810 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b821c4bb60 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.858+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b821dde380 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff53401ea30 con 0x55b821ed6fc0 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 10 client.26810 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 10 client.26810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 20 client.26810 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821ee6bc0 0x55b821ee8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 4 client.26810 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:15.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b821e16700 con 0x55b821ee6bc0 2024-03-30T04:06:15.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.859+0000 7ff539e00700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821ee6bc0 0x55b821ee8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff539e00700 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821ee6bc0 0x55b821ee8fe0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69ca9d7b43292b50 server_cookie=1cf93c26e9d3b7a5 in_seq=0 out_seq=0 2024-03-30T04:06:15.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff51b400700 10 client.26810 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff51b400700 1 -- 172.21.15.23:0/3673157806 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b821e16700 con 0x55b821ee6bc0 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff51b400700 10 client.26810 handle_command_reply: tid=0 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 client.26810 shutdown 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 msgr2=0x55b821ed4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff53400bcd0 tx=0x7ff53400b3e0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 shutdown_connections 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ed6fc0 0x55b821ed4450 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821ee6bc0 0x55b821ee8fe0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ed6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 --2- 172.21.15.23:0/3673157806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ecc0c0 0x55b821ed3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff51b400700 0 client.26810 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:15.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.860+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 >> 172.21.15.23:0/3673157806 conn(0x55b821e2b6a0 msgr2=0x55b821ecd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.861+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 shutdown_connections 2024-03-30T04:06:15.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.861+0000 7ff554556b80 1 -- 172.21.15.23:0/3673157806 wait complete. 2024-03-30T04:06:15.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.861+0000 7ff554556b80 20 client.26810 trim_cache size 0 max 16384 2024-03-30T04:06:15.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 Processor -- start 2024-03-30T04:06:15.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 -- start start 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ec7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821eddc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821ede1b0 con 0x55b821ed6fc0 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821ee3010 con 0x55b821ecc0c0 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff554556b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821ecbdd0 con 0x55b821ecb7a0 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff53a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821eddc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46464/0 (socket says 172.21.15.23:46464) 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff539400700 1 -- 172.21.15.23:0/3121174352 learned_addr learned my addr 172.21.15.23:0/3121174352 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.877+0000 7ff539e00700 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ec7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 -- 172.21.15.23:0/3121174352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 msgr2=0x55b821ec7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ec7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 -- 172.21.15.23:0/3121174352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 msgr2=0x55b821eddc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821eddc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 -- 172.21.15.23:0/3121174352 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b821de3ea0 con 0x55b821ecc0c0 2024-03-30T04:06:15.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff539400700 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff52801c040 tx=0x7ff528005ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cdeab924a2bfc4d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff527a00700 1 -- 172.21.15.23:0/3121174352 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528019630 con 0x55b821ecc0c0 2024-03-30T04:06:15.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff527a00700 1 -- 172.21.15.23:0/3121174352 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528005040 con 0x55b821ecc0c0 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.878+0000 7ff527a00700 1 -- 172.21.15.23:0/3121174352 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528003aa0 con 0x55b821ecc0c0 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 -- 172.21.15.23:0/3121174352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 msgr2=0x55b821ed42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff52801c040 tx=0x7ff528005ce0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 -- 172.21.15.23:0/3121174352 shutdown_connections 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821eddc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821ec7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 --2- 172.21.15.23:0/3121174352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821ed42f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 -- 172.21.15.23:0/3121174352 >> 172.21.15.23:0/3121174352 conn(0x55b821e2b6a0 msgr2=0x55b821eed940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 -- 172.21.15.23:0/3121174352 shutdown_connections 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.879+0000 7ff554556b80 1 -- 172.21.15.23:0/3121174352 wait complete. 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 -- start start 2024-03-30T04:06:15.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 Processor -- start 2024-03-30T04:06:15.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821edbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821f339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821f33f00 con 0x55b821ed6fc0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821f34070 con 0x55b821ecc0c0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.880+0000 7ff554556b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b821f34230 con 0x55b821ecb7a0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff554556b80 10 client.0 _ll_register_callbacks cb 0x55b821dc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46468/0 (socket says 172.21.15.23:46468) 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 -- 172.21.15.23:0/1246822134 learned_addr learned my addr 172.21.15.23:0/1246822134 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff53a800700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821f339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539e00700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821edbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 -- 172.21.15.23:0/1246822134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 msgr2=0x55b821edbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821edbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 -- 172.21.15.23:0/1246822134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 msgr2=0x55b821f339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821f339c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.881+0000 7ff539400700 1 -- 172.21.15.23:0/1246822134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b821c9f3b0 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff539400700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff528000c00 tx=0x7ff528005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bf0e54598f6df9d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff51b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528019770 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff528027070 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff528019c50 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5280198d0 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff554556b80 10 client.26816 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b821ca3cd0 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.882+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff528019c50 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff5280235d0 con 0x55b821ecc0c0 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 10 client.26816 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:15.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 10 client.26816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 20 client.26816 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821f3b6c0 0x55b821f3dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 4 client.26816 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b821c6ccf0 con 0x55b821f3b6c0 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff539e00700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821f3b6c0 0x55b821f3dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff539e00700 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821f3b6c0 0x55b821f3dab0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3282e5d01570953c server_cookie=29cf880c489c25a7 in_seq=0 out_seq=0 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.883+0000 7ff51b400700 10 client.26816 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff51b400700 1 -- 172.21.15.23:0/1246822134 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b821c6ccf0 con 0x55b821f3b6c0 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff51b400700 10 client.26816 handle_command_reply: tid=0 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 client.26816 shutdown 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 msgr2=0x55b821edc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff528000c00 tx=0x7ff528005fa0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 shutdown_connections 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b821ed6fc0 0x55b821f339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b821f3b6c0 0x55b821f3dab0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b821ecb7a0 0x55b821edbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 --2- 172.21.15.23:0/1246822134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b821ecc0c0 0x55b821edc090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 >> 172.21.15.23:0/1246822134 conn(0x55b821e2b6a0 msgr2=0x55b821ecf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff51b400700 0 client.26816 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 shutdown_connections 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 1 -- 172.21.15.23:0/1246822134 wait complete. 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.884+0000 7ff554556b80 20 client.26816 trim_cache size 0 max 16384 2024-03-30T04:06:15.886 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:15.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 msgr2=0x7ff5200a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 0x7ff5200a3750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff534006260 tx=0x7ff5340061f0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 msgr2=0x7ff53c199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff52c00e9a0 tx=0x7ff52c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 shutdown_connections 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff53c106c10 0x7ff53c19dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff5200a1290 0x7ff5200a3750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff53c100b50 0x7ff53c199340 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 --2- 172.21.15.23:0/3376067869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff53c101510 0x7ff53c199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 >> 172.21.15.23:0/3376067869 conn(0x7ff53c0fc620 msgr2=0x7ff53c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 shutdown_connections 2024-03-30T04:06:15.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:15.885+0000 7ff53b200700 1 -- 172.21.15.23:0/3376067869 wait complete. 2024-03-30T04:06:16.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 372512 11.74 MB/sec execute 53 sec latency 14.923 ms 2024-03-30T04:06:16.904 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:06:16.904 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:06:17.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 -- 172.21.15.23:0/3366101684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 msgr2=0x7fae001011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 --2- 172.21.15.23:0/3366101684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001011c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadf00098f0 tx=0x7fadf0009c00 comp rx=0 tx=0).stop 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 -- 172.21.15.23:0/3366101684 shutdown_connections 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 --2- 172.21.15.23:0/3366101684 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 0x7fae0010da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 --2- 172.21.15.23:0/3366101684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001011c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 --2- 172.21.15.23:0/3366101684 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae00101790 0x7fae001057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 -- 172.21.15.23:0/3366101684 >> 172.21.15.23:0/3366101684 conn(0x7fae000fc680 msgr2=0x7fae000feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 -- 172.21.15.23:0/3366101684 shutdown_connections 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.134+0000 7fae05400700 1 -- 172.21.15.23:0/3366101684 wait complete. 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 Processor -- start 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 -- start start 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 0x7fae00197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0019c3e0 con 0x7fae00105f00 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0019c550 con 0x7fae0019b960 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.136+0000 7fae05400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0019c710 con 0x7fae00100de0 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fadffe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46494/0 (socket says 172.21.15.23:46494) 2024-03-30T04:06:17.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 -- 172.21.15.23:0/982780085 learned_addr learned my addr 172.21.15.23:0/982780085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fadff400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 0x7fae00197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 -- 172.21.15.23:0/982780085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 msgr2=0x7fae001970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001970d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 -- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 msgr2=0x7fae00197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 0x7fae00197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fae04a00700 1 -- 172.21.15.23:0/982780085 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec0095f0 con 0x7fae0019b960 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.137+0000 7fadffe00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:06:17.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fae04a00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fadf800ef30 tx=0x7fadf800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fadfcc00700 1 -- 172.21.15.23:0/982780085 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf8018070 con 0x7fae0019b960 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf0009520 con 0x7fae0019b960 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fadfcc00700 1 -- 172.21.15.23:0/982780085 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadf800f040 con 0x7fae0019b960 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fadfcc00700 1 -- 172.21.15.23:0/982780085 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf800cdb0 con 0x7fae0019b960 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.138+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae001a3fb0 con 0x7fae0019b960 2024-03-30T04:06:17.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.141+0000 7fadfcc00700 1 -- 172.21.15.23:0/982780085 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fadf8003a00 con 0x7fae0019b960 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.142+0000 7fae1a44bb80 1 Processor -- start 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.142+0000 7fae1a44bb80 1 -- start start 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ef820 0x55d30b9efc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba01840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b948a10 con 0x55d30b9f0140 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b9463d0 con 0x55d30b9ff450 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadfcc00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 0x7fade40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b941cc0 con 0x55d30b9ef820 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadffe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadfcc00700 1 -- 172.21.15.23:0/982780085 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fadf80ccbd0 con 0x7fae0019b960 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadffe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33678/0 (socket says 172.21.15.23:33678) 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadffe00700 1 -- 172.21.15.23:0/3225188535 learned_addr learned my addr 172.21.15.23:0/3225188535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fae04a00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 0x7fade40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadff400700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ef820 0x55d30b9efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.143+0000 7fadff400700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba01840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 -- 172.21.15.23:0/3225188535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ef820 msgr2=0x55d30b9efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ef820 0x55d30b9efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 -- 172.21.15.23:0/3225188535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 msgr2=0x55d30ba01840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba01840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 -- 172.21.15.23:0/3225188535 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d30b7c3430 con 0x55d30b9f0140 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadff400700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba01840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fae04a00700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 0x7fade40a3750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fadf8009c10 tx=0x7fadf80092e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadffe00700 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009fd0 tx=0x7fadf000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9463da447d90890e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadf5200700 1 -- 172.21.15.23:0/3225188535 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf001c070 con 0x55d30b9f0140 2024-03-30T04:06:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadf5200700 1 -- 172.21.15.23:0/3225188535 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadf000f460 con 0x55d30b9f0140 2024-03-30T04:06:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.144+0000 7fadf5200700 1 -- 172.21.15.23:0/3225188535 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf0005040 con 0x55d30b9f0140 2024-03-30T04:06:17.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 -- 172.21.15.23:0/3225188535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 msgr2=0x55d30b9fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009fd0 tx=0x7fadf000be90 comp rx=0 tx=0).stop 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 -- 172.21.15.23:0/3225188535 shutdown_connections 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba01840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ef820 0x55d30b9efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/3225188535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fef10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 -- 172.21.15.23:0/3225188535 >> 172.21.15.23:0/3225188535 conn(0x55d30b94f720 msgr2=0x55d30b9ebce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.145+0000 7fae1a44bb80 1 -- 172.21.15.23:0/3225188535 shutdown_connections 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.146+0000 7fae1a44bb80 1 -- 172.21.15.23:0/3225188535 wait complete. 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.146+0000 7fae1a44bb80 1 -- start start 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.146+0000 7fae1a44bb80 1 Processor -- start 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ff450 0x55d30b9fc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b8dde00 con 0x55d30b9f0140 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b8ddaa0 con 0x55d30b9ef820 2024-03-30T04:06:17.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b7c5720 con 0x55d30b9ff450 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fae1a44bb80 10 client.0 _ll_register_callbacks cb 0x55d30b8e74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fadff400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fadff400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46514/0 (socket says 172.21.15.23:46514) 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fadff400700 1 -- 172.21.15.23:0/1277259420 learned_addr learned my addr 172.21.15.23:0/1277259420 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fadffe00700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.147+0000 7fadff400700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ff450 0x55d30b9fc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 -- 172.21.15.23:0/1277259420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ff450 msgr2=0x55d30b9fc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ff450 0x55d30b9fc590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 -- 172.21.15.23:0/1277259420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 msgr2=0x55d30b9fc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 -- 172.21.15.23:0/1277259420 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0010e2f0 con 0x55d30b9ef820 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadffe00700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fadff400700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fadec011fd0 tx=0x7fadec004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=856560a4efc4b312 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fade9600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec016040 con 0x55d30b9ef820 2024-03-30T04:06:17.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d30b902400 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.148+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fadec00fe30 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadec00e5d0 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 10 client.26828 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d30b76fbe0 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d30b902400 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fadec021550 con 0x55d30b9ef820 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 10 client.26828 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:17.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 20 client.26828 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba0efb0 0x55d30ba113d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 4 client.26828 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.149+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d30b93a780 con 0x55d30ba0efb0 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.150+0000 7fadffe00700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba0efb0 0x55d30ba113d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.150+0000 7fadffe00700 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba0efb0 0x55d30ba113d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d411a2495327efc4 server_cookie=e135b43214bf6599 in_seq=0 out_seq=0 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.150+0000 7fade9600700 10 client.26828 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fade9600700 1 -- 172.21.15.23:0/1277259420 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d30b93a780 con 0x55d30ba0efb0 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fade9600700 10 client.26828 handle_command_reply: tid=0 2024-03-30T04:06:17.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 client.26828 shutdown 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 msgr2=0x55d30b9fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fadec011fd0 tx=0x7fadec004750 comp rx=0 tx=0).stop 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 shutdown_connections 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba0efb0 0x55d30ba113d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9ff450 0x55d30b9fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ef820 0x55d30b9fed10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1277259420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9f0140 0x55d30b9fc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 >> 172.21.15.23:0/1277259420 conn(0x55d30b94f720 msgr2=0x55d30b9f1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fade9600700 0 client.26828 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 shutdown_connections 2024-03-30T04:06:17.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1277259420 wait complete. 2024-03-30T04:06:17.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.151+0000 7fae1a44bb80 20 client.26828 trim_cache size 0 max 16384 2024-03-30T04:06:17.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 Processor -- start 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 -- start start 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba0af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30b9f1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b9f0aa0 con 0x55d30b9ef820 2024-03-30T04:06:17.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30ba11760 con 0x55d30b9ff450 2024-03-30T04:06:17.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30b9fd0f0 con 0x55d30b9f0140 2024-03-30T04:06:17.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fadffe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fadff400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba0af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fadffe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35914/0 (socket says 172.21.15.23:35914) 2024-03-30T04:06:17.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.171+0000 7fadffe00700 1 -- 172.21.15.23:0/138578458 learned_addr learned my addr 172.21.15.23:0/138578458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadff400700 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30b9f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 -- 172.21.15.23:0/138578458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 msgr2=0x55d30b9f1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30b9f1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 -- 172.21.15.23:0/138578458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 msgr2=0x55d30ba0af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba0af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 -- 172.21.15.23:0/138578458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d30ba15fc0 con 0x55d30b9f0140 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadffe00700 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b8c0 tx=0x7fadf0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c443dd1bf5c697e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadf5c00700 1 -- 172.21.15.23:0/138578458 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf001c040 con 0x55d30b9f0140 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadf5c00700 1 -- 172.21.15.23:0/138578458 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadf000b500 con 0x55d30b9f0140 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.172+0000 7fadf5c00700 1 -- 172.21.15.23:0/138578458 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf002b380 con 0x55d30b9f0140 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 -- 172.21.15.23:0/138578458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 msgr2=0x55d30ba060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b8c0 tx=0x7fadf0009c00 comp rx=0 tx=0).stop 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 -- 172.21.15.23:0/138578458 shutdown_connections 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30b9f1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba0af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/138578458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba060d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 -- 172.21.15.23:0/138578458 >> 172.21.15.23:0/138578458 conn(0x55d30b94f720 msgr2=0x55d30ba0a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 -- 172.21.15.23:0/138578458 shutdown_connections 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.173+0000 7fae1a44bb80 1 -- 172.21.15.23:0/138578458 wait complete. 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.174+0000 7fae1a44bb80 1 -- start start 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.174+0000 7fae1a44bb80 1 Processor -- start 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.174+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba56f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.174+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.174+0000 7fae1a44bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba5b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30ba5b5b0 con 0x55d30b9ef820 2024-03-30T04:06:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30ba5b720 con 0x55d30b9ff450 2024-03-30T04:06:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fae1a44bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d30ba5b8e0 con 0x55d30b9f0140 2024-03-30T04:06:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fae1a44bb80 10 client.0 _ll_register_callbacks cb 0x55d30b8e74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadff400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba56f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadff400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba56f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33698/0 (socket says 172.21.15.23:33698) 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35918/0 (socket says 172.21.15.23:35918) 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadff400700 1 -- 172.21.15.23:0/1922363445 learned_addr learned my addr 172.21.15.23:0/1922363445 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 -- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 msgr2=0x55d30ba5b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadff400700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba5b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba5b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 -- 172.21.15.23:0/1922363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 msgr2=0x55d30ba56f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba56f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadffe00700 1 -- 172.21.15.23:0/1922363445 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d30b7c3430 con 0x55d30b9f0140 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.175+0000 7fadff400700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba5b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fadffe00700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55d30ba0c7d0 tx=0x7fadf002bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51236c5694e50a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fade9600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf001c050 con 0x55d30b9f0140 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fadf000f460 con 0x55d30b9f0140 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fadf000fab0 con 0x55d30b9f0140 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.176+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadf000f890 con 0x55d30b9f0140 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 10 client.26824 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:17.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d30b907f20 con 0x55d30b9f0140 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fadf003b020 con 0x55d30b9f0140 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fadf0038070 con 0x55d30b9f0140 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 10 client.26824 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 10 client.26824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 20 client.26824 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:17.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba63e70 0x55d30ba66260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 4 client.26824 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d30b790d70 con 0x55d30ba63e70 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.177+0000 7fadff400700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba63e70 0x55d30ba66260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.178+0000 7fadff400700 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba63e70 0x55d30ba66260 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7b8cb790b8d7bcd server_cookie=c972ff1593c91bd7 in_seq=0 out_seq=0 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.178+0000 7fade9600700 10 client.26824 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fade9600700 1 -- 172.21.15.23:0/1922363445 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d30b790d70 con 0x55d30ba63e70 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fade9600700 10 client.26824 handle_command_reply: tid=0 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 client.26824 shutdown 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 msgr2=0x55d30ba57460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55d30ba0c7d0 tx=0x7fadf002bef0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 shutdown_connections 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d30b9ff450 0x55d30ba5b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d30b9ef820 0x55d30ba56f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d30ba63e70 0x55d30ba66260 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 --2- 172.21.15.23:0/1922363445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d30b9f0140 0x55d30ba57460 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 >> 172.21.15.23:0/1922363445 conn(0x55d30b94f720 msgr2=0x55d30b9f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.179+0000 7fade9600700 0 client.26824 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 shutdown_connections 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae1a44bb80 1 -- 172.21.15.23:0/1922363445 wait complete. 2024-03-30T04:06:17.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae1a44bb80 20 client.26824 trim_cache size 0 max 16384 2024-03-30T04:06:17.193 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:17.193 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:06:17.193 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "e82e4652-345b-4ea0-bb94-87df42e0365c", 2024-03-30T04:06:17.193 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:06:17.193 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:17.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 msgr2=0x7fade40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 0x7fade40a3750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fadf8009c10 tx=0x7fadf80092e0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 msgr2=0x7fae0019bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.180+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fadf800ef30 tx=0x7fadf800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 shutdown_connections 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fade40a1290 0x7fade40a3750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae0019b960 0x7fae0019bde0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae00100de0 0x7fae001970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 --2- 172.21.15.23:0/982780085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae00105f00 0x7fae00197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 >> 172.21.15.23:0/982780085 conn(0x7fae000fc680 msgr2=0x7fae000fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 shutdown_connections 2024-03-30T04:06:17.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.181+0000 7fae05400700 1 -- 172.21.15.23:0/982780085 wait complete. 2024-03-30T04:06:17.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 374321 11.65 MB/sec execute 54 sec latency 11.482 ms 2024-03-30T04:06:17.200 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:06:17.263 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:17 smithi023 ceph-mon[38254]: pgmap v133: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:06:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 -- 172.21.15.23:0/3651980419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390105ca0 msgr2=0x7f6390067e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 --2- 172.21.15.23:0/3651980419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390105ca0 0x7f6390067e30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6384009990 tx=0x7f6384009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 -- 172.21.15.23:0/3651980419 shutdown_connections 2024-03-30T04:06:17.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 --2- 172.21.15.23:0/3651980419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390103740 0x7f6390103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 --2- 172.21.15.23:0/3651980419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390105ca0 0x7f6390067e30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 --2- 172.21.15.23:0/3651980419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390068370 0x7f63900687f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 -- 172.21.15.23:0/3651980419 >> 172.21.15.23:0/3651980419 conn(0x7f6390076d40 msgr2=0x7f6390077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 -- 172.21.15.23:0/3651980419 shutdown_connections 2024-03-30T04:06:17.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.430+0000 7f6397e00700 1 -- 172.21.15.23:0/3651980419 wait complete. 2024-03-30T04:06:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:17 smithi154 ceph-mon[79161]: pgmap v133: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:06:17.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 Processor -- start 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 -- start start 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390103740 0x7f63901a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390105ca0 0x7f639019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639019dcd0 con 0x7f6390068370 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639019de40 con 0x7f6390105ca0 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.432+0000 7f6397e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639019e000 con 0x7f6390103740 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33720/0 (socket says 172.21.15.23:33720) 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 -- 172.21.15.23:0/3467789911 learned_addr learned my addr 172.21.15.23:0/3467789911 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6394a00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390103740 0x7f63901a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390105ca0 0x7f639019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 -- 172.21.15.23:0/3467789911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390103740 msgr2=0x7f63901a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390103740 0x7f63901a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 -- 172.21.15.23:0/3467789911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390105ca0 msgr2=0x7f639019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390105ca0 0x7f639019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.433+0000 7f6395400700 1 -- 172.21.15.23:0/3467789911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63840095f0 con 0x7f6390068370 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f6395400700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6384005910 tx=0x7f6384004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f638e000700 1 -- 172.21.15.23:0/3467789911 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638401d070 con 0x7f6390068370 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f638e000700 1 -- 172.21.15.23:0/3467789911 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6384022470 con 0x7f6390068370 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f638e000700 1 -- 172.21.15.23:0/3467789911 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384017650 con 0x7f6390068370 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639019e280 con 0x7f6390068370 2024-03-30T04:06:17.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.434+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63901aa530 con 0x7f6390068370 2024-03-30T04:06:17.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.437+0000 7f638e000700 1 -- 172.21.15.23:0/3467789911 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f63840225e0 con 0x7f6390068370 2024-03-30T04:06:17.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.439+0000 7f63ab8f4b80 1 Processor -- start 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.439+0000 7f63ab8f4b80 1 -- start start 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.439+0000 7f638e000700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 0x7f63780a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f638e000700 1 -- 172.21.15.23:0/3467789911 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f63840ce240 con 0x7f6390068370 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc2b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc2e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bb79990 con 0x560a6bc2bdf0 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bb77350 con 0x560a6bc205d0 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bb72c40 con 0x560a6bc20ef0 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6395400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6395e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 0x7f63780a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6395400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46570/0 (socket says 172.21.15.23:46570) 2024-03-30T04:06:17.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6395400700 1 -- 172.21.15.23:0/1543629897 learned_addr learned my addr 172.21.15.23:0/1543629897 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6394a00700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc2b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.440+0000 7f6395e00700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc2e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 -- 172.21.15.23:0/1543629897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 msgr2=0x560a6bc2b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc2b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 -- 172.21.15.23:0/1543629897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 msgr2=0x560a6bc2e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc2e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 -- 172.21.15.23:0/1543629897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a6b9f43b0 con 0x560a6bc205d0 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395e00700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc2e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395400700 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63840a3000 tx=0x7f6384009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac2f19ccdfd41a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6395e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 0x7f63780a37d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6388009ec0 tx=0x7f6388009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6382600700 1 -- 172.21.15.23:0/1543629897 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638409b840 con 0x560a6bc205d0 2024-03-30T04:06:17.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6382600700 1 -- 172.21.15.23:0/1543629897 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f638409fe90 con 0x560a6bc205d0 2024-03-30T04:06:17.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.441+0000 7f6382600700 1 -- 172.21.15.23:0/1543629897 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63840ce0c0 con 0x560a6bc205d0 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/1543629897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 msgr2=0x560a6bc209b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63840a3000 tx=0x7f6384009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/1543629897 shutdown_connections 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc2e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc209b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/1543629897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc2b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/1543629897 >> 172.21.15.23:0/1543629897 conn(0x560a6bb806a0 msgr2=0x560a6bc1ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/1543629897 shutdown_connections 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.442+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/1543629897 wait complete. 2024-03-30T04:06:17.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.443+0000 7f63ab8f4b80 1 -- start start 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.443+0000 7f63ab8f4b80 1 Processor -- start 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.443+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc2b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc28f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bb0ed80 con 0x560a6bc2bdf0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bb0ea20 con 0x560a6bc205d0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6b9f66a0 con 0x560a6bc20ef0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f63ab8f4b80 10 client.0 _ll_register_callbacks cb 0x560a6bb18470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f6394a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f6394a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35948/0 (socket says 172.21.15.23:35948) 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f6394a00700 1 -- 172.21.15.23:0/823202860 learned_addr learned my addr 172.21.15.23:0/823202860 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f6395e00700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc28f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.444+0000 7f6395400700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc2b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 -- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 msgr2=0x560a6bc2b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc2b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 -- 172.21.15.23:0/823202860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 msgr2=0x560a6bc28f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc28f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 -- 172.21.15.23:0/823202860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63901aa530 con 0x560a6bc20ef0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6395400700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc2b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6394a00700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f637c012840 tx=0x7f637c012c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9447486f5700f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6376a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c017070 con 0x560a6bc20ef0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f637c00f640 con 0x560a6bc20ef0 2024-03-30T04:06:17.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.445+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a6bb33380 con 0x560a6bc20ef0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f637c0122e0 con 0x560a6bc20ef0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 10 client.26836 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x560a6b9a0b60 con 0x560a6bc20ef0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560a6bb33380 con 0x560a6bc20ef0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f637c0109c0 con 0x560a6bc20ef0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 10 client.26836 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 10 client.26836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 20 client.26836 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc3b750 0x560a6bc3db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 4 client.26836 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a6bb6b700 con 0x560a6bc3b750 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.446+0000 7f6395400700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc3b750 0x560a6bc3db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.447+0000 7f6395400700 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc3b750 0x560a6bc3db70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac1c69b6ca51affc server_cookie=ca27534a0f00513d in_seq=0 out_seq=0 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.447+0000 7f6376a00700 10 client.26836 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.447+0000 7f6376a00700 1 -- 172.21.15.23:0/823202860 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560a6bb6b700 con 0x560a6bc3b750 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.447+0000 7f6376a00700 10 client.26836 handle_command_reply: tid=0 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.447+0000 7f63ab8f4b80 1 client.26836 shutdown 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 msgr2=0x560a6bc289e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f637c012840 tx=0x7f637c012c00 comp rx=0 tx=0).stop 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 shutdown_connections 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc28f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc205d0 0x560a6bc2b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc3b750 0x560a6bc3db70 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/823202860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc20ef0 0x560a6bc289e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f6376a00700 0 client.26836 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 >> 172.21.15.23:0/823202860 conn(0x560a6bb806a0 msgr2=0x560a6bc22510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 shutdown_connections 2024-03-30T04:06:17.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/823202860 wait complete. 2024-03-30T04:06:17.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.448+0000 7f63ab8f4b80 20 client.26836 trim_cache size 0 max 16384 2024-03-30T04:06:17.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 Processor -- start 2024-03-30T04:06:17.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 -- start start 2024-03-30T04:06:17.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc3db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc23d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc32d20 con 0x560a6bc2bdf0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc23400 con 0x560a6bc20ef0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.468+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc18480 con 0x560a6bc205d0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6394a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6394a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:46586/0 (socket says 172.21.15.23:46586) 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc23d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc3db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6394a00700 1 -- 172.21.15.23:0/3663863974 learned_addr learned my addr 172.21.15.23:0/3663863974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 -- 172.21.15.23:0/3663863974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 msgr2=0x560a6bc327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 -- 172.21.15.23:0/3663863974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 msgr2=0x560a6bc23d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc23d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 -- 172.21.15.23:0/3663863974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a6bb38ea0 con 0x560a6bc205d0 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.469+0000 7f6395400700 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc3db40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6384006010 tx=0x7f638401a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e49d9542b9087a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f6383000700 1 -- 172.21.15.23:0/3663863974 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638409ef70 con 0x560a6bc205d0 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f6383000700 1 -- 172.21.15.23:0/3663863974 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63840b7070 con 0x560a6bc205d0 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f6383000700 1 -- 172.21.15.23:0/3663863974 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63840a10b0 con 0x560a6bc205d0 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/3663863974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 msgr2=0x560a6bc3db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc3db40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6384006010 tx=0x7f638401a040 comp rx=0 tx=0).stop 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/3663863974 shutdown_connections 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc23d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc3db40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/3663863974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.470+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/3663863974 >> 172.21.15.23:0/3663863974 conn(0x560a6bb806a0 msgr2=0x560a6bc155a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.471+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/3663863974 shutdown_connections 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.471+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/3663863974 wait complete. 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.471+0000 7f63ab8f4b80 1 -- start start 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.471+0000 7f63ab8f4b80 1 Processor -- start 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc30ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc8e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc31690 con 0x560a6bc2bdf0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc8eab0 con 0x560a6bc20ef0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a6bc8ec70 con 0x560a6bc205d0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f63ab8f4b80 10 client.0 _ll_register_callbacks cb 0x560a6bb18470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35958/0 (socket says 172.21.15.23:35958) 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 -- 172.21.15.23:0/470223784 learned_addr learned my addr 172.21.15.23:0/470223784 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6394a00700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc30ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395e00700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc8e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 -- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 msgr2=0x560a6bc30ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc30ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 -- 172.21.15.23:0/470223784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 msgr2=0x560a6bc8e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:17.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc8e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.472+0000 7f6395400700 1 -- 172.21.15.23:0/470223784 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560a6b9f43b0 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6395400700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6384099490 tx=0x7f6384005c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a292511fdde3fdf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638409af10 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f638409b140 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f638409fe90 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f638409ef70 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 10 client.26842 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x560a6b9f8cd0 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f638409fe90 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6384083030 con 0x560a6bc205d0 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 10 client.26842 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:17.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 10 client.26842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 20 client.26842 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc8ff10 0x560a6bc92300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 4 client.26842 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:06:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.473+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560a6b9c1cf0 con 0x560a6bc8ff10 2024-03-30T04:06:17.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f6394a00700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc8ff10 0x560a6bc92300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f6394a00700 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc8ff10 0x560a6bc92300 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afb24cb586cdce60 server_cookie=cd8745a1a1bf287e in_seq=0 out_seq=0 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f6376a00700 10 client.26842 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f6376a00700 1 -- 172.21.15.23:0/470223784 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560a6b9c1cf0 con 0x560a6bc8ff10 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f6376a00700 10 client.26842 handle_command_reply: tid=0 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f63ab8f4b80 1 client.26842 shutdown 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 msgr2=0x560a6bc309b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.474+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6384099490 tx=0x7f6384005c20 comp rx=0 tx=0).stop 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 shutdown_connections 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560a6bc2bdf0 0x560a6bc8e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560a6bc8ff10 0x560a6bc92300 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a6bc205d0 0x560a6bc309b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 --2- 172.21.15.23:0/470223784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560a6bc20ef0 0x560a6bc30ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 >> 172.21.15.23:0/470223784 conn(0x560a6bb806a0 msgr2=0x560a6bc23d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f6376a00700 0 client.26842 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 shutdown_connections 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 1 -- 172.21.15.23:0/470223784 wait complete. 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f63ab8f4b80 20 client.26842 trim_cache size 0 max 16384 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:06:17.479 INFO:teuthology.orchestra.run.smithi023.stdout: "e82e4652-345b-4ea0-bb94-87df42e0365c": { 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "e82e4652-345b-4ea0-bb94-87df42e0365c", 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:06:17.480 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 msgr2=0x7f63780a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 0x7f63780a37d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6388009ec0 tx=0x7f6388009920 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 msgr2=0x7f63901a3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.475+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6384005910 tx=0x7f6384004730 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 shutdown_connections 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f63780a1310 0x7f63780a37d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6390105ca0 0x7f639019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6390068370 0x7f63901a3540 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 --2- 172.21.15.23:0/3467789911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6390103740 0x7f63901a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 >> 172.21.15.23:0/3467789911 conn(0x7f6390076d40 msgr2=0x7f6390100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 shutdown_connections 2024-03-30T04:06:17.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:17.476+0000 7f6397e00700 1 -- 172.21.15.23:0/3467789911 wait complete. 2024-03-30T04:06:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:17 smithi193 ceph-mon[79581]: pgmap v133: 129 pgs: 129 active+clean; 201 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 207 op/s 2024-03-30T04:06:17.493 INFO:tasks.cephfs.filesystem:scrub status for tag:e82e4652-345b-4ea0-bb94-87df42e0365c - {'path': '/', 'tag': 'e82e4652-345b-4ea0-bb94-87df42e0365c', 'options': 'recursive,force'} 2024-03-30T04:06:18.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 376401 11.69 MB/sec execute 55 sec latency 53.447 ms 2024-03-30T04:06:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:06:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:06:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:06:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:18 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:06:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:06:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:06:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:18 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:06:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:06:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:06:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:18.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:18 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:19.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 378595 11.62 MB/sec execute 56 sec latency 25.587 ms 2024-03-30T04:06:19.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:19 smithi154 ceph-mon[79161]: pgmap v134: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-30T04:06:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:19 smithi193 ceph-mon[79581]: pgmap v134: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-30T04:06:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:19 smithi023 ceph-mon[38254]: pgmap v134: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 309 op/s 2024-03-30T04:06:20.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 380436 11.63 MB/sec execute 57 sec latency 9.559 ms 2024-03-30T04:06:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:20 smithi193 ceph-mon[79581]: pgmap v135: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-30T04:06:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:20 smithi023 ceph-mon[38254]: pgmap v135: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-30T04:06:20.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:20 smithi154 ceph-mon[79161]: pgmap v135: 129 pgs: 129 active+clean; 207 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.2 MiB/s wr, 207 op/s 2024-03-30T04:06:21.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 383316 11.74 MB/sec execute 58 sec latency 10.083 ms 2024-03-30T04:06:22.196 INFO:tasks.workunit.client.0.smithi023.stdout: 1 385498 11.66 MB/sec execute 59 sec latency 11.956 ms 2024-03-30T04:06:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:22 smithi193 ceph-mon[79581]: pgmap v136: 129 pgs: 129 active+clean; 211 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 274 op/s 2024-03-30T04:06:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:22 smithi023 ceph-mon[38254]: pgmap v136: 129 pgs: 129 active+clean; 211 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 274 op/s 2024-03-30T04:06:22.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:22 smithi154 ceph-mon[79161]: pgmap v136: 129 pgs: 129 active+clean; 211 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 274 op/s 2024-03-30T04:06:23.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 387353 11.68 MB/sec execute 60 sec latency 43.947 ms 2024-03-30T04:06:23.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:23 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:06:23.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:23 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:06:23.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:23 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:06:24.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 389599 11.69 MB/sec execute 61 sec latency 69.029 ms 2024-03-30T04:06:24.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:24 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:06:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:24 smithi154 ceph-mon[79161]: pgmap v137: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 212 op/s 2024-03-30T04:06:24.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:24 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:06:24.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:24 smithi193 ceph-mon[79581]: pgmap v137: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 212 op/s 2024-03-30T04:06:24.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:24 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:06:24.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:24 smithi023 ceph-mon[38254]: pgmap v137: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 212 op/s 2024-03-30T04:06:25.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 391292 11.64 MB/sec execute 62 sec latency 10.805 ms 2024-03-30T04:06:26.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 394035 11.72 MB/sec execute 63 sec latency 19.529 ms 2024-03-30T04:06:27.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 396197 11.65 MB/sec execute 64 sec latency 18.862 ms 2024-03-30T04:06:27.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:27 smithi154 ceph-mon[79161]: pgmap v138: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-30T04:06:27.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:27 smithi193 ceph-mon[79581]: pgmap v138: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-30T04:06:27.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:27 smithi023 ceph-mon[38254]: pgmap v138: 129 pgs: 129 active+clean; 215 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 205 op/s 2024-03-30T04:06:28.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 398066 11.66 MB/sec execute 65 sec latency 16.482 ms 2024-03-30T04:06:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:28 smithi193 ceph-mon[79581]: pgmap v139: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-03-30T04:06:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:28 smithi023 ceph-mon[38254]: pgmap v139: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-03-30T04:06:28.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:28 smithi154 ceph-mon[79161]: pgmap v139: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-03-30T04:06:29.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 400483 11.68 MB/sec execute 66 sec latency 84.179 ms 2024-03-30T04:06:30.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 402057 11.63 MB/sec execute 67 sec latency 15.869 ms 2024-03-30T04:06:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:30 smithi193 ceph-mon[79581]: pgmap v140: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-30T04:06:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:30 smithi023 ceph-mon[38254]: pgmap v140: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-30T04:06:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:30 smithi154 ceph-mon[79161]: pgmap v140: 129 pgs: 129 active+clean; 221 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 206 op/s 2024-03-30T04:06:31.197 INFO:tasks.workunit.client.0.smithi023.stdout: 1 404680 11.70 MB/sec execute 68 sec latency 44.034 ms 2024-03-30T04:06:32.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 406747 11.64 MB/sec execute 69 sec latency 43.534 ms 2024-03-30T04:06:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:32 smithi193 ceph-mon[79581]: pgmap v141: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 301 op/s 2024-03-30T04:06:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:32 smithi023 ceph-mon[38254]: pgmap v141: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 301 op/s 2024-03-30T04:06:32.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:32 smithi154 ceph-mon[79161]: pgmap v141: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 301 op/s 2024-03-30T04:06:33.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 408632 11.65 MB/sec execute 70 sec latency 12.965 ms 2024-03-30T04:06:34.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 410825 11.64 MB/sec execute 71 sec latency 35.215 ms 2024-03-30T04:06:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:34 smithi193 ceph-mon[79581]: pgmap v142: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-03-30T04:06:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:34 smithi023 ceph-mon[38254]: pgmap v142: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-03-30T04:06:34.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:34 smithi154 ceph-mon[79161]: pgmap v142: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-03-30T04:06:35.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 412435 11.60 MB/sec execute 72 sec latency 9.748 ms 2024-03-30T04:06:36.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 415252 11.68 MB/sec execute 73 sec latency 36.097 ms 2024-03-30T04:06:37.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 417376 11.62 MB/sec execute 74 sec latency 33.441 ms 2024-03-30T04:06:37.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:37 smithi154 ceph-mon[79161]: pgmap v143: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-30T04:06:37.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:37 smithi193 ceph-mon[79581]: pgmap v143: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-30T04:06:37.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:37 smithi023 ceph-mon[38254]: pgmap v143: 129 pgs: 129 active+clean; 227 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 203 op/s 2024-03-30T04:06:38.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 419147 11.63 MB/sec execute 75 sec latency 61.681 ms 2024-03-30T04:06:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:38 smithi193 ceph-mon[79581]: pgmap v144: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 299 op/s 2024-03-30T04:06:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:38 smithi023 ceph-mon[38254]: pgmap v144: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 299 op/s 2024-03-30T04:06:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:38 smithi154 ceph-mon[79161]: pgmap v144: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 299 op/s 2024-03-30T04:06:39.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 421239 11.60 MB/sec execute 76 sec latency 78.305 ms 2024-03-30T04:06:40.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 423015 11.59 MB/sec execute 77 sec latency 14.815 ms 2024-03-30T04:06:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:40 smithi193 ceph-mon[79581]: pgmap v145: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-03-30T04:06:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:40 smithi023 ceph-mon[38254]: pgmap v145: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-03-30T04:06:40.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:40 smithi154 ceph-mon[79161]: pgmap v145: 129 pgs: 129 active+clean; 234 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 196 op/s 2024-03-30T04:06:41.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 425807 11.66 MB/sec execute 78 sec latency 10.195 ms 2024-03-30T04:06:42.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 427967 11.61 MB/sec execute 79 sec latency 15.792 ms 2024-03-30T04:06:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:42 smithi193 ceph-mon[79581]: pgmap v146: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-30T04:06:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:42 smithi023 ceph-mon[38254]: pgmap v146: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-30T04:06:42.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:42 smithi154 ceph-mon[79161]: pgmap v146: 129 pgs: 129 active+clean; 240 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 291 op/s 2024-03-30T04:06:43.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 429834 11.62 MB/sec execute 80 sec latency 14.939 ms 2024-03-30T04:06:44.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 432141 11.63 MB/sec execute 81 sec latency 67.127 ms 2024-03-30T04:06:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:44 smithi193 ceph-mon[79581]: pgmap v147: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:06:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:44 smithi023 ceph-mon[38254]: pgmap v147: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:06:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:44 smithi154 ceph-mon[79161]: pgmap v147: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:06:45.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 433689 11.58 MB/sec execute 82 sec latency 13.919 ms 2024-03-30T04:06:46.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 436550 11.66 MB/sec execute 83 sec latency 10.434 ms 2024-03-30T04:06:47.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 438743 11.61 MB/sec execute 84 sec latency 11.789 ms 2024-03-30T04:06:47.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:47 smithi154 ceph-mon[79161]: pgmap v148: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-03-30T04:06:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:47 smithi193 ceph-mon[79581]: pgmap v148: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-03-30T04:06:47.495 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:06:47.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:47 smithi023 ceph-mon[38254]: pgmap v148: 129 pgs: 129 active+clean; 240 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 195 op/s 2024-03-30T04:06:47.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 -- 172.21.15.23:0/3726191524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244101910 msgr2=0x7fd244105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 --2- 172.21.15.23:0/3726191524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244101910 0x7fd244105950 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd230009990 tx=0x7fd230009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 -- 172.21.15.23:0/3726191524 shutdown_connections 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 --2- 172.21.15.23:0/3726191524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244106080 0x7fd24410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 --2- 172.21.15.23:0/3726191524 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244100f60 0x7fd244101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 --2- 172.21.15.23:0/3726191524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244101910 0x7fd244105950 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 -- 172.21.15.23:0/3726191524 >> 172.21.15.23:0/3726191524 conn(0x7fd2440fc7c0 msgr2=0x7fd2440febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 -- 172.21.15.23:0/3726191524 shutdown_connections 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.722+0000 7fd24a000700 1 -- 172.21.15.23:0/3726191524 wait complete. 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.723+0000 7fd24a000700 1 Processor -- start 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.723+0000 7fd24a000700 1 -- start start 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244106080 0x7fd2440735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244073b30 0x7fd2441a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441a59f0 con 0x7fd244073b30 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441a5b60 con 0x7fd244100f60 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd24a000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2441a5d20 con 0x7fd244106080 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58182/0 (socket says 172.21.15.23:58182) 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244073b30 0x7fd2441a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 -- 172.21.15.23:0/3213427238 learned_addr learned my addr 172.21.15.23:0/3213427238 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd242a00700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244106080 0x7fd2440735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 -- 172.21.15.23:0/3213427238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244106080 msgr2=0x7fd2440735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244106080 0x7fd2440735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 -- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244073b30 msgr2=0x7fd2441a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244073b30 0x7fd2441a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.724+0000 7fd243400700 1 -- 172.21.15.23:0/3213427238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd234009520 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd243400700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd23400eb30 tx=0x7fd23400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd23be00700 1 -- 172.21.15.23:0/3213427238 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234009880 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd23be00700 1 -- 172.21.15.23:0/3213427238 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2340044e0 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2300095f0 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd23be00700 1 -- 172.21.15.23:0/3213427238 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234003e90 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.725+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2441a62f0 con 0x7fd244100f60 2024-03-30T04:06:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.728+0000 7fd23be00700 1 -- 172.21.15.23:0/3213427238 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd234004000 con 0x7fd244100f60 2024-03-30T04:06:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 Processor -- start 2024-03-30T04:06:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 -- start start 2024-03-30T04:06:47.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd23be00700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 0x7fd2280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd23be00700 1 -- 172.21.15.23:0/3213427238 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd234014070 con 0x7fd244100f60 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd01d990 con 0x55ddbd0c45d0 2024-03-30T04:06:47.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd01b350 con 0x55ddbd0cfdf0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd243e00700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 0x7fd2280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.730+0000 7fd25db73b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd016c40 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:49400/0 (socket says 172.21.15.23:49400) 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 -- 172.21.15.23:0/497071057 learned_addr learned my addr 172.21.15.23:0/497071057 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd243e00700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd243400700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 -- 172.21.15.23:0/497071057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 msgr2=0x55ddbd0d21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 -- 172.21.15.23:0/497071057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 msgr2=0x55ddbd0c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 -- 172.21.15.23:0/497071057 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddbce983b0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd243e00700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd243400700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.731+0000 7fd242a00700 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b2e0 tx=0x7fd230005ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13bed7df5da21b03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.732+0000 7fd243e00700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 0x7fd2280a3750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009ec0 tx=0x7fd23c009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.732+0000 7fd22fe00700 1 -- 172.21.15.23:0/497071057 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2300056a0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.732+0000 7fd22fe00700 1 -- 172.21.15.23:0/497071057 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23001e040 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.732+0000 7fd22fe00700 1 -- 172.21.15.23:0/497071057 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230004a00 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- 172.21.15.23:0/497071057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 msgr2=0x55ddbd0cf8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd23000b2e0 tx=0x7fd230005ea0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- 172.21.15.23:0/497071057 shutdown_connections 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 --2- 172.21.15.23:0/497071057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cf8b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- 172.21.15.23:0/497071057 >> 172.21.15.23:0/497071057 conn(0x55ddbd0246a0 msgr2=0x55ddbd0c0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- 172.21.15.23:0/497071057 shutdown_connections 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- 172.21.15.23:0/497071057 wait complete. 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.733+0000 7fd25db73b80 1 -- start start 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 Processor -- start 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbcfb2d80 con 0x55ddbd0c45d0 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbcfb2a20 con 0x55ddbd0cfdf0 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbce9a6a0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd25db73b80 10 client.0 _ll_register_callbacks cb 0x55ddbcfbc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd242a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd243400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd242a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:49404/0 (socket says 172.21.15.23:49404) 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.734+0000 7fd242a00700 1 -- 172.21.15.23:0/4056754394 learned_addr learned my addr 172.21.15.23:0/4056754394 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd243e00700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 -- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 msgr2=0x55ddbd0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 -- 172.21.15.23:0/4056754394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 msgr2=0x55ddbd0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 -- 172.21.15.23:0/4056754394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24410e470 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd242a00700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd230000c00 tx=0x7fd23000b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448034e8e8239191 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd22cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:47.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.735+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230004840 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ddbcfd7380 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd23000bdc0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd2300049a0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 10 client.26854 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ddbce44b60 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ddbcfd7380 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd23000fcd0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 10 client.26854 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 10 client.26854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 20 client.26854 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd0df430 0x55ddbd0e1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 4 client.26854 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.736+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ddbd00f700 con 0x55ddbd0df430 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.737+0000 7fd243400700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd0df430 0x55ddbd0e1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.737+0000 7fd22cc00700 10 client.26854 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.737+0000 7fd243400700 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd0df430 0x55ddbd0e1820 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530396dcd53e0ea2 server_cookie=afc7c94b0bc4522d in_seq=0 out_seq=0 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd22cc00700 1 -- 172.21.15.23:0/4056754394 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ddbd00f700 con 0x55ddbd0df430 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd22cc00700 10 client.26854 handle_command_reply: tid=0 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 client.26854 shutdown 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 msgr2=0x55ddbd0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd230000c00 tx=0x7fd23000b980 comp rx=0 tx=0).stop 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 shutdown_connections 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd0df430 0x55ddbd0e1820 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 --2- 172.21.15.23:0/4056754394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0ccb10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 >> 172.21.15.23:0/4056754394 conn(0x55ddbd0246a0 msgr2=0x55ddbd0c66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd22cc00700 0 client.26854 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 shutdown_connections 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 1 -- 172.21.15.23:0/4056754394 wait complete. 2024-03-30T04:06:47.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.738+0000 7fd25db73b80 20 client.26854 trim_cache size 0 max 16384 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.758+0000 7fd25db73b80 1 Processor -- start 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.758+0000 7fd25db73b80 1 -- start start 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.758+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.758+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd25db73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd0c79a0 con 0x55ddbd0c45d0 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd25db73b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd0d6f50 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd25db73b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd0cb060 con 0x55ddbd0cfdf0 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd243400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58200/0 (socket says 172.21.15.23:58200) 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 -- 172.21.15.23:0/1335064852 learned_addr learned my addr 172.21.15.23:0/1335064852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd243400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50572/0 (socket says 172.21.15.23:50572) 2024-03-30T04:06:47.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd243e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 -- 172.21.15.23:0/1335064852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 msgr2=0x55ddbd0cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 -- 172.21.15.23:0/1335064852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 msgr2=0x55ddbd0d6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd242a00700 1 -- 172.21.15.23:0/1335064852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ddbcfdcea0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd243400700 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.759+0000 7fd243e00700 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.760+0000 7fd242a00700 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd230009990 tx=0x7fd230009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3248c459c5b753f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.760+0000 7fd239200700 1 -- 172.21.15.23:0/1335064852 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23000b9f0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.760+0000 7fd239200700 1 -- 172.21.15.23:0/1335064852 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23002c690 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.760+0000 7fd239200700 1 -- 172.21.15.23:0/1335064852 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23002cbc0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- 172.21.15.23:0/1335064852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 msgr2=0x55ddbd0cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd230009990 tx=0x7fd230009550 comp rx=0 tx=0).stop 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- 172.21.15.23:0/1335064852 shutdown_connections 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 --2- 172.21.15.23:0/1335064852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0cc3a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- 172.21.15.23:0/1335064852 >> 172.21.15.23:0/1335064852 conn(0x55ddbd0246a0 msgr2=0x55ddbd0b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- 172.21.15.23:0/1335064852 shutdown_connections 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- 172.21.15.23:0/1335064852 wait complete. 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.761+0000 7fd25db73b80 1 -- start start 2024-03-30T04:06:47.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 Processor -- start 2024-03-30T04:06:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0d4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd0d5cc0 con 0x55ddbd0c4ef0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd0d5e30 con 0x55ddbd0c45d0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ddbd133240 con 0x55ddbd0cfdf0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.762+0000 7fd25db73b80 10 client.0 _ll_register_callbacks cb 0x55ddbcfbc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd242a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0d4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58206/0 (socket says 172.21.15.23:58206) 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 -- 172.21.15.23:0/2867031997 learned_addr learned my addr 172.21.15.23:0/2867031997 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:47.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243e00700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 -- 172.21.15.23:0/2867031997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 msgr2=0x55ddbd0d55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 -- 172.21.15.23:0/2867031997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c4ef0 msgr2=0x55ddbd0d4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0d4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 -- 172.21.15.23:0/2867031997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2441a62f0 con 0x55ddbd0c45d0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd243400700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd234096730 tx=0x7fd23409a120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ab48e2edcf4ecb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd22cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23409a230 con 0x55ddbd0c45d0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd23001e040 con 0x55ddbd0c45d0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.763+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd234094e40 con 0x55ddbd0c45d0 2024-03-30T04:06:47.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd23409e060 con 0x55ddbd0c45d0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 10 client.26864 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fd23c0090f0 con 0x55ddbd0c45d0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd23001e040 con 0x55ddbd0c45d0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd234096940 con 0x55ddbd0c45d0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 10 client.26864 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 10 client.26864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 20 client.26864 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd134a30 0x55ddbd136e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 4 client.26864 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ddbce65cf0 con 0x55ddbd134a30 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.764+0000 7fd242a00700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd134a30 0x55ddbd136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd242a00700 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd134a30 0x55ddbd136e50 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7e7d252a9aceaf server_cookie=47e5cbf15fca9702 in_seq=0 out_seq=0 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd22cc00700 10 client.26864 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd22cc00700 1 -- 172.21.15.23:0/2867031997 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ddbce65cf0 con 0x55ddbd134a30 2024-03-30T04:06:47.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd22cc00700 10 client.26864 handle_command_reply: tid=0 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 client.26864 shutdown 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 msgr2=0x55ddbd0d6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd234096730 tx=0x7fd23409a120 comp rx=0 tx=0).stop 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 shutdown_connections 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ddbd134a30 0x55ddbd136e50 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ddbd0cfdf0 0x55ddbd0d55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ddbd0c45d0 0x55ddbd0d6ae0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd22cc00700 0 client.26864 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 --2- 172.21.15.23:0/2867031997 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ddbd0c4ef0 0x55ddbd0d4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.765+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 >> 172.21.15.23:0/2867031997 conn(0x55ddbd0246a0 msgr2=0x55ddbd0c02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 shutdown_connections 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd25db73b80 1 -- 172.21.15.23:0/2867031997 wait complete. 2024-03-30T04:06:47.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd25db73b80 20 client.26864 trim_cache size 0 max 16384 2024-03-30T04:06:47.770 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:47.770 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:06:47.770 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:06:47.770 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 msgr2=0x7fd2280a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 0x7fd2280a3750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009ec0 tx=0x7fd23c009920 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 msgr2=0x7fd2440730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.766+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd23400eb30 tx=0x7fd23400ee40 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 shutdown_connections 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd244073b30 0x7fd2441a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd2280a1290 0x7fd2280a3750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd244100f60 0x7fd2440730b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 --2- 172.21.15.23:0/3213427238 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd244106080 0x7fd2440735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:47.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 >> 172.21.15.23:0/3213427238 conn(0x7fd2440fc7c0 msgr2=0x7fd2440fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:47.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 shutdown_connections 2024-03-30T04:06:47.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:47.767+0000 7fd24a000700 1 -- 172.21.15.23:0/3213427238 wait complete. 2024-03-30T04:06:47.784 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:06:47.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:06:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 -- 172.21.15.23:0/3026721707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 msgr2=0x7f093010db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 --2- 172.21.15.23:0/3026721707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f093010db00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f092800b0a0 tx=0x7f092800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 -- 172.21.15.23:0/3026721707 shutdown_connections 2024-03-30T04:06:48.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 --2- 172.21.15.23:0/3026721707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f093010db00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 --2- 172.21.15.23:0/3026721707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 0x7f0930101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 --2- 172.21.15.23:0/3026721707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0930101810 0x7f0930105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 -- 172.21.15.23:0/3026721707 >> 172.21.15.23:0/3026721707 conn(0x7f09300fc700 msgr2=0x7f09300feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 -- 172.21.15.23:0/3026721707 shutdown_connections 2024-03-30T04:06:48.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.045+0000 7f0937c00700 1 -- 172.21.15.23:0/3026721707 wait complete. 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.046+0000 7f0937c00700 1 Processor -- start 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.046+0000 7f0937c00700 1 -- start start 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 0x7f09301994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093019dda0 0x7f093019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093019e820 con 0x7f093019dda0 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093019e990 con 0x7f0930100e60 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0937c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093019eb50 con 0x7f0930105f80 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0935200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 0x7f09301994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f092fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f092fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44100/0 (socket says 172.21.15.23:44100) 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f092fe00700 1 -- 172.21.15.23:0/3082164526 learned_addr learned my addr 172.21.15.23:0/3082164526 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f0935c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093019dda0 0x7f093019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f092fe00700 1 -- 172.21.15.23:0/3082164526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 msgr2=0x7f09301994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.047+0000 7f092fe00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 0x7f09301994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092fe00700 1 -- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093019dda0 msgr2=0x7f093019e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092fe00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093019dda0 0x7f093019e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092fe00700 1 -- 172.21.15.23:0/3082164526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0928009d00 con 0x7f0930105f80 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092fe00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d980 tx=0x7f091c00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00f890 con 0x7f0930105f80 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09301a60b0 con 0x7f0930105f80 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09301a65e0 con 0x7f0930105f80 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091c010460 con 0x7f0930105f80 2024-03-30T04:06:48.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.048+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00e610 con 0x7f0930105f80 2024-03-30T04:06:48.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.049+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09100052f0 con 0x7f0930105f80 2024-03-30T04:06:48.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.052+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f091c00f9f0 con 0x7f0930105f80 2024-03-30T04:06:48.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.053+0000 7f092d600700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 0x7f09180a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.053+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f091c0cc8e0 con 0x7f0930105f80 2024-03-30T04:06:48.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.053+0000 7f0935200700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 0x7f09180a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.054+0000 7f0935200700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 0x7f09180a3750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0924009c60 tx=0x7f0924009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.057+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f091c095970 con 0x7f0930105f80 2024-03-30T04:06:48.198 INFO:tasks.workunit.client.0.smithi023.stdout: 1 440597 11.62 MB/sec execute 85 sec latency 9.050 ms 2024-03-30T04:06:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.379+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0910005160 con 0x7f0930105f80 2024-03-30T04:06:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.380+0000 7f092d600700 1 -- 172.21.15.23:0/3082164526 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f091c098f90 con 0x7f0930105f80 2024-03-30T04:06:48.378 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:06:48.378 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:06:48.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 msgr2=0x7f09180a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 0x7f09180a3750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0924009c60 tx=0x7f0924009260 comp rx=0 tx=0).stop 2024-03-30T04:06:48.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 msgr2=0x7f0930199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f091c00d980 tx=0x7f091c00dc90 comp rx=0 tx=0).stop 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 shutdown_connections 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f093019dda0 0x7f093019e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f09180a1290 0x7f09180a3750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0930100e60 0x7f09301994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 --2- 172.21.15.23:0/3082164526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0930105f80 0x7f0930199a00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 >> 172.21.15.23:0/3082164526 conn(0x7f09300fc700 msgr2=0x7f09300fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.382+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 shutdown_connections 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.383+0000 7f0937c00700 1 -- 172.21.15.23:0/3082164526 wait complete. 2024-03-30T04:06:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:06:48.402 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:06:48.462 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:48 smithi023 ceph-mon[38254]: pgmap v149: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 295 op/s 2024-03-30T04:06:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:48 smithi193 ceph-mon[79581]: pgmap v149: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 295 op/s 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1227528061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 msgr2=0x7fc0b01035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1227528061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01035b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0009990 tx=0x7fc0a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1227528061 shutdown_connections 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1227528061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 0x7fc0b010fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1227528061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01035b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1227528061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b0103b80 0x7fc0b0107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1227528061 >> 172.21.15.23:0/1227528061 conn(0x7fc0b00fea10 msgr2=0x7fc0b0100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1227528061 shutdown_connections 2024-03-30T04:06:48.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.631+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1227528061 wait complete. 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.632+0000 7fc0b6c00700 1 Processor -- start 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.632+0000 7fc0b6c00700 1 -- start start 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 0x7fc0b0199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b019ddc0 0x7fc0b019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b019e840 con 0x7fc0b019ddc0 2024-03-30T04:06:48.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b019e9b0 con 0x7fc0b01031d0 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b6c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b019eb70 con 0x7fc0b01082f0 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38050/0 (socket says 172.21.15.23:38050) 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 -- 172.21.15.23:0/1602263642 learned_addr learned my addr 172.21.15.23:0/1602263642 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0af400700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 0x7fc0b0199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0b4c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b019ddc0 0x7fc0b019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 -- 172.21.15.23:0/1602263642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 msgr2=0x7fc0b0199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 0x7fc0b0199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 -- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b019ddc0 msgr2=0x7fc0b019e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b019ddc0 0x7fc0b019e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.633+0000 7fc0afe00700 1 -- 172.21.15.23:0/1602263642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0a00095f0 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0afe00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0005910 tx=0x7fc0a0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0acc00700 1 -- 172.21.15.23:0/1602263642 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a001d070 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0acc00700 1 -- 172.21.15.23:0/1602263642 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0a0022470 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0acc00700 1 -- 172.21.15.23:0/1602263642 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a0017650 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b01a6070 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.634+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b01a64f0 con 0x7fc0b01031d0 2024-03-30T04:06:48.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.637+0000 7fc0acc00700 1 -- 172.21.15.23:0/1602263642 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc0a00225e0 con 0x7fc0b01031d0 2024-03-30T04:06:48.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.638+0000 7fc0acc00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 0x7fc0940a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.639+0000 7fc0acc00700 1 -- 172.21.15.23:0/1602263642 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc0a00ce1d0 con 0x7fc0b01031d0 2024-03-30T04:06:48.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.639+0000 7fc0af400700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 0x7fc0940a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.639+0000 7fc0ca70bb80 1 Processor -- start 2024-03-30T04:06:48.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.639+0000 7fc0ca70bb80 1 -- start start 2024-03-30T04:06:48.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0af400700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 0x7fc0940a37d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0b019aa30 tx=0x7fc09c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc49d990 con 0x5590cc544ef0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc49b350 con 0x5590cc5445d0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc496c40 con 0x5590cc54fdf0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0b4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0b4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44126/0 (socket says 172.21.15.23:44126) 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60996/0 (socket says 172.21.15.23:60996) 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0b4c00700 1 -- 172.21.15.23:0/3682613119 learned_addr learned my addr 172.21.15.23:0/3682613119 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38064/0 (socket says 172.21.15.23:38064) 2024-03-30T04:06:48.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 -- 172.21.15.23:0/3682613119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 msgr2=0x5590cc5521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 -- 172.21.15.23:0/3682613119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 msgr2=0x5590cc54f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0afe00700 1 -- 172.21.15.23:0/3682613119 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590cc3183b0 con 0x5590cc5445d0 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0b4c00700 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.640+0000 7fc0af400700 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.641+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00cb560 tx=0x7fc0a0022c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fa8a8c0351d1456 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.641+0000 7fc0a5200700 1 -- 172.21.15.23:0/3682613119 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a009b6b0 con 0x5590cc5445d0 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.641+0000 7fc0a5200700 1 -- 172.21.15.23:0/3682613119 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0a009afb0 con 0x5590cc5445d0 2024-03-30T04:06:48.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.641+0000 7fc0a5200700 1 -- 172.21.15.23:0/3682613119 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a009b450 con 0x5590cc5445d0 2024-03-30T04:06:48.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3682613119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 msgr2=0x5590cc5449b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00cb560 tx=0x7fc0a0022c90 comp rx=0 tx=0).stop 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3682613119 shutdown_connections 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc5521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc5445d0 0x5590cc5449b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3682613119 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc54f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3682613119 >> 172.21.15.23:0/3682613119 conn(0x5590cc4a46a0 msgr2=0x5590cc540a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3682613119 shutdown_connections 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.642+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3682613119 wait complete. 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 -- start start 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 Processor -- start 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc5445d0 0x5590cc54f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc54d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc432d80 con 0x5590cc5445d0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc432a20 con 0x5590cc544ef0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc31a6a0 con 0x5590cc54fdf0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.643+0000 7fc0ca70bb80 10 client.0 _ll_register_callbacks cb 0x5590cc43c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0b4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc54d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38074/0 (socket says 172.21.15.23:38074) 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 -- 172.21.15.23:0/74808188 learned_addr learned my addr 172.21.15.23:0/74808188 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0afe00700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc5445d0 0x5590cc54f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 -- 172.21.15.23:0/74808188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 msgr2=0x5590cc54d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc54d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 -- 172.21.15.23:0/74808188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc5445d0 msgr2=0x5590cc54f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc5445d0 0x5590cc54f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 -- 172.21.15.23:0/74808188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0a8011330 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0b4c00700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc54d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.644+0000 7fc0af400700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc09c01d8f0 tx=0x7fc09c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=395b41ca470c622b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc099600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09c022070 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc09c004240 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5590cc457380 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc09c004400 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc0ca70bb80 10 client.26882 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fc0b01a64f0 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.645+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5590cc457380 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc09c00e700 con 0x5590cc544ef0 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 10 client.26882 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:48.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 10 client.26882 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 20 client.26882 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc567f10 0x5590cc56a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 4 client.26882 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590cc48f700 con 0x5590cc567f10 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.646+0000 7fc0afe00700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc567f10 0x5590cc56a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc099600700 10 client.26882 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc0afe00700 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc567f10 0x5590cc56a330 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e65f0b8bba657743 server_cookie=2515f272736955f0 in_seq=0 out_seq=0 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc099600700 1 -- 172.21.15.23:0/74808188 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5590cc48f700 con 0x5590cc567f10 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc099600700 10 client.26882 handle_command_reply: tid=0 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc0ca70bb80 1 client.26882 shutdown 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 msgr2=0x5590cc54cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.647+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc09c01d8f0 tx=0x7fc09c0047c0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 shutdown_connections 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc567f10 0x5590cc56a330 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc54fdf0 0x5590cc54d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc5445d0 0x5590cc54f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/74808188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc544ef0 0x5590cc54cb10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc099600700 0 client.26882 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 >> 172.21.15.23:0/74808188 conn(0x5590cc4a46a0 msgr2=0x5590cc5466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 shutdown_connections 2024-03-30T04:06:48.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/74808188 wait complete. 2024-03-30T04:06:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.648+0000 7fc0ca70bb80 20 client.26882 trim_cache size 0 max 16384 2024-03-30T04:06:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.670+0000 7fc0ca70bb80 1 Processor -- start 2024-03-30T04:06:48.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.670+0000 7fc0ca70bb80 1 -- start start 2024-03-30T04:06:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.670+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc5395a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc54de60 con 0x5590cc544ef0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc546130 con 0x5590cc54fdf0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc545f60 con 0x5590cc5445d0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44132/0 (socket says 172.21.15.23:44132) 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0afe00700 1 -- 172.21.15.23:0/945995321 learned_addr learned my addr 172.21.15.23:0/945995321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0af400700 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.671+0000 7fc0b4c00700 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc5395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 -- 172.21.15.23:0/945995321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 msgr2=0x5590cc5395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc5395a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 -- 172.21.15.23:0/945995321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 msgr2=0x5590cc55ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 -- 172.21.15.23:0/945995321 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590cc2c4b60 con 0x5590cc5445d0 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0afe00700 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0099300 tx=0x7fc0a0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdf1a4008ec8bb51 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0a5c00700 1 -- 172.21.15.23:0/945995321 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00b9070 con 0x5590cc5445d0 2024-03-30T04:06:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0a5c00700 1 -- 172.21.15.23:0/945995321 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0a009ad80 con 0x5590cc5445d0 2024-03-30T04:06:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.672+0000 7fc0a5c00700 1 -- 172.21.15.23:0/945995321 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a009a4a0 con 0x5590cc5445d0 2024-03-30T04:06:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/945995321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 msgr2=0x5590cc54cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0099300 tx=0x7fc0a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/945995321 shutdown_connections 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc5395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc54cad0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/945995321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/945995321 >> 172.21.15.23:0/945995321 conn(0x5590cc4a46a0 msgr2=0x5590cc54c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.674+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/945995321 shutdown_connections 2024-03-30T04:06:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.675+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/945995321 wait complete. 2024-03-30T04:06:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.675+0000 7fc0ca70bb80 1 -- start start 2024-03-30T04:06:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.675+0000 7fc0ca70bb80 1 Processor -- start 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc55cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc55d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc5bad00 con 0x5590cc544ef0 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc5bae70 con 0x5590cc54fdf0 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.676+0000 7fc0ca70bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590cc5cc040 con 0x5590cc5445d0 2024-03-30T04:06:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.677+0000 7fc0ca70bb80 10 client.0 _ll_register_callbacks cb 0x5590cc43c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.677+0000 7fc0afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc55cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.677+0000 7fc0af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32782/0 (socket says 172.21.15.23:32782) 2024-03-30T04:06:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0af400700 1 -- 172.21.15.23:0/3834475839 learned_addr learned my addr 172.21.15.23:0/3834475839 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0b4c00700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc55d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0afe00700 1 -- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 msgr2=0x5590cc55d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc55d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0afe00700 1 -- 172.21.15.23:0/3834475839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 msgr2=0x5590cc55d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.678+0000 7fc0afe00700 1 -- 172.21.15.23:0/3834475839 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590cc3183b0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.679+0000 7fc0afe00700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc55cf40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00974b0 tx=0x7fc0a0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78296dc1ef25c394 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.679+0000 7fc099600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.679+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a00b9040 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.679+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc0a00a33a0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.679+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0a009afb0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.680+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0a00228c0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.680+0000 7fc0ca70bb80 10 client.26884 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.680+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5590cc45cea0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.681+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0a009afb0 con 0x5590cc5445d0 2024-03-30T04:06:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.681+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc0a009a5b0 con 0x5590cc5445d0 2024-03-30T04:06:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 10 client.26884 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 10 client.26884 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:06:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 20 client.26884 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc5cd330 0x5590cc5cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 4 client.26884 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5590cc2e5cf0 con 0x5590cc5cd330 2024-03-30T04:06:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:48 smithi154 ceph-mon[79161]: pgmap v149: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 295 op/s 2024-03-30T04:06:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.682+0000 7fc0af400700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc5cd330 0x5590cc5cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.683+0000 7fc099600700 10 client.26884 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.683+0000 7fc0af400700 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc5cd330 0x5590cc5cf750 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ea6baabadba6800 server_cookie=2b164b31dced01f3 in_seq=0 out_seq=0 2024-03-30T04:06:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.684+0000 7fc099600700 1 -- 172.21.15.23:0/3834475839 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5590cc2e5cf0 con 0x5590cc5cd330 2024-03-30T04:06:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.684+0000 7fc099600700 10 client.26884 handle_command_reply: tid=0 2024-03-30T04:06:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.684+0000 7fc0ca70bb80 1 client.26884 shutdown 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.684+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 msgr2=0x5590cc55cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.684+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc55cf40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc0a00974b0 tx=0x7fc0a0009310 comp rx=0 tx=0).stop 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 shutdown_connections 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5590cc54fdf0 0x5590cc55d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5590cc5cd330 0x5590cc5cf750 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590cc5445d0 0x5590cc55cf40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 --2- 172.21.15.23:0/3834475839 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5590cc544ef0 0x5590cc55d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc099600700 0 client.26884 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:06:48.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.685+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 >> 172.21.15.23:0/3834475839 conn(0x5590cc4a46a0 msgr2=0x5590cc54f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.686+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 shutdown_connections 2024-03-30T04:06:48.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.686+0000 7fc0ca70bb80 1 -- 172.21.15.23:0/3834475839 wait complete. 2024-03-30T04:06:48.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.687+0000 7fc0ca70bb80 20 client.26884 trim_cache size 0 max 16384 2024-03-30T04:06:48.685 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 msgr2=0x7fc0940a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 0x7fc0940a37d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0b019aa30 tx=0x7fc09c009260 comp rx=0 tx=0).stop 2024-03-30T04:06:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 msgr2=0x7fc0b01994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc0a0005910 tx=0x7fc0a0004730 comp rx=0 tx=0).stop 2024-03-30T04:06:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 shutdown_connections 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc0b019ddc0 0x7fc0b019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc0940a1310 0x7fc0940a37d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0b01031d0 0x7fc0b01994e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 --2- 172.21.15.23:0/1602263642 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc0b01082f0 0x7fc0b0199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.688+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 >> 172.21.15.23:0/1602263642 conn(0x7fc0b00fea10 msgr2=0x7fc0b00ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.689+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 shutdown_connections 2024-03-30T04:06:48.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:48.689+0000 7fc0b6c00700 1 -- 172.21.15.23:0/1602263642 wait complete. 2024-03-30T04:06:48.718 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:06:49.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 -- 172.21.15.23:0/499709120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101910 msgr2=0x7f54fc105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 --2- 172.21.15.23:0/499709120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101910 0x7f54fc105950 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f54e80098f0 tx=0x7f54e8009c00 comp rx=0 tx=0).stop 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 -- 172.21.15.23:0/499709120 shutdown_connections 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 --2- 172.21.15.23:0/499709120 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc106080 0x7f54fc10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 --2- 172.21.15.23:0/499709120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc100f60 0x7f54fc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 --2- 172.21.15.23:0/499709120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc101910 0x7f54fc105950 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 -- 172.21.15.23:0/499709120 >> 172.21.15.23:0/499709120 conn(0x7f54fc0fc7c0 msgr2=0x7f54fc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 -- 172.21.15.23:0/499709120 shutdown_connections 2024-03-30T04:06:49.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.003+0000 7f5502000700 1 -- 172.21.15.23:0/499709120 wait complete. 2024-03-30T04:06:49.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 Processor -- start 2024-03-30T04:06:49.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 -- start start 2024-03-30T04:06:49.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 0x7f54fc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc073b30 0x7f54fc1a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a5ac0 con 0x7f54fc100f60 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a5c00 con 0x7f54fc106080 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.004+0000 7f5502000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a5dc0 con 0x7f54fc073b30 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54fb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 0x7f54fc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54fb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 0x7f54fc0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32790/0 (socket says 172.21.15.23:32790) 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38110/0 (socket says 172.21.15.23:38110) 2024-03-30T04:06:49.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54fb400700 1 -- 172.21.15.23:0/2335956683 learned_addr learned my addr 172.21.15.23:0/2335956683 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54fbe00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc073b30 0x7f54fc1a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 -- 172.21.15.23:0/2335956683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc073b30 msgr2=0x7f54fc1a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc073b30 0x7f54fc1a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 -- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 msgr2=0x7f54fc0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 0x7f54fc0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 -- 172.21.15.23:0/2335956683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e8009520 con 0x7f54fc106080 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.005+0000 7f54faa00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f54e8000c00 tx=0x7f54e800f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.006+0000 7f54f3e00700 1 -- 172.21.15.23:0/2335956683 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e801c070 con 0x7f54fc106080 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.006+0000 7f54f3e00700 1 -- 172.21.15.23:0/2335956683 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54e800baa0 con 0x7f54fc106080 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.006+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc1a6040 con 0x7f54fc106080 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.006+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc1a64c0 con 0x7f54fc106080 2024-03-30T04:06:49.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.006+0000 7f54f3e00700 1 -- 172.21.15.23:0/2335956683 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e80038e0 con 0x7f54fc106080 2024-03-30T04:06:49.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.009+0000 7f54f3e00700 1 -- 172.21.15.23:0/2335956683 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f54e8020030 con 0x7f54fc106080 2024-03-30T04:06:49.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.010+0000 7f5515aa3b80 1 Processor -- start 2024-03-30T04:06:49.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 -- start start 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x5555858f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859031e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558584e990 con 0x5555858f55d0 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558584c350 con 0x555585900df0 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555585847c40 con 0x5555858f5ef0 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54f3e00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 0x7f54e00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54fb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54fb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44172/0 (socket says 172.21.15.23:44172) 2024-03-30T04:06:49.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54fb400700 1 -- 172.21.15.23:0/1149045153 learned_addr learned my addr 172.21.15.23:0/1149045153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54fbe00700 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x5555858f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54f3e00700 1 -- 172.21.15.23:0/2335956683 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f54e80cdde0 con 0x7f54fc106080 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.011+0000 7f54faa00700 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859031e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fbe00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 0x7f54e00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 -- 172.21.15.23:0/1149045153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 msgr2=0x5555859031e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859031e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 -- 172.21.15.23:0/1149045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 msgr2=0x5555858f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x5555858f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 -- 172.21.15.23:0/1149045153 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5555856c93b0 con 0x5555858f5ef0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fb400700 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009e30 tx=0x7f54ec011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e031ced44b54c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54fbe00700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 0x7f54e00a3750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009790 tx=0x7f54f4009700 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54e7e00700 1 -- 172.21.15.23:0/1149045153 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec019630 con 0x5555858f5ef0 2024-03-30T04:06:49.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54e7e00700 1 -- 172.21.15.23:0/1149045153 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54ec019c50 con 0x5555858f5ef0 2024-03-30T04:06:49.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.012+0000 7f54e7e00700 1 -- 172.21.15.23:0/1149045153 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec00ebf0 con 0x5555858f5ef0 2024-03-30T04:06:49.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 -- 172.21.15.23:0/1149045153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 msgr2=0x5555859008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f54ec009e30 tx=0x7f54ec011720 comp rx=0 tx=0).stop 2024-03-30T04:06:49.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 -- 172.21.15.23:0/1149045153 shutdown_connections 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x5555858f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/1149045153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x5555859008b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 -- 172.21.15.23:0/1149045153 >> 172.21.15.23:0/1149045153 conn(0x5555858556a0 msgr2=0x5555858f1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 -- 172.21.15.23:0/1149045153 shutdown_connections 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.013+0000 7f5515aa3b80 1 -- 172.21.15.23:0/1149045153 wait complete. 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.014+0000 7f5515aa3b80 1 -- start start 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.014+0000 7f5515aa3b80 1 Processor -- start 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x5555858fdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858fe150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555857e3d80 con 0x5555858f5ef0 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555857e3a20 con 0x5555858f55d0 2024-03-30T04:06:49.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555856cb6a0 con 0x555585900df0 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f5515aa3b80 10 client.0 _ll_register_callbacks cb 0x5555857ed470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54faa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x5555858fdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38132/0 (socket says 172.21.15.23:38132) 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 -- 172.21.15.23:0/3302820427 learned_addr learned my addr 172.21.15.23:0/3302820427 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fbe00700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858fe150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 -- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 msgr2=0x5555858fe150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858fe150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 -- 172.21.15.23:0/3302820427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 msgr2=0x5555858fdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x5555858fdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54fb400700 1 -- 172.21.15.23:0/3302820427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54ec009a60 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.015+0000 7f54faa00700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x5555858fdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54fb400700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0098f0 tx=0x7f54ec00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d454543c878e5de server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec019640 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f54ec0197a0 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555585808380 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54ec019960 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f5515aa3b80 10 client.26897 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555585675b60 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.016+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555585808380 con 0x5555858f55d0 2024-03-30T04:06:49.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54ec02a060 con 0x5555858f55d0 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 10 client.26897 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 10 client.26897 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 20 client.26897 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5555859108a0 0x555585912c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 4 client.26897 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555585840700 con 0x5555859108a0 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f54faa00700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5555859108a0 0x555585912c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f54faa00700 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5555859108a0 0x555585912c90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=88b3c83f037276fa server_cookie=4a7c5953a9a8dfa3 in_seq=0 out_seq=0 2024-03-30T04:06:49.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.017+0000 7f54e4c00700 10 client.26897 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:49.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f54e4c00700 1 -- 172.21.15.23:0/3302820427 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555585840700 con 0x5555859108a0 2024-03-30T04:06:49.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f54e4c00700 10 client.26897 handle_command_reply: tid=0 2024-03-30T04:06:49.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 client.26897 shutdown 2024-03-30T04:06:49.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 msgr2=0x5555859007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f54ec0098f0 tx=0x7f54ec00ee70 comp rx=0 tx=0).stop 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 shutdown_connections 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5555859108a0 0x555585912c90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858fe150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555859007d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/3302820427 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x5555858fdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f54e4c00700 0 client.26897 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 >> 172.21.15.23:0/3302820427 conn(0x5555858556a0 msgr2=0x5555858f76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 shutdown_connections 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 1 -- 172.21.15.23:0/3302820427 wait complete. 2024-03-30T04:06:49.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.019+0000 7f5515aa3b80 20 client.26897 trim_cache size 0 max 16384 2024-03-30T04:06:49.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 Processor -- start 2024-03-30T04:06:49.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 -- start start 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x555585907a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858eaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555858f2660 con 0x5555858f5ef0 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555585907fb0 con 0x5555858f55d0 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555858f9740 con 0x555585900df0 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f54fb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f54fb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38136/0 (socket says 172.21.15.23:38136) 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f54fb400700 1 -- 172.21.15.23:0/2828660546 learned_addr learned my addr 172.21.15.23:0/2828660546 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f54faa00700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x555585907a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.034+0000 7f54fbe00700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858eaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 -- 172.21.15.23:0/2828660546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 msgr2=0x5555858eaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858eaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 -- 172.21.15.23:0/2828660546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 msgr2=0x555585907a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x555585907a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 -- 172.21.15.23:0/2828660546 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558580dea0 con 0x5555858f55d0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54faa00700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x555585907a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54fb400700 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005850 tx=0x7f54ec00eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9eca7af183738cbb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54f1200700 1 -- 172.21.15.23:0/2828660546 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec00e5d0 con 0x5555858f55d0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54f1200700 1 -- 172.21.15.23:0/2828660546 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54ec005a00 con 0x5555858f55d0 2024-03-30T04:06:49.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.035+0000 7f54f1200700 1 -- 172.21.15.23:0/2828660546 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ec022660 con 0x5555858f55d0 2024-03-30T04:06:49.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 -- 172.21.15.23:0/2828660546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 msgr2=0x5555858f8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f54ec005850 tx=0x7f54ec00eee0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 -- 172.21.15.23:0/2828660546 shutdown_connections 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555585900df0 0x5555858eaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5555858f55d0 0x5555858f8610 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/2828660546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f5ef0 0x555585907a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 -- 172.21.15.23:0/2828660546 >> 172.21.15.23:0/2828660546 conn(0x5555858556a0 msgr2=0x5555858ea5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 -- 172.21.15.23:0/2828660546 shutdown_connections 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.036+0000 7f5515aa3b80 1 -- 172.21.15.23:0/2828660546 wait complete. 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 -- start start 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 Processor -- start 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x555585905910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x555585905e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5555859065c0 con 0x5555858f55d0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555585963be0 con 0x555585900df0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555585963da0 con 0x5555858f5ef0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.037+0000 7f5515aa3b80 10 client.0 _ll_register_callbacks cb 0x5555857ed470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x555585905910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38146/0 (socket says 172.21.15.23:38146) 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 -- 172.21.15.23:0/468768457 learned_addr learned my addr 172.21.15.23:0/468768457 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54faa00700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x555585905e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 -- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 msgr2=0x555585905e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x555585905e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 -- 172.21.15.23:0/468768457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 msgr2=0x555585905910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x555585905910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 -- 172.21.15.23:0/468768457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555585675b60 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54fbe00700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f54f4003a90 tx=0x7f54f400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1ab39c1fa10c21b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.038+0000 7f54e4c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4003e50 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54ec019c50 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f54f4004030 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54f4004bb0 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 10 client.26903 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5555856cdcd0 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54ec019c50 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54f40284f0 con 0x555585900df0 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 10 client.26903 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 10 client.26903 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 20 client.26903 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555585965590 0x5555859679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 4 client.26903 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.039+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555585696cf0 con 0x555585965590 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f54fb400700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555585965590 0x5555859679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f54fb400700 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555585965590 0x5555859679b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4b677d3240fc1a65 server_cookie=93d3d0d29f187786 in_seq=0 out_seq=0 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f54e4c00700 10 client.26903 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f54e4c00700 1 -- 172.21.15.23:0/468768457 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555585696cf0 con 0x555585965590 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f54e4c00700 10 client.26903 handle_command_reply: tid=0 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.040+0000 7f5515aa3b80 1 client.26903 shutdown 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 msgr2=0x5555859636a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f54f4003a90 tx=0x7f54f400b040 comp rx=0 tx=0).stop 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 shutdown_connections 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555585965590 0x5555859679b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555585900df0 0x5555859636a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5555858f55d0 0x555585905910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f54e4c00700 0 client.26903 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 --2- 172.21.15.23:0/468768457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5555858f5ef0 0x555585905e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 >> 172.21.15.23:0/468768457 conn(0x5555858556a0 msgr2=0x5555858fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 shutdown_connections 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 1 -- 172.21.15.23:0/468768457 wait complete. 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.041+0000 7f5515aa3b80 20 client.26903 trim_cache size 0 max 16384 2024-03-30T04:06:49.045 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 msgr2=0x7f54e00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 0x7f54e00a3750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009790 tx=0x7f54f4009700 comp rx=0 tx=0).stop 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 msgr2=0x7f54fc0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f54e8000c00 tx=0x7f54e800f9e0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 shutdown_connections 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54fc073b30 0x7f54fc1a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54fc100f60 0x7f54fc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54e00a1290 0x7f54e00a3750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 --2- 172.21.15.23:0/2335956683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54fc106080 0x7f54fc0735f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 >> 172.21.15.23:0/2335956683 conn(0x7f54fc0fc7c0 msgr2=0x7f54fc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 shutdown_connections 2024-03-30T04:06:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.042+0000 7f5502000700 1 -- 172.21.15.23:0/2335956683 wait complete. 2024-03-30T04:06:49.058 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:06:49.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 443050 11.63 MB/sec execute 86 sec latency 42.688 ms 2024-03-30T04:06:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 -- 172.21.15.23:0/3242195481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40100800 msgr2=0x7f2c40104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 --2- 172.21.15.23:0/3242195481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40100800 0x7f2c40104830 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c0098f0 tx=0x7f2c2c009c00 comp rx=0 tx=0).stop 2024-03-30T04:06:49.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 -- 172.21.15.23:0/3242195481 shutdown_connections 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 --2- 172.21.15.23:0/3242195481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40104f60 0x7f2c4010cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 --2- 172.21.15.23:0/3242195481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c40100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 --2- 172.21.15.23:0/3242195481 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40100800 0x7f2c40104830 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 -- 172.21.15.23:0/3242195481 >> 172.21.15.23:0/3242195481 conn(0x7f2c40076d40 msgr2=0x7f2c40077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.286+0000 7f2c47000700 1 -- 172.21.15.23:0/3242195481 shutdown_connections 2024-03-30T04:06:49.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.287+0000 7f2c47000700 1 -- 172.21.15.23:0/3242195481 wait complete. 2024-03-30T04:06:49.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.287+0000 7f2c47000700 1 Processor -- start 2024-03-30T04:06:49.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 -- start start 2024-03-30T04:06:49.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40104f60 0x7f2c40101e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40102350 0x7f2c401a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401a5a00 con 0x7f2c40104f60 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401a5b70 con 0x7f2c400ffe50 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c47000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c401a5d30 con 0x7f2c40102350 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38170/0 (socket says 172.21.15.23:38170) 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c3fe00700 1 -- 172.21.15.23:0/2041399199 learned_addr learned my addr 172.21.15.23:0/2041399199 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c3f400700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40104f60 0x7f2c40101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.288+0000 7f2c45000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40102350 0x7f2c401a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 -- 172.21.15.23:0/2041399199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40102350 msgr2=0x7f2c401a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40102350 0x7f2c401a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 -- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40104f60 msgr2=0x7f2c40101e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40104f60 0x7f2c40101e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 -- 172.21.15.23:0/2041399199 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300095f0 con 0x7f2c400ffe50 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ec10 tx=0x7f2c3000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3cc00700 1 -- 172.21.15.23:0/2041399199 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000cc90 con 0x7f2c400ffe50 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3cc00700 1 -- 172.21.15.23:0/2041399199 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c300044e0 con 0x7f2c400ffe50 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c009520 con 0x7f2c400ffe50 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c3cc00700 1 -- 172.21.15.23:0/2041399199 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30010430 con 0x7f2c400ffe50 2024-03-30T04:06:49.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.289+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c401a62a0 con 0x7f2c400ffe50 2024-03-30T04:06:49.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.294+0000 7f2c3cc00700 1 -- 172.21.15.23:0/2041399199 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2c30003f90 con 0x7f2c400ffe50 2024-03-30T04:06:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.295+0000 7f2c5aabeb80 1 Processor -- start 2024-03-30T04:06:49.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.295+0000 7f2c5aabeb80 1 -- start start 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d22d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 0x55ec2d23b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d186990 con 0x55ec2d22d5d0 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d184350 con 0x55ec2d238df0 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d17fc40 con 0x55ec2d22def0 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3cc00700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 0x7f2c240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44216/0 (socket says 172.21.15.23:44216) 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3fe00700 1 -- 172.21.15.23:0/553548686 learned_addr learned my addr 172.21.15.23:0/553548686 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c45000700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d22d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3f400700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 0x55ec2d23b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c45000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 0x7f2c240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3fe00700 1 -- 172.21.15.23:0/553548686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 msgr2=0x55ec2d23b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.296+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 0x55ec2d23b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3fe00700 1 -- 172.21.15.23:0/553548686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 msgr2=0x55ec2d22d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d22d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3fe00700 1 -- 172.21.15.23:0/553548686 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec2d0013b0 con 0x55ec2d22def0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3f400700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 0x55ec2d23b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c45000700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d22d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3cc00700 1 -- 172.21.15.23:0/2041399199 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2c30014070 con 0x7f2c400ffe50 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2c300ca8d0 tx=0x7f2c3000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974ee17f1a339275 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c35600700 1 -- 172.21.15.23:0/553548686 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3009aa50 con 0x55ec2d22def0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c35600700 1 -- 172.21.15.23:0/553548686 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c30016330 con 0x55ec2d22def0 2024-03-30T04:06:49.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.297+0000 7f2c35600700 1 -- 172.21.15.23:0/553548686 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c30004240 con 0x55ec2d22def0 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/553548686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 msgr2=0x55ec2d2388b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2c300ca8d0 tx=0x7f2c3000ef90 comp rx=0 tx=0).stop 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c45000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 0x7f2c240a3750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009ec0 tx=0x7f2c3800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/553548686 shutdown_connections 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d238df0 0x55ec2d23b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d22d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/553548686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d22def0 0x55ec2d2388b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/553548686 >> 172.21.15.23:0/553548686 conn(0x55ec2d18d6a0 msgr2=0x55ec2d229a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/553548686 shutdown_connections 2024-03-30T04:06:49.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.298+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/553548686 wait complete. 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.299+0000 7f2c5aabeb80 1 -- start start 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.299+0000 7f2c5aabeb80 1 Processor -- start 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d235a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d236090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d11bd80 con 0x55ec2d22def0 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d11ba20 con 0x55ec2d22d5d0 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d0036a0 con 0x55ec2d238df0 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c5aabeb80 10 client.0 _ll_register_callbacks cb 0x55ec2d125470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38192/0 (socket says 172.21.15.23:38192) 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 -- 172.21.15.23:0/1459460075 learned_addr learned my addr 172.21.15.23:0/1459460075 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3f400700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d235a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c45000700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d236090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 -- 172.21.15.23:0/1459460075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 msgr2=0x55ec2d236090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d236090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 -- 172.21.15.23:0/1459460075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 msgr2=0x55ec2d235a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d235a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c3fe00700 1 -- 172.21.15.23:0/1459460075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c401a62a0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.300+0000 7f2c45000700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d236090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009b40 tx=0x7f2c30096700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d9aa958b3830dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c300a27d0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2c3009e180 con 0x55ec2d22d5d0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec2d140380 con 0x55ec2d22d5d0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c3007d080 con 0x55ec2d22d5d0 2024-03-30T04:06:49.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 10 client.26915 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ec2cfadb60 con 0x55ec2d22d5d0 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec2d140380 con 0x55ec2d22d5d0 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2c3009a5b0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 10 client.26915 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 10 client.26915 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.301+0000 7f2c5aabeb80 20 client.26915 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.302+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d248660 0x55ec2d24aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.302+0000 7f2c5aabeb80 4 client.26915 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.302+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec2d178700 con 0x55ec2d248660 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.302+0000 7f2c3f400700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d248660 0x55ec2d24aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.303+0000 7f2c3f400700 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d248660 0x55ec2d24aa50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=464c646169743764 server_cookie=fc0a0c50ff38d714 in_seq=0 out_seq=0 2024-03-30T04:06:49.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.303+0000 7f2c2a000700 10 client.26915 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:49.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.303+0000 7f2c2a000700 1 -- 172.21.15.23:0/1459460075 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec2d178700 con 0x55ec2d248660 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c2a000700 10 client.26915 handle_command_reply: tid=0 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 client.26915 shutdown 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 msgr2=0x55ec2d238770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2c30009b40 tx=0x7f2c30096700 comp rx=0 tx=0).stop 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 shutdown_connections 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d248660 0x55ec2d24aa50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d236090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d238770 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c2a000700 0 client.26915 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/1459460075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d235a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 >> 172.21.15.23:0/1459460075 conn(0x55ec2d18d6a0 msgr2=0x55ec2d22f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 shutdown_connections 2024-03-30T04:06:49.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/1459460075 wait complete. 2024-03-30T04:06:49.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.304+0000 7f2c5aabeb80 20 client.26915 trim_cache size 0 max 16384 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.319+0000 7f2c5aabeb80 1 Processor -- start 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.319+0000 7f2c5aabeb80 1 -- start start 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.319+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.319+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d234fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d230d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d240050 con 0x55ec2d22def0 2024-03-30T04:06:49.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d235de0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d236970 con 0x55ec2d238df0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c3f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d234fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d230d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38208/0 (socket says 172.21.15.23:38208) 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c3fe00700 1 -- 172.21.15.23:0/3629143677 learned_addr learned my addr 172.21.15.23:0/3629143677 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 -- 172.21.15.23:0/3629143677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 msgr2=0x55ec2d23faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 -- 172.21.15.23:0/3629143677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 msgr2=0x55ec2d234fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d234fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c45000700 1 -- 172.21.15.23:0/3629143677 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec2d145ea0 con 0x55ec2d238df0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.320+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c45000700 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d230d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c38018650 tx=0x7f2c38011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85377178e119ca85 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c36000700 1 -- 172.21.15.23:0/3629143677 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3800e070 con 0x55ec2d238df0 2024-03-30T04:06:49.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c36000700 1 -- 172.21.15.23:0/3629143677 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c3800f040 con 0x55ec2d238df0 2024-03-30T04:06:49.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c36000700 1 -- 172.21.15.23:0/3629143677 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c380255e0 con 0x55ec2d238df0 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/3629143677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 msgr2=0x55ec2d230d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.321+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d230d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2c38018650 tx=0x7f2c38011000 comp rx=0 tx=0).stop 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/3629143677 shutdown_connections 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d230d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/3629143677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22def0 0x55ec2d234fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/3629143677 >> 172.21.15.23:0/3629143677 conn(0x55ec2d18d6a0 msgr2=0x55ec2d2225a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/3629143677 shutdown_connections 2024-03-30T04:06:49.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/3629143677 wait complete. 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 -- start start 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.322+0000 7f2c5aabeb80 1 Processor -- start 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 0x55ec2d23db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d23e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d23e5c0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d23e730 con 0x55ec2d22def0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2d23e8f0 con 0x55ec2d238df0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c5aabeb80 10 client.0 _ll_register_callbacks cb 0x55ec2d125470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32880/0 (socket says 172.21.15.23:32880) 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 0x55ec2d23db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 -- 172.21.15.23:0/510588092 learned_addr learned my addr 172.21.15.23:0/510588092 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c45000700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d23e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 -- 172.21.15.23:0/510588092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 msgr2=0x55ec2d23e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d23e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 -- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 msgr2=0x55ec2d23db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 0x55ec2d23db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3fe00700 1 -- 172.21.15.23:0/510588092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec2cfadb60 con 0x55ec2d22d5d0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.323+0000 7f2c3f400700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 0x55ec2d23db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c3fe00700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2c30005cb0 tx=0x7f2c3000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9814be029395da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c2a000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3009aa50 con 0x55ec2d22d5d0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f2c3009e180 con 0x55ec2d22d5d0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c30016330 con 0x55ec2d22d5d0 2024-03-30T04:06:49.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c300a27d0 con 0x55ec2d22d5d0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c5aabeb80 10 client.16935 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.324+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f2c38020040 con 0x55ec2d22d5d0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c30016330 con 0x55ec2d22d5d0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2c30083b50 con 0x55ec2d22d5d0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 10 client.16935 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 10 client.16935 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 20 client.16935 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d29d480 0x55ec2d29f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 4 client.16935 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec2cfcecf0 con 0x55ec2d29d480 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.325+0000 7f2c3f400700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d29d480 0x55ec2d29f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.326+0000 7f2c3f400700 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d29d480 0x55ec2d29f870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=539fecf6827dff21 server_cookie=a51547461ba222c2 in_seq=0 out_seq=0 2024-03-30T04:06:49.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.326+0000 7f2c2a000700 10 client.16935 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:49.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c2a000700 1 -- 172.21.15.23:0/510588092 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec2cfcecf0 con 0x55ec2d29d480 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c2a000700 10 client.16935 handle_command_reply: tid=0 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 client.16935 shutdown 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 msgr2=0x55ec2d23f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2c30005cb0 tx=0x7f2c3000ef90 comp rx=0 tx=0).stop 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 shutdown_connections 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec2d29d480 0x55ec2d29f870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec2d238df0 0x55ec2d23e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec2d22d5d0 0x55ec2d23f770 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 --2- 172.21.15.23:0/510588092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec2d22def0 0x55ec2d23db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c2a000700 0 client.16935 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 >> 172.21.15.23:0/510588092 conn(0x55ec2d18d6a0 msgr2=0x55ec2d234fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 shutdown_connections 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 1 -- 172.21.15.23:0/510588092 wait complete. 2024-03-30T04:06:49.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.327+0000 7f2c5aabeb80 20 client.16935 trim_cache size 0 max 16384 2024-03-30T04:06:49.330 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:49.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 msgr2=0x7f2c240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 0x7f2c240a3750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009ec0 tx=0x7f2c3800b040 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 msgr2=0x7f2c401018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ec10 tx=0x7f2c3000ef20 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 shutdown_connections 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c40102350 0x7f2c401a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2c240a1290 0x7f2c240a3750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c400ffe50 0x7f2c401018d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 --2- 172.21.15.23:0/2041399199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2c40104f60 0x7f2c40101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 >> 172.21.15.23:0/2041399199 conn(0x7f2c40076d40 msgr2=0x7f2c40068290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 shutdown_connections 2024-03-30T04:06:49.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.328+0000 7f2c47000700 1 -- 172.21.15.23:0/2041399199 wait complete. 2024-03-30T04:06:49.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:49 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3082164526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:49.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:06:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:49 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3082164526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 -- 172.21.15.23:0/586728694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 msgr2=0x7f8b70101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 --2- 172.21.15.23:0/586728694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b70101350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b60009990 tx=0x7f8b60009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 -- 172.21.15.23:0/586728694 shutdown_connections 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 --2- 172.21.15.23:0/586728694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b70106090 0x7f8b7010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 --2- 172.21.15.23:0/586728694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b70101350 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 --2- 172.21.15.23:0/586728694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70101920 0x7f8b70105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 -- 172.21.15.23:0/586728694 >> 172.21.15.23:0/586728694 conn(0x7f8b700fc7f0 msgr2=0x7f8b700fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 -- 172.21.15.23:0/586728694 shutdown_connections 2024-03-30T04:06:49.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.556+0000 7f8b76000700 1 -- 172.21.15.23:0/586728694 wait complete. 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 Processor -- start 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 -- start start 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 0x7f8b70199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b7019dcd0 0x7f8b7019e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7019e750 con 0x7f8b70106090 2024-03-30T04:06:49.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7019e8c0 con 0x7f8b70100f70 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.557+0000 7f8b76000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7019ea80 con 0x7f8b7019dcd0 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 0x7f8b70199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b7019dcd0 0x7f8b7019e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38244/0 (socket says 172.21.15.23:38244) 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 -- 172.21.15.23:0/676216494 learned_addr learned my addr 172.21.15.23:0/676216494 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 -- 172.21.15.23:0/676216494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b7019dcd0 msgr2=0x7f8b7019e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b7019dcd0 0x7f8b7019e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 -- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 msgr2=0x7f8b70199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 0x7f8b70199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6f400700 1 -- 172.21.15.23:0/676216494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5c009520 con 0x7f8b70100f70 2024-03-30T04:06:49.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.558+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 0x7f8b70199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b6f400700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8b60004760 tx=0x7f8b60004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b67e00700 1 -- 172.21.15.23:0/676216494 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6001d070 con 0x7f8b70100f70 2024-03-30T04:06:49.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b67e00700 1 -- 172.21.15.23:0/676216494 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b60022470 con 0x7f8b70100f70 2024-03-30T04:06:49.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b600095f0 con 0x7f8b70100f70 2024-03-30T04:06:49.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b67e00700 1 -- 172.21.15.23:0/676216494 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b60017650 con 0x7f8b70100f70 2024-03-30T04:06:49.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.559+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b70107e50 con 0x7f8b70100f70 2024-03-30T04:06:49.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.562+0000 7f8b67e00700 1 -- 172.21.15.23:0/676216494 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8b600225e0 con 0x7f8b70100f70 2024-03-30T04:06:49.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.563+0000 7f8b67e00700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 0x7f8b540a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.563+0000 7f8b67e00700 1 -- 172.21.15.23:0/676216494 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8b600ce140 con 0x7f8b70100f70 2024-03-30T04:06:49.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.564+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 0x7f8b540a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.564+0000 7f8b89ab4b80 1 Processor -- start 2024-03-30T04:06:49.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.564+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 0x7f8b540a37d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8b7019a940 tx=0x7f8b5c016040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.564+0000 7f8b89ab4b80 1 -- start start 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e0a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e0d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61d58990 con 0x563a61dffef0 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61d56350 con 0x563a61e0adf0 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61d51c40 con 0x563a61dff5d0 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e0a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44264/0 (socket says 172.21.15.23:44264) 2024-03-30T04:06:49.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 -- 172.21.15.23:0/1676376733 learned_addr learned my addr 172.21.15.23:0/1676376733 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e0d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 -- 172.21.15.23:0/1676376733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 msgr2=0x563a61e0d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e0d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 -- 172.21.15.23:0/1676376733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 msgr2=0x563a61e0a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e0a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.565+0000 7f8b6f400700 1 -- 172.21.15.23:0/1676376733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a61bd33b0 con 0x563a61dff5d0 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.566+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000c010 tx=0x7f8b6009ad70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aad8f8cb6a0ac9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.566+0000 7f8b5be00700 1 -- 172.21.15.23:0/1676376733 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b600a1070 con 0x563a61dff5d0 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.566+0000 7f8b5be00700 1 -- 172.21.15.23:0/1676376733 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b6009b620 con 0x563a61dff5d0 2024-03-30T04:06:49.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.566+0000 7f8b5be00700 1 -- 172.21.15.23:0/1676376733 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6009b3e0 con 0x563a61dff5d0 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1676376733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 msgr2=0x563a61dff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8b6000c010 tx=0x7f8b6009ad70 comp rx=0 tx=0).stop 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1676376733 shutdown_connections 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e0d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61dff9b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1676376733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e0a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1676376733 >> 172.21.15.23:0/1676376733 conn(0x563a61d5f6a0 msgr2=0x563a61dfba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1676376733 shutdown_connections 2024-03-30T04:06:49.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.567+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1676376733 wait complete. 2024-03-30T04:06:49.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.568+0000 7f8b89ab4b80 1 -- start start 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.568+0000 7f8b89ab4b80 1 Processor -- start 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e0b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61e0adf0 0x563a61e16250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61cedd80 con 0x563a61e0adf0 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61ceda20 con 0x563a61dffef0 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61bd56a0 con 0x563a61dff5d0 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b89ab4b80 10 client.0 _ll_register_callbacks cb 0x563a61cf7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38266/0 (socket says 172.21.15.23:38266) 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b6ea00700 1 -- 172.21.15.23:0/2034765565 learned_addr learned my addr 172.21.15.23:0/2034765565 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61e0adf0 0x563a61e16250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.569+0000 7f8b6f400700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e0b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 -- 172.21.15.23:0/2034765565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 msgr2=0x563a61e0b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e0b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 -- 172.21.15.23:0/2034765565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61e0adf0 msgr2=0x563a61e16250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61e0adf0 0x563a61e16250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 -- 172.21.15.23:0/2034765565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b70107e50 con 0x563a61dffef0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c020770 tx=0x7f8b5c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd41907fc1e81452 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b58c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c025040 con 0x563a61dffef0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8b5c004020 con 0x563a61dffef0 2024-03-30T04:06:49.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.570+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563a61d12380 con 0x563a61dffef0 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b5c0041e0 con 0x563a61dffef0 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 10 client.26933 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563a61b7fb60 con 0x563a61dffef0 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563a61d12380 con 0x563a61dffef0 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8b5c0051d0 con 0x563a61dffef0 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 10 client.26933 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 10 client.26933 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 20 client.26933 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e1a720 0x563a61e1cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 4 client.26933 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a61d4a700 con 0x563a61e1a720 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.571+0000 7f8b6f400700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e1a720 0x563a61e1cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.572+0000 7f8b6f400700 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e1a720 0x563a61e1cb40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e9f260fca7b91b0 server_cookie=a3300558bea2554 in_seq=0 out_seq=0 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.572+0000 7f8b58c00700 10 client.26933 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.572+0000 7f8b58c00700 1 -- 172.21.15.23:0/2034765565 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563a61d4a700 con 0x563a61e1a720 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.572+0000 7f8b58c00700 10 client.26933 handle_command_reply: tid=0 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.572+0000 7f8b89ab4b80 1 client.26933 shutdown 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 msgr2=0x563a61e0b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c020770 tx=0x7f8b5c004750 comp rx=0 tx=0).stop 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 shutdown_connections 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61e0adf0 0x563a61e16250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e1a720 0x563a61e1cb40 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e0b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/2034765565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dffef0 0x563a61e0b710 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 >> 172.21.15.23:0/2034765565 conn(0x563a61d5f6a0 msgr2=0x563a61e016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b58c00700 0 client.26933 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:49.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 shutdown_connections 2024-03-30T04:06:49.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/2034765565 wait complete. 2024-03-30T04:06:49.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.573+0000 7f8b89ab4b80 20 client.26933 trim_cache size 0 max 16384 2024-03-30T04:06:49.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.587+0000 7f8b89ab4b80 1 Processor -- start 2024-03-30T04:06:49.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.587+0000 7f8b89ab4b80 1 -- start start 2024-03-30T04:06:49.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e05360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e058a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61e16bc0 con 0x563a61dffef0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61df7480 con 0x563a61e0adf0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61e01130 con 0x563a61dff5d0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e058a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44280/0 (socket says 172.21.15.23:44280) 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6f400700 1 -- 172.21.15.23:0/3983481422 learned_addr learned my addr 172.21.15.23:0/3983481422 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e05360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 -- 172.21.15.23:0/3983481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 msgr2=0x563a61e10210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 -- 172.21.15.23:0/3983481422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 msgr2=0x563a61e05360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e05360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6fe00700 1 -- 172.21.15.23:0/3983481422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a61d17ea0 con 0x563a61e0adf0 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6f400700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.588+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e05360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.589+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e058a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b68011930 tx=0x7f8b68009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50ecdd011990bee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.589+0000 7f8b65200700 1 -- 172.21.15.23:0/3983481422 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800e040 con 0x563a61e0adf0 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.589+0000 7f8b65200700 1 -- 172.21.15.23:0/3983481422 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b6800f040 con 0x563a61e0adf0 2024-03-30T04:06:49.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.589+0000 7f8b65200700 1 -- 172.21.15.23:0/3983481422 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b68004730 con 0x563a61e0adf0 2024-03-30T04:06:49.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/3983481422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 msgr2=0x563a61e058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e058a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8b68011930 tx=0x7f8b68009920 comp rx=0 tx=0).stop 2024-03-30T04:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/3983481422 shutdown_connections 2024-03-30T04:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61e0adf0 0x563a61e058a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61dff5d0 0x563a61e10210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/3983481422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e05360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/3983481422 >> 172.21.15.23:0/3983481422 conn(0x563a61d5f6a0 msgr2=0x563a61e04860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/3983481422 shutdown_connections 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/3983481422 wait complete. 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.590+0000 7f8b89ab4b80 1 -- start start 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 Processor -- start 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 0x563a61e67a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 0x563a61e624e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61e62bf0 con 0x563a61dffef0 2024-03-30T04:06:49.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61e62d60 con 0x563a61dff5d0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a61e62f20 con 0x563a61e0adf0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b89ab4b80 10 client.0 _ll_register_callbacks cb 0x563a61cf7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b6f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 0x563a61e67a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.591+0000 7f8b6ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32932/0 (socket says 172.21.15.23:32932) 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6ea00700 1 -- 172.21.15.23:0/1052548073 learned_addr learned my addr 172.21.15.23:0/1052548073 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 0x563a61e624e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 -- 172.21.15.23:0/1052548073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 msgr2=0x563a61e624e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 0x563a61e624e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 -- 172.21.15.23:0/1052548073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 msgr2=0x563a61e67fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 -- 172.21.15.23:0/1052548073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a61bd33b0 con 0x563a61dff5d0 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6fe00700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 0x563a61e624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b6f400700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 0x563a61e67a90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8b60022890 tx=0x7f8b60009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd1a98cdc5e56b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b58c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.592+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b600a3310 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8b6009ed50 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b6009b620 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b6009a410 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 10 client.26939 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f8b680115b0 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b6009b620 con 0x563a61dff5d0 2024-03-30T04:06:49.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8b6009b180 con 0x563a61dff5d0 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 10 client.26939 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 10 client.26939 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 20 client.26939 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e6f4d0 0x563a61e718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 4 client.26939 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.593+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563a61ba0cf0 con 0x563a61e6f4d0 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e6f4d0 0x563a61e718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b6ea00700 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e6f4d0 0x563a61e718c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6cd1c947cea5beae server_cookie=6ea3f3d3612b6f5f in_seq=0 out_seq=0 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b58c00700 10 client.26939 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b58c00700 1 -- 172.21.15.23:0/1052548073 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563a61ba0cf0 con 0x563a61e6f4d0 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b58c00700 10 client.26939 handle_command_reply: tid=0 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.594+0000 7f8b89ab4b80 1 client.26939 shutdown 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 msgr2=0x563a61e67a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 0x563a61e67a90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8b60022890 tx=0x7f8b60009310 comp rx=0 tx=0).stop 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 shutdown_connections 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x563a61e6f4d0 0x563a61e718c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563a61e0adf0 0x563a61e624e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563a61dff5d0 0x563a61e67a90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b58c00700 0 client.26939 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 --2- 172.21.15.23:0/1052548073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563a61dffef0 0x563a61e67fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 >> 172.21.15.23:0/1052548073 conn(0x563a61d5f6a0 msgr2=0x563a61e05360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 shutdown_connections 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 1 -- 172.21.15.23:0/1052548073 wait complete. 2024-03-30T04:06:49.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.595+0000 7f8b89ab4b80 20 client.26939 trim_cache size 0 max 16384 2024-03-30T04:06:49.603 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:49.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 msgr2=0x7f8b540a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 0x7f8b540a37d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8b7019a940 tx=0x7f8b5c016040 comp rx=0 tx=0).stop 2024-03-30T04:06:49.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 msgr2=0x7f8b701993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8b60004760 tx=0x7f8b60004790 comp rx=0 tx=0).stop 2024-03-30T04:06:49.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 shutdown_connections 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b7019dcd0 0x7f8b7019e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8b540a1310 0x7f8b540a37d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b70100f70 0x7f8b701993f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 --2- 172.21.15.23:0/676216494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b70106090 0x7f8b70199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 >> 172.21.15.23:0/676216494 conn(0x7f8b700fc7f0 msgr2=0x7f8b700fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 shutdown_connections 2024-03-30T04:06:49.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.596+0000 7f8b76000700 1 -- 172.21.15.23:0/676216494 wait complete. 2024-03-30T04:06:49.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:06:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:49 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3082164526' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:06:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 -- 172.21.15.23:0/3030480800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548101920 msgr2=0x7f9548105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 --2- 172.21.15.23:0/3030480800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548101920 0x7f9548105960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9534009990 tx=0x7f9534009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 -- 172.21.15.23:0/3030480800 shutdown_connections 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 --2- 172.21.15.23:0/3030480800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548106090 0x7f954810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 --2- 172.21.15.23:0/3030480800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548100f70 0x7f9548101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 --2- 172.21.15.23:0/3030480800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548101920 0x7f9548105960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 -- 172.21.15.23:0/3030480800 >> 172.21.15.23:0/3030480800 conn(0x7f95480fc7f0 msgr2=0x7f95480fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 -- 172.21.15.23:0/3030480800 shutdown_connections 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.899+0000 7f954e400700 1 -- 172.21.15.23:0/3030480800 wait complete. 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 Processor -- start 2024-03-30T04:06:49.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 -- start start 2024-03-30T04:06:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548100f70 0x7f95480730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 0x7f95481a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95481a5900 con 0x7f9548073b30 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95481a5a70 con 0x7f9548106090 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.900+0000 7f954e400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95481a5c30 con 0x7f9548100f70 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38290/0 (socket says 172.21.15.23:38290) 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 -- 172.21.15.23:0/3675176771 learned_addr learned my addr 172.21.15.23:0/3675176771 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9547400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548100f70 0x7f95480730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9547e00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 0x7f95481a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 -- 172.21.15.23:0/3675176771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548100f70 msgr2=0x7f95480730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548100f70 0x7f95480730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 -- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 msgr2=0x7f95481a5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 0x7f95481a5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 -- 172.21.15.23:0/3675176771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9538009520 con 0x7f9548106090 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9547e00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 0x7f95481a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.901+0000 7f9546a00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9534005770 tx=0x7f95340057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.902+0000 7f953fe00700 1 -- 172.21.15.23:0/3675176771 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953401d070 con 0x7f9548106090 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.902+0000 7f953fe00700 1 -- 172.21.15.23:0/3675176771 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9534021410 con 0x7f9548106090 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.902+0000 7f953fe00700 1 -- 172.21.15.23:0/3675176771 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9534017650 con 0x7f9548106090 2024-03-30T04:06:49.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.902+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95340095f0 con 0x7f9548106090 2024-03-30T04:06:49.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.902+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95481a61d0 con 0x7f9548106090 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.905+0000 7f953fe00700 1 -- 172.21.15.23:0/3675176771 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f953400fb00 con 0x7f9548106090 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.906+0000 7f9561ec0b80 1 Processor -- start 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 -- start start 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e6a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 0x564532e781e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532dc3990 con 0x564532e75df0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532dc1350 con 0x564532e6aef0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532dbcc40 con 0x564532e6a5d0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f953fe00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 0x7f952c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.907+0000 7f9547400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38296/0 (socket says 172.21.15.23:38296) 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 -- 172.21.15.23:0/3787984658 learned_addr learned my addr 172.21.15.23:0/3787984658 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9546a00700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 0x564532e781e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547e00700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e6a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f953fe00700 1 -- 172.21.15.23:0/3675176771 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f95340ce120 con 0x7f9548106090 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547e00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 0x7f952c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 -- 172.21.15.23:0/3787984658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 msgr2=0x564532e6a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e6a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 -- 172.21.15.23:0/3787984658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 msgr2=0x564532e781e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 0x564532e781e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 -- 172.21.15.23:0/3787984658 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564532c3e3b0 con 0x564532e6aef0 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9546a00700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 0x564532e781e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.908+0000 7f9547400700 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9538009f10 tx=0x7f9538011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6bb2321b515a885 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9547e00700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 0x7f952c0a37d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9540009fd0 tx=0x7f954000b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f953cc00700 1 -- 172.21.15.23:0/3787984658 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9538019630 con 0x564532e6aef0 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f953cc00700 1 -- 172.21.15.23:0/3787984658 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9538019c50 con 0x564532e6aef0 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f953cc00700 1 -- 172.21.15.23:0/3787984658 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953800ece0 con 0x564532e6aef0 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3787984658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 msgr2=0x564532e758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9538009f10 tx=0x7f9538011720 comp rx=0 tx=0).stop 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3787984658 shutdown_connections 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e75df0 0x564532e781e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e6a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3787984658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e758b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3787984658 >> 172.21.15.23:0/3787984658 conn(0x564532dca6a0 msgr2=0x564532e66a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3787984658 shutdown_connections 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.909+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3787984658 wait complete. 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.910+0000 7f9561ec0b80 1 -- start start 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.910+0000 7f9561ec0b80 1 Processor -- start 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e75800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6aef0 0x564532e72af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532d58d80 con 0x564532e6a5d0 2024-03-30T04:06:49.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532d58a20 con 0x564532e75df0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532c406a0 con 0x564532e6aef0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9561ec0b80 10 client.0 _ll_register_callbacks cb 0x564532d62470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e75800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38312/0 (socket says 172.21.15.23:38312) 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547e00700 1 -- 172.21.15.23:0/3626445972 learned_addr learned my addr 172.21.15.23:0/3626445972 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9546a00700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6aef0 0x564532e72af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 -- 172.21.15.23:0/3626445972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6aef0 msgr2=0x564532e72af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6aef0 0x564532e72af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 -- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 msgr2=0x564532e73120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547400700 1 -- 172.21.15.23:0/3626445972 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9538009b70 con 0x564532e6a5d0 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.911+0000 7f9547e00700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9547400700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e75800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9538000c00 tx=0x7f953800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528cd334ecca8798 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9531600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9538019630 con 0x564532e6a5d0 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f9538019790 con 0x564532e6a5d0 2024-03-30T04:06:49.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564532d7d380 con 0x564532e6a5d0 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f953800e680 con 0x564532e6a5d0 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9561ec0b80 10 client.16965 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.912+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564532beab60 con 0x564532e6a5d0 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564532d7d380 con 0x564532e6a5d0 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9538028020 con 0x564532e6a5d0 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 10 client.16965 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 20 client.16965 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532e85750 0x564532e87b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 4 client.16965 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564532db5700 con 0x564532e85750 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.913+0000 7f9546a00700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532e85750 0x564532e87b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.914+0000 7f9546a00700 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532e85750 0x564532e87b70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d03f7b0eae798c4 server_cookie=c34d47c48013a51f in_seq=0 out_seq=0 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.914+0000 7f9531600700 10 client.16965 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.914+0000 7f9531600700 1 -- 172.21.15.23:0/3626445972 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564532db5700 con 0x564532e85750 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.914+0000 7f9531600700 10 client.16965 handle_command_reply: tid=0 2024-03-30T04:06:49.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.914+0000 7f9561ec0b80 1 client.16965 shutdown 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 msgr2=0x564532e75800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e75800 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9538000c00 tx=0x7f953800edb0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 shutdown_connections 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e73120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e75800 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532e85750 0x564532e87b70 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/3626445972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6aef0 0x564532e72af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 >> 172.21.15.23:0/3626445972 conn(0x564532dca6a0 msgr2=0x564532e6c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9531600700 0 client.16965 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:49.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 shutdown_connections 2024-03-30T04:06:49.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 1 -- 172.21.15.23:0/3626445972 wait complete. 2024-03-30T04:06:49.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.915+0000 7f9561ec0b80 20 client.16965 trim_cache size 0 max 16384 2024-03-30T04:06:49.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 Processor -- start 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 -- start start 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532e6b640 con 0x564532e6aef0 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532e7cfe0 con 0x564532e75df0 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.929+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532e6a3c0 con 0x564532e6a5d0 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9547e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32972/0 (socket says 172.21.15.23:32972) 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 -- 172.21.15.23:0/2160916782 learned_addr learned my addr 172.21.15.23:0/2160916782 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9547e00700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38320/0 (socket says 172.21.15.23:38320) 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9547400700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 -- 172.21.15.23:0/2160916782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 msgr2=0x564532e730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 -- 172.21.15.23:0/2160916782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 msgr2=0x564532e713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 -- 172.21.15.23:0/2160916782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564532d82ea0 con 0x564532e6aef0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9547400700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9547e00700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.930+0000 7f9546a00700 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9534097400 tx=0x7f9534027f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa580c1a4bb3d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f953d600700 1 -- 172.21.15.23:0/2160916782 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953409acd0 con 0x564532e6aef0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f953d600700 1 -- 172.21.15.23:0/2160916782 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f953409aa40 con 0x564532e6aef0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f953d600700 1 -- 172.21.15.23:0/2160916782 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340a32f0 con 0x564532e6aef0 2024-03-30T04:06:49.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 -- 172.21.15.23:0/2160916782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 msgr2=0x564532e7caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9534097400 tx=0x7f9534027f90 comp rx=0 tx=0).stop 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 -- 172.21.15.23:0/2160916782 shutdown_connections 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e75df0 0x564532e713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e6a5d0 0x564532e730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/2160916782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6aef0 0x564532e7caa0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 -- 172.21.15.23:0/2160916782 >> 172.21.15.23:0/2160916782 conn(0x564532dca6a0 msgr2=0x564532e5ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 -- 172.21.15.23:0/2160916782 shutdown_connections 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.931+0000 7f9561ec0b80 1 -- 172.21.15.23:0/2160916782 wait complete. 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 -- start start 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 Processor -- start 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e79f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e75df0 0x564532e7aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532e7ce40 con 0x564532e6a5d0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532ed8540 con 0x564532e6aef0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.932+0000 7f9561ec0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564532ed8680 con 0x564532e75df0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9561ec0b80 10 client.0 _ll_register_callbacks cb 0x564532d62470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38334/0 (socket says 172.21.15.23:38334) 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 -- 172.21.15.23:0/1693406250 learned_addr learned my addr 172.21.15.23:0/1693406250 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9547400700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e79f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9547e00700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e75df0 0x564532e7aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 -- 172.21.15.23:0/1693406250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e75df0 msgr2=0x564532e7aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e75df0 0x564532e7aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 -- 172.21.15.23:0/1693406250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 msgr2=0x564532e79f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e79f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 -- 172.21.15.23:0/1693406250 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564532beab60 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9546a00700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f95340973b0 tx=0x7f9534005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e65bfe7b29d43efb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9531600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95340b7070 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.933+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9538019c50 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f953409b600 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95340a32f0 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 10 client.26957 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f9540007500 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9538019c50 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f953409b160 con 0x564532e6aef0 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 10 client.26957 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 10 client.26957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 20 client.26957 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532ed99a0 0x564532edbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 4 client.26957 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.934+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564532c0bcf0 con 0x564532ed99a0 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9547400700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532ed99a0 0x564532edbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9547400700 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532ed99a0 0x564532edbd90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=333f12dd96e38464 server_cookie=6d59f1da22c42f6d in_seq=0 out_seq=0 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9531600700 10 client.26957 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9531600700 1 -- 172.21.15.23:0/1693406250 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564532c0bcf0 con 0x564532ed99a0 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9531600700 10 client.26957 handle_command_reply: tid=0 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 client.26957 shutdown 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 msgr2=0x564532e7a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f95340973b0 tx=0x7f9534005fd0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 shutdown_connections 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564532ed99a0 0x564532edbd90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564532e75df0 0x564532e7aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9531600700 0 client.26957 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564532e6a5d0 0x564532e79f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 --2- 172.21.15.23:0/1693406250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564532e6aef0 0x564532e7a4c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.935+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 >> 172.21.15.23:0/1693406250 conn(0x564532dca6a0 msgr2=0x564532e71300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 shutdown_connections 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f9561ec0b80 1 -- 172.21.15.23:0/1693406250 wait complete. 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f9561ec0b80 20 client.26957 trim_cache size 0 max 16384 2024-03-30T04:06:49.942 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 msgr2=0x7f952c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 0x7f952c0a37d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9540009fd0 tx=0x7f954000b040 comp rx=0 tx=0).stop 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 msgr2=0x7f95480735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9534005770 tx=0x7f95340057a0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 shutdown_connections 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9548073b30 0x7f95481a5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9548100f70 0x7f95480730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f952c0a1310 0x7f952c0a37d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 --2- 172.21.15.23:0/3675176771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9548106090 0x7f95480735f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:49.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.936+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 >> 172.21.15.23:0/3675176771 conn(0x7f95480fc7f0 msgr2=0x7f95480fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:49.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.937+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 shutdown_connections 2024-03-30T04:06:49.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:49.937+0000 7f954e400700 1 -- 172.21.15.23:0/3675176771 wait complete. 2024-03-30T04:06:50.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 444397 11.58 MB/sec execute 87 sec latency 20.714 ms 2024-03-30T04:06:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:50 smithi193 ceph-mon[79581]: pgmap v150: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-30T04:06:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:50 smithi023 ceph-mon[38254]: pgmap v150: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-30T04:06:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:50 smithi154 ceph-mon[79161]: pgmap v150: 129 pgs: 129 active+clean; 247 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-03-30T04:06:50.955 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:06:50.956 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 -- 172.21.15.23:0/3716280602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 msgr2=0x7f0c24069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 --2- 172.21.15.23:0/3716280602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24069640 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009990 tx=0x7f0c14009ca0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 -- 172.21.15.23:0/3716280602 shutdown_connections 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 --2- 172.21.15.23:0/3716280602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 0x7f0c2410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 --2- 172.21.15.23:0/3716280602 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24069640 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 --2- 172.21.15.23:0/3716280602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c24069c10 0x7f0c24105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 -- 172.21.15.23:0/3716280602 >> 172.21.15.23:0/3716280602 conn(0x7f0c24076d80 msgr2=0x7f0c24077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 -- 172.21.15.23:0/3716280602 shutdown_connections 2024-03-30T04:06:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.171+0000 7f0c29600700 1 -- 172.21.15.23:0/3716280602 wait complete. 2024-03-30T04:06:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.173+0000 7f0c29600700 1 Processor -- start 2024-03-30T04:06:51.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.173+0000 7f0c29600700 1 -- start start 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 0x7f0c24197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c2419bae0 0x7f0c2419bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419c560 con 0x7f0c241060b0 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419c6d0 con 0x7f0c24069260 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c29600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2419c890 con 0x7f0c2419bae0 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38360/0 (socket says 172.21.15.23:38360) 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 -- 172.21.15.23:0/2903665881 learned_addr learned my addr 172.21.15.23:0/2903665881 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22000700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 0x7f0c24197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c23400700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c2419bae0 0x7f0c2419bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 -- 172.21.15.23:0/2903665881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c2419bae0 msgr2=0x7f0c2419bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c2419bae0 0x7f0c2419bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 -- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 msgr2=0x7f0c24197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 0x7f0c24197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.174+0000 7f0c22a00700 1 -- 172.21.15.23:0/2903665881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140095f0 con 0x7f0c24069260 2024-03-30T04:06:51.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0c14000c00 tx=0x7f0c14003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c1b400700 1 -- 172.21.15.23:0/2903665881 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1401d070 con 0x7f0c24069260 2024-03-30T04:06:51.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c241a3d90 con 0x7f0c24069260 2024-03-30T04:06:51.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c1b400700 1 -- 172.21.15.23:0/2903665881 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c14005580 con 0x7f0c24069260 2024-03-30T04:06:51.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c1b400700 1 -- 172.21.15.23:0/2903665881 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1400fc50 con 0x7f0c24069260 2024-03-30T04:06:51.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.175+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c241a41b0 con 0x7f0c24069260 2024-03-30T04:06:51.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.178+0000 7f0c1b400700 1 -- 172.21.15.23:0/2903665881 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0c14005050 con 0x7f0c24069260 2024-03-30T04:06:51.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c1b400700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 0x7f0c080a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 Processor -- start 2024-03-30T04:06:51.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c1b400700 1 -- 172.21.15.23:0/2903665881 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0c140ce030 con 0x7f0c24069260 2024-03-30T04:06:51.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c22000700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 0x7f0c080a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 -- start start 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc6782a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc678d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.181+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc66dba10 con 0x562dc6782650 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc66d93d0 con 0x562dc678de70 2024-03-30T04:06:51.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc66d4cc0 con 0x562dc6782f70 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc6782a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38372/0 (socket says 172.21.15.23:38372) 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c22000700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 0x7f0c080a37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c24198750 tx=0x7f0c100073d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 -- 172.21.15.23:0/3772172428 learned_addr learned my addr 172.21.15.23:0/3772172428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c22000700 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc678d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 -- 172.21.15.23:0/3772172428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 msgr2=0x562dc678d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc678d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 -- 172.21.15.23:0/3772172428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 msgr2=0x562dc6782a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc6782a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 -- 172.21.15.23:0/3772172428 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562dc6556430 con 0x562dc678de70 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c22a00700 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc6782a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.182+0000 7f0c23400700 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c003f30 tx=0x7f0c1c009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fca96809b944906 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.183+0000 7f0c0f400700 1 -- 172.21.15.23:0/3772172428 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c003a00 con 0x562dc678de70 2024-03-30T04:06:51.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.183+0000 7f0c0f400700 1 -- 172.21.15.23:0/3772172428 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c1c00f040 con 0x562dc678de70 2024-03-30T04:06:51.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.183+0000 7f0c0f400700 1 -- 172.21.15.23:0/3772172428 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0156e0 con 0x562dc678de70 2024-03-30T04:06:51.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.183+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3772172428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 msgr2=0x562dc6790260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.183+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c003f30 tx=0x7f0c1c009c20 comp rx=0 tx=0).stop 2024-03-30T04:06:51.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3772172428 shutdown_connections 2024-03-30T04:06:51.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc678de70 0x562dc6790260 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc6782a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3772172428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc678d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3772172428 >> 172.21.15.23:0/3772172428 conn(0x562dc66e2720 msgr2=0x562dc677eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3772172428 shutdown_connections 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3772172428 wait complete. 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 -- start start 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.184+0000 7f0c3d039b80 1 Processor -- start 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782f70 0x562dc678abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc678b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc6670e00 con 0x562dc6782f70 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc6670aa0 con 0x562dc6782650 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc6558720 con 0x562dc678de70 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c3d039b80 10 client.0 _ll_register_callbacks cb 0x562dc667a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:51.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38386/0 (socket says 172.21.15.23:38386) 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.185+0000 7f0c22a00700 1 -- 172.21.15.23:0/3794382239 learned_addr learned my addr 172.21.15.23:0/3794382239 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22000700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782f70 0x562dc678abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c23400700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc678b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 -- 172.21.15.23:0/3794382239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 msgr2=0x562dc678b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc678b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 -- 172.21.15.23:0/3794382239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782f70 msgr2=0x562dc678abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782f70 0x562dc678abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 -- 172.21.15.23:0/3794382239 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c241a41b0 con 0x562dc6782650 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c23400700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc678b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c22a00700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c1409ab10 tx=0x7f0c1409a770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d92c450d6e254da8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c03e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1409b510 con 0x562dc6782650 2024-03-30T04:06:51.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0c1409ae10 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.186+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562dc6695400 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c1409ec40 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 10 client.26969 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f0c1c003b60 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c140b9030 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0c1409a300 con 0x562dc6782650 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 10 client.26969 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 10 client.26969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 20 client.26969 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67a1ac0 0x562dc67a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 4 client.26969 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.187+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562dc66cd780 con 0x562dc67a1ac0 2024-03-30T04:06:51.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.188+0000 7f0c22000700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67a1ac0 0x562dc67a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.188+0000 7f0c22000700 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67a1ac0 0x562dc67a3ee0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29f3a817cf734b9f server_cookie=46fe098e0453bc77 in_seq=0 out_seq=0 2024-03-30T04:06:51.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.188+0000 7f0c03e00700 10 client.26969 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c03e00700 1 -- 172.21.15.23:0/3794382239 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562dc66cd780 con 0x562dc67a1ac0 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c03e00700 10 client.26969 handle_command_reply: tid=0 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 client.26969 shutdown 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 msgr2=0x562dc678d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c1409ab10 tx=0x7f0c1409a770 comp rx=0 tx=0).stop 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 shutdown_connections 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67a1ac0 0x562dc67a3ee0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc678b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678d870 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/3794382239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782f70 0x562dc678abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 >> 172.21.15.23:0/3794382239 conn(0x562dc66e2720 msgr2=0x562dc6784740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c03e00700 0 client.26969 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 shutdown_connections 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 1 -- 172.21.15.23:0/3794382239 wait complete. 2024-03-30T04:06:51.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.189+0000 7f0c3d039b80 20 client.26969 trim_cache size 0 max 16384 2024-03-30T04:06:51.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 447209 11.65 MB/sec execute 88 sec latency 9.057 ms 2024-03-30T04:06:51.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 Processor -- start 2024-03-30T04:06:51.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 -- start start 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc6798f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc678de70 0x562dc679d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc67841c0 con 0x562dc678de70 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc67838d0 con 0x562dc6782650 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.210+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc6785a20 con 0x562dc6782f70 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38398/0 (socket says 172.21.15.23:38398) 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 -- 172.21.15.23:0/1925491265 learned_addr learned my addr 172.21.15.23:0/1925491265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c23400700 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc678de70 0x562dc679d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22000700 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc6798f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 -- 172.21.15.23:0/1925491265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 msgr2=0x562dc6798f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc6798f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 -- 172.21.15.23:0/1925491265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc678de70 msgr2=0x562dc679d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc678de70 0x562dc679d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 -- 172.21.15.23:0/1925491265 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562dc67a8b10 con 0x562dc6782650 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c22a00700 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c14011040 tx=0x7f0c1400f720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c55076733608652 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.211+0000 7f0c0fe00700 1 -- 172.21.15.23:0/1925491265 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1409b510 con 0x562dc6782650 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c0fe00700 1 -- 172.21.15.23:0/1925491265 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c140b9040 con 0x562dc6782650 2024-03-30T04:06:51.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c0fe00700 1 -- 172.21.15.23:0/1925491265 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c140a3200 con 0x562dc6782650 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 -- 172.21.15.23:0/1925491265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 msgr2=0x562dc678ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c14011040 tx=0x7f0c1400f720 comp rx=0 tx=0).stop 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 -- 172.21.15.23:0/1925491265 shutdown_connections 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc678de70 0x562dc679d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782650 0x562dc678ab50 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/1925491265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc6782f70 0x562dc6798f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 -- 172.21.15.23:0/1925491265 >> 172.21.15.23:0/1925491265 conn(0x562dc66e2720 msgr2=0x562dc678a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.212+0000 7f0c3d039b80 1 -- 172.21.15.23:0/1925491265 shutdown_connections 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.213+0000 7f0c3d039b80 1 -- 172.21.15.23:0/1925491265 wait complete. 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.213+0000 7f0c3d039b80 1 -- start start 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.213+0000 7f0c3d039b80 1 Processor -- start 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 0x562dc67985e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc67ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc67eff20 con 0x562dc6782650 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc67f0090 con 0x562dc6782f70 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562dc67f0250 con 0x562dc678de70 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c3d039b80 10 client.0 _ll_register_callbacks cb 0x562dc667a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 0x562dc67985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33030/0 (socket says 172.21.15.23:33030) 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22a00700 1 -- 172.21.15.23:0/2973856517 learned_addr learned my addr 172.21.15.23:0/2973856517 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c23400700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc67ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 -- 172.21.15.23:0/2973856517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 msgr2=0x562dc67ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc67ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 -- 172.21.15.23:0/2973856517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 msgr2=0x562dc67980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22000700 1 -- 172.21.15.23:0/2973856517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1c003b60 con 0x562dc6782f70 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.214+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c22000700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 0x562dc67985e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x562dc67a47c0 tx=0x7f0c100047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72c08c31b39ae28 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c03e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10010d50 con 0x562dc6782f70 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c1c00f040 con 0x562dc6782f70 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0c10017040 con 0x562dc6782f70 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c10010eb0 con 0x562dc6782f70 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c3d039b80 10 client.26975 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:51.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.215+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562dc6502be0 con 0x562dc6782f70 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c1c00f040 con 0x562dc6782f70 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0c10003980 con 0x562dc6782f70 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 10 client.26975 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 10 client.26975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 20 client.26975 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67f7a90 0x562dc67f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 4 client.26975 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562dc6523d70 con 0x562dc67f7a90 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.216+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67f7a90 0x562dc67f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.217+0000 7f0c22a00700 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67f7a90 0x562dc67f9e80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d7be2664aab3b35 server_cookie=e757414485a2aabc in_seq=0 out_seq=0 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.217+0000 7f0c03e00700 10 client.26975 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c03e00700 1 -- 172.21.15.23:0/2973856517 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562dc6523d70 con 0x562dc67f7a90 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c03e00700 10 client.26975 handle_command_reply: tid=0 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 client.26975 shutdown 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 msgr2=0x562dc67985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 0x562dc67985e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x562dc67a47c0 tx=0x7f0c100047c0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 shutdown_connections 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562dc67f7a90 0x562dc67f9e80 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562dc678de70 0x562dc67ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562dc6782650 0x562dc67980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c03e00700 0 client.26975 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 --2- 172.21.15.23:0/2973856517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562dc6782f70 0x562dc67985e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 >> 172.21.15.23:0/2973856517 conn(0x562dc66e2720 msgr2=0x562dc679d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.218+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 shutdown_connections 2024-03-30T04:06:51.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c3d039b80 1 -- 172.21.15.23:0/2973856517 wait complete. 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c3d039b80 20 client.26975 trim_cache size 0 max 16384 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "5fb4ac02-cb8b-4a8e-963d-bf2b9c183f33", 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:06:51.220 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 msgr2=0x7f0c080a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 0x7f0c080a37d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c24198750 tx=0x7f0c100073d0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 msgr2=0x7f0c24197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0c14000c00 tx=0x7f0c14003a90 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 shutdown_connections 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0c2419bae0 0x7f0c2419bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0c080a1310 0x7f0c080a37d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c24069260 0x7f0c24197200 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 --2- 172.21.15.23:0/2903665881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c241060b0 0x7f0c24197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 >> 172.21.15.23:0/2903665881 conn(0x7f0c24076d80 msgr2=0x7f0c241071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.219+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 shutdown_connections 2024-03-30T04:06:51.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.220+0000 7f0c29600700 1 -- 172.21.15.23:0/2903665881 wait complete. 2024-03-30T04:06:51.239 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 -- 172.21.15.23:0/80114781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc106080 msgr2=0x7ff8bc10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/80114781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc106080 0x7ff8bc10dc00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009f90 tx=0x7ff8b400b330 comp rx=0 tx=0).stop 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 -- 172.21.15.23:0/80114781 shutdown_connections 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/80114781 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc106080 0x7ff8bc10dc00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/80114781 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/80114781 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc101910 0x7ff8bc105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 -- 172.21.15.23:0/80114781 >> 172.21.15.23:0/80114781 conn(0x7ff8bc0fc7c0 msgr2=0x7ff8bc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 -- 172.21.15.23:0/80114781 shutdown_connections 2024-03-30T04:06:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.456+0000 7ff8c2c00700 1 -- 172.21.15.23:0/80114781 wait complete. 2024-03-30T04:06:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.457+0000 7ff8c2c00700 1 Processor -- start 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.457+0000 7ff8c2c00700 1 -- start start 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc106080 0x7ff8bc1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc19c500 con 0x7ff8bc100f60 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc19c670 con 0x7ff8bc19ba80 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c2c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8bc19c830 con 0x7ff8bc106080 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38422/0 (socket says 172.21.15.23:38422) 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8c0c00700 1 -- 172.21.15.23:0/856585882 learned_addr learned my addr 172.21.15.23:0/856585882 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33044/0 (socket says 172.21.15.23:33044) 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8bb400700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc106080 0x7ff8bc1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8bbe00700 1 -- 172.21.15.23:0/856585882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc106080 msgr2=0x7ff8bc1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.458+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc106080 0x7ff8bc1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8bbe00700 1 -- 172.21.15.23:0/856585882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 msgr2=0x7ff8bc19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8bbe00700 1 -- 172.21.15.23:0/856585882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80095f0 con 0x7ff8bc100f60 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8c0c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9f0 tx=0x7ff8ac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8b8c00700 1 -- 172.21.15.23:0/856585882 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00c9c0 con 0x7ff8bc100f60 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8b8c00700 1 -- 172.21.15.23:0/856585882 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8ac010430 con 0x7ff8bc100f60 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b4009c40 con 0x7ff8bc100f60 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8b8c00700 1 -- 172.21.15.23:0/856585882 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac018680 con 0x7ff8bc100f60 2024-03-30T04:06:51.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.459+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc1a4130 con 0x7ff8bc100f60 2024-03-30T04:06:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.462+0000 7ff8b8c00700 1 -- 172.21.15.23:0/856585882 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff8ac010860 con 0x7ff8bc100f60 2024-03-30T04:06:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.464+0000 7ff8b8c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 0x7ff8a00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.464+0000 7ff8b8c00700 1 -- 172.21.15.23:0/856585882 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff8ac014070 con 0x7ff8bc100f60 2024-03-30T04:06:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.464+0000 7ff8bb400700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 0x7ff8a00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.464+0000 7ff8d671db80 1 Processor -- start 2024-03-30T04:06:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.464+0000 7ff8d671db80 1 -- start start 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 0x557f9472b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f947391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f94684990 con 0x557f9472b5d0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f94682350 con 0x557f9472bef0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8d671db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9467dc40 con 0x557f94736df0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8bb400700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 0x7ff8a00a37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc1986f0 tx=0x7ff8a8009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8bb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 0x557f9472b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8bb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38426/0 (socket says 172.21.15.23:38426) 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8bb400700 1 -- 172.21.15.23:0/2981146789 learned_addr learned my addr 172.21.15.23:0/2981146789 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.465+0000 7ff8c0c00700 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f947391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 -- 172.21.15.23:0/2981146789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 msgr2=0x557f947391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f947391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 -- 172.21.15.23:0/2981146789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 msgr2=0x557f947368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 -- 172.21.15.23:0/2981146789 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f944ff3b0 con 0x557f9472b5d0 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bb400700 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 0x557f9472b9b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0ca530 tx=0x7ff8ac0998d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f069343f5431131 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8b1200700 1 -- 172.21.15.23:0/2981146789 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac09dd20 con 0x557f9472b5d0 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8b1200700 1 -- 172.21.15.23:0/2981146789 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8ac0a2370 con 0x557f9472b5d0 2024-03-30T04:06:51.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.466+0000 7ff8b1200700 1 -- 172.21.15.23:0/2981146789 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac099640 con 0x557f9472b5d0 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 -- 172.21.15.23:0/2981146789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 msgr2=0x557f9472b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 0x557f9472b9b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0ca530 tx=0x7ff8ac0998d0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 -- 172.21.15.23:0/2981146789 shutdown_connections 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f947391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472b5d0 0x557f9472b9b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 --2- 172.21.15.23:0/2981146789 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472bef0 0x557f947368b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 -- 172.21.15.23:0/2981146789 >> 172.21.15.23:0/2981146789 conn(0x557f9468b6a0 msgr2=0x557f94727a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 -- 172.21.15.23:0/2981146789 shutdown_connections 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.467+0000 7ff8d671db80 1 -- 172.21.15.23:0/2981146789 wait complete. 2024-03-30T04:06:51.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 -- start start 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 Processor -- start 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94733bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f94734310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f94619d80 con 0x557f9472bef0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f94619a20 con 0x557f9472b5d0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.468+0000 7ff8d671db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f945016a0 con 0x557f94736df0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8d671db80 10 client.0 _ll_register_callbacks cb 0x557f94623470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38438/0 (socket says 172.21.15.23:38438) 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 -- 172.21.15.23:0/4130359643 learned_addr learned my addr 172.21.15.23:0/4130359643 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bb400700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94733bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8c0c00700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f94734310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 -- 172.21.15.23:0/4130359643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 msgr2=0x557f94734310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f94734310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 -- 172.21.15.23:0/4130359643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 msgr2=0x557f94733bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94733bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 -- 172.21.15.23:0/4130359643 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a8017070 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bb400700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94733bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0cd200 tx=0x7ff8ac0107d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e591bd47f01825c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.469+0000 7ff8a5600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac09a360 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff8ac0a1ee0 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f9463e380 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8ac0b8250 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8d671db80 10 client.26987 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7ff8bc1a4130 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557f9463e380 con 0x557f9472b5d0 2024-03-30T04:06:51.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff8ac094b00 con 0x557f9472b5d0 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8d671db80 10 client.26987 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.470+0000 7ff8d671db80 10 client.26987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8d671db80 20 client.26987 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f94746580 0x557f94748970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8d671db80 4 client.26987 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f94676700 con 0x557f94746580 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8bb400700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f94746580 0x557f94748970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8bb400700 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f94746580 0x557f94748970 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2414a70effb224b server_cookie=938777d2f77bfedd in_seq=0 out_seq=0 2024-03-30T04:06:51.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.471+0000 7ff8a5600700 10 client.26987 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.472+0000 7ff8a5600700 1 -- 172.21.15.23:0/4130359643 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557f94676700 con 0x557f94746580 2024-03-30T04:06:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.472+0000 7ff8a5600700 10 client.26987 handle_command_reply: tid=0 2024-03-30T04:06:51.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.472+0000 7ff8d671db80 1 client.26987 shutdown 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 msgr2=0x557f94736840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0cd200 tx=0x7ff8ac0107d0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 shutdown_connections 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f94746580 0x557f94748970 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f94734310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f94736840 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8a5600700 0 client.26987 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 --2- 172.21.15.23:0/4130359643 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94733bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 >> 172.21.15.23:0/4130359643 conn(0x557f9468b6a0 msgr2=0x557f9472d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 shutdown_connections 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 1 -- 172.21.15.23:0/4130359643 wait complete. 2024-03-30T04:06:51.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.473+0000 7ff8d671db80 20 client.26987 trim_cache size 0 max 16384 2024-03-30T04:06:51.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 Processor -- start 2024-03-30T04:06:51.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 -- start start 2024-03-30T04:06:51.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94726f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f94736df0 0x557f9473da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f94729160 con 0x557f9472bef0 2024-03-30T04:06:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9472c850 con 0x557f94736df0 2024-03-30T04:06:51.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8d671db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9472c2d0 con 0x557f9472b5d0 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8bb400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94726f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44420/0 (socket says 172.21.15.23:44420) 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.493+0000 7ff8bbe00700 1 -- 172.21.15.23:0/3738108170 learned_addr learned my addr 172.21.15.23:0/3738108170 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8c0c00700 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f94736df0 0x557f9473da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 -- 172.21.15.23:0/3738108170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f94736df0 msgr2=0x557f9473da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f94736df0 0x557f9473da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 -- 172.21.15.23:0/3738108170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 msgr2=0x557f94726f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94726f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 -- 172.21.15.23:0/3738108170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f944abb60 con 0x557f9472b5d0 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009800 tx=0x7ff8ac010c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da863cd1dbf07b58 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8b1c00700 1 -- 172.21.15.23:0/3738108170 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac09a5f0 con 0x557f9472b5d0 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8b1c00700 1 -- 172.21.15.23:0/3738108170 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8ac095d30 con 0x557f9472b5d0 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.494+0000 7ff8b1c00700 1 -- 172.21.15.23:0/3738108170 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0a1ee0 con 0x557f9472b5d0 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 -- 172.21.15.23:0/3738108170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 msgr2=0x557f94734310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009800 tx=0x7ff8ac010c60 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 -- 172.21.15.23:0/3738108170 shutdown_connections 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f94736df0 0x557f9473da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f9472b5d0 0x557f94734310 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 --2- 172.21.15.23:0/3738108170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f94726f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 -- 172.21.15.23:0/3738108170 >> 172.21.15.23:0/3738108170 conn(0x557f9468b6a0 msgr2=0x557f947205a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 -- 172.21.15.23:0/3738108170 shutdown_connections 2024-03-30T04:06:51.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.495+0000 7ff8d671db80 1 -- 172.21.15.23:0/3738108170 wait complete. 2024-03-30T04:06:51.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 -- start start 2024-03-30T04:06:51.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 Processor -- start 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f9473d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f9473bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9473c240 con 0x557f9472bef0 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9473c3b0 con 0x557f9472b5d0 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f9473c570 con 0x557f94736df0 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.496+0000 7ff8d671db80 10 client.0 _ll_register_callbacks cb 0x557f94623470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:06:51.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38462/0 (socket says 172.21.15.23:38462) 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 -- 172.21.15.23:0/365748153 learned_addr learned my addr 172.21.15.23:0/365748153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bb400700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f9473d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8c0c00700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f9473bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 -- 172.21.15.23:0/365748153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 msgr2=0x557f9473bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f9473bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 -- 172.21.15.23:0/365748153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 msgr2=0x557f9473d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f9473d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 -- 172.21.15.23:0/365748153 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8bc1a4130 con 0x557f9472b5d0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8bbe00700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000c00 tx=0x7ff8ac09a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a991af0f10b40276 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8a5600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac09a5f0 con 0x557f9472b5d0 2024-03-30T04:06:51.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7ff8ac0833a0 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.497+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8ac0a2370 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8ac094b00 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 10 client.26993 fetch_fsmap learned FSMap version 30 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557f94643ea0 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8ac0a2370 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff8ac0836f0 con 0x557f9472b5d0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 10 client.26993 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 10 client.26993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 20 client.26993 populate_metadata read hostname 'smithi023' 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f9479f4a0 0x557f947a1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 4 client.26993 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557f944cccf0 con 0x557f9479f4a0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.498+0000 7ff8bb400700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f9479f4a0 0x557f947a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8bb400700 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f9479f4a0 0x557f947a1890 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c9154e1807342d2 server_cookie=ba1cf56f1df9ccd4 in_seq=0 out_seq=0 2024-03-30T04:06:51.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8a5600700 10 client.26993 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8a5600700 1 -- 172.21.15.23:0/365748153 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557f944cccf0 con 0x557f9479f4a0 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8a5600700 10 client.26993 handle_command_reply: tid=0 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 client.26993 shutdown 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 msgr2=0x557f9473d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac000c00 tx=0x7ff8ac09a070 comp rx=0 tx=0).stop 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 shutdown_connections 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557f9479f4a0 0x557f947a1890 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557f94736df0 0x557f9473bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557f9472b5d0 0x557f9473d490 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 --2- 172.21.15.23:0/365748153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557f9472bef0 0x557f9473d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8a5600700 0 client.26993 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.499+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 >> 172.21.15.23:0/365748153 conn(0x557f9468b6a0 msgr2=0x557f947421c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 shutdown_connections 2024-03-30T04:06:51.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8d671db80 1 -- 172.21.15.23:0/365748153 wait complete. 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8d671db80 20 client.26993 trim_cache size 0 max 16384 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "5fb4ac02-cb8b-4a8e-963d-bf2b9c183f33": { 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "5fb4ac02-cb8b-4a8e-963d-bf2b9c183f33", 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:06:51.503 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 msgr2=0x7ff8a00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 0x7ff8a00a37d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc1986f0 tx=0x7ff8a8009310 comp rx=0 tx=0).stop 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 msgr2=0x7ff8bc1971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00e9f0 tx=0x7ff8ac00ed00 comp rx=0 tx=0).stop 2024-03-30T04:06:51.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 shutdown_connections 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff8a00a1310 0x7ff8a00a37d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8bc19ba80 0x7ff8bc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff8bc100f60 0x7ff8bc1971a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 --2- 172.21.15.23:0/856585882 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8bc106080 0x7ff8bc1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.500+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 >> 172.21.15.23:0/856585882 conn(0x7ff8bc0fc7c0 msgr2=0x7ff8bc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.501+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 shutdown_connections 2024-03-30T04:06:51.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:06:51.501+0000 7ff8c2c00700 1 -- 172.21.15.23:0/856585882 wait complete. 2024-03-30T04:06:51.518 INFO:tasks.cephfs.filesystem:scrub status for tag:5fb4ac02-cb8b-4a8e-963d-bf2b9c183f33 - {'path': '/', 'tag': '5fb4ac02-cb8b-4a8e-963d-bf2b9c183f33', 'options': 'recursive,force'} 2024-03-30T04:06:52.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 449348 11.60 MB/sec execute 89 sec latency 9.356 ms 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:52 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:06:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:06:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:06:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:52 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:06:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:06:52.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:06:52.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:52.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:52 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:06:53.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 451178 11.61 MB/sec execute 90 sec latency 9.203 ms 2024-03-30T04:06:53.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:53 smithi154 ceph-mon[79161]: pgmap v151: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 296 op/s 2024-03-30T04:06:53.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:53 smithi193 ceph-mon[79581]: pgmap v151: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 296 op/s 2024-03-30T04:06:53.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:53 smithi023 ceph-mon[38254]: pgmap v151: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 296 op/s 2024-03-30T04:06:54.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 453474 11.62 MB/sec execute 91 sec latency 54.111 ms 2024-03-30T04:06:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:54 smithi193 ceph-mon[79581]: pgmap v152: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-03-30T04:06:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:54 smithi023 ceph-mon[38254]: pgmap v152: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-03-30T04:06:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:54 smithi154 ceph-mon[79161]: pgmap v152: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-03-30T04:06:55.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 454981 11.57 MB/sec execute 92 sec latency 13.987 ms 2024-03-30T04:06:56.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 457846 11.64 MB/sec execute 93 sec latency 11.074 ms 2024-03-30T04:06:57.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 459158 11.75 MB/sec execute 94 sec latency 78.495 ms 2024-03-30T04:06:57.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:57 smithi154 ceph-mon[79161]: pgmap v153: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-03-30T04:06:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:57 smithi193 ceph-mon[79581]: pgmap v153: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-03-30T04:06:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:57 smithi023 ceph-mon[38254]: pgmap v153: 129 pgs: 129 active+clean; 253 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 199 op/s 2024-03-30T04:06:58.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 462229 11.82 MB/sec execute 95 sec latency 6.269 ms 2024-03-30T04:06:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:58 smithi193 ceph-mon[79581]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 298 op/s 2024-03-30T04:06:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:58 smithi023 ceph-mon[38254]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 298 op/s 2024-03-30T04:06:58.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:58 smithi154 ceph-mon[79161]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.5 MiB/s wr, 298 op/s 2024-03-30T04:06:59.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 464286 11.77 MB/sec execute 96 sec latency 52.673 ms 2024-03-30T04:06:59.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:06:59.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:06:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:06:59.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:06:59.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:06:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:06:59.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:06:59.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:06:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:07:00.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 465998 11.78 MB/sec execute 97 sec latency 21.278 ms 2024-03-30T04:07:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:00 smithi193 ceph-mon[79581]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-03-30T04:07:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:00 smithi023 ceph-mon[38254]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-03-30T04:07:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:00 smithi154 ceph-mon[79161]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-03-30T04:07:01.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 466480 11.67 MB/sec execute 98 sec latency 67.855 ms 2024-03-30T04:07:02.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 469294 11.73 MB/sec execute 99 sec latency 56.705 ms 2024-03-30T04:07:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:02 smithi193 ceph-mon[79581]: pgmap v156: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 325 op/s 2024-03-30T04:07:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:02 smithi023 ceph-mon[38254]: pgmap v156: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 325 op/s 2024-03-30T04:07:02.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:02 smithi154 ceph-mon[79161]: pgmap v156: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.6 MiB/s wr, 325 op/s 2024-03-30T04:07:03.199 INFO:tasks.workunit.client.0.smithi023.stdout: 1 471790 11.72 MB/sec execute 100 sec latency 6.777 ms 2024-03-30T04:07:04.200 INFO:tasks.workunit.client.0.smithi023.stdout: 1 473340 11.69 MB/sec execute 101 sec latency 82.230 ms 2024-03-30T04:07:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:04 smithi193 ceph-mon[79581]: pgmap v157: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:07:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:04 smithi023 ceph-mon[38254]: pgmap v157: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:07:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:04 smithi154 ceph-mon[79161]: pgmap v157: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-03-30T04:07:05.200 INFO:tasks.workunit.client.0.smithi023.stdout: 1 476073 11.74 MB/sec execute 102 sec latency 150.187 ms 2024-03-30T04:07:06.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 478056 11.70 MB/sec execute 103 sec latency 12.806 ms 2024-03-30T04:07:07.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 480391 11.72 MB/sec execute 104 sec latency 7.148 ms 2024-03-30T04:07:07.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:07 smithi154 ceph-mon[79161]: pgmap v158: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-03-30T04:07:07.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:07 smithi193 ceph-mon[79581]: pgmap v158: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-03-30T04:07:07.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:07 smithi023 ceph-mon[38254]: pgmap v158: 129 pgs: 129 active+clean; 266 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-03-30T04:07:08.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 483278 11.77 MB/sec execute 105 sec latency 8.296 ms 2024-03-30T04:07:08.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:08 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:07:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:08 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:07:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:08 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:07:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:08 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:07:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:08 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:07:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:08 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:07:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:08 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:07:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:08 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:07:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:08 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:07:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:08 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:07:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:08 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:07:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:08 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:07:09.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 484995 11.73 MB/sec execute 106 sec latency 20.565 ms 2024-03-30T04:07:09.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:09 smithi154 ceph-mon[79161]: pgmap v159: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 327 op/s 2024-03-30T04:07:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:09 smithi193 ceph-mon[79581]: pgmap v159: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 327 op/s 2024-03-30T04:07:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:09 smithi023 ceph-mon[38254]: pgmap v159: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 4.6 MiB/s wr, 327 op/s 2024-03-30T04:07:10.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 487135 11.75 MB/sec execute 107 sec latency 34.983 ms 2024-03-30T04:07:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:10 smithi193 ceph-mon[79581]: pgmap v160: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 228 op/s 2024-03-30T04:07:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:10 smithi023 ceph-mon[38254]: pgmap v160: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 228 op/s 2024-03-30T04:07:10.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:10 smithi154 ceph-mon[79161]: pgmap v160: 129 pgs: 129 active+clean; 274 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 228 op/s 2024-03-30T04:07:11.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 489680 11.75 MB/sec execute 108 sec latency 8.342 ms 2024-03-30T04:07:12.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 491599 11.73 MB/sec execute 109 sec latency 7.072 ms 2024-03-30T04:07:13.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 494301 11.77 MB/sec execute 110 sec latency 10.485 ms 2024-03-30T04:07:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:13 smithi193 ceph-mon[79581]: pgmap v161: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-30T04:07:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:13 smithi023 ceph-mon[38254]: pgmap v161: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-30T04:07:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:13 smithi154 ceph-mon[79161]: pgmap v161: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.7 MiB/s wr, 327 op/s 2024-03-30T04:07:14.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 496779 11.78 MB/sec execute 111 sec latency 16.957 ms 2024-03-30T04:07:15.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 498134 11.74 MB/sec execute 112 sec latency 39.293 ms 2024-03-30T04:07:15.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:15 smithi193 ceph-mon[79581]: pgmap v162: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-03-30T04:07:15.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:15 smithi023 ceph-mon[38254]: pgmap v162: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-03-30T04:07:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:15 smithi154 ceph-mon[79161]: pgmap v162: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.3 MiB/s wr, 204 op/s 2024-03-30T04:07:16.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 501165 11.80 MB/sec execute 113 sec latency 8.953 ms 2024-03-30T04:07:17.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 503359 11.76 MB/sec execute 114 sec latency 13.280 ms 2024-03-30T04:07:18.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 505404 11.77 MB/sec execute 115 sec latency 10.285 ms 2024-03-30T04:07:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:18 smithi154 ceph-mon[79161]: pgmap v163: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-03-30T04:07:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:18 smithi193 ceph-mon[79581]: pgmap v163: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-03-30T04:07:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:18 smithi023 ceph-mon[38254]: pgmap v163: 129 pgs: 129 active+clean; 282 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 202 op/s 2024-03-30T04:07:19.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 508296 11.82 MB/sec execute 116 sec latency 7.662 ms 2024-03-30T04:07:20.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 509540 11.77 MB/sec execute 117 sec latency 53.660 ms 2024-03-30T04:07:20.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:20 smithi154 ceph-mon[79161]: pgmap v164: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 307 op/s 2024-03-30T04:07:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:20 smithi193 ceph-mon[79581]: pgmap v164: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 307 op/s 2024-03-30T04:07:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:20 smithi023 ceph-mon[38254]: pgmap v164: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.1 MiB/s wr, 307 op/s 2024-03-30T04:07:21.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 512113 11.79 MB/sec execute 118 sec latency 6.978 ms 2024-03-30T04:07:21.519 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:07:21.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 -- 172.21.15.23:0/3730041735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8101920 msgr2=0x7f16f8105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 --2- 172.21.15.23:0/3730041735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8101920 0x7f16f8105960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f16e4009990 tx=0x7f16e4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 -- 172.21.15.23:0/3730041735 shutdown_connections 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 --2- 172.21.15.23:0/3730041735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f8106090 0x7f16f810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 --2- 172.21.15.23:0/3730041735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 0x7f16f8101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 --2- 172.21.15.23:0/3730041735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8101920 0x7f16f8105960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.745+0000 7f16fd200700 1 -- 172.21.15.23:0/3730041735 >> 172.21.15.23:0/3730041735 conn(0x7f16f80fc7f0 msgr2=0x7f16f80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.746+0000 7f16fd200700 1 -- 172.21.15.23:0/3730041735 shutdown_connections 2024-03-30T04:07:21.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.746+0000 7f16fd200700 1 -- 172.21.15.23:0/3730041735 wait complete. 2024-03-30T04:07:21.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.747+0000 7f16fd200700 1 Processor -- start 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.747+0000 7f16fd200700 1 -- start start 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.747+0000 7f16fd200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 0x7f16f81993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.747+0000 7f16fd200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16fd200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f819dcc0 0x7f16f819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16fd200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819e740 con 0x7f16f8100f70 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16fd200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819e8b0 con 0x7f16f8106090 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16fd200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f819ea70 con 0x7f16f819dcc0 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 0x7f16f81993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37292/0 (socket says 172.21.15.23:37292) 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 -- 172.21.15.23:0/1998777224 learned_addr learned my addr 172.21.15.23:0/1998777224 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f6800700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f819dcc0 0x7f16f819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 -- 172.21.15.23:0/1998777224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f819dcc0 msgr2=0x7f16f819e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f819dcc0 0x7f16f819e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 -- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 msgr2=0x7f16f81993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 0x7f16f81993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.748+0000 7f16f5400700 1 -- 172.21.15.23:0/1998777224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e40095f0 con 0x7f16f8106090 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16f5400700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f16e4000c00 tx=0x7f16e4005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16eea00700 1 -- 172.21.15.23:0/1998777224 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e401d070 con 0x7f16f8106090 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16eea00700 1 -- 172.21.15.23:0/1998777224 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e4022470 con 0x7f16f8106090 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f81a5f70 con 0x7f16f8106090 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16eea00700 1 -- 172.21.15.23:0/1998777224 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4017650 con 0x7f16f8106090 2024-03-30T04:07:21.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.749+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a6390 con 0x7f16f8106090 2024-03-30T04:07:21.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.754+0000 7f16eea00700 1 -- 172.21.15.23:0/1998777224 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f16e400fad0 con 0x7f16f8106090 2024-03-30T04:07:21.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.755+0000 7f1710d11b80 1 Processor -- start 2024-03-30T04:07:21.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.755+0000 7f1710d11b80 1 -- start start 2024-03-30T04:07:21.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.755+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44e5d0 0x55cc1c44e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.755+0000 7f16eea00700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 0x7f16dc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.755+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c459df0 0x55cc1c45c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f1710d11b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c3a7990 con 0x55cc1c44e5d0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f1710d11b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c3a5350 con 0x55cc1c44eef0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f1710d11b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c3a0c40 con 0x55cc1c459df0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16eea00700 1 -- 172.21.15.23:0/1998777224 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f16e40ce3a0 con 0x7f16f8106090 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37296/0 (socket says 172.21.15.23:37296) 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 -- 172.21.15.23:0/2915492971 learned_addr learned my addr 172.21.15.23:0/2915492971 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f6800700 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44e5d0 0x55cc1c44e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5e00700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 0x7f16dc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f6800700 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c459df0 0x55cc1c45c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 -- 172.21.15.23:0/2915492971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c459df0 msgr2=0x55cc1c45c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c459df0 0x55cc1c45c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 -- 172.21.15.23:0/2915492971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44e5d0 msgr2=0x55cc1c44e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44e5d0 0x55cc1c44e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.756+0000 7f16f5400700 1 -- 172.21.15.23:0/2915492971 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc1c2223b0 con 0x55cc1c44eef0 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.757+0000 7f16f5400700 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f16e4097680 tx=0x7f16e4009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d852136007a72821 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.757+0000 7f16e3400700 1 -- 172.21.15.23:0/2915492971 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e40999d0 con 0x55cc1c44eef0 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.757+0000 7f16e3400700 1 -- 172.21.15.23:0/2915492971 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e40997e0 con 0x55cc1c44eef0 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.757+0000 7f16e3400700 1 -- 172.21.15.23:0/2915492971 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e40995b0 con 0x55cc1c44eef0 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 -- 172.21.15.23:0/2915492971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 msgr2=0x55cc1c4598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f16f5e00700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 0x7f16dc0a3750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009c60 tx=0x7f16e8009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f16e4097680 tx=0x7f16e4009310 comp rx=0 tx=0).stop 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 -- 172.21.15.23:0/2915492971 shutdown_connections 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c459df0 0x55cc1c45c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44e5d0 0x55cc1c44e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 --2- 172.21.15.23:0/2915492971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4598b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 -- 172.21.15.23:0/2915492971 >> 172.21.15.23:0/2915492971 conn(0x55cc1c3ae6a0 msgr2=0x55cc1c44aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 -- 172.21.15.23:0/2915492971 shutdown_connections 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.758+0000 7f1710d11b80 1 -- 172.21.15.23:0/2915492971 wait complete. 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.759+0000 7f1710d11b80 1 -- start start 2024-03-30T04:07:21.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.759+0000 7f1710d11b80 1 Processor -- start 2024-03-30T04:07:21.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.759+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c459780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.759+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 0x55cc1c456bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.759+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f1710d11b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c33cd80 con 0x55cc1c44eef0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f1710d11b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c33ca20 con 0x55cc1c459df0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f1710d11b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c2246a0 con 0x55cc1c44e5d0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f1710d11b80 10 client.0 _ll_register_callbacks cb 0x55cc1c346470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37298/0 (socket says 172.21.15.23:37298) 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 -- 172.21.15.23:0/4281712991 learned_addr learned my addr 172.21.15.23:0/4281712991 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f5e00700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c459780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f5400700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 0x55cc1c456bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 -- 172.21.15.23:0/4281712991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 msgr2=0x55cc1c459780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c459780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 -- 172.21.15.23:0/4281712991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 msgr2=0x55cc1c456bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 0x55cc1c456bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 -- 172.21.15.23:0/4281712991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a6390 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f5400700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 0x55cc1c456bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16f6800700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f16f0007a60 tx=0x7f16f0009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa82b7744dad7846 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.760+0000 7f16d7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f000e040 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f16f0004bb0 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc1c361380 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f0004d70 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f1710d11b80 10 client.27005 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.761+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cc1c1ceb60 con 0x55cc1c459df0 2024-03-30T04:07:21.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc1c361380 con 0x55cc1c459df0 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f16f0019a10 con 0x55cc1c459df0 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 10 client.27005 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 10 client.27005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 20 client.27005 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c469680 0x55cc1c46baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 4 client.27005 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc1c399700 con 0x55cc1c469680 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.762+0000 7f16f5e00700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c469680 0x55cc1c46baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f16f5e00700 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c469680 0x55cc1c46baa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e834c7792158544c server_cookie=9a6f1b837f352bd in_seq=0 out_seq=0 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f16d7e00700 10 client.27005 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f16d7e00700 1 -- 172.21.15.23:0/4281712991 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cc1c399700 con 0x55cc1c469680 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f16d7e00700 10 client.27005 handle_command_reply: tid=0 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f1710d11b80 1 client.27005 shutdown 2024-03-30T04:07:21.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 msgr2=0x55cc1c457150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f16f0007a60 tx=0x7f16f0009c20 comp rx=0 tx=0).stop 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 shutdown_connections 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.763+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c459df0 0x55cc1c457150 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c469680 0x55cc1c46baa0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c459780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 --2- 172.21.15.23:0/4281712991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c44eef0 0x55cc1c456bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 >> 172.21.15.23:0/4281712991 conn(0x55cc1c3ae6a0 msgr2=0x55cc1c450530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f16d7e00700 0 client.27005 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 shutdown_connections 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 1 -- 172.21.15.23:0/4281712991 wait complete. 2024-03-30T04:07:21.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.764+0000 7f1710d11b80 20 client.27005 trim_cache size 0 max 16384 2024-03-30T04:07:21.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.783+0000 7f1710d11b80 1 Processor -- start 2024-03-30T04:07:21.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.783+0000 7f1710d11b80 1 -- start start 2024-03-30T04:07:21.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c4607e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c451ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c44b660 con 0x55cc1c459df0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c460d20 con 0x55cc1c44eef0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f1710d11b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c44c300 con 0x55cc1c44e5d0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f16f6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c451ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f16f5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37302/0 (socket says 172.21.15.23:37302) 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f16f5400700 1 -- 172.21.15.23:0/1765182493 learned_addr learned my addr 172.21.15.23:0/1765182493 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.784+0000 7f16f5e00700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c4607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 -- 172.21.15.23:0/1765182493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 msgr2=0x55cc1c4607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c4607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 -- 172.21.15.23:0/1765182493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 msgr2=0x55cc1c451ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c451ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 -- 172.21.15.23:0/1765182493 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc1c366ea0 con 0x55cc1c44eef0 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f6800700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c451ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5e00700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c4607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16f5400700 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f16e4005cd0 tx=0x7f16e4099cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145eb2bc58d67689 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16e3e00700 1 -- 172.21.15.23:0/1765182493 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e409b880 con 0x55cc1c44eef0 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16e3e00700 1 -- 172.21.15.23:0/1765182493 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e4099980 con 0x55cc1c44eef0 2024-03-30T04:07:21.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.785+0000 7f16e3e00700 1 -- 172.21.15.23:0/1765182493 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e4099520 con 0x55cc1c44eef0 2024-03-30T04:07:21.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 -- 172.21.15.23:0/1765182493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 msgr2=0x55cc1c4592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f16e4005cd0 tx=0x7f16e4099cc0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 -- 172.21.15.23:0/1765182493 shutdown_connections 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c451ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44e5d0 0x55cc1c4607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 --2- 172.21.15.23:0/1765182493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44eef0 0x55cc1c4592b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 -- 172.21.15.23:0/1765182493 >> 172.21.15.23:0/1765182493 conn(0x55cc1c3ae6a0 msgr2=0x55cc1c4435a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 -- 172.21.15.23:0/1765182493 shutdown_connections 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.786+0000 7f1710d11b80 1 -- 172.21.15.23:0/1765182493 wait complete. 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 -- start start 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 Processor -- start 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44eef0 0x55cc1c45de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c45e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c45ec50 con 0x55cc1c459df0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c4c4cc0 con 0x55cc1c44e5d0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc1c4c4e00 con 0x55cc1c44eef0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.787+0000 7f1710d11b80 10 client.0 _ll_register_callbacks cb 0x55cc1c346470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37314/0 (socket says 172.21.15.23:37314) 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 -- 172.21.15.23:0/144009681 learned_addr learned my addr 172.21.15.23:0/144009681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f6800700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c45e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5400700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44eef0 0x55cc1c45de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 -- 172.21.15.23:0/144009681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44eef0 msgr2=0x55cc1c45de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44eef0 0x55cc1c45de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 -- 172.21.15.23:0/144009681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 msgr2=0x55cc1c45e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c45e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 -- 172.21.15.23:0/144009681 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc1c1ceb60 con 0x55cc1c44e5d0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16f5e00700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f16e8011ab0 tx=0x7f16e80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0ec8ecf7178dd46 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:21.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.788+0000 7f16d7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8022040 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16e40997e0 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f16e8028420 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16e8003680 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 10 client.27011 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cc1c2223b0 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f16e40997e0 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f16e801d660 con 0x55cc1c44e5d0 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 10 client.27011 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 10 client.27011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 20 client.27011 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c4c6620 0x55cc1c4c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 4 client.27011 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:07:21.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.789+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cc1c1efcf0 con 0x55cc1c4c6620 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16f5400700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c4c6620 0x55cc1c4c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16f5400700 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c4c6620 0x55cc1c4c8a40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9230f9a75215d41 server_cookie=1b196a6456ca8c00 in_seq=0 out_seq=0 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16d7e00700 10 client.27011 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16d7e00700 1 -- 172.21.15.23:0/144009681 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cc1c1efcf0 con 0x55cc1c4c6620 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16d7e00700 10 client.27011 handle_command_reply: tid=0 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 client.27011 shutdown 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 msgr2=0x55cc1c460910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f16e8011ab0 tx=0x7f16e80047c0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 shutdown_connections 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cc1c459df0 0x55cc1c45e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cc1c44e5d0 0x55cc1c460910 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55cc1c4c6620 0x55cc1c4c8a40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 --2- 172.21.15.23:0/144009681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cc1c44eef0 0x55cc1c45de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f16d7e00700 0 client.27011 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.790+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 >> 172.21.15.23:0/144009681 conn(0x55cc1c3ae6a0 msgr2=0x55cc1c451ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 shutdown_connections 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f1710d11b80 1 -- 172.21.15.23:0/144009681 wait complete. 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f1710d11b80 20 client.27011 trim_cache size 0 max 16384 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:07:21.794 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 msgr2=0x7f16dc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 0x7f16dc0a3750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009c60 tx=0x7f16e8009260 comp rx=0 tx=0).stop 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 msgr2=0x7f16f8199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f16e4000c00 tx=0x7f16e4005740 comp rx=0 tx=0).stop 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 shutdown_connections 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f16f819dcc0 0x7f16f819e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f16f8100f70 0x7f16f81993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f16dc0a1290 0x7f16dc0a3750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 --2- 172.21.15.23:0/1998777224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16f8106090 0x7f16f8199920 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:21.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.791+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 >> 172.21.15.23:0/1998777224 conn(0x7f16f80fc7f0 msgr2=0x7f16f80febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:21.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.792+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 shutdown_connections 2024-03-30T04:07:21.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:21.792+0000 7f16fd200700 1 -- 172.21.15.23:0/1998777224 wait complete. 2024-03-30T04:07:21.809 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:07:21.809 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:07:22.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 -- 172.21.15.23:0/1573640511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 msgr2=0x7f064410cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 --2- 172.21.15.23:0/1573640511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f064410cef0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0630009990 tx=0x7f0630009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 -- 172.21.15.23:0/1573640511 shutdown_connections 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 --2- 172.21.15.23:0/1573640511 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064410f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 --2- 172.21.15.23:0/1573640511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f06440735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 --2- 172.21.15.23:0/1573640511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f064410cef0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 -- 172.21.15.23:0/1573640511 >> 172.21.15.23:0/1573640511 conn(0x7f06440fc760 msgr2=0x7f06440feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 -- 172.21.15.23:0/1573640511 shutdown_connections 2024-03-30T04:07:22.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.032+0000 7f0649800700 1 -- 172.21.15.23:0/1573640511 wait complete. 2024-03-30T04:07:22.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.033+0000 7f0649800700 1 Processor -- start 2024-03-30T04:07:22.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.033+0000 7f0649800700 1 -- start start 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f0644199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f06441998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064419dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064419e280 con 0x7f06440731e0 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064419e3f0 con 0x7f064410d430 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0649800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f064419e5b0 con 0x7f0644073b00 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0642a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f0644199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0643400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064419dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0642a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f0644199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46060/0 (socket says 172.21.15.23:46060) 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0642a00700 1 -- 172.21.15.23:0/1325631224 learned_addr learned my addr 172.21.15.23:0/1325631224 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.034+0000 7f0642000700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f06441998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 -- 172.21.15.23:0/1325631224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 msgr2=0x7f06441998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f06441998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 -- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 msgr2=0x7f0644199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f0644199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 -- 172.21.15.23:0/1325631224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06300095f0 con 0x7f064410d430 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0642000700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f06441998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0643400700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064419dc40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f063c00efc0 tx=0x7f063c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c018070 con 0x7f064410d430 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f064419e830 con 0x7f064410d430 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f063c00f040 con 0x7f064410d430 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00cd40 con 0x7f064410d430 2024-03-30T04:07:22.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.035+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06441a60d0 con 0x7f064410d430 2024-03-30T04:07:22.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.036+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064419eaa0 con 0x7f064410d430 2024-03-30T04:07:22.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.038+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f063c003a00 con 0x7f064410d430 2024-03-30T04:07:22.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.040+0000 7f063b400700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 0x7f06280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.040+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f063c0cd520 con 0x7f064410d430 2024-03-30T04:07:22.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.040+0000 7f0642a00700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 0x7f06280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.040+0000 7f0642a00700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 0x7f06280a3750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0634009d20 tx=0x7f0634009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.042+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f063c096680 con 0x7f064410d430 2024-03-30T04:07:22.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 514788 11.80 MB/sec execute 119 sec latency 14.756 ms 2024-03-30T04:07:22.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:22 smithi023 ceph-mon[38254]: pgmap v165: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-03-30T04:07:22.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.350+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0644108640 con 0x7f064410d430 2024-03-30T04:07:22.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.350+0000 7f063b400700 1 -- 172.21.15.23:0/1325631224 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f063c099ca0 con 0x7f064410d430 2024-03-30T04:07:22.349 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:07:22.349 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:07:22.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 msgr2=0x7f06280a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 0x7f06280a3750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0634009d20 tx=0x7f0634009260 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 msgr2=0x7f064419dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064419dc40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f063c00efc0 tx=0x7f063c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 shutdown_connections 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f06280a1290 0x7f06280a3750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f064410d430 0x7f064419dc40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06440731e0 0x7f0644199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 --2- 172.21.15.23:0/1325631224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0644073b00 0x7f06441998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 >> 172.21.15.23:0/1325631224 conn(0x7f06440fc760 msgr2=0x7f06440fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 shutdown_connections 2024-03-30T04:07:22.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.353+0000 7f0649800700 1 -- 172.21.15.23:0/1325631224 wait complete. 2024-03-30T04:07:22.353 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:07:22.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:07:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:22 smithi154 ceph-mon[79161]: pgmap v165: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-03-30T04:07:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:22 smithi193 ceph-mon[79581]: pgmap v165: 129 pgs: 129 active+clean; 292 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.5 MiB/s wr, 206 op/s 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 -- 172.21.15.23:0/3495307694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 msgr2=0x7fabc01060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 --2- 172.21.15.23:0/3495307694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01060c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fabb40099c0 tx=0x7fabb4009cd0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 -- 172.21.15.23:0/3495307694 shutdown_connections 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 --2- 172.21.15.23:0/3495307694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 --2- 172.21.15.23:0/3495307694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01060c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 --2- 172.21.15.23:0/3495307694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 0x7fabc00682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 -- 172.21.15.23:0/3495307694 >> 172.21.15.23:0/3495307694 conn(0x7fabc0076d40 msgr2=0x7fabc0077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 -- 172.21.15.23:0/3495307694 shutdown_connections 2024-03-30T04:07:22.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.589+0000 7fabc7600700 1 -- 172.21.15.23:0/3495307694 wait complete. 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 Processor -- start 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 -- start start 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 0x7fabc00ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01a5620 con 0x7fabc00687f0 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01a5790 con 0x7fabc0105ce0 2024-03-30T04:07:22.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.591+0000 7fabc7600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01a5950 con 0x7fabc0067e30 2024-03-30T04:07:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37356/0 (socket says 172.21.15.23:37356) 2024-03-30T04:07:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 -- 172.21.15.23:0/61917635 learned_addr learned my addr 172.21.15.23:0/61917635 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc4c00700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 0x7fabc00ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 -- 172.21.15.23:0/61917635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 msgr2=0x7fabc00ffc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 0x7fabc00ffc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 -- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 msgr2=0x7fabc01001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc01001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 -- 172.21.15.23:0/61917635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb8009c40 con 0x7fabc0105ce0 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabbfe00700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.592+0000 7fabc5600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fabb800c370 tx=0x7fabb800c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.593+0000 7fabbd600700 1 -- 172.21.15.23:0/61917635 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8017070 con 0x7fabc0105ce0 2024-03-30T04:07:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.593+0000 7fabbd600700 1 -- 172.21.15.23:0/61917635 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb800f040 con 0x7fabc0105ce0 2024-03-30T04:07:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.593+0000 7fabbd600700 1 -- 172.21.15.23:0/61917635 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb800e050 con 0x7fabc0105ce0 2024-03-30T04:07:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.593+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb40095f0 con 0x7fabc0105ce0 2024-03-30T04:07:22.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.593+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc01a5f50 con 0x7fabc0105ce0 2024-03-30T04:07:22.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.596+0000 7fabbd600700 1 -- 172.21.15.23:0/61917635 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fabb800cb60 con 0x7fabc0105ce0 2024-03-30T04:07:22.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.598+0000 7fabdb1dfb80 1 Processor -- start 2024-03-30T04:07:22.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.598+0000 7fabdb1dfb80 1 -- start start 2024-03-30T04:07:22.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.598+0000 7fabbd600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 0x7faba80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabbd600700 1 -- 172.21.15.23:0/61917635 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fabb80cc970 con 0x7fabc0105ce0 2024-03-30T04:07:22.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b8374b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabbfe00700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 0x7faba80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b837568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b836a4990 con 0x555b8374b5d0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b836a2350 con 0x555b8374bef0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b8369dc40 con 0x555b83756df0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b837568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabc4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b8374b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b837568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37362/0 (socket says 172.21.15.23:37362) 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabbfe00700 1 -- 172.21.15.23:0/560117460 learned_addr learned my addr 172.21.15.23:0/560117460 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.599+0000 7fabc5600700 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 -- 172.21.15.23:0/560117460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 msgr2=0x555b837591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 -- 172.21.15.23:0/560117460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 msgr2=0x555b837568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b837568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 -- 172.21.15.23:0/560117460 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b8351f3b0 con 0x555b8374b5d0 2024-03-30T04:07:22.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc5600700 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabbfe00700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 0x7faba80a37d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fabac014a10 tx=0x7fabac014dd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabc4c00700 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b8374b9b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fabb40098e0 tx=0x7fabb4005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7931e5ed9142df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabb1c00700 1 -- 172.21.15.23:0/560117460 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb401d030 con 0x555b8374b5d0 2024-03-30T04:07:22.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabb1c00700 1 -- 172.21.15.23:0/560117460 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb401e040 con 0x555b8374b5d0 2024-03-30T04:07:22.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.600+0000 7fabb1c00700 1 -- 172.21.15.23:0/560117460 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40036e0 con 0x555b8374b5d0 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/560117460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 msgr2=0x555b8374b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b8374b9b0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fabb40098e0 tx=0x7fabb4005c70 comp rx=0 tx=0).stop 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/560117460 shutdown_connections 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b8374b9b0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/560117460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b837568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/560117460 >> 172.21.15.23:0/560117460 conn(0x555b836ab6a0 msgr2=0x555b83747a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/560117460 shutdown_connections 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.601+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/560117460 wait complete. 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.602+0000 7fabdb1dfb80 1 -- start start 2024-03-30T04:07:22.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.602+0000 7fabdb1dfb80 1 Processor -- start 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b837567a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374bef0 0x555b83753ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b83639d80 con 0x555b8374b5d0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b83639a20 con 0x555b83756df0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b835216a0 con 0x555b8374bef0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabdb1dfb80 10 client.0 _ll_register_callbacks cb 0x555b83643470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b837567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37374/0 (socket says 172.21.15.23:37374) 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b837567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46094/0 (socket says 172.21.15.23:46094) 2024-03-30T04:07:22.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 -- 172.21.15.23:0/589077137 learned_addr learned my addr 172.21.15.23:0/589077137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabbfe00700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374bef0 0x555b83753ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 -- 172.21.15.23:0/589077137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374bef0 msgr2=0x555b83753ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374bef0 0x555b83753ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 -- 172.21.15.23:0/589077137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 msgr2=0x555b837567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b837567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.603+0000 7fabc5600700 1 -- 172.21.15.23:0/589077137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc01a5f50 con 0x555b83756df0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7fabc5600700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fabb8095bc0 tx=0x7fabb80990f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7320023ba866d608 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8099dc0 con 0x555b83756df0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fabb80996c0 con 0x555b83756df0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555b8365e380 con 0x555b83756df0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb809d4f0 con 0x555b83756df0 2024-03-30T04:07:22.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7fabdb1dfb80 10 client.27029 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fabc010e550 con 0x555b83756df0 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555b8365e380 con 0x555b83756df0 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.604+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fabb8099a40 con 0x555b83756df0 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 10 client.27029 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 10 client.27029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 20 client.27029 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b83766a00 0x555b83768e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 4 client.27029 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b83696700 con 0x555b83766a00 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.605+0000 7fabc4c00700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b83766a00 0x555b83768e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.606+0000 7faba6000700 10 client.27029 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:22.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.606+0000 7fabc4c00700 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b83766a00 0x555b83768e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a5540ea3cdcc9c89 server_cookie=c26f8ea9ca43516d in_seq=0 out_seq=0 2024-03-30T04:07:22.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.606+0000 7faba6000700 1 -- 172.21.15.23:0/589077137 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555b83696700 con 0x555b83766a00 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.606+0000 7faba6000700 10 client.27029 handle_command_reply: tid=0 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.606+0000 7fabdb1dfb80 1 client.27029 shutdown 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 msgr2=0x555b83754110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fabb8095bc0 tx=0x7fabb80990f0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 shutdown_connections 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b83756df0 0x555b83754110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b837567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b83766a00 0x555b83768e20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/589077137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374bef0 0x555b83753ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 >> 172.21.15.23:0/589077137 conn(0x555b836ab6a0 msgr2=0x555b8374d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7faba6000700 0 client.27029 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 shutdown_connections 2024-03-30T04:07:22.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/589077137 wait complete. 2024-03-30T04:07:22.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.607+0000 7fabdb1dfb80 20 client.27029 trim_cache size 0 max 16384 2024-03-30T04:07:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.621+0000 7fabdb1dfb80 1 Processor -- start 2024-03-30T04:07:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 -- start start 2024-03-30T04:07:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b83754030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b8374c850 con 0x555b8374b5d0 2024-03-30T04:07:22.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b837691b0 con 0x555b8374bef0 2024-03-30T04:07:22.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b8375df50 con 0x555b83756df0 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabc4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b83754030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37382/0 (socket says 172.21.15.23:37382) 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabbfe00700 1 -- 172.21.15.23:0/215102339 learned_addr learned my addr 172.21.15.23:0/215102339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.622+0000 7fabc5600700 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 -- 172.21.15.23:0/215102339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 msgr2=0x555b837405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 -- 172.21.15.23:0/215102339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 msgr2=0x555b8375da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 -- 172.21.15.23:0/215102339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b834cbb60 con 0x555b8374b5d0 2024-03-30T04:07:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabc4c00700 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b83754030 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fabb4000c00 tx=0x7fabb4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efc3ccfe80a6256 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabb2600700 1 -- 172.21.15.23:0/215102339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb401d040 con 0x555b8374b5d0 2024-03-30T04:07:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabb2600700 1 -- 172.21.15.23:0/215102339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb400b540 con 0x555b8374b5d0 2024-03-30T04:07:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.623+0000 7fabb2600700 1 -- 172.21.15.23:0/215102339 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4003be0 con 0x555b8374b5d0 2024-03-30T04:07:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/215102339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 msgr2=0x555b83754030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b83754030 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fabb4000c00 tx=0x7fabb4009310 comp rx=0 tx=0).stop 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/215102339 shutdown_connections 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b83756df0 0x555b837405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b8374b5d0 0x555b83754030 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/215102339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/215102339 >> 172.21.15.23:0/215102339 conn(0x555b836ab6a0 msgr2=0x555b83740f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/215102339 shutdown_connections 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.624+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/215102339 wait complete. 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 -- start start 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 Processor -- start 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 0x555b8375b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 0x555b837b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b837b9d30 con 0x555b83756df0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b837b9e70 con 0x555b8374bef0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b837b9fb0 con 0x555b8374b5d0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.625+0000 7fabdb1dfb80 10 client.0 _ll_register_callbacks cb 0x555b83643470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabc5600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 0x555b837b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37386/0 (socket says 172.21.15.23:37386) 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 -- 172.21.15.23:0/1355853162 learned_addr learned my addr 172.21.15.23:0/1355853162 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabc4c00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 0x555b8375b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 -- 172.21.15.23:0/1355853162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 msgr2=0x555b8375b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 0x555b8375b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 -- 172.21.15.23:0/1355853162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 msgr2=0x555b837b97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 0x555b837b97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 -- 172.21.15.23:0/1355853162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b8351f3b0 con 0x555b8374bef0 2024-03-30T04:07:22.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabc4c00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 0x555b8375b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabc5600700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 0x555b837b97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7fabbfe00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fabac00be40 tx=0x7fabac01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a38402245143bf52 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.626+0000 7faba6000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac026070 con 0x555b8374bef0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fabac004020 con 0x555b8374bef0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabb401e040 con 0x555b8374bef0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabac0041e0 con 0x555b8374bef0 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7fabdb1dfb80 10 client.27035 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:22.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.627+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555b83663ea0 con 0x555b8374bef0 2024-03-30T04:07:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb401e040 con 0x555b8374bef0 2024-03-30T04:07:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fabac020440 con 0x555b8374bef0 2024-03-30T04:07:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 10 client.27035 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 10 client.27035 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:07:22.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 20 client.27035 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b837bb7d0 0x555b837bdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 4 client.27035 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555b834eccf0 con 0x555b837bb7d0 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.628+0000 7fabc4c00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b837bb7d0 0x555b837bdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabc4c00700 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b837bb7d0 0x555b837bdbf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2f1f75c1385a806c server_cookie=46155cf9b0c1db1b in_seq=0 out_seq=0 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7faba6000700 10 client.27035 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7faba6000700 1 -- 172.21.15.23:0/1355853162 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555b834eccf0 con 0x555b837bb7d0 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7faba6000700 10 client.27035 handle_command_reply: tid=0 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 client.27035 shutdown 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 msgr2=0x555b8375bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fabac00be40 tx=0x7fabac01a000 comp rx=0 tx=0).stop 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 shutdown_connections 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b83756df0 0x555b837b97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555b837bb7d0 0x555b837bdbf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b8374b5d0 0x555b8375b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 --2- 172.21.15.23:0/1355853162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555b8374bef0 0x555b8375bc50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 >> 172.21.15.23:0/1355853162 conn(0x555b836ab6a0 msgr2=0x555b837405a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7faba6000700 0 client.27035 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 shutdown_connections 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 1 -- 172.21.15.23:0/1355853162 wait complete. 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.629+0000 7fabdb1dfb80 20 client.27035 trim_cache size 0 max 16384 2024-03-30T04:07:22.632 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 msgr2=0x7faba80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 0x7faba80a37d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fabac014a10 tx=0x7fabac014dd0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 msgr2=0x7fabc01006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fabb800c370 tx=0x7fabb800c680 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 shutdown_connections 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7faba80a1310 0x7faba80a37d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabc0105ce0 0x7fabc01006f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fabc0067e30 0x7fabc00ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 --2- 172.21.15.23:0/61917635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabc00687f0 0x7fabc01001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 >> 172.21.15.23:0/61917635 conn(0x7fabc0076d40 msgr2=0x7fabc00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 shutdown_connections 2024-03-30T04:07:22.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.630+0000 7fabc7600700 1 -- 172.21.15.23:0/61917635 wait complete. 2024-03-30T04:07:22.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:07:22.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.896+0000 7f008fe00700 1 -- 172.21.15.23:0/3924562264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088106070 msgr2=0x7f008810dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.896+0000 7f008fe00700 1 --2- 172.21.15.23:0/3924562264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088106070 0x7f008810dbf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f008000b0a0 tx=0x7f008000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 -- 172.21.15.23:0/3924562264 shutdown_connections 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 --2- 172.21.15.23:0/3924562264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088106070 0x7f008810dbf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 --2- 172.21.15.23:0/3924562264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088100f50 0x7f0088101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 --2- 172.21.15.23:0/3924562264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0088101900 0x7f0088105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 -- 172.21.15.23:0/3924562264 >> 172.21.15.23:0/3924562264 conn(0x7f00880fc7f0 msgr2=0x7f00880fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 -- 172.21.15.23:0/3924562264 shutdown_connections 2024-03-30T04:07:22.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.897+0000 7f008fe00700 1 -- 172.21.15.23:0/3924562264 wait complete. 2024-03-30T04:07:22.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.898+0000 7f008fe00700 1 Processor -- start 2024-03-30T04:07:22.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.898+0000 7f008fe00700 1 -- start start 2024-03-30T04:07:22.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 0x7f0088197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 0x7f008819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008819c670 con 0x7f0088106070 2024-03-30T04:07:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008819c7e0 con 0x7f008819bbf0 2024-03-30T04:07:22.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008819c9a0 con 0x7f0088100f50 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59206/0 (socket says 172.21.15.23:59206) 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 -- 172.21.15.23:0/2022430880 learned_addr learned my addr 172.21.15.23:0/2022430880 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008de00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 0x7f008819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008ca00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 0x7f0088197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 -- 172.21.15.23:0/2022430880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 msgr2=0x7f008819c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 0x7f008819c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.899+0000 7f008d400700 1 -- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 msgr2=0x7f0088197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008d400700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 0x7f0088197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008d400700 1 -- 172.21.15.23:0/2022430880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0080009d00 con 0x7f0088100f50 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008de00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 0x7f008819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008ca00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 0x7f0088197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008d400700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f007c00b570 tx=0x7f007c00b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f0086000700 1 -- 172.21.15.23:0/2022430880 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c019070 con 0x7f0088100f50 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f0086000700 1 -- 172.21.15.23:0/2022430880 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f007c014e40 con 0x7f0088100f50 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f0086000700 1 -- 172.21.15.23:0/2022430880 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c01d940 con 0x7f0088100f50 2024-03-30T04:07:22.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.900+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0088073040 con 0x7f0088100f50 2024-03-30T04:07:22.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.901+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0088073500 con 0x7f0088100f50 2024-03-30T04:07:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.906+0000 7f0086000700 1 -- 172.21.15.23:0/2022430880 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f007c01b520 con 0x7f0088100f50 2024-03-30T04:07:22.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.906+0000 7f00a3a06b80 1 Processor -- start 2024-03-30T04:07:22.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 -- start start 2024-03-30T04:07:22.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f49701e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f48bb990 con 0x5633f496ddf0 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f48b9350 con 0x5633f49625d0 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f48b4c40 con 0x5633f4962ef0 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f008de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f008de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37402/0 (socket says 172.21.15.23:37402) 2024-03-30T04:07:22.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.907+0000 7f008de00700 1 -- 172.21.15.23:0/1001141762 learned_addr learned my addr 172.21.15.23:0/1001141762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008d400700 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008de00700 1 -- 172.21.15.23:0/1001141762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 msgr2=0x5633f496d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008de00700 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008de00700 1 -- 172.21.15.23:0/1001141762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 msgr2=0x5633f49701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008de00700 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f49701e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.908+0000 7f008de00700 1 -- 172.21.15.23:0/1001141762 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633f47363b0 con 0x5633f49625d0 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f008de00700 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f008000b0a0 tx=0x7f0080007990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e06fed4ea6cd7c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f007a600700 1 -- 172.21.15.23:0/1001141762 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008000e040 con 0x5633f49625d0 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f0086000700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 0x7f00700a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f007a600700 1 -- 172.21.15.23:0/1001141762 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f008000ba50 con 0x5633f49625d0 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f007a600700 1 -- 172.21.15.23:0/1001141762 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008001c9c0 con 0x5633f49625d0 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f0086000700 1 -- 172.21.15.23:0/2022430880 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f007c0d4b90 con 0x7f0088100f50 2024-03-30T04:07:22.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.909+0000 7f008d400700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 0x7f00700a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1001141762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 msgr2=0x5633f49629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f008000b0a0 tx=0x7f0080007990 comp rx=0 tx=0).stop 2024-03-30T04:07:22.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1001141762 shutdown_connections 2024-03-30T04:07:22.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f49701e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f49629b0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1001141762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1001141762 >> 172.21.15.23:0/1001141762 conn(0x5633f48c26a0 msgr2=0x5633f495ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f008d400700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 0x7f00700a38a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f007c0a18b0 tx=0x7f007c01c000 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1001141762 shutdown_connections 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.910+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1001141762 wait complete. 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.911+0000 7f00a3a06b80 1 -- start start 2024-03-30T04:07:22.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.911+0000 7f00a3a06b80 1 Processor -- start 2024-03-30T04:07:22.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f496ddf0 0x5633f496b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4850d80 con 0x5633f49625d0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4850a20 con 0x5633f496ddf0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f47386a0 con 0x5633f4962ef0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.912+0000 7f00a3a06b80 10 client.0 _ll_register_callbacks cb 0x5633f485a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.913+0000 7f008de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.913+0000 7f008de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46140/0 (socket says 172.21.15.23:46140) 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.913+0000 7f008de00700 1 -- 172.21.15.23:0/1800590691 learned_addr learned my addr 172.21.15.23:0/1800590691 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.913+0000 7f008ca00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.913+0000 7f008de00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f496ddf0 0x5633f496b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 -- 172.21.15.23:0/1800590691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f496ddf0 msgr2=0x5633f496b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f496ddf0 0x5633f496b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 -- 172.21.15.23:0/1800590691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 msgr2=0x5633f496d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 -- 172.21.15.23:0/1800590691 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008810e460 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008de00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f008ca00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496ab10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0088198860 tx=0x7f00740106b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd9b47e944b215ad server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.914+0000 7f006ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0074004050 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f00740041b0 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5633f4875380 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0074004370 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f00a3a06b80 10 client.27046 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5633f46e2b60 con 0x5633f4962ef0 2024-03-30T04:07:22.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.915+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5633f4875380 con 0x5633f4962ef0 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0074011560 con 0x5633f4962ef0 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 10 client.27046 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 10 client.27046 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 20 client.27046 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f497da70 0x5633f497fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 4 client.27046 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5633f48ad700 con 0x5633f497da70 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f008de00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f497da70 0x5633f497fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f008de00700 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f497da70 0x5633f497fe90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6a9418089750ce9 server_cookie=d788ca744d2790c0 in_seq=0 out_seq=0 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.916+0000 7f006ea00700 10 client.27046 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.917+0000 7f006ea00700 1 -- 172.21.15.23:0/1800590691 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5633f48ad700 con 0x5633f497da70 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.917+0000 7f006ea00700 10 client.27046 handle_command_reply: tid=0 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.917+0000 7f00a3a06b80 1 client.27046 shutdown 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.917+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 msgr2=0x5633f496ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.917+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496ab10 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0088198860 tx=0x7f00740106b0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 shutdown_connections 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f497da70 0x5633f497fe90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f496ddf0 0x5633f496b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f49625d0 0x5633f496d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1800590691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f496ab10 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 >> 172.21.15.23:0/1800590691 conn(0x5633f48c26a0 msgr2=0x5633f49646c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f006ea00700 0 client.27046 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:22.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 shutdown_connections 2024-03-30T04:07:22.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1800590691 wait complete. 2024-03-30T04:07:22.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.918+0000 7f00a3a06b80 20 client.27046 trim_cache size 0 max 16384 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.933+0000 7f00a3a06b80 1 Processor -- start 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.933+0000 7f00a3a06b80 1 -- start start 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.933+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f4974a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4957f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4963850 con 0x5633f496ddf0 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4974f50 con 0x5633f49625d0 2024-03-30T04:07:22.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4969080 con 0x5633f4962ef0 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37426/0 (socket says 172.21.15.23:37426) 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 -- 172.21.15.23:0/1365709421 learned_addr learned my addr 172.21.15.23:0/1365709421 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4957f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008ca00700 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f4974a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 -- 172.21.15.23:0/1365709421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 msgr2=0x5633f4974a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f4974a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 -- 172.21.15.23:0/1365709421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 msgr2=0x5633f4957f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4957f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 -- 172.21.15.23:0/1365709421 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5633f487aea0 con 0x5633f49625d0 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.934+0000 7f008de00700 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f00800062a0 tx=0x7f008001c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a7891e5398a591 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f007b000700 1 -- 172.21.15.23:0/1365709421 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008003c040 con 0x5633f49625d0 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f007b000700 1 -- 172.21.15.23:0/1365709421 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0080040890 con 0x5633f49625d0 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f007b000700 1 -- 172.21.15.23:0/1365709421 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0080040730 con 0x5633f49625d0 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1365709421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 msgr2=0x5633f4965550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f00800062a0 tx=0x7f008001c580 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1365709421 shutdown_connections 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4957f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f49625d0 0x5633f4965550 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/1365709421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f4962ef0 0x5633f4974a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.935+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1365709421 >> 172.21.15.23:0/1365709421 conn(0x5633f48c26a0 msgr2=0x5633f49575a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.936+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1365709421 shutdown_connections 2024-03-30T04:07:22.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.936+0000 7f00a3a06b80 1 -- 172.21.15.23:0/1365709421 wait complete. 2024-03-30T04:07:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.936+0000 7f00a3a06b80 1 -- start start 2024-03-30T04:07:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.936+0000 7f00a3a06b80 1 Processor -- start 2024-03-30T04:07:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f49625d0 0x5633f4974a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4972f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4973550 con 0x5633f496ddf0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f49736c0 con 0x5633f4962ef0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5633f4973880 con 0x5633f49625d0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f00a3a06b80 10 client.0 _ll_register_callbacks cb 0x5633f485a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f008ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f008ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37430/0 (socket says 172.21.15.23:37430) 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f008ca00700 1 -- 172.21.15.23:0/3319720559 learned_addr learned my addr 172.21.15.23:0/3319720559 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f008de00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f49625d0 0x5633f4974a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.937+0000 7f008de00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4972f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 -- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f49625d0 msgr2=0x5633f4974a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f49625d0 0x5633f4974a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 -- 172.21.15.23:0/3319720559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 msgr2=0x5633f4972f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4972f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 -- 172.21.15.23:0/3319720559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5633f47363b0 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008de00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4972f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f008ca00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0088198860 tx=0x7f00740114f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b387a341cdc2bc6c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f006ea00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00740050c0 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f008000ba50 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0074000e40 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0074005220 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f00a3a06b80 10 client.27053 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.938+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5633f473acd0 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f008000ba50 con 0x5633f4962ef0 2024-03-30T04:07:22.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0074003b80 con 0x5633f4962ef0 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 10 client.27053 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 10 client.27053 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 20 client.27053 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f49d1f30 0x5633f49d4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 4 client.27053 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5633f4703cf0 con 0x5633f49d1f30 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.939+0000 7f008de00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f49d1f30 0x5633f49d4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f008de00700 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f49d1f30 0x5633f49d4350 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=22ff4f22c34c2eab server_cookie=8ab3a3740493f559 in_seq=0 out_seq=0 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f006ea00700 10 client.27053 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f006ea00700 1 -- 172.21.15.23:0/3319720559 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5633f4703cf0 con 0x5633f49d1f30 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f006ea00700 10 client.27053 handle_command_reply: tid=0 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 client.27053 shutdown 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 msgr2=0x5633f49729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0088198860 tx=0x7f00740114f0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 shutdown_connections 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5633f49d1f30 0x5633f49d4350 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5633f496ddf0 0x5633f4972f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f006ea00700 0 client.27053 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5633f49625d0 0x5633f4974a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 --2- 172.21.15.23:0/3319720559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5633f4962ef0 0x5633f49729f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 >> 172.21.15.23:0/3319720559 conn(0x5633f48c26a0 msgr2=0x5633f4957f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 shutdown_connections 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.940+0000 7f00a3a06b80 1 -- 172.21.15.23:0/3319720559 wait complete. 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f00a3a06b80 20 client.27053 trim_cache size 0 max 16384 2024-03-30T04:07:22.943 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:22.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 msgr2=0x7f00700a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 0x7f00700a38a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f007c0a18b0 tx=0x7f007c01c000 comp rx=0 tx=0).stop 2024-03-30T04:07:22.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 msgr2=0x7f0088197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f007c00b570 tx=0x7f007c00b880 comp rx=0 tx=0).stop 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 shutdown_connections 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f00700a13e0 0x7f00700a38a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f008819bbf0 0x7f008819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0088100f50 0x7f0088197310 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 --2- 172.21.15.23:0/2022430880 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0088106070 0x7f0088197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.941+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 >> 172.21.15.23:0/2022430880 conn(0x7f00880fc7f0 msgr2=0x7f00880feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.942+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 shutdown_connections 2024-03-30T04:07:22.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:22.942+0000 7f008fe00700 1 -- 172.21.15.23:0/2022430880 wait complete. 2024-03-30T04:07:22.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:07:23.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 -- 172.21.15.23:0/4128771070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8028068fd0 msgr2=0x7f80280693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 --2- 172.21.15.23:0/4128771070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8028068fd0 0x7f80280693b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f80180098f0 tx=0x7f8018009c00 comp rx=0 tx=0).stop 2024-03-30T04:07:23.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 -- 172.21.15.23:0/4128771070 shutdown_connections 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 --2- 172.21.15.23:0/4128771070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f802810d750 0x7f802810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 --2- 172.21.15.23:0/4128771070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8028068fd0 0x7f80280693b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 --2- 172.21.15.23:0/4128771070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f80280698f0 0x7f802810d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.191+0000 7f802e200700 1 -- 172.21.15.23:0/4128771070 >> 172.21.15.23:0/4128771070 conn(0x7f8028076d80 msgr2=0x7f8028077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.192+0000 7f802e200700 1 -- 172.21.15.23:0/4128771070 shutdown_connections 2024-03-30T04:07:23.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.192+0000 7f802e200700 1 -- 172.21.15.23:0/4128771070 wait complete. 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 Processor -- start 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 -- start start 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8028068fd0 0x7f8028199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f802810d750 0x7f802819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802819e320 con 0x7f8028068fd0 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802819e490 con 0x7f80280698f0 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f802e200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802819e650 con 0x7f802810d750 2024-03-30T04:07:23.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37442/0 (socket says 172.21.15.23:37442) 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f8027400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8028068fd0 0x7f8028199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f8026a00700 1 -- 172.21.15.23:0/1889092801 learned_addr learned my addr 172.21.15.23:0/1889092801 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.193+0000 7f8027e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f802810d750 0x7f802819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 -- 172.21.15.23:0/1889092801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f802810d750 msgr2=0x7f802819dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f802810d750 0x7f802819dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 -- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8028068fd0 msgr2=0x7f8028199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8028068fd0 0x7f8028199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 -- 172.21.15.23:0/1889092801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80140095f0 con 0x7f80280698f0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f8026a00700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f801400ec10 tx=0x7f801400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f801fe00700 1 -- 172.21.15.23:0/1889092801 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801400cc90 con 0x7f80280698f0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f801fe00700 1 -- 172.21.15.23:0/1889092801 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f80140044e0 con 0x7f80280698f0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8018009520 con 0x7f80280698f0 2024-03-30T04:07:23.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.194+0000 7f801fe00700 1 -- 172.21.15.23:0/1889092801 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014010430 con 0x7f80280698f0 2024-03-30T04:07:23.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.195+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802819eb20 con 0x7f80280698f0 2024-03-30T04:07:23.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.198+0000 7f801fe00700 1 -- 172.21.15.23:0/1889092801 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8014003fb0 con 0x7f80280698f0 2024-03-30T04:07:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.199+0000 7f801fe00700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 0x7f800c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.199+0000 7f801fe00700 1 -- 172.21.15.23:0/1889092801 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8014014070 con 0x7f80280698f0 2024-03-30T04:07:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 Processor -- start 2024-03-30T04:07:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8027400700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 0x7f800c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 -- start start 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe045d0 0x5616bbe049b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 0x5616bbe0f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbd5d990 con 0x5616bbe045d0 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbd5b350 con 0x5616bbe0fdf0 2024-03-30T04:07:23.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8041d97b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbd56c40 con 0x5616bbe04ef0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.200+0000 7f8027e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37456/0 (socket says 172.21.15.23:37456) 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 -- 172.21.15.23:0/3716159589 learned_addr learned my addr 172.21.15.23:0/3716159589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8026a00700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 0x5616bbe0f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027400700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe045d0 0x5616bbe049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027400700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 0x7f800c0a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f80180098c0 tx=0x7f8018019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 -- 172.21.15.23:0/3716159589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 msgr2=0x5616bbe0f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 0x5616bbe0f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 -- 172.21.15.23:0/3716159589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe045d0 msgr2=0x5616bbe049b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe045d0 0x5616bbe049b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 -- 172.21.15.23:0/3716159589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616bbbd83b0 con 0x5616bbe0fdf0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8026a00700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 0x5616bbe0f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.201+0000 7f8027e00700 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8020011600 tx=0x7f8020009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fee87125211abcf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.202+0000 7f8013e00700 1 -- 172.21.15.23:0/3716159589 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000e040 con 0x5616bbe0fdf0 2024-03-30T04:07:23.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.202+0000 7f8013e00700 1 -- 172.21.15.23:0/3716159589 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802000f040 con 0x5616bbe0fdf0 2024-03-30T04:07:23.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.202+0000 7f8013e00700 1 -- 172.21.15.23:0/3716159589 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80200047a0 con 0x5616bbe0fdf0 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.202+0000 7f8041d97b80 1 -- 172.21.15.23:0/3716159589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 msgr2=0x5616bbe121e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.202+0000 7f8041d97b80 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8020011600 tx=0x7f8020009920 comp rx=0 tx=0).stop 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 -- 172.21.15.23:0/3716159589 shutdown_connections 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe0fdf0 0x5616bbe121e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe045d0 0x5616bbe049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 --2- 172.21.15.23:0/3716159589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe04ef0 0x5616bbe0f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 -- 172.21.15.23:0/3716159589 >> 172.21.15.23:0/3716159589 conn(0x5616bbd646a0 msgr2=0x5616bbe00a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 -- 172.21.15.23:0/3716159589 shutdown_connections 2024-03-30T04:07:23.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 -- 172.21.15.23:0/3716159589 wait complete. 2024-03-30T04:07:23.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 516559 11.79 MB/sec execute 120 sec latency 18.470 ms 2024-03-30T04:07:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.203+0000 7f8041d97b80 1 -- start start 2024-03-30T04:07:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 Processor -- start 2024-03-30T04:07:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe0f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe0d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbcf2d80 con 0x5616bbe0fdf0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbcf2a20 con 0x5616bbe04ef0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbbda6a0 con 0x5616bbe045d0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.204+0000 7f8041d97b80 10 client.0 _ll_register_callbacks cb 0x5616bbcfc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe0f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe0d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe0f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59288/0 (socket says 172.21.15.23:59288) 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027400700 1 -- 172.21.15.23:0/1137446245 learned_addr learned my addr 172.21.15.23:0/1137446245 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 -- 172.21.15.23:0/1137446245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 msgr2=0x5616bbe0f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe0f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 -- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 msgr2=0x5616bbe0cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 -- 172.21.15.23:0/1137446245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802819eb20 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8026a00700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8027e00700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe0d0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f802000b010 tx=0x7f8020011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311e177398b39791 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.205+0000 7f8010c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000e070 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5616bbd17380 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f80200048b0 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8020003c10 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8041d97b80 10 client.17076 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f80200112b0 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5616bbd17380 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8020027460 con 0x5616bbe0fdf0 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8041d97b80 10 client.17076 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.206+0000 7f8041d97b80 10 client.17076 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8041d97b80 20 client.17076 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe1f870 0x5616bbe21c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8041d97b80 4 client.17076 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616bbd4f700 con 0x5616bbe1f870 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8027400700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe1f870 0x5616bbe21c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8027400700 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe1f870 0x5616bbe21c90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7fc53e0a5dc519d7 server_cookie=2d91fe896174a24a in_seq=0 out_seq=0 2024-03-30T04:07:23.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.207+0000 7f8010c00700 10 client.17076 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:23.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.209+0000 7f8010c00700 1 -- 172.21.15.23:0/1137446245 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5616bbd4f700 con 0x5616bbe1f870 2024-03-30T04:07:23.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.209+0000 7f8010c00700 10 client.17076 handle_command_reply: tid=0 2024-03-30T04:07:23.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.209+0000 7f8041d97b80 1 client.17076 shutdown 2024-03-30T04:07:23.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.209+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 msgr2=0x5616bbe0d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.209+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe0d0e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f802000b010 tx=0x7f8020011f60 comp rx=0 tx=0).stop 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 shutdown_connections 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe1f870 0x5616bbe21c90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe0d0e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe0f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1137446245 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8010c00700 0 client.17076 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 >> 172.21.15.23:0/1137446245 conn(0x5616bbd646a0 msgr2=0x5616bbe06530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 shutdown_connections 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 1 -- 172.21.15.23:0/1137446245 wait complete. 2024-03-30T04:07:23.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.210+0000 7f8041d97b80 20 client.17076 trim_cache size 0 max 16384 2024-03-30T04:07:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.226+0000 7f8041d97b80 1 Processor -- start 2024-03-30T04:07:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 -- start start 2024-03-30T04:07:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe21c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe16a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe05850 con 0x5616bbe0fdf0 2024-03-30T04:07:23.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe01660 con 0x5616bbe04ef0 2024-03-30T04:07:23.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8041d97b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe0c960 con 0x5616bbe045d0 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8027400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe21c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37466/0 (socket says 172.21.15.23:37466) 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.227+0000 7f8026a00700 1 -- 172.21.15.23:0/260054176 learned_addr learned my addr 172.21.15.23:0/260054176 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8027e00700 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe16a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 -- 172.21.15.23:0/260054176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 msgr2=0x5616bbe21c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe21c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 -- 172.21.15.23:0/260054176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 msgr2=0x5616bbe16a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe16a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 -- 172.21.15.23:0/260054176 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616bbb84b60 con 0x5616bbe04ef0 2024-03-30T04:07:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f8026a00700 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8014085100 tx=0x7f8014099d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c91575a61b857ae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f801d200700 1 -- 172.21.15.23:0/260054176 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8014016330 con 0x5616bbe04ef0 2024-03-30T04:07:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f801d200700 1 -- 172.21.15.23:0/260054176 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f801409a9d0 con 0x5616bbe04ef0 2024-03-30T04:07:23.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.228+0000 7f801d200700 1 -- 172.21.15.23:0/260054176 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801409a530 con 0x5616bbe04ef0 2024-03-30T04:07:23.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 -- 172.21.15.23:0/260054176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 msgr2=0x5616bbe0c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f8014085100 tx=0x7f8014099d00 comp rx=0 tx=0).stop 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 -- 172.21.15.23:0/260054176 shutdown_connections 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe16a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe21c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 --2- 172.21.15.23:0/260054176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe0c420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.229+0000 7f8041d97b80 1 -- 172.21.15.23:0/260054176 >> 172.21.15.23:0/260054176 conn(0x5616bbd646a0 msgr2=0x5616bbdf95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.230+0000 7f8041d97b80 1 -- 172.21.15.23:0/260054176 shutdown_connections 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.230+0000 7f8041d97b80 1 -- 172.21.15.23:0/260054176 wait complete. 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.230+0000 7f8041d97b80 1 -- start start 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.230+0000 7f8041d97b80 1 Processor -- start 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe14860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe74eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe753f0 con 0x5616bbe0fdf0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe75560 con 0x5616bbe04ef0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bbe75720 con 0x5616bbe045d0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8041d97b80 10 client.0 _ll_register_callbacks cb 0x5616bbcfc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8027400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe14860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8027e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe74eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37478/0 (socket says 172.21.15.23:37478) 2024-03-30T04:07:23.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 -- 172.21.15.23:0/1282790057 learned_addr learned my addr 172.21.15.23:0/1282790057 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 -- 172.21.15.23:0/1282790057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 msgr2=0x5616bbe14860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe14860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 -- 172.21.15.23:0/1282790057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 msgr2=0x5616bbe74eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.231+0000 7f8026a00700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe74eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8026a00700 1 -- 172.21.15.23:0/1282790057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8018009520 con 0x5616bbe04ef0 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8027400700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe14860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8026a00700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5616bbe017a0 tx=0x7f8014016330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6965e97b9df1648e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8010c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f801409a2d0 con 0x5616bbe04ef0 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f80140a2750 con 0x5616bbe04ef0 2024-03-30T04:07:23.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f802000f040 con 0x5616bbe04ef0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80140997c0 con 0x5616bbe04ef0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8041d97b80 10 client.27071 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.232+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5616bbd1cea0 con 0x5616bbe04ef0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f802000f040 con 0x5616bbe04ef0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f80140b1030 con 0x5616bbe04ef0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 10 client.27071 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 10 client.27071 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 20 client.27071 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe7caf0 0x5616bbe7ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 4 client.27071 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5616bbba5cf0 con 0x5616bbe7caf0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8027400700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe7caf0 0x5616bbe7ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8010c00700 10 client.27071 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:23.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.233+0000 7f8027400700 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe7caf0 0x5616bbe7ef10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d129fb02f1a3c77 server_cookie=ff36348a8291e774 in_seq=0 out_seq=0 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8010c00700 1 -- 172.21.15.23:0/1282790057 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5616bbba5cf0 con 0x5616bbe7caf0 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8010c00700 10 client.27071 handle_command_reply: tid=0 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 client.27071 shutdown 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 msgr2=0x5616bbe14da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x5616bbe017a0 tx=0x7f8014016330 comp rx=0 tx=0).stop 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 shutdown_connections 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5616bbe7caf0 0x5616bbe7ef10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5616bbe0fdf0 0x5616bbe74eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8010c00700 0 client.27071 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616bbe045d0 0x5616bbe14860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 --2- 172.21.15.23:0/1282790057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5616bbe04ef0 0x5616bbe14da0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.234+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 >> 172.21.15.23:0/1282790057 conn(0x5616bbd646a0 msgr2=0x5616bbe07df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 shutdown_connections 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f8041d97b80 1 -- 172.21.15.23:0/1282790057 wait complete. 2024-03-30T04:07:23.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f8041d97b80 20 client.27071 trim_cache size 0 max 16384 2024-03-30T04:07:23.237 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:23.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 msgr2=0x7f800c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 0x7f800c0a37d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f80180098c0 tx=0x7f8018019040 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 msgr2=0x7f8028199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f801400ec10 tx=0x7f801400ef20 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 shutdown_connections 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.235+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f802810d750 0x7f802819dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8028068fd0 0x7f8028199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f800c0a1310 0x7f800c0a37d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 --2- 172.21.15.23:0/1889092801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80280698f0 0x7f8028199990 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 >> 172.21.15.23:0/1889092801 conn(0x7f8028076d80 msgr2=0x7f802810edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 shutdown_connections 2024-03-30T04:07:23.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.236+0000 7f802e200700 1 -- 172.21.15.23:0/1889092801 wait complete. 2024-03-30T04:07:23.252 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:23 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1325631224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:23.254 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:07:23.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:23 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1325631224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 -- 172.21.15.23:0/1803047706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8058073aa0 msgr2=0x7f805810d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 --2- 172.21.15.23:0/1803047706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8058073aa0 0x7f805810d040 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8044009990 tx=0x7f8044009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 -- 172.21.15.23:0/1803047706 shutdown_connections 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 --2- 172.21.15.23:0/1803047706 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805810d580 0x7f805810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 --2- 172.21.15.23:0/1803047706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 --2- 172.21.15.23:0/1803047706 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8058073aa0 0x7f805810d040 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 -- 172.21.15.23:0/1803047706 >> 172.21.15.23:0/1803047706 conn(0x7f80580fc7f0 msgr2=0x7f80580fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 -- 172.21.15.23:0/1803047706 shutdown_connections 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.490+0000 7f805ca00700 1 -- 172.21.15.23:0/1803047706 wait complete. 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.491+0000 7f805ca00700 1 Processor -- start 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.491+0000 7f805ca00700 1 -- start start 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 0x7f80581999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805810d580 0x7f805819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805819e390 con 0x7f805810d580 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805819e500 con 0x7f8058073aa0 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f805ca00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805819e6c0 con 0x7f8058073180 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f8055e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f8055400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 0x7f80581999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f8055e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59308/0 (socket says 172.21.15.23:59308) 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f8055e00700 1 -- 172.21.15.23:0/2821653719 learned_addr learned my addr 172.21.15.23:0/2821653719 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.492+0000 7f8056800700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805810d580 0x7f805819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 -- 172.21.15.23:0/2821653719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 msgr2=0x7f8058199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 -- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805810d580 msgr2=0x7f805819dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805810d580 0x7f805819dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 -- 172.21.15.23:0/2821653719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80440095f0 con 0x7f8058073aa0 2024-03-30T04:07:23.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:23 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1325631224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f8055400700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 0x7f80581999b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f80440098e0 tx=0x7f8044005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f804ea00700 1 -- 172.21.15.23:0/2821653719 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804401d070 con 0x7f8058073aa0 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805819e940 con 0x7f8058073aa0 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f804ea00700 1 -- 172.21.15.23:0/2821653719 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8044022470 con 0x7f8058073aa0 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f804ea00700 1 -- 172.21.15.23:0/2821653719 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044017650 con 0x7f8058073aa0 2024-03-30T04:07:23.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.493+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80581a61e0 con 0x7f8058073aa0 2024-03-30T04:07:23.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.496+0000 7f804ea00700 1 -- 172.21.15.23:0/2821653719 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f80440225e0 con 0x7f8058073aa0 2024-03-30T04:07:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.498+0000 7f804ea00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 0x7f803c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.498+0000 7f804ea00700 1 -- 172.21.15.23:0/2821653719 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f80440ce3c0 con 0x7f8058073aa0 2024-03-30T04:07:23.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.498+0000 7f8055e00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 0x7f803c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.498+0000 7f807056fb80 1 Processor -- start 2024-03-30T04:07:23.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.498+0000 7f807056fb80 1 -- start start 2024-03-30T04:07:23.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52159b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8055e00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 0x7f803c0a37d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8048009d20 tx=0x7f8048009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52208b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c516e990 con 0x5589c52155d0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c516c350 con 0x5589c5220df0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f807056fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5167c40 con 0x5589c5215ef0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8056800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8056800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37502/0 (socket says 172.21.15.23:37502) 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8056800700 1 -- 172.21.15.23:0/2886939592 learned_addr learned my addr 172.21.15.23:0/2886939592 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8055400700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52208b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.499+0000 7f8055e00700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 -- 172.21.15.23:0/2886939592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 msgr2=0x5589c52231e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 -- 172.21.15.23:0/2886939592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 msgr2=0x5589c52159b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52159b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 -- 172.21.15.23:0/2886939592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589c4fe93b0 con 0x5589c5215ef0 2024-03-30T04:07:23.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055e00700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:23.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8056800700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8055400700 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52208b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f80440cb750 tx=0x7f804409b090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=481251f2b53c181f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.500+0000 7f8043000700 1 -- 172.21.15.23:0/2886939592 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80440cbea0 con 0x5589c5215ef0 2024-03-30T04:07:23.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f8043000700 1 -- 172.21.15.23:0/2886939592 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f804409b9c0 con 0x5589c5215ef0 2024-03-30T04:07:23.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f8043000700 1 -- 172.21.15.23:0/2886939592 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044099770 con 0x5589c5215ef0 2024-03-30T04:07:23.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 -- 172.21.15.23:0/2886939592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 msgr2=0x5589c52208b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52208b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f80440cb750 tx=0x7f804409b090 comp rx=0 tx=0).stop 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 -- 172.21.15.23:0/2886939592 shutdown_connections 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c52231e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52159b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 --2- 172.21.15.23:0/2886939592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52208b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.501+0000 7f807056fb80 1 -- 172.21.15.23:0/2886939592 >> 172.21.15.23:0/2886939592 conn(0x5589c51756a0 msgr2=0x5589c5211a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.502+0000 7f807056fb80 1 -- 172.21.15.23:0/2886939592 shutdown_connections 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.502+0000 7f807056fb80 1 -- 172.21.15.23:0/2886939592 wait complete. 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.502+0000 7f807056fb80 1 -- start start 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.502+0000 7f807056fb80 1 Processor -- start 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52207c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c521e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5103d80 con 0x5589c52155d0 2024-03-30T04:07:23.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5103a20 con 0x5589c5220df0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c4feb6a0 con 0x5589c5215ef0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f807056fb80 10 client.0 _ll_register_callbacks cb 0x5589c510d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f8055400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f8055e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52207c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f8055400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59320/0 (socket says 172.21.15.23:59320) 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f8055400700 1 -- 172.21.15.23:0/508262012 learned_addr learned my addr 172.21.15.23:0/508262012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.503+0000 7f8056800700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c521e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 -- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 msgr2=0x5589c521e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c521e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 -- 172.21.15.23:0/508262012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 msgr2=0x5589c52207c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52207c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 -- 172.21.15.23:0/508262012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80500112b0 con 0x5589c5215ef0 2024-03-30T04:07:23.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055e00700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52207c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8055400700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f80440a3180 tx=0x7f8044009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ab2ae8c06f96bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8037400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80440cbea0 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f804409f0f0 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5589c5128380 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f804409b730 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f807056fb80 10 client.27082 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f80581a61e0 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.504+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5589c5128380 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f80440a1080 con 0x5589c5215ef0 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 10 client.27082 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 10 client.27082 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 20 client.27082 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c52301e0 0x5589c52325d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 4 client.27082 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:23.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5589c5160700 con 0x5589c52301e0 2024-03-30T04:07:23.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.505+0000 7f8055e00700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c52301e0 0x5589c52325d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.506+0000 7f8037400700 10 client.27082 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:23.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.506+0000 7f8055e00700 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c52301e0 0x5589c52325d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ea909a23ccd0cb4 server_cookie=526133130d3cd797 in_seq=0 out_seq=0 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.506+0000 7f8037400700 1 -- 172.21.15.23:0/508262012 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5589c5160700 con 0x5589c52301e0 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.506+0000 7f8037400700 10 client.27082 handle_command_reply: tid=0 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.506+0000 7f807056fb80 1 client.27082 shutdown 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 msgr2=0x5589c521db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f80440a3180 tx=0x7f8044009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 shutdown_connections 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c5220df0 0x5589c521e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c52155d0 0x5589c52207c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c52301e0 0x5589c52325d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 --2- 172.21.15.23:0/508262012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c521db00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 >> 172.21.15.23:0/508262012 conn(0x5589c51756a0 msgr2=0x5589c52176c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f8037400700 0 client.27082 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 shutdown_connections 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 1 -- 172.21.15.23:0/508262012 wait complete. 2024-03-30T04:07:23.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.507+0000 7f807056fb80 20 client.27082 trim_cache size 0 max 16384 2024-03-30T04:07:23.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.522+0000 7f807056fb80 1 Processor -- start 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 -- start start 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c520a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c521d8d0 con 0x5589c5220df0 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5227d20 con 0x5589c52155d0 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f807056fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c521c050 con 0x5589c5215ef0 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f8055e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f8056800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c520a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f8055e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37528/0 (socket says 172.21.15.23:37528) 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f8055e00700 1 -- 172.21.15.23:0/1628396778 learned_addr learned my addr 172.21.15.23:0/1628396778 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.523+0000 7f8055400700 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 -- 172.21.15.23:0/1628396778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 msgr2=0x5589c52277e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52277e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 -- 172.21.15.23:0/1628396778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 msgr2=0x5589c520a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c520a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 -- 172.21.15.23:0/1628396778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589c4f95b60 con 0x5589c52155d0 2024-03-30T04:07:23.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055400700 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8055e00700 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5589c5232e40 tx=0x7f8048014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=996416270e46c528 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8043a00700 1 -- 172.21.15.23:0/1628396778 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048023040 con 0x5589c52155d0 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8043a00700 1 -- 172.21.15.23:0/1628396778 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f804801e460 con 0x5589c52155d0 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.524+0000 7f8043a00700 1 -- 172.21.15.23:0/1628396778 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804801d670 con 0x5589c52155d0 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 -- 172.21.15.23:0/1628396778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 msgr2=0x5589c521e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5589c5232e40 tx=0x7f8048014000 comp rx=0 tx=0).stop 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 -- 172.21.15.23:0/1628396778 shutdown_connections 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c520a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c521e180 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 --2- 172.21.15.23:0/1628396778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 -- 172.21.15.23:0/1628396778 >> 172.21.15.23:0/1628396778 conn(0x5589c51756a0 msgr2=0x5589c520af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 -- 172.21.15.23:0/1628396778 shutdown_connections 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.525+0000 7f807056fb80 1 -- 172.21.15.23:0/1628396778 wait complete. 2024-03-30T04:07:23.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 -- start start 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 Processor -- start 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52258b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c5225df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5226330 con 0x5589c5220df0 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c52264a0 con 0x5589c52155d0 2024-03-30T04:07:23.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589c5226660 con 0x5589c5215ef0 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f807056fb80 10 client.0 _ll_register_callbacks cb 0x5589c510d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.526+0000 7f8055e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37534/0 (socket says 172.21.15.23:37534) 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8056800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c5225df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 -- 172.21.15.23:0/3271840563 learned_addr learned my addr 172.21.15.23:0/3271840563 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055400700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 -- 172.21.15.23:0/3271840563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 msgr2=0x5589c52258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52258b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 -- 172.21.15.23:0/3271840563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 msgr2=0x5589c5225df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c5225df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 -- 172.21.15.23:0/3271840563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8048017070 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8055e00700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f804801e610 tx=0x7f80480286b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1da69debd4f9ca49 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8037400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048023050 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f804409b9c0 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.527+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f8048028b00 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8048028cc0 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 10 client.27089 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5589c512dea0 con 0x5589c52155d0 2024-03-30T04:07:23.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f804409b9c0 con 0x5589c52155d0 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8048004c80 con 0x5589c52155d0 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 10 client.27089 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 10 client.27089 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 20 client.27089 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c5285430 0x5589c5287850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 4 client.27089 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.528+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5589c4fb6cf0 con 0x5589c5285430 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f8055400700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c5285430 0x5589c5287850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f8055400700 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c5285430 0x5589c5287850 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3ec6124f80dd009 server_cookie=84f8854e1c0fc67c in_seq=0 out_seq=0 2024-03-30T04:07:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f8037400700 10 client.27089 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f8037400700 1 -- 172.21.15.23:0/3271840563 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5589c4fb6cf0 con 0x5589c5285430 2024-03-30T04:07:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f8037400700 10 client.27089 handle_command_reply: tid=0 2024-03-30T04:07:23.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.529+0000 7f807056fb80 1 client.27089 shutdown 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 msgr2=0x5589c52273c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f804801e610 tx=0x7f80480286b0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 shutdown_connections 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5589c5220df0 0x5589c5225df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5589c52155d0 0x5589c52273c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5589c5285430 0x5589c5287850 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 --2- 172.21.15.23:0/3271840563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5589c5215ef0 0x5589c52258b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 >> 172.21.15.23:0/3271840563 conn(0x5589c51756a0 msgr2=0x5589c5218db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f8037400700 0 client.27089 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 shutdown_connections 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 1 -- 172.21.15.23:0/3271840563 wait complete. 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.530+0000 7f807056fb80 20 client.27089 trim_cache size 0 max 16384 2024-03-30T04:07:23.537 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 msgr2=0x7f803c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 0x7f803c0a37d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8048009d20 tx=0x7f8048009260 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 msgr2=0x7f80581999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 0x7f80581999b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f80440098e0 tx=0x7f8044005740 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 shutdown_connections 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805810d580 0x7f805819dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8058073180 0x7f8058199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f803c0a1310 0x7f803c0a37d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 --2- 172.21.15.23:0/2821653719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8058073aa0 0x7f80581999b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 >> 172.21.15.23:0/2821653719 conn(0x7f80580fc7f0 msgr2=0x7f80580fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 shutdown_connections 2024-03-30T04:07:23.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.531+0000 7f805ca00700 1 -- 172.21.15.23:0/2821653719 wait complete. 2024-03-30T04:07:23.551 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:07:23.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 -- 172.21.15.23:0/2017194779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c100f60 msgr2=0x7fe36c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 --2- 172.21.15.23:0/2017194779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c100f60 0x7fe36c101340 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009990 tx=0x7fe35c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 -- 172.21.15.23:0/2017194779 shutdown_connections 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 --2- 172.21.15.23:0/2017194779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c106080 0x7fe36c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 --2- 172.21.15.23:0/2017194779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c100f60 0x7fe36c101340 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 --2- 172.21.15.23:0/2017194779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c101910 0x7fe36c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.771+0000 7fe371a00700 1 -- 172.21.15.23:0/2017194779 >> 172.21.15.23:0/2017194779 conn(0x7fe36c0fc7c0 msgr2=0x7fe36c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.772+0000 7fe371a00700 1 -- 172.21.15.23:0/2017194779 shutdown_connections 2024-03-30T04:07:23.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.772+0000 7fe371a00700 1 -- 172.21.15.23:0/2017194779 wait complete. 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 Processor -- start 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 -- start start 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 0x7fe36c199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c19dd30 0x7fe36c19e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c19e7b0 con 0x7fe36c19dd30 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c19e920 con 0x7fe36c100f60 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe371a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c19eae0 con 0x7fe36c106080 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.773+0000 7fe36a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 0x7fe36c199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59342/0 (socket says 172.21.15.23:59342) 2024-03-30T04:07:23.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 -- 172.21.15.23:0/282994350 learned_addr learned my addr 172.21.15.23:0/282994350 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36b400700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c19dd30 0x7fe36c19e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 -- 172.21.15.23:0/282994350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 msgr2=0x7fe36c199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 0x7fe36c199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 -- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c19dd30 msgr2=0x7fe36c19e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c19dd30 0x7fe36c19e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 -- 172.21.15.23:0/282994350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c0095f0 con 0x7fe36c106080 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36aa00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 0x7fe36c199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.774+0000 7fe36a000700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe35800d950 tx=0x7fe35800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.775+0000 7fe363400700 1 -- 172.21.15.23:0/282994350 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358004220 con 0x7fe36c106080 2024-03-30T04:07:23.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.775+0000 7fe363400700 1 -- 172.21.15.23:0/282994350 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe358007620 con 0x7fe36c106080 2024-03-30T04:07:23.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.775+0000 7fe363400700 1 -- 172.21.15.23:0/282994350 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358005670 con 0x7fe36c106080 2024-03-30T04:07:23.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.775+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c1a5fe0 con 0x7fe36c106080 2024-03-30T04:07:23.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.775+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c1a64c0 con 0x7fe36c106080 2024-03-30T04:07:23.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.778+0000 7fe363400700 1 -- 172.21.15.23:0/282994350 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe358003680 con 0x7fe36c106080 2024-03-30T04:07:23.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.779+0000 7fe363400700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 0x7fe3500a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.779+0000 7fe363400700 1 -- 172.21.15.23:0/282994350 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe3580ccce0 con 0x7fe36c106080 2024-03-30T04:07:23.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.779+0000 7fe3855fab80 1 Processor -- start 2024-03-30T04:07:23.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe36aa00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 0x7fe3500a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 -- start start 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca3169b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca3241e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca26f990 con 0x55abca321df0 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca26d350 con 0x55abca3165d0 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe3855fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca268c40 con 0x55abca316ef0 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe36a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.780+0000 7fe36aa00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 0x7fe3500a37d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006010 tx=0x7fe35c00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca3169b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59344/0 (socket says 172.21.15.23:59344) 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 -- 172.21.15.23:0/3750268527 learned_addr learned my addr 172.21.15.23:0/3750268527 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca3241e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 -- 172.21.15.23:0/3750268527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 msgr2=0x55abca3169b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca3169b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 -- 172.21.15.23:0/3750268527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 msgr2=0x55abca3241e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca3241e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 -- 172.21.15.23:0/3750268527 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abca0ea3b0 con 0x55abca316ef0 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe36a000700 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe358000c00 tx=0x7fe35809a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=724fb340d3c5d14a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe357e00700 1 -- 172.21.15.23:0/3750268527 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35809d860 con 0x55abca316ef0 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe357e00700 1 -- 172.21.15.23:0/3750268527 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3580a1eb0 con 0x55abca316ef0 2024-03-30T04:07:23.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.781+0000 7fe357e00700 1 -- 172.21.15.23:0/3750268527 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe358099180 con 0x55abca316ef0 2024-03-30T04:07:23.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 -- 172.21.15.23:0/3750268527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 msgr2=0x55abca3218b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe358000c00 tx=0x7fe35809a130 comp rx=0 tx=0).stop 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 -- 172.21.15.23:0/3750268527 shutdown_connections 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca3241e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca3169b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 --2- 172.21.15.23:0/3750268527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca3218b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 -- 172.21.15.23:0/3750268527 >> 172.21.15.23:0/3750268527 conn(0x55abca2766a0 msgr2=0x55abca312a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 -- 172.21.15.23:0/3750268527 shutdown_connections 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.782+0000 7fe3855fab80 1 -- 172.21.15.23:0/3750268527 wait complete. 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.783+0000 7fe3855fab80 1 -- start start 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.783+0000 7fe3855fab80 1 Processor -- start 2024-03-30T04:07:23.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca321870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca31ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca204d80 con 0x55abca316ef0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca204a20 con 0x55abca3165d0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca0ec6a0 con 0x55abca321df0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe3855fab80 10 client.0 _ll_register_callbacks cb 0x55abca20e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca31ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca321870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca321870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37562/0 (socket says 172.21.15.23:37562) 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36aa00700 1 -- 172.21.15.23:0/2629956412 learned_addr learned my addr 172.21.15.23:0/2629956412 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca31ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46298/0 (socket says 172.21.15.23:46298) 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 -- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 msgr2=0x55abca321870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca321870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 -- 172.21.15.23:0/2629956412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 msgr2=0x55abca31ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca31ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.784+0000 7fe36b400700 1 -- 172.21.15.23:0/2629956412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c1a64c0 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe36b400700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31f2a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe364012dc0 tx=0x7fe364014600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=549292c7737be184 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe354c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400e040 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe364014a80 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55abca229380 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe364014c40 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe3855fab80 10 client.27100 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55abca096b60 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.785+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55abca229380 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe36401c020 con 0x55abca321df0 2024-03-30T04:07:23.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 10 client.27100 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 10 client.27100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 20 client.27100 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca331ae0 0x55abca333f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 4 client.27100 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abca261700 con 0x55abca331ae0 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.786+0000 7fe36aa00700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca331ae0 0x55abca333f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe354c00700 10 client.27100 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe36aa00700 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca331ae0 0x55abca333f00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723ef8bc3f7e196c server_cookie=eb3fe565b2e193d in_seq=0 out_seq=0 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe354c00700 1 -- 172.21.15.23:0/2629956412 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55abca261700 con 0x55abca331ae0 2024-03-30T04:07:23.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe354c00700 10 client.27100 handle_command_reply: tid=0 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe3855fab80 1 client.27100 shutdown 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 msgr2=0x55abca31f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.787+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31f2a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe364012dc0 tx=0x7fe364014600 comp rx=0 tx=0).stop 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 shutdown_connections 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca331ae0 0x55abca333f00 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31f2a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca321870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 --2- 172.21.15.23:0/2629956412 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca31ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 >> 172.21.15.23:0/2629956412 conn(0x55abca2766a0 msgr2=0x55abca3186c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe354c00700 0 client.27100 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 shutdown_connections 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 1 -- 172.21.15.23:0/2629956412 wait complete. 2024-03-30T04:07:23.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.788+0000 7fe3855fab80 20 client.27100 trim_cache size 0 max 16384 2024-03-30T04:07:23.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 Processor -- start 2024-03-30T04:07:23.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 -- start start 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca319eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca32f190 con 0x55abca316ef0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca320160 con 0x55abca3165d0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.805+0000 7fe3855fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca3317d0 con 0x55abca321df0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca319eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37568/0 (socket says 172.21.15.23:37568) 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 -- 172.21.15.23:0/1104926764 learned_addr learned my addr 172.21.15.23:0/1104926764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36b400700 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 -- 172.21.15.23:0/1104926764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 msgr2=0x55abca31a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 -- 172.21.15.23:0/1104926764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 msgr2=0x55abca319eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca319eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 -- 172.21.15.23:0/1104926764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abca22eea0 con 0x55abca3165d0 2024-03-30T04:07:23.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36b400700 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.806+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe35c015000 tx=0x7fe35c020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b282dfd61261fa28 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.807+0000 7fe360c00700 1 -- 172.21.15.23:0/1104926764 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c0343d0 con 0x55abca3165d0 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.807+0000 7fe360c00700 1 -- 172.21.15.23:0/1104926764 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe35c01e040 con 0x55abca3165d0 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.807+0000 7fe360c00700 1 -- 172.21.15.23:0/1104926764 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35c004be0 con 0x55abca3165d0 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- 172.21.15.23:0/1104926764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 msgr2=0x55abca328aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe35c015000 tx=0x7fe35c020000 comp rx=0 tx=0).stop 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- 172.21.15.23:0/1104926764 shutdown_connections 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca321df0 0x55abca31a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca328aa0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1104926764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca316ef0 0x55abca319eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- 172.21.15.23:0/1104926764 >> 172.21.15.23:0/1104926764 conn(0x55abca2766a0 msgr2=0x55abca30b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- 172.21.15.23:0/1104926764 shutdown_connections 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- 172.21.15.23:0/1104926764 wait complete. 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 -- start start 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.808+0000 7fe3855fab80 1 Processor -- start 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca326a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca326fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca384ad0 con 0x55abca321df0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca384c40 con 0x55abca3165d0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abca384e00 con 0x55abca316ef0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe3855fab80 10 client.0 _ll_register_callbacks cb 0x55abca20e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe36b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca326fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe36aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37570/0 (socket says 172.21.15.23:37570) 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe36aa00700 1 -- 172.21.15.23:0/1580781675 learned_addr learned my addr 172.21.15.23:0/1580781675 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.809+0000 7fe36a000700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca326a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 -- 172.21.15.23:0/1580781675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 msgr2=0x55abca326a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca326a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 -- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 msgr2=0x55abca326540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 -- 172.21.15.23:0/1580781675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c10e470 con 0x55abca321df0 2024-03-30T04:07:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe36b400700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca326fe0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe364012d90 tx=0x7fe364004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bfdbf34503e431 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe354c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:07:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe36400e040 con 0x55abca321df0 2024-03-30T04:07:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fe364014b50 con 0x55abca321df0 2024-03-30T04:07:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.810+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3580a1eb0 con 0x55abca321df0 2024-03-30T04:07:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe364014d50 con 0x55abca321df0 2024-03-30T04:07:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 10 client.17118 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:23.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55abca0eecd0 con 0x55abca321df0 2024-03-30T04:07:23.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3580a1eb0 con 0x55abca321df0 2024-03-30T04:07:23.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe364014320 con 0x55abca321df0 2024-03-30T04:07:23.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 10 client.17118 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:23.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 20 client.17118 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca386530 0x55abca388950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 4 client.17118 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55abca0b7cf0 con 0x55abca386530 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.811+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca386530 0x55abca388950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe36aa00700 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca386530 0x55abca388950 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6710b685583c8368 server_cookie=e0c223a9d061c42d in_seq=0 out_seq=0 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe354c00700 10 client.17118 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe354c00700 1 -- 172.21.15.23:0/1580781675 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55abca0b7cf0 con 0x55abca386530 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe354c00700 10 client.17118 handle_command_reply: tid=0 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 client.17118 shutdown 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 msgr2=0x55abca326fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca326fe0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe364012d90 tx=0x7fe364004b10 comp rx=0 tx=0).stop 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 shutdown_connections 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55abca321df0 0x55abca326fe0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55abca3165d0 0x55abca326540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55abca386530 0x55abca388950 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 --2- 172.21.15.23:0/1580781675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55abca316ef0 0x55abca326a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe354c00700 0 client.17118 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.812+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 >> 172.21.15.23:0/1580781675 conn(0x55abca2766a0 msgr2=0x55abca319eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 shutdown_connections 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe3855fab80 1 -- 172.21.15.23:0/1580781675 wait complete. 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe3855fab80 20 client.17118 trim_cache size 0 max 16384 2024-03-30T04:07:23.818 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 msgr2=0x7fe3500a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 0x7fe3500a37d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006010 tx=0x7fe35c00b260 comp rx=0 tx=0).stop 2024-03-30T04:07:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 msgr2=0x7fe36c199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe35800d950 tx=0x7fe35800dc60 comp rx=0 tx=0).stop 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 shutdown_connections 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.813+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe36c19dd30 0x7fe36c19e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe3500a1310 0x7fe3500a37d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe36c100f60 0x7fe36c199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 --2- 172.21.15.23:0/282994350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe36c106080 0x7fe36c199990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 >> 172.21.15.23:0/282994350 conn(0x7fe36c0fc7c0 msgr2=0x7fe36c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 shutdown_connections 2024-03-30T04:07:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:23.814+0000 7fe371a00700 1 -- 172.21.15.23:0/282994350 wait complete. 2024-03-30T04:07:24.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 519110 11.81 MB/sec execute 121 sec latency 10.446 ms 2024-03-30T04:07:24.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:24 smithi023 ceph-mon[38254]: pgmap v166: 129 pgs: 129 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-30T04:07:24.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:24 smithi154 ceph-mon[79161]: pgmap v166: 129 pgs: 129 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-30T04:07:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:24 smithi193 ceph-mon[79581]: pgmap v166: 129 pgs: 129 active+clean; 299 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 303 op/s 2024-03-30T04:07:24.832 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:07:24.833 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:07:25.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 -- 172.21.15.23:0/2394076408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8100f50 msgr2=0x7fcfe8101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 --2- 172.21.15.23:0/2394076408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8100f50 0x7fcfe8101330 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009990 tx=0x7fcfd8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 -- 172.21.15.23:0/2394076408 shutdown_connections 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 --2- 172.21.15.23:0/2394076408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8106070 0x7fcfe810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 --2- 172.21.15.23:0/2394076408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8100f50 0x7fcfe8101330 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 --2- 172.21.15.23:0/2394076408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe8101900 0x7fcfe8105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 -- 172.21.15.23:0/2394076408 >> 172.21.15.23:0/2394076408 conn(0x7fcfe80fc7f0 msgr2=0x7fcfe80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 -- 172.21.15.23:0/2394076408 shutdown_connections 2024-03-30T04:07:25.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.105+0000 7fcfedc00700 1 -- 172.21.15.23:0/2394076408 wait complete. 2024-03-30T04:07:25.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.106+0000 7fcfedc00700 1 Processor -- start 2024-03-30T04:07:25.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.106+0000 7fcfedc00700 1 -- start start 2024-03-30T04:07:25.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.106+0000 7fcfedc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8100f50 0x7fcfe8199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.106+0000 7fcfedc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfedc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfedc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe819e760 con 0x7fcfe819dce0 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfedc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe819e8d0 con 0x7fcfe8106070 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfedc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe819ea90 con 0x7fcfe8100f50 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46334/0 (socket says 172.21.15.23:46334) 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe7400700 1 -- 172.21.15.23:0/772131743 learned_addr learned my addr 172.21.15.23:0/772131743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37586/0 (socket says 172.21.15.23:37586) 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8100f50 0x7fcfe8199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 -- 172.21.15.23:0/772131743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8100f50 msgr2=0x7fcfe8199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8100f50 0x7fcfe8199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 -- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 msgr2=0x7fcfe819e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe6000700 1 -- 172.21.15.23:0/772131743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd80095f0 con 0x7fcfe8106070 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.107+0000 7fcfe7400700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfe6000700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9c0 tx=0x7fcfd400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfdf400700 1 -- 172.21.15.23:0/772131743 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400c9c0 con 0x7fcfe8106070 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfdf400700 1 -- 172.21.15.23:0/772131743 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfd4010430 con 0x7fcfe8106070 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe81a5ff0 con 0x7fcfe8106070 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfdf400700 1 -- 172.21.15.23:0/772131743 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4018680 con 0x7fcfe8106070 2024-03-30T04:07:25.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.108+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe81a64c0 con 0x7fcfe8106070 2024-03-30T04:07:25.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.111+0000 7fcfdf400700 1 -- 172.21.15.23:0/772131743 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fcfd400cb20 con 0x7fcfe8106070 2024-03-30T04:07:25.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.112+0000 7fcfdf400700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 0x7fcfcc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fcfdf400700 1 -- 172.21.15.23:0/772131743 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcfd4014070 con 0x7fcfe8106070 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 Processor -- start 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 0x7fcfcc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 -- start start 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 0x55a4740eaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 0x55a4740f5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a474043a10 con 0x55a4740f5e70 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740413d0 con 0x55a4740ea650 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a47403ccc0 con 0x55a4740eaf70 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.113+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 0x7fcfcc0a37d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8006010 tx=0x7fcfd800b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 0x55a4740eaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 0x55a4740eaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37598/0 (socket says 172.21.15.23:37598) 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6a00700 1 -- 172.21.15.23:0/3982533749 learned_addr learned my addr 172.21.15.23:0/3982533749 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 0x55a4740f5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe7400700 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 -- 172.21.15.23:0/3982533749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 msgr2=0x55a4740eaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 0x55a4740eaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 -- 172.21.15.23:0/3982533749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 msgr2=0x55a4740f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 -- 172.21.15.23:0/3982533749 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a473ebe430 con 0x55a4740eaf70 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfe6000700 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 0x55a4740f5930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40ca4a0 tx=0x7fcfd40962d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b057d7dbecf34cc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfd3e00700 1 -- 172.21.15.23:0/3982533749 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4099e60 con 0x55a4740eaf70 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.114+0000 7fcfd3e00700 1 -- 172.21.15.23:0/3982533749 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfd409dc90 con 0x55a4740eaf70 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fcfd3e00700 1 -- 172.21.15.23:0/3982533749 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd409a0c0 con 0x55a4740eaf70 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3982533749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 msgr2=0x55a4740f5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 0x55a4740f5930 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40ca4a0 tx=0x7fcfd40962d0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3982533749 shutdown_connections 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740ea650 0x55a4740eaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3982533749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740eaf70 0x55a4740f5930 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3982533749 >> 172.21.15.23:0/3982533749 conn(0x55a47404a720 msgr2=0x55a4740e6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3982533749 shutdown_connections 2024-03-30T04:07:25.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.115+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3982533749 wait complete. 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.116+0000 7fd0017f4b80 1 -- start start 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.116+0000 7fd0017f4b80 1 Processor -- start 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a4740f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a473fd8e00 con 0x55a4740f5e70 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a473fd8aa0 con 0x55a4740eaf70 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a473ec0720 con 0x55a4740ea650 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fd0017f4b80 10 client.0 _ll_register_callbacks cb 0x55a473fe24f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:25.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37606/0 (socket says 172.21.15.23:37606) 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 -- 172.21.15.23:0/2789415102 learned_addr learned my addr 172.21.15.23:0/2789415102 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a4740f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe7400700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 -- 172.21.15.23:0/2789415102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 msgr2=0x55a4740f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a4740f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 -- 172.21.15.23:0/2789415102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 msgr2=0x55a4740f3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.117+0000 7fcfe6000700 1 -- 172.21.15.23:0/2789415102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe81a64c0 con 0x55a4740eaf70 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfe6000700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cd170 tx=0x7fcfd40ca410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a4fcc8bd0f11f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:25.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4099e60 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcfd40105a0 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a473ffd400 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd40b8250 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fd0017f4b80 10 client.27119 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a473e6abe0 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a473ffd400 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.118+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcfd40a1fa0 con 0x55a4740eaf70 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 10 client.27119 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 10 client.27119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 20 client.27119 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:25.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4741056e0 0x55a474107ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 4 client.27119 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a474035780 con 0x55a4741056e0 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4741056e0 0x55a474107ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4741056e0 0x55a474107ad0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcc64bf2ca32c031 server_cookie=3255a75de168321a in_seq=0 out_seq=0 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.119+0000 7fcfd0c00700 10 client.27119 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fcfd0c00700 1 -- 172.21.15.23:0/2789415102 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a474035780 con 0x55a4741056e0 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fcfd0c00700 10 client.27119 handle_command_reply: tid=0 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 client.27119 shutdown 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 msgr2=0x55a4740f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fcfd40cd170 tx=0x7fcfd40ca410 comp rx=0 tx=0).stop 2024-03-30T04:07:25.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 shutdown_connections 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740f3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4741056e0 0x55a474107ad0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a4740f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/2789415102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2c30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fcfd0c00700 0 client.27119 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.120+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 >> 172.21.15.23:0/2789415102 conn(0x55a47404a720 msgr2=0x55a4740ec740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.121+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 shutdown_connections 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.121+0000 7fd0017f4b80 1 -- 172.21.15.23:0/2789415102 wait complete. 2024-03-30T04:07:25.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.121+0000 7fd0017f4b80 20 client.27119 trim_cache size 0 max 16384 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 Processor -- start 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 -- start start 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 0x55a4740fca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 0x55a4740f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740ea4b0 con 0x55a4740ea650 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740f3710 con 0x55a4740eaf70 2024-03-30T04:07:25.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740eb8d0 con 0x55a4740f5e70 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 0x55a4740fca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fcfe7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 0x55a4740f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 0x55a4740fca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46352/0 (socket says 172.21.15.23:46352) 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.142+0000 7fcfe6a00700 1 -- 172.21.15.23:0/3324995418 learned_addr learned my addr 172.21.15.23:0/3324995418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 -- 172.21.15.23:0/3324995418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 msgr2=0x55a4740f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 -- 172.21.15.23:0/3324995418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 msgr2=0x55a4740fca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 0x55a4740fca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 -- 172.21.15.23:0/3324995418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a474002f20 con 0x55a4740f5e70 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe6000700 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfe7400700 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 0x55a4740f2a50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009fd0 tx=0x7fcfe0009710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9ca5eaba30486a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfdcc00700 1 -- 172.21.15.23:0/3324995418 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe000e040 con 0x55a4740f5e70 2024-03-30T04:07:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfdcc00700 1 -- 172.21.15.23:0/3324995418 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfe000f040 con 0x55a4740f5e70 2024-03-30T04:07:25.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.143+0000 7fcfdcc00700 1 -- 172.21.15.23:0/3324995418 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe001db90 con 0x55a4740f5e70 2024-03-30T04:07:25.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3324995418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 msgr2=0x55a4740f2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 0x55a4740f2a50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009fd0 tx=0x7fcfe0009710 comp rx=0 tx=0).stop 2024-03-30T04:07:25.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3324995418 shutdown_connections 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740f5e70 0x55a4740f2a50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740ea650 0x55a4740fca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/3324995418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a4740f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3324995418 >> 172.21.15.23:0/3324995418 conn(0x55a47404a720 msgr2=0x55a4740edf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3324995418 shutdown_connections 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.144+0000 7fd0017f4b80 1 -- 172.21.15.23:0/3324995418 wait complete. 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.145+0000 7fd0017f4b80 1 -- start start 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.145+0000 7fd0017f4b80 1 Processor -- start 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.145+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.145+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.145+0000 7fd0017f4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740fc2a0 con 0x55a4740f5e70 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740fc410 con 0x55a4740eaf70 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fd0017f4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4740fc5d0 con 0x55a4740ea650 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fd0017f4b80 10 client.0 _ll_register_callbacks cb 0x55a473fe24f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37616/0 (socket says 172.21.15.23:37616) 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59440/0 (socket says 172.21.15.23:59440) 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 -- 172.21.15.23:0/567949887 learned_addr learned my addr 172.21.15.23:0/567949887 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe7400700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 -- 172.21.15.23:0/567949887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 msgr2=0x55a474153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 -- 172.21.15.23:0/567949887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 msgr2=0x55a4740fbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740fbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6000700 1 -- 172.21.15.23:0/567949887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a473e6abe0 con 0x55a4740eaf70 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.146+0000 7fcfe7400700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfe6000700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c5b0 tx=0x7fcfd4010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3363a463d7ed68eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd40b8250 con 0x55a4740eaf70 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfd409dc90 con 0x55a4740eaf70 2024-03-30T04:07:25.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fcfd4098f30 con 0x55a4740eaf70 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd4099170 con 0x55a4740eaf70 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fd0017f4b80 10 client.27125 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fcfe00143e0 con 0x55a4740eaf70 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfd409dc90 con 0x55a4740eaf70 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcfd4083150 con 0x55a4740eaf70 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fd0017f4b80 10 client.27125 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.147+0000 7fd0017f4b80 10 client.27125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fd0017f4b80 20 client.27125 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a47415b5e0 0x55a47415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fd0017f4b80 4 client.27125 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a473e8bd70 con 0x55a47415b5e0 2024-03-30T04:07:25.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a47415b5e0 0x55a47415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fcfe6a00700 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a47415b5e0 0x55a47415d9d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d33d9b5196caed server_cookie=e2002b756a4d28c6 in_seq=0 out_seq=0 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.148+0000 7fcfd0c00700 10 client.27125 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.149+0000 7fcfd0c00700 1 -- 172.21.15.23:0/567949887 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a473e8bd70 con 0x55a47415b5e0 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.149+0000 7fcfd0c00700 10 client.27125 handle_command_reply: tid=0 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.149+0000 7fd0017f4b80 1 client.27125 shutdown 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.149+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 msgr2=0x55a474154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.149+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c5b0 tx=0x7fcfd4010a90 comp rx=0 tx=0).stop 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 shutdown_connections 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4740f5e70 0x55a4740fbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a47415b5e0 0x55a47415d9d0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4740ea650 0x55a474153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 --2- 172.21.15.23:0/567949887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4740eaf70 0x55a474154170 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 >> 172.21.15.23:0/567949887 conn(0x55a47404a720 msgr2=0x55a4740e5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fcfd0c00700 0 client.27125 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 shutdown_connections 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 1 -- 172.21.15.23:0/567949887 wait complete. 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fd0017f4b80 20 client.27125 trim_cache size 0 max 16384 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:07:25.154 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "3003f8a0-f41c-487f-8c77-2ec361aa6615", 2024-03-30T04:07:25.155 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:07:25.155 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:25.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 msgr2=0x7fcfcc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.150+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 0x7fcfcc0a37d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8006010 tx=0x7fcfd800b260 comp rx=0 tx=0).stop 2024-03-30T04:07:25.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 msgr2=0x7fcfe8199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400e9c0 tx=0x7fcfd400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 shutdown_connections 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcfe819dce0 0x7fcfe819e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe8100f50 0x7fcfe8199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcfcc0a1310 0x7fcfcc0a37d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 --2- 172.21.15.23:0/772131743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8106070 0x7fcfe8199940 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 >> 172.21.15.23:0/772131743 conn(0x7fcfe80fc7f0 msgr2=0x7fcfe80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 shutdown_connections 2024-03-30T04:07:25.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.151+0000 7fcfedc00700 1 -- 172.21.15.23:0/772131743 wait complete. 2024-03-30T04:07:25.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:07:25.201 INFO:tasks.workunit.client.0.smithi023.stdout: 1 521063 11.77 MB/sec execute 122 sec latency 38.889 ms 2024-03-30T04:07:25.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 -- 172.21.15.23:0/2644822725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60101910 msgr2=0x7fea60105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 --2- 172.21.15.23:0/2644822725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60101910 0x7fea60105950 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009990 tx=0x7fea4c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 -- 172.21.15.23:0/2644822725 shutdown_connections 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 --2- 172.21.15.23:0/2644822725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea6010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 --2- 172.21.15.23:0/2644822725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea60100f60 0x7fea60101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 --2- 172.21.15.23:0/2644822725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60101910 0x7fea60105950 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 -- 172.21.15.23:0/2644822725 >> 172.21.15.23:0/2644822725 conn(0x7fea600fc7c0 msgr2=0x7fea600febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 -- 172.21.15.23:0/2644822725 shutdown_connections 2024-03-30T04:07:25.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.382+0000 7fea66a00700 1 -- 172.21.15.23:0/2644822725 wait complete. 2024-03-30T04:07:25.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.383+0000 7fea66a00700 1 Processor -- start 2024-03-30T04:07:25.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.383+0000 7fea66a00700 1 -- start start 2024-03-30T04:07:25.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.383+0000 7fea66a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.383+0000 7fea66a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea60197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.383+0000 7fea66a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 0x7fea6019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea66a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6019c540 con 0x7fea6019bac0 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea66a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6019c6b0 con 0x7fea60100f60 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea66a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6019c870 con 0x7fea60106080 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea64a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 0x7fea6019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37638/0 (socket says 172.21.15.23:37638) 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 -- 172.21.15.23:0/1680912201 learned_addr learned my addr 172.21.15.23:0/1680912201 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5f400700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea60197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 -- 172.21.15.23:0/1680912201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 msgr2=0x7fea60197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea60197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 -- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 msgr2=0x7fea6019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 0x7fea6019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 -- 172.21.15.23:0/1680912201 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea4c0095f0 con 0x7fea60100f60 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5f400700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea60197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea64a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 0x7fea6019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.384+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fea5000e9a0 tx=0x7fea5000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.385+0000 7fea5cc00700 1 -- 172.21.15.23:0/1680912201 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50009790 con 0x7fea60100f60 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.385+0000 7fea5cc00700 1 -- 172.21.15.23:0/1680912201 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea50010430 con 0x7fea60100f60 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.385+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea601a3dd0 con 0x7fea60100f60 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.385+0000 7fea5cc00700 1 -- 172.21.15.23:0/1680912201 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50018680 con 0x7fea60100f60 2024-03-30T04:07:25.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.385+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea601a4260 con 0x7fea60100f60 2024-03-30T04:07:25.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.389+0000 7fea5cc00700 1 -- 172.21.15.23:0/1680912201 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fea500105a0 con 0x7fea60100f60 2024-03-30T04:07:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.390+0000 7fea7a493b80 1 Processor -- start 2024-03-30T04:07:25.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 -- start start 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae4fd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae50f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae456990 con 0x564bae50d200 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae454350 con 0x564bae4fd5d0 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea7a493b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae44fc40 con 0x564bae4fdef0 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.391+0000 7fea5cc00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 0x7fea440a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59462/0 (socket says 172.21.15.23:59462) 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 -- 172.21.15.23:0/4008335013 learned_addr learned my addr 172.21.15.23:0/4008335013 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea64a00700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae4fd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5f400700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae50f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea64a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 0x7fea440a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5cc00700 1 -- 172.21.15.23:0/1680912201 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fea50014070 con 0x7fea60100f60 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 -- 172.21.15.23:0/4008335013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 msgr2=0x564bae4fd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae4fd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 -- 172.21.15.23:0/4008335013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 msgr2=0x564bae50f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae50f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 -- 172.21.15.23:0/4008335013 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bae2d13b0 con 0x564bae4fdef0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea64a00700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae4fd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.392+0000 7fea5fe00700 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fea50012020 tx=0x7fea50069eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cb6cd65b7f3a520 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea55200700 1 -- 172.21.15.23:0/4008335013 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5009a630 con 0x564bae4fdef0 2024-03-30T04:07:25.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea64a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 0x7fea440a37d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea58009fd0 tx=0x7fea5800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea55200700 1 -- 172.21.15.23:0/4008335013 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea50099f30 con 0x564bae4fdef0 2024-03-30T04:07:25.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea55200700 1 -- 172.21.15.23:0/4008335013 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea500af3c0 con 0x564bae4fdef0 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 -- 172.21.15.23:0/4008335013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 msgr2=0x564bae50ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fea50012020 tx=0x7fea50069eb0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 -- 172.21.15.23:0/4008335013 shutdown_connections 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae50f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae4fd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 --2- 172.21.15.23:0/4008335013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fdef0 0x564bae50ccc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.393+0000 7fea7a493b80 1 -- 172.21.15.23:0/4008335013 >> 172.21.15.23:0/4008335013 conn(0x564bae45d6a0 msgr2=0x564bae4f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.394+0000 7fea7a493b80 1 -- 172.21.15.23:0/4008335013 shutdown_connections 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.394+0000 7fea7a493b80 1 -- 172.21.15.23:0/4008335013 wait complete. 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.394+0000 7fea7a493b80 1 -- start start 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.394+0000 7fea7a493b80 1 Processor -- start 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae509f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 0x564bae50a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae3ebd80 con 0x564bae4fdef0 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae3eba20 con 0x564bae4fd5d0 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae2d36a0 con 0x564bae50d200 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea7a493b80 10 client.0 _ll_register_callbacks cb 0x564bae3f5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37662/0 (socket says 172.21.15.23:37662) 2024-03-30T04:07:25.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea5fe00700 1 -- 172.21.15.23:0/1972796857 learned_addr learned my addr 172.21.15.23:0/1972796857 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea5f400700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae509f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea64a00700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 0x564bae50a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.395+0000 7fea5fe00700 1 -- 172.21.15.23:0/1972796857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 msgr2=0x564bae50a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 0x564bae50a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5fe00700 1 -- 172.21.15.23:0/1972796857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 msgr2=0x564bae509f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae509f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5fe00700 1 -- 172.21.15.23:0/1972796857 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea601a4260 con 0x564bae4fd5d0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea64a00700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 0x564bae50a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5f400700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae509f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea5fe00700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fea50099c30 tx=0x7fea5009a6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cf3b478f3cd6b82 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea49600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5009dde0 con 0x564bae4fd5d0 2024-03-30T04:07:25.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fea5009a190 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564bae410380 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea5009a350 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea7a493b80 10 client.27137 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.396+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564bae27db60 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564bae410380 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fea50095d20 con 0x564bae4fd5d0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 10 client.27137 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 10 client.27137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 20 client.27137 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae51ca00 0x564bae51edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 4 client.27137 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bae448700 con 0x564bae51ca00 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.397+0000 7fea5f400700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae51ca00 0x564bae51edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.398+0000 7fea5f400700 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae51ca00 0x564bae51edf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb80c0bd1616365 server_cookie=deb116da7de5df93 in_seq=0 out_seq=0 2024-03-30T04:07:25.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.398+0000 7fea49600700 10 client.27137 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.398+0000 7fea49600700 1 -- 172.21.15.23:0/1972796857 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564bae448700 con 0x564bae51ca00 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.398+0000 7fea49600700 10 client.27137 handle_command_reply: tid=0 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 client.27137 shutdown 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 msgr2=0x564bae50cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fea50099c30 tx=0x7fea5009a6e0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 shutdown_connections 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae51ca00 0x564bae51edf0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae50d200 0x564bae50a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fd5d0 0x564bae50cc20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 --2- 172.21.15.23:0/1972796857 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae509f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea49600700 0 client.27137 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 >> 172.21.15.23:0/1972796857 conn(0x564bae45d6a0 msgr2=0x564bae4ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 shutdown_connections 2024-03-30T04:07:25.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 1 -- 172.21.15.23:0/1972796857 wait complete. 2024-03-30T04:07:25.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.399+0000 7fea7a493b80 20 client.27137 trim_cache size 0 max 16384 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.420+0000 7fea7a493b80 1 Processor -- start 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.420+0000 7fea7a493b80 1 -- start start 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.420+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.420+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae513e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae50d200 0x564bae4f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea7a493b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae500570 con 0x564bae4fdef0 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea7a493b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae4fa660 con 0x564bae50d200 2024-03-30T04:07:25.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea7a493b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae514360 con 0x564bae4fd5d0 2024-03-30T04:07:25.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59472/0 (socket says 172.21.15.23:59472) 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 -- 172.21.15.23:0/2631101173 learned_addr learned my addr 172.21.15.23:0/2631101173 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5f400700 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae513e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea64a00700 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae50d200 0x564bae4f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 -- 172.21.15.23:0/2631101173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae50d200 msgr2=0x564bae4f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae50d200 0x564bae4f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 -- 172.21.15.23:0/2631101173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 msgr2=0x564bae513e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae513e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.421+0000 7fea5fe00700 1 -- 172.21.15.23:0/2631101173 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bae4fd440 con 0x564bae4fd5d0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.422+0000 7fea5fe00700 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea5000c2d0 tx=0x7fea50010a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5d12c7f783fd93a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.422+0000 7fea55c00700 1 -- 172.21.15.23:0/2631101173 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5009dd60 con 0x564bae4fd5d0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.422+0000 7fea55c00700 1 -- 172.21.15.23:0/2631101173 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea500af0e0 con 0x564bae4fd5d0 2024-03-30T04:07:25.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.422+0000 7fea55c00700 1 -- 172.21.15.23:0/2631101173 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50099680 con 0x564bae4fd5d0 2024-03-30T04:07:25.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- 172.21.15.23:0/2631101173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 msgr2=0x564bae50a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fea5000c2d0 tx=0x7fea50010a90 comp rx=0 tx=0).stop 2024-03-30T04:07:25.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- 172.21.15.23:0/2631101173 shutdown_connections 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae50d200 0x564bae4f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae50a540 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 --2- 172.21.15.23:0/2631101173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae4fdef0 0x564bae513e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- 172.21.15.23:0/2631101173 >> 172.21.15.23:0/2631101173 conn(0x564bae45d6a0 msgr2=0x564bae4f25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- 172.21.15.23:0/2631101173 shutdown_connections 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- 172.21.15.23:0/2631101173 wait complete. 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.423+0000 7fea7a493b80 1 -- start start 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.424+0000 7fea7a493b80 1 Processor -- start 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.424+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.424+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 0x564bae569ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea7a493b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae511e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea7a493b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae512340 con 0x564bae50d200 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea7a493b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae5124b0 con 0x564bae4fdef0 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea7a493b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bae512670 con 0x564bae4fd5d0 2024-03-30T04:07:25.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea7a493b80 10 client.0 _ll_register_callbacks cb 0x564bae3f5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:25.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:59478/0 (socket says 172.21.15.23:59478) 2024-03-30T04:07:25.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 -- 172.21.15.23:0/3635316604 learned_addr learned my addr 172.21.15.23:0/3635316604 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:25.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5f400700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 0x564bae569ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea64a00700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae511e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 -- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 msgr2=0x564bae569ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 0x564bae569ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 -- 172.21.15.23:0/3635316604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 msgr2=0x564bae511e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae511e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.425+0000 7fea5fe00700 1 -- 172.21.15.23:0/3635316604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bae27db60 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea5f400700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 0x564bae569ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea5fe00700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea5009a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74c41a3f9a1fd4df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea49600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5009dd60 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fea50099420 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea50099f30 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea500b82b0 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea7a493b80 10 client.27139 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.426+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564bae415ea0 con 0x564bae4fd5d0 2024-03-30T04:07:25.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea500a8080 con 0x564bae4fd5d0 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fea50098e90 con 0x564bae4fd5d0 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 10 client.27139 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 10 client.27139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 20 client.27139 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae571320 0x564bae573710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 4 client.27139 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564bae29ecf0 con 0x564bae571320 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.427+0000 7fea5f400700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae571320 0x564bae573710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea49600700 10 client.27139 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea5f400700 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae571320 0x564bae573710 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39d89fa692511843 server_cookie=22b37533dcd9e247 in_seq=0 out_seq=0 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea49600700 1 -- 172.21.15.23:0/3635316604 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564bae29ecf0 con 0x564bae571320 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea49600700 10 client.27139 handle_command_reply: tid=0 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 client.27139 shutdown 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 msgr2=0x564bae5699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea5009a540 comp rx=0 tx=0).stop 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 shutdown_connections 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564bae50d200 0x564bae511e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564bae571320 0x564bae573710 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564bae4fd5d0 0x564bae5699b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 --2- 172.21.15.23:0/3635316604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564bae4fdef0 0x564bae569ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea49600700 0 client.27139 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 >> 172.21.15.23:0/3635316604 conn(0x564bae45d6a0 msgr2=0x564bae4f2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 shutdown_connections 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.428+0000 7fea7a493b80 1 -- 172.21.15.23:0/3635316604 wait complete. 2024-03-30T04:07:25.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea7a493b80 20 client.27139 trim_cache size 0 max 16384 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "3003f8a0-f41c-487f-8c77-2ec361aa6615": { 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "3003f8a0-f41c-487f-8c77-2ec361aa6615", 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:07:25.431 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 msgr2=0x7fea440a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 0x7fea440a37d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea58009fd0 tx=0x7fea5800b040 comp rx=0 tx=0).stop 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 msgr2=0x7fea601971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fea5000e9a0 tx=0x7fea5000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 shutdown_connections 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fea6019bac0 0x7fea6019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fea440a1310 0x7fea440a37d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea60100f60 0x7fea601971e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 --2- 172.21.15.23:0/1680912201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea60106080 0x7fea60197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:25.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 >> 172.21.15.23:0/1680912201 conn(0x7fea600fc7c0 msgr2=0x7fea600fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:25.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 shutdown_connections 2024-03-30T04:07:25.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:25.429+0000 7fea66a00700 1 -- 172.21.15.23:0/1680912201 wait complete. 2024-03-30T04:07:25.446 INFO:tasks.cephfs.filesystem:scrub status for tag:3003f8a0-f41c-487f-8c77-2ec361aa6615 - {'path': '/', 'tag': '3003f8a0-f41c-487f-8c77-2ec361aa6615', 'options': 'recursive,force'} 2024-03-30T04:07:26.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 523003 11.78 MB/sec execute 123 sec latency 11.434 ms 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: pgmap v167: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:26 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: pgmap v167: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-30T04:07:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:07:26.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:07:26.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:07:26.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:26 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: pgmap v167: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.7 MiB/s wr, 207 op/s 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:26 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:07:27.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 526027 11.83 MB/sec execute 124 sec latency 9.475 ms 2024-03-30T04:07:28.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 528345 11.80 MB/sec execute 125 sec latency 12.820 ms 2024-03-30T04:07:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:28 smithi154 ceph-mon[79161]: pgmap v168: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 204 op/s 2024-03-30T04:07:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:28 smithi193 ceph-mon[79581]: pgmap v168: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 204 op/s 2024-03-30T04:07:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:28 smithi023 ceph-mon[38254]: pgmap v168: 129 pgs: 129 active+clean; 298 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 204 op/s 2024-03-30T04:07:29.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 530470 11.81 MB/sec execute 126 sec latency 6.718 ms 2024-03-30T04:07:30.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 532909 11.84 MB/sec execute 127 sec latency 131.687 ms 2024-03-30T04:07:30.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:30 smithi154 ceph-mon[79161]: pgmap v169: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-30T04:07:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:30 smithi193 ceph-mon[79581]: pgmap v169: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-30T04:07:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:30 smithi023 ceph-mon[38254]: pgmap v169: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.1 MiB/s wr, 304 op/s 2024-03-30T04:07:31.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 534171 11.78 MB/sec execute 128 sec latency 14.921 ms 2024-03-30T04:07:32.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 536829 11.82 MB/sec execute 129 sec latency 9.110 ms 2024-03-30T04:07:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:32 smithi154 ceph-mon[79161]: pgmap v170: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 199 op/s 2024-03-30T04:07:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:32 smithi193 ceph-mon[79581]: pgmap v170: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 199 op/s 2024-03-30T04:07:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:32 smithi023 ceph-mon[38254]: pgmap v170: 129 pgs: 129 active+clean; 306 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 199 op/s 2024-03-30T04:07:33.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 539307 11.81 MB/sec execute 130 sec latency 13.284 ms 2024-03-30T04:07:34.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 540959 11.79 MB/sec execute 131 sec latency 10.708 ms 2024-03-30T04:07:34.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:34 smithi154 ceph-mon[79161]: pgmap v171: 129 pgs: 129 active+clean; 330 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-03-30T04:07:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:34 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:07:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:34 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:07:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:34 smithi193 ceph-mon[79581]: pgmap v171: 129 pgs: 129 active+clean; 330 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-03-30T04:07:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:34 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:07:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:34 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:07:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:34 smithi023 ceph-mon[38254]: pgmap v171: 129 pgs: 129 active+clean; 330 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.9 MiB/s wr, 294 op/s 2024-03-30T04:07:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:34 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:07:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:34 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:07:35.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 543606 11.83 MB/sec execute 132 sec latency 67.533 ms 2024-03-30T04:07:36.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 544945 11.79 MB/sec execute 133 sec latency 16.408 ms 2024-03-30T04:07:36.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:36 smithi154 ceph-mon[79161]: pgmap v172: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 199 op/s 2024-03-30T04:07:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:36 smithi193 ceph-mon[79581]: pgmap v172: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 199 op/s 2024-03-30T04:07:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:36 smithi023 ceph-mon[38254]: pgmap v172: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 199 op/s 2024-03-30T04:07:37.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 547212 11.80 MB/sec execute 134 sec latency 18.262 ms 2024-03-30T04:07:38.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 548628 11.76 MB/sec execute 135 sec latency 21.565 ms 2024-03-30T04:07:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:38 smithi154 ceph-mon[79161]: pgmap v173: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 197 op/s 2024-03-30T04:07:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:38 smithi193 ceph-mon[79581]: pgmap v173: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 197 op/s 2024-03-30T04:07:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:38 smithi023 ceph-mon[38254]: pgmap v173: 129 pgs: 129 active+clean; 332 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.5 MiB/s wr, 197 op/s 2024-03-30T04:07:39.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 550886 11.77 MB/sec execute 136 sec latency 17.827 ms 2024-03-30T04:07:40.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 552136 11.72 MB/sec execute 137 sec latency 99.693 ms 2024-03-30T04:07:40.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:40 smithi154 ceph-mon[79161]: pgmap v174: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 292 op/s 2024-03-30T04:07:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:40 smithi193 ceph-mon[79581]: pgmap v174: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 292 op/s 2024-03-30T04:07:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:40 smithi023 ceph-mon[38254]: pgmap v174: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 5.8 MiB/s wr, 292 op/s 2024-03-30T04:07:41.202 INFO:tasks.workunit.client.0.smithi023.stdout: 1 554324 11.73 MB/sec execute 138 sec latency 19.168 ms 2024-03-30T04:07:42.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 555828 11.69 MB/sec execute 139 sec latency 20.929 ms 2024-03-30T04:07:42.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:42 smithi154 ceph-mon[79161]: pgmap v175: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 193 op/s 2024-03-30T04:07:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:42 smithi193 ceph-mon[79581]: pgmap v175: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 193 op/s 2024-03-30T04:07:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:42 smithi023 ceph-mon[38254]: pgmap v175: 129 pgs: 129 active+clean; 339 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 4.3 MiB/s wr, 193 op/s 2024-03-30T04:07:43.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 558032 11.71 MB/sec execute 140 sec latency 18.208 ms 2024-03-30T04:07:44.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 559566 11.67 MB/sec execute 141 sec latency 22.665 ms 2024-03-30T04:07:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:44 smithi154 ceph-mon[79161]: pgmap v176: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.7 MiB/s wr, 282 op/s 2024-03-30T04:07:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:44 smithi193 ceph-mon[79581]: pgmap v176: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.7 MiB/s wr, 282 op/s 2024-03-30T04:07:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:44 smithi023 ceph-mon[38254]: pgmap v176: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 5.7 MiB/s wr, 282 op/s 2024-03-30T04:07:45.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 561643 11.68 MB/sec execute 142 sec latency 87.759 ms 2024-03-30T04:07:46.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 562958 11.64 MB/sec execute 143 sec latency 40.139 ms 2024-03-30T04:07:46.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:46 smithi154 ceph-mon[79161]: pgmap v177: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-03-30T04:07:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:46 smithi193 ceph-mon[79581]: pgmap v177: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-03-30T04:07:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:46 smithi023 ceph-mon[38254]: pgmap v177: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 190 op/s 2024-03-30T04:07:47.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 565221 11.65 MB/sec execute 144 sec latency 31.534 ms 2024-03-30T04:07:48.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 566885 11.61 MB/sec execute 145 sec latency 30.152 ms 2024-03-30T04:07:48.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:48 smithi154 ceph-mon[79161]: pgmap v178: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:48 smithi193 ceph-mon[79581]: pgmap v178: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:48 smithi023 ceph-mon[38254]: pgmap v178: 129 pgs: 129 active+clean; 345 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:49.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 568938 11.62 MB/sec execute 146 sec latency 14.261 ms 2024-03-30T04:07:50.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 570902 11.60 MB/sec execute 147 sec latency 21.945 ms 2024-03-30T04:07:50.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:50 smithi154 ceph-mon[79161]: pgmap v179: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:50 smithi193 ceph-mon[79581]: pgmap v179: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:50 smithi023 ceph-mon[38254]: pgmap v179: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:51.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 572432 11.59 MB/sec execute 148 sec latency 50.630 ms 2024-03-30T04:07:52.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 574317 11.56 MB/sec execute 149 sec latency 22.038 ms 2024-03-30T04:07:52.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:52 smithi154 ceph-mon[79161]: pgmap v180: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-03-30T04:07:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:52 smithi193 ceph-mon[79581]: pgmap v180: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-03-30T04:07:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:52 smithi023 ceph-mon[38254]: pgmap v180: 129 pgs: 129 active+clean; 351 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 181 op/s 2024-03-30T04:07:53.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 576155 11.57 MB/sec execute 150 sec latency 22.366 ms 2024-03-30T04:07:54.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 578331 11.57 MB/sec execute 151 sec latency 19.786 ms 2024-03-30T04:07:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:54 smithi154 ceph-mon[79161]: pgmap v181: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:54 smithi193 ceph-mon[79581]: pgmap v181: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:54 smithi023 ceph-mon[38254]: pgmap v181: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:07:55.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 579901 11.55 MB/sec execute 152 sec latency 19.116 ms 2024-03-30T04:07:55.447 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:07:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 -- 172.21.15.23:0/697065881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c105ce0 msgr2=0x7fd63c1060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 --2- 172.21.15.23:0/697065881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c105ce0 0x7fd63c1060c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd630009fc0 tx=0x7fd63000b330 comp rx=0 tx=0).stop 2024-03-30T04:07:55.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 -- 172.21.15.23:0/697065881 shutdown_connections 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 --2- 172.21.15.23:0/697065881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c0687f0 0x7fd63c10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 --2- 172.21.15.23:0/697065881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c105ce0 0x7fd63c1060c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 --2- 172.21.15.23:0/697065881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c067e30 0x7fd63c0682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 -- 172.21.15.23:0/697065881 >> 172.21.15.23:0/697065881 conn(0x7fd63c076d40 msgr2=0x7fd63c077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 -- 172.21.15.23:0/697065881 shutdown_connections 2024-03-30T04:07:55.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.687+0000 7fd643e00700 1 -- 172.21.15.23:0/697065881 wait complete. 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.688+0000 7fd643e00700 1 Processor -- start 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 -- start start 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c067e30 0x7fd63c1973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c105ce0 0x7fd63c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19c2c0 con 0x7fd63c105ce0 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19c430 con 0x7fd63c067e30 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd643e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c19c5f0 con 0x7fd63c0687f0 2024-03-30T04:07:55.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd640a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd641400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c067e30 0x7fd63c1973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd640a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41252/0 (socket says 172.21.15.23:41252) 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd640a00700 1 -- 172.21.15.23:0/628162918 learned_addr learned my addr 172.21.15.23:0/628162918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.689+0000 7fd641e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c105ce0 0x7fd63c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 -- 172.21.15.23:0/628162918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c067e30 msgr2=0x7fd63c1973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c067e30 0x7fd63c1973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 -- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c105ce0 msgr2=0x7fd63c19bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c105ce0 0x7fd63c19bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 -- 172.21.15.23:0/628162918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6280095f0 con 0x7fd63c0687f0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd640a00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd62800ec10 tx=0x7fd62800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd63a000700 1 -- 172.21.15.23:0/628162918 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800cd40 con 0x7fd63c0687f0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd63a000700 1 -- 172.21.15.23:0/628162918 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd6280044e0 con 0x7fd63c0687f0 2024-03-30T04:07:55.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd630009c40 con 0x7fd63c0687f0 2024-03-30T04:07:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd63a000700 1 -- 172.21.15.23:0/628162918 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628010690 con 0x7fd63c0687f0 2024-03-30T04:07:55.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.690+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c1a3f30 con 0x7fd63c0687f0 2024-03-30T04:07:55.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.695+0000 7fd63a000700 1 -- 172.21.15.23:0/628162918 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd628003710 con 0x7fd63c0687f0 2024-03-30T04:07:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 Processor -- start 2024-03-30T04:07:55.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 -- start start 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d36925d0 0x5614d36929b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd63a000700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 0x7fd6240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d36a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d35eb990 con 0x5614d3692ef0 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d35e9350 con 0x5614d36a2200 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd657847b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d35e4c40 con 0x5614d36925d0 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd63a000700 1 -- 172.21.15.23:0/628162918 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd628014070 con 0x7fd63c0687f0 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd641e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d36925d0 0x5614d36929b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd641400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd641400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58636/0 (socket says 172.21.15.23:58636) 2024-03-30T04:07:55.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd641400700 1 -- 172.21.15.23:0/1324529641 learned_addr learned my addr 172.21.15.23:0/1324529641 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.697+0000 7fd640a00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 0x7fd6240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641e00700 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d36a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 -- 172.21.15.23:0/1324529641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d36925d0 msgr2=0x5614d36929b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d36925d0 0x5614d36929b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 -- 172.21.15.23:0/1324529641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 msgr2=0x5614d36a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d36a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 -- 172.21.15.23:0/1324529641 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614d34663b0 con 0x5614d3692ef0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641e00700 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d36a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd640a00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 0x7fd6240a3750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6280caa50 tx=0x7fd628003680 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd641400700 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd6300093d0 tx=0x7fd630009400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed8ebac8b5025e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.698+0000 7fd62e600700 1 -- 172.21.15.23:0/1324529641 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000e040 con 0x5614d3692ef0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd62e600700 1 -- 172.21.15.23:0/1324529641 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd630009ac0 con 0x5614d3692ef0 2024-03-30T04:07:55.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd62e600700 1 -- 172.21.15.23:0/1324529641 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6300079d0 con 0x5614d3692ef0 2024-03-30T04:07:55.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 -- 172.21.15.23:0/1324529641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 msgr2=0x5614d36a1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd6300093d0 tx=0x7fd630009400 comp rx=0 tx=0).stop 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 -- 172.21.15.23:0/1324529641 shutdown_connections 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d36a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d36925d0 0x5614d36929b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 --2- 172.21.15.23:0/1324529641 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d3692ef0 0x5614d36a1cc0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 -- 172.21.15.23:0/1324529641 >> 172.21.15.23:0/1324529641 conn(0x5614d35f26a0 msgr2=0x5614d368ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.699+0000 7fd657847b80 1 -- 172.21.15.23:0/1324529641 shutdown_connections 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.700+0000 7fd657847b80 1 -- 172.21.15.23:0/1324529641 wait complete. 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.700+0000 7fd657847b80 1 -- start start 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.700+0000 7fd657847b80 1 Processor -- start 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d369f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d3580d80 con 0x5614d36925d0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d3580a20 con 0x5614d36a2200 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d34686a0 con 0x5614d3692ef0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd657847b80 10 client.0 _ll_register_callbacks cb 0x5614d358a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41280/0 (socket says 172.21.15.23:41280) 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36a1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58638/0 (socket says 172.21.15.23:58638) 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 -- 172.21.15.23:0/3936376513 learned_addr learned my addr 172.21.15.23:0/3936376513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641e00700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d369f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 -- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 msgr2=0x5614d369f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d369f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 -- 172.21.15.23:0/3936376513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 msgr2=0x5614d36a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.701+0000 7fd641400700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd641400700 1 -- 172.21.15.23:0/3936376513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c10e550 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd641400700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd63000bad0 tx=0x7fd63000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21829b0d8546409 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd622a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000e040 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd6300077b0 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614d35dd700 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6300079d0 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd657847b80 10 client.27151 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.702+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5614d3412b60 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5614d35dd700 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd630019070 con 0x5614d3692ef0 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 10 client.27151 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 10 client.27151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:55.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 20 client.27151 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:55.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d36b18d0 0x5614d36b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 4 client.27151 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:55.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5614d3433cf0 con 0x5614d36b18d0 2024-03-30T04:07:55.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.703+0000 7fd641e00700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d36b18d0 0x5614d36b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.704+0000 7fd641e00700 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d36b18d0 0x5614d36b3cf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde8bc157d89354c server_cookie=8f9d82d11d7b2245 in_seq=0 out_seq=0 2024-03-30T04:07:55.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.704+0000 7fd622a00700 10 client.27151 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:55.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.704+0000 7fd622a00700 1 -- 172.21.15.23:0/3936376513 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5614d3433cf0 con 0x5614d36b18d0 2024-03-30T04:07:55.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.704+0000 7fd622a00700 10 client.27151 handle_command_reply: tid=0 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 client.27151 shutdown 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 msgr2=0x5614d369ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd63000bad0 tx=0x7fd63000be80 comp rx=0 tx=0).stop 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 shutdown_connections 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d369f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d36b18d0 0x5614d36b3cf0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 --2- 172.21.15.23:0/3936376513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d369ee90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 >> 172.21.15.23:0/3936376513 conn(0x5614d35f26a0 msgr2=0x5614d36946c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd622a00700 0 client.27151 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 shutdown_connections 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 1 -- 172.21.15.23:0/3936376513 wait complete. 2024-03-30T04:07:55.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.705+0000 7fd657847b80 20 client.27151 trim_cache size 0 max 16384 2024-03-30T04:07:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.726+0000 7fd657847b80 1 Processor -- start 2024-03-30T04:07:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.726+0000 7fd657847b80 1 -- start start 2024-03-30T04:07:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.726+0000 7fd657847b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 0x5614d36a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd657847b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd657847b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36a2200 0x5614d36adae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd657847b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d35dd700 con 0x5614d36a2200 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd657847b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d36944d0 con 0x5614d36925d0 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd657847b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d36ae810 con 0x5614d3692ef0 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 0x5614d36a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 0x5614d36a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34522/0 (socket says 172.21.15.23:34522) 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641e00700 1 -- 172.21.15.23:0/112198663 learned_addr learned my addr 172.21.15.23:0/112198663 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641e00700 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36a2200 0x5614d36adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 -- 172.21.15.23:0/112198663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 msgr2=0x5614d36a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 0x5614d36a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 -- 172.21.15.23:0/112198663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36a2200 msgr2=0x5614d36adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36a2200 0x5614d36adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.727+0000 7fd641400700 1 -- 172.21.15.23:0/112198663 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614d35aaea0 con 0x5614d3692ef0 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.728+0000 7fd641400700 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36ad5a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd630003fd0 tx=0x7fd6300059f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3a18a0bf658c081 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.728+0000 7fd62f000700 1 -- 172.21.15.23:0/112198663 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000e040 con 0x5614d3692ef0 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.728+0000 7fd62f000700 1 -- 172.21.15.23:0/112198663 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd630007e90 con 0x5614d3692ef0 2024-03-30T04:07:55.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.728+0000 7fd62f000700 1 -- 172.21.15.23:0/112198663 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630005310 con 0x5614d3692ef0 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- 172.21.15.23:0/112198663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 msgr2=0x5614d36ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36ad5a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd630003fd0 tx=0x7fd6300059f0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- 172.21.15.23:0/112198663 shutdown_connections 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36a2200 0x5614d36adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36925d0 0x5614d36a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 --2- 172.21.15.23:0/112198663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36ad5a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- 172.21.15.23:0/112198663 >> 172.21.15.23:0/112198663 conn(0x5614d35f26a0 msgr2=0x5614d36875a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- 172.21.15.23:0/112198663 shutdown_connections 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- 172.21.15.23:0/112198663 wait complete. 2024-03-30T04:07:55.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.729+0000 7fd657847b80 1 -- start start 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.730+0000 7fd657847b80 1 Processor -- start 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36f9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d37046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d3704c10 con 0x5614d36925d0 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d3704d50 con 0x5614d36a2200 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614d3704ee0 con 0x5614d3692ef0 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd657847b80 10 client.0 _ll_register_callbacks cb 0x5614d358a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41304/0 (socket says 172.21.15.23:41304) 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 -- 172.21.15.23:0/2325934753 learned_addr learned my addr 172.21.15.23:0/2325934753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641e00700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641e00700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d37046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 -- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 msgr2=0x5614d37046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d37046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 -- 172.21.15.23:0/2325934753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 msgr2=0x5614d36f9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36f9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.731+0000 7fd641400700 1 -- 172.21.15.23:0/2325934753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5614d3412b60 con 0x5614d3692ef0 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd641400700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd630003ed0 tx=0x7fd63000be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45f995f271e1bdf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:55.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd622a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63000e040 con 0x5614d3692ef0 2024-03-30T04:07:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fd6300052d0 con 0x5614d3692ef0 2024-03-30T04:07:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614d35a5380 con 0x5614d3692ef0 2024-03-30T04:07:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6300077b0 con 0x5614d3692ef0 2024-03-30T04:07:55.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd657847b80 10 client.27157 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.732+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5614d346acd0 con 0x5614d3692ef0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5614d35a5380 con 0x5614d3692ef0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd630019070 con 0x5614d3692ef0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 10 client.27157 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 10 client.27157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 20 client.27157 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d3696a30 0x5614d3698e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 4 client.27157 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5614d3468ce0 con 0x5614d3696a30 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.733+0000 7fd641e00700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d3696a30 0x5614d3698e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd622a00700 10 client.27157 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd641e00700 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d3696a30 0x5614d3698e50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba8df40490e9a359 server_cookie=b2d6635b66d34f95 in_seq=0 out_seq=0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd622a00700 1 -- 172.21.15.23:0/2325934753 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5614d3468ce0 con 0x5614d3696a30 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd622a00700 10 client.27157 handle_command_reply: tid=0 2024-03-30T04:07:55.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd657847b80 1 client.27157 shutdown 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 msgr2=0x5614d36f9d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.734+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd630003ed0 tx=0x7fd63000be80 comp rx=0 tx=0).stop 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 shutdown_connections 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5614d36a2200 0x5614d37046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5614d36925d0 0x5614d36f9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5614d3696a30 0x5614d3698e50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 --2- 172.21.15.23:0/2325934753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614d3692ef0 0x5614d36f9d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 >> 172.21.15.23:0/2325934753 conn(0x5614d35f26a0 msgr2=0x5614d36ad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd622a00700 0 client.27157 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 shutdown_connections 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.735+0000 7fd657847b80 1 -- 172.21.15.23:0/2325934753 wait complete. 2024-03-30T04:07:55.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.736+0000 7fd657847b80 20 client.27157 trim_cache size 0 max 16384 2024-03-30T04:07:55.737 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:55.737 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:07:55.737 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:07:55.737 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:55.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.736+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 msgr2=0x7fd6240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.736+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 0x7fd6240a3750 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd6280caa50 tx=0x7fd628003680 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.736+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 msgr2=0x7fd63c1978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.736+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd62800ec10 tx=0x7fd62800ef20 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 shutdown_connections 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd63c105ce0 0x7fd63c19bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd6240a1290 0x7fd6240a3750 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd63c067e30 0x7fd63c1973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 --2- 172.21.15.23:0/628162918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd63c0687f0 0x7fd63c1978e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 >> 172.21.15.23:0/628162918 conn(0x7fd63c076d40 msgr2=0x7fd63c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 shutdown_connections 2024-03-30T04:07:55.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:55.737+0000 7fd643e00700 1 -- 172.21.15.23:0/628162918 wait complete. 2024-03-30T04:07:55.754 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:07:55.755 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:07:56.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.037+0000 7fc3dec00700 1 -- 172.21.15.23:0/338874182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 msgr2=0x7fc3d8102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.037+0000 7fc3dec00700 1 --2- 172.21.15.23:0/338874182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8102190 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009990 tx=0x7fc3c8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.038+0000 7fc3dec00700 1 -- 172.21.15.23:0/338874182 shutdown_connections 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.038+0000 7fc3dec00700 1 --2- 172.21.15.23:0/338874182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3d810b1d0 0x7fc3d810d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.038+0000 7fc3dec00700 1 --2- 172.21.15.23:0/338874182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8102190 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.038+0000 7fc3dec00700 1 --2- 172.21.15.23:0/338874182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.038+0000 7fc3dec00700 1 -- 172.21.15.23:0/338874182 >> 172.21.15.23:0/338874182 conn(0x7fc3d80fb5f0 msgr2=0x7fc3d80fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.039+0000 7fc3dec00700 1 -- 172.21.15.23:0/338874182 shutdown_connections 2024-03-30T04:07:56.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.039+0000 7fc3dec00700 1 -- 172.21.15.23:0/338874182 wait complete. 2024-03-30T04:07:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.040+0000 7fc3dec00700 1 Processor -- start 2024-03-30T04:07:56.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.041+0000 7fc3dec00700 1 -- start start 2024-03-30T04:07:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.042+0000 7fc3dec00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8109d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.042+0000 7fc3dec00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.042+0000 7fc3dec00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3d810b1d0 0x7fc3d81997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3dec00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8199e20 con 0x7fc3d810b1d0 2024-03-30T04:07:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3dec00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8199f90 con 0x7fc3d8101db0 2024-03-30T04:07:56.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3dec00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d819a150 con 0x7fc3d81026d0 2024-03-30T04:07:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3d7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3d7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41324/0 (socket says 172.21.15.23:41324) 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3d7400700 1 -- 172.21.15.23:0/959444741 learned_addr learned my addr 172.21.15.23:0/959444741 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.043+0000 7fc3d7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.044+0000 7fc3d7400700 1 -- 172.21.15.23:0/959444741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 msgr2=0x7fc3d8109d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.044+0000 7fc3d7400700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8109d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.044+0000 7fc3d7400700 1 -- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3d810b1d0 msgr2=0x7fc3d81997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.044+0000 7fc3d7400700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3d810b1d0 0x7fc3d81997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.044+0000 7fc3d7400700 1 -- 172.21.15.23:0/959444741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c80095f0 con 0x7fc3d81026d0 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.045+0000 7fc3d7400700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400d950 tx=0x7fc3c400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.045+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4009930 con 0x7fc3d81026d0 2024-03-30T04:07:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.045+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d819a430 con 0x7fc3d81026d0 2024-03-30T04:07:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.046+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d8196960 con 0x7fc3d81026d0 2024-03-30T04:07:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.046+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3c4010460 con 0x7fc3d81026d0 2024-03-30T04:07:56.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.046+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c400f620 con 0x7fc3d81026d0 2024-03-30T04:07:56.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.047+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b80052f0 con 0x7fc3d81026d0 2024-03-30T04:07:56.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.052+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc3c4009a90 con 0x7fc3d81026d0 2024-03-30T04:07:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.056+0000 7fc3d4c00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 0x7fc3bc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.056+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc3c40cccc0 con 0x7fc3d81026d0 2024-03-30T04:07:56.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.057+0000 7fc3d7e00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 0x7fc3bc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.058+0000 7fc3d7e00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 0x7fc3bc0a37d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006010 tx=0x7fc3c800b540 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.060+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fc3c4095d90 con 0x7fc3d81026d0 2024-03-30T04:07:56.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 581379 11.51 MB/sec execute 153 sec latency 54.837 ms 2024-03-30T04:07:56.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.388+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3b8005160 con 0x7fc3d81026d0 2024-03-30T04:07:56.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.389+0000 7fc3d4c00700 1 -- 172.21.15.23:0/959444741 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fc3c4016090 con 0x7fc3d81026d0 2024-03-30T04:07:56.387 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:07:56.388 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:07:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 msgr2=0x7fc3bc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 0x7fc3bc0a37d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8006010 tx=0x7fc3c800b540 comp rx=0 tx=0).stop 2024-03-30T04:07:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 msgr2=0x7fc3d810a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400d950 tx=0x7fc3c400dc60 comp rx=0 tx=0).stop 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 shutdown_connections 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc3d810b1d0 0x7fc3d81997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc3bc0a1310 0x7fc3bc0a37d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3d8101db0 0x7fc3d8109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 --2- 172.21.15.23:0/959444741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc3d81026d0 0x7fc3d810a2f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 >> 172.21.15.23:0/959444741 conn(0x7fc3d80fb5f0 msgr2=0x7fc3d8100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.392+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 shutdown_connections 2024-03-30T04:07:56.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.393+0000 7fc3dec00700 1 -- 172.21.15.23:0/959444741 wait complete. 2024-03-30T04:07:56.392 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:07:56.410 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:56 smithi023 ceph-mon[38254]: pgmap v182: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-03-30T04:07:56.413 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:07:56.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:56 smithi154 ceph-mon[79161]: pgmap v182: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-03-30T04:07:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:56 smithi193 ceph-mon[79581]: pgmap v182: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.1 MiB/s wr, 190 op/s 2024-03-30T04:07:56.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 -- 172.21.15.23:0/2459780888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc106080 msgr2=0x7f25fc10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 --2- 172.21.15.23:0/2459780888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc106080 0x7f25fc10dc00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009f90 tx=0x7f25f400b330 comp rx=0 tx=0).stop 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 -- 172.21.15.23:0/2459780888 shutdown_connections 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 --2- 172.21.15.23:0/2459780888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc106080 0x7f25fc10dc00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 --2- 172.21.15.23:0/2459780888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc100f60 0x7f25fc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 --2- 172.21.15.23:0/2459780888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc101910 0x7f25fc105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 -- 172.21.15.23:0/2459780888 >> 172.21.15.23:0/2459780888 conn(0x7f25fc0fc7c0 msgr2=0x7f25fc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 -- 172.21.15.23:0/2459780888 shutdown_connections 2024-03-30T04:07:56.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.637+0000 7f2601400700 1 -- 172.21.15.23:0/2459780888 wait complete. 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.638+0000 7f2601400700 1 Processor -- start 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.638+0000 7f2601400700 1 -- start start 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.638+0000 7f2601400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f2601400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 0x7f25fc1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f2601400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 0x7f25fc19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f2601400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc19c500 con 0x7f25fc106080 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f2601400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc19c670 con 0x7f25fc19ba80 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f2601400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25fc19c830 con 0x7f25fc100f60 2024-03-30T04:07:56.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41340/0 (socket says 172.21.15.23:41340) 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 -- 172.21.15.23:0/4069941719 learned_addr learned my addr 172.21.15.23:0/4069941719 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25f9e00700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 0x7f25fc1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fb200700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 0x7f25fc19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 -- 172.21.15.23:0/4069941719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 msgr2=0x7f25fc19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 0x7f25fc19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 -- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 msgr2=0x7f25fc1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 0x7f25fc1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fa800700 1 -- 172.21.15.23:0/4069941719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec0095f0 con 0x7f25fc100f60 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25f9e00700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 0x7f25fc1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.639+0000 7f25fb200700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 0x7f25fc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f25fa800700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ec10 tx=0x7f25ec00c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f25f3400700 1 -- 172.21.15.23:0/4069941719 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec009a10 con 0x7f25fc100f60 2024-03-30T04:07:56.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f25f3400700 1 -- 172.21.15.23:0/4069941719 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25ec0044e0 con 0x7f25fc100f60 2024-03-30T04:07:56.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e8009520 con 0x7f25fc100f60 2024-03-30T04:07:56.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f25f3400700 1 -- 172.21.15.23:0/4069941719 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec010490 con 0x7f25fc100f60 2024-03-30T04:07:56.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.640+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f4009c40 con 0x7f25fc100f60 2024-03-30T04:07:56.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.644+0000 7f25f3400700 1 -- 172.21.15.23:0/4069941719 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f25ec003680 con 0x7f25fc100f60 2024-03-30T04:07:56.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f25f3400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 0x7f25e00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f2614fccb80 1 Processor -- start 2024-03-30T04:07:56.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f25f3400700 1 -- 172.21.15.23:0/4069941719 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f25ec014070 con 0x7f25fc100f60 2024-03-30T04:07:56.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f2614fccb80 1 -- start start 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f25f9e00700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 0x7f25e00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.646+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cc58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f2614fccb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714c13990 con 0x563714cbaef0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f2614fccb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714c11350 con 0x563714cba5d0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f2614fccb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714c0cc40 con 0x563714cc5df0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34560/0 (socket says 172.21.15.23:34560) 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 -- 172.21.15.23:0/1604797220 learned_addr learned my addr 172.21.15.23:0/1604797220 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fb200700 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25f9e00700 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cc58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 -- 172.21.15.23:0/1604797220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 msgr2=0x563714cc81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 -- 172.21.15.23:0/1604797220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 msgr2=0x563714cc58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cc58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25f9e00700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 0x7f25e00a37d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f25fc1986f0 tx=0x7f25e80106b0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 -- 172.21.15.23:0/1604797220 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563714a8e3b0 con 0x563714cba5d0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.647+0000 7f25fa800700 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005d70 tx=0x7f25ec099df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e05befcbe662396 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.648+0000 7f25e7400700 1 -- 172.21.15.23:0/1604797220 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec09e1c0 con 0x563714cba5d0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.648+0000 7f25e7400700 1 -- 172.21.15.23:0/1604797220 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25ec0a2810 con 0x563714cba5d0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.648+0000 7f25e7400700 1 -- 172.21.15.23:0/1604797220 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25ec096c80 con 0x563714cba5d0 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- 172.21.15.23:0/1604797220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 msgr2=0x563714cba9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005d70 tx=0x7f25ec099df0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- 172.21.15.23:0/1604797220 shutdown_connections 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cba9b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 --2- 172.21.15.23:0/1604797220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cc58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- 172.21.15.23:0/1604797220 >> 172.21.15.23:0/1604797220 conn(0x563714c1a6a0 msgr2=0x563714cb6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- 172.21.15.23:0/1604797220 shutdown_connections 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- 172.21.15.23:0/1604797220 wait complete. 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.649+0000 7f2614fccb80 1 -- start start 2024-03-30T04:07:56.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.650+0000 7f2614fccb80 1 Processor -- start 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.650+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cba5d0 0x563714cc5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.650+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f2614fccb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714ba8d80 con 0x563714cba5d0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f2614fccb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714ba8a20 con 0x563714cbaef0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f2614fccb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714a906a0 con 0x563714cc5df0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f2614fccb80 10 client.0 _ll_register_callbacks cb 0x563714bb2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fa800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cba5d0 0x563714cc5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34572/0 (socket says 172.21.15.23:34572) 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25f9e00700 1 -- 172.21.15.23:0/2023276820 learned_addr learned my addr 172.21.15.23:0/2023276820 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 -- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 msgr2=0x563714cc2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 -- 172.21.15.23:0/2023276820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cba5d0 msgr2=0x563714cc5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cba5d0 0x563714cc5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25fb200700 1 -- 172.21.15.23:0/2023276820 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25f4009c40 con 0x563714cc5df0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.651+0000 7f25f9e00700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25fb200700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc3250 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009c10 tx=0x7f25f4004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1326a834d140139e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f400e040 con 0x563714cc5df0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f25f4022070 con 0x563714cc5df0 2024-03-30T04:07:56.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563714bcd380 con 0x563714cc5df0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25f4004e50 con 0x563714cc5df0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f2614fccb80 10 client.27175 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563714a3ab60 con 0x563714cc5df0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563714bcd380 con 0x563714cc5df0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f25f400b050 con 0x563714cc5df0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.652+0000 7f2614fccb80 10 client.27175 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f2614fccb80 10 client.27175 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f2614fccb80 20 client.27175 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714cd9dc0 0x563714cdc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f2614fccb80 4 client.27175 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563714c05700 con 0x563714cd9dc0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f25fa800700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714cd9dc0 0x563714cdc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f25dbe00700 10 client.27175 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:56.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.653+0000 7f25fa800700 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714cd9dc0 0x563714cdc1e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9e8c14e5835d814 server_cookie=3f51e219345f2c1f in_seq=0 out_seq=0 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f25dbe00700 1 -- 172.21.15.23:0/2023276820 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563714c05700 con 0x563714cd9dc0 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f25dbe00700 10 client.27175 handle_command_reply: tid=0 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 client.27175 shutdown 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 msgr2=0x563714cc3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc3250 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25f4009c10 tx=0x7f25f4004c60 comp rx=0 tx=0).stop 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 shutdown_connections 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714cd9dc0 0x563714cdc1e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714cc3250 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cba5d0 0x563714cc5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2023276820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cbaef0 0x563714cc2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 >> 172.21.15.23:0/2023276820 conn(0x563714c1a6a0 msgr2=0x563714cbc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f25dbe00700 0 client.27175 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:56.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.654+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 shutdown_connections 2024-03-30T04:07:56.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.655+0000 7f2614fccb80 1 -- 172.21.15.23:0/2023276820 wait complete. 2024-03-30T04:07:56.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.655+0000 7f2614fccb80 20 client.27175 trim_cache size 0 max 16384 2024-03-30T04:07:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 Processor -- start 2024-03-30T04:07:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 -- start start 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cc3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714caff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714cbbf90 con 0x563714cbaef0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714cb8160 con 0x563714cba5d0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f2614fccb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714cb6fa0 con 0x563714cc5df0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cc3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cc3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34580/0 (socket says 172.21.15.23:34580) 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f25fa800700 1 -- 172.21.15.23:0/2574889346 learned_addr learned my addr 172.21.15.23:0/2574889346 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.671+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58708/0 (socket says 172.21.15.23:58708) 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25fb200700 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714caff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 -- 172.21.15.23:0/2574889346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 msgr2=0x563714caff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714caff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 -- 172.21.15.23:0/2574889346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 msgr2=0x563714cc3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cc3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 -- 172.21.15.23:0/2574889346 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563714bd2ea0 con 0x563714cbaef0 2024-03-30T04:07:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25f9e00700 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x563714cc40e0 tx=0x7f25e8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cfac595d39a9764 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.672+0000 7f25e7e00700 1 -- 172.21.15.23:0/2574889346 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8024070 con 0x563714cbaef0 2024-03-30T04:07:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f25e7e00700 1 -- 172.21.15.23:0/2574889346 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25e8017070 con 0x563714cbaef0 2024-03-30T04:07:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f25e7e00700 1 -- 172.21.15.23:0/2574889346 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8028410 con 0x563714cbaef0 2024-03-30T04:07:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 -- 172.21.15.23:0/2574889346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 msgr2=0x563714cd0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x563714cc40e0 tx=0x7f25e8015000 comp rx=0 tx=0).stop 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 -- 172.21.15.23:0/2574889346 shutdown_connections 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cc5df0 0x563714caff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cc3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 --2- 172.21.15.23:0/2574889346 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cbaef0 0x563714cd0e80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.673+0000 7f2614fccb80 1 -- 172.21.15.23:0/2574889346 >> 172.21.15.23:0/2574889346 conn(0x563714c1a6a0 msgr2=0x563714caf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.674+0000 7f2614fccb80 1 -- 172.21.15.23:0/2574889346 shutdown_connections 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.674+0000 7f2614fccb80 1 -- 172.21.15.23:0/2574889346 wait complete. 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.674+0000 7f2614fccb80 1 -- start start 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.674+0000 7f2614fccb80 1 Processor -- start 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cd0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 0x563714ccf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714ccf750 con 0x563714cc5df0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714ccf8c0 con 0x563714cba5d0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563714ccfa80 con 0x563714cbaef0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f2614fccb80 10 client.0 _ll_register_callbacks cb 0x563714bb2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cd0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25fb200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 0x563714ccf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41378/0 (socket says 172.21.15.23:41378) 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25fb200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 0x563714ccf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58722/0 (socket says 172.21.15.23:58722) 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25fb200700 1 -- 172.21.15.23:0/3261895768 learned_addr learned my addr 172.21.15.23:0/3261895768 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25fa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cd0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34588/0 (socket says 172.21.15.23:34588) 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 -- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 msgr2=0x563714cd0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cd0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 -- 172.21.15.23:0/3261895768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 msgr2=0x563714ccf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.675+0000 7f25f9e00700 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 0x563714ccf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25f9e00700 1 -- 172.21.15.23:0/3261895768 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e8010d90 con 0x563714cbaef0 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25f9e00700 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f25e8011d10 tx=0x7f25e8003e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79ded5d2420e330 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25dbe00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e8024070 con 0x563714cbaef0 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f25ec0a2810 con 0x563714cbaef0 2024-03-30T04:07:56.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f25e8019070 con 0x563714cbaef0 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25e8004510 con 0x563714cbaef0 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f2614fccb80 10 client.27181 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.676+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563714a92cd0 con 0x563714cbaef0 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25ec0a2810 con 0x563714cbaef0 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f25e8004b30 con 0x563714cbaef0 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 10 client.27181 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 10 client.27181 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:07:56.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 20 client.27181 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714d2ef00 0x563714d31320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 4 client.27181 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563714a5bcf0 con 0x563714d2ef00 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.677+0000 7f25fa800700 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714d2ef00 0x563714d31320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f25fa800700 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714d2ef00 0x563714d31320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1a01d47ef444c60a server_cookie=b11c3c98252e0ecc in_seq=0 out_seq=0 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f25dbe00700 10 client.27181 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f25dbe00700 1 -- 172.21.15.23:0/3261895768 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563714a5bcf0 con 0x563714d2ef00 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f25dbe00700 10 client.27181 handle_command_reply: tid=0 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 client.27181 shutdown 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 msgr2=0x563714cd0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f25e8011d10 tx=0x7f25e8003e90 comp rx=0 tx=0).stop 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 shutdown_connections 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563714cc5df0 0x563714ccf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563714cba5d0 0x563714cd0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x563714d2ef00 0x563714d31320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 --2- 172.21.15.23:0/3261895768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563714cbaef0 0x563714cd0e60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f25dbe00700 0 client.27181 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 >> 172.21.15.23:0/3261895768 conn(0x563714c1a6a0 msgr2=0x563714cc53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 shutdown_connections 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.678+0000 7f2614fccb80 1 -- 172.21.15.23:0/3261895768 wait complete. 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2614fccb80 20 client.27181 trim_cache size 0 max 16384 2024-03-30T04:07:56.681 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 msgr2=0x7f25e00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 0x7f25e00a37d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f25fc1986f0 tx=0x7f25e80106b0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 msgr2=0x7f25fc1971a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ec10 tx=0x7f25ec00c2d0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 shutdown_connections 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f25e00a1310 0x7f25e00a37d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f25fc19ba80 0x7f25fc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25fc100f60 0x7f25fc1971a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 --2- 172.21.15.23:0/4069941719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25fc106080 0x7f25fc1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 >> 172.21.15.23:0/4069941719 conn(0x7f25fc0fc7c0 msgr2=0x7f25fc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 shutdown_connections 2024-03-30T04:07:56.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.679+0000 7f2601400700 1 -- 172.21.15.23:0/4069941719 wait complete. 2024-03-30T04:07:56.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:07:56.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.911+0000 7f59b7800700 1 -- 172.21.15.23:0/2302185771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101910 msgr2=0x7f59b0105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.911+0000 7f59b7800700 1 --2- 172.21.15.23:0/2302185771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101910 0x7f59b0105950 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f599c009990 tx=0x7f599c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 -- 172.21.15.23:0/2302185771 shutdown_connections 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 --2- 172.21.15.23:0/2302185771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0106080 0x7f59b010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 --2- 172.21.15.23:0/2302185771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0100f60 0x7f59b0101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 --2- 172.21.15.23:0/2302185771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0101910 0x7f59b0105950 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 -- 172.21.15.23:0/2302185771 >> 172.21.15.23:0/2302185771 conn(0x7f59b00fc7c0 msgr2=0x7f59b00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 -- 172.21.15.23:0/2302185771 shutdown_connections 2024-03-30T04:07:56.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.912+0000 7f59b7800700 1 -- 172.21.15.23:0/2302185771 wait complete. 2024-03-30T04:07:56.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.913+0000 7f59b7800700 1 Processor -- start 2024-03-30T04:07:56.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.913+0000 7f59b7800700 1 -- start start 2024-03-30T04:07:56.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0106080 0x7f59b00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0073b30 0x7f59b01a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01a59f0 con 0x7f59b0100f60 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01a5b60 con 0x7f59b0073b30 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b7800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b01a5d20 con 0x7f59b0106080 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58732/0 (socket says 172.21.15.23:58732) 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b4e00700 1 -- 172.21.15.23:0/4161001777 learned_addr learned my addr 172.21.15.23:0/4161001777 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59afe00700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0106080 0x7f59b00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.914+0000 7f59b5800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0073b30 0x7f59b01a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 -- 172.21.15.23:0/4161001777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0106080 msgr2=0x7f59b00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0106080 0x7f59b00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 -- 172.21.15.23:0/4161001777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0073b30 msgr2=0x7f59b01a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0073b30 0x7f59b01a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 -- 172.21.15.23:0/4161001777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4009520 con 0x7f59b0100f60 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b4e00700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f59a400eb30 tx=0x7f59a400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59ad600700 1 -- 172.21.15.23:0/4161001777 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4009880 con 0x7f59b0100f60 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f599c0095f0 con 0x7f59b0100f60 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59ad600700 1 -- 172.21.15.23:0/4161001777 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59a40044e0 con 0x7f59b0100f60 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59ad600700 1 -- 172.21.15.23:0/4161001777 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4003d80 con 0x7f59b0100f60 2024-03-30T04:07:56.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.915+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b01a6290 con 0x7f59b0100f60 2024-03-30T04:07:56.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.920+0000 7f59ad600700 1 -- 172.21.15.23:0/4161001777 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f59a4004000 con 0x7f59b0100f60 2024-03-30T04:07:56.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.921+0000 7f59cb383b80 1 Processor -- start 2024-03-30T04:07:56.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.921+0000 7f59cb383b80 1 -- start start 2024-03-30T04:07:56.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf29635d0 0x55aaf29639b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 0x55aaf296e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf29711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf28bc990 con 0x55aaf2963ef0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf28ba350 con 0x55aaf296edf0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59cb383b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf28b5c40 con 0x55aaf29635d0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 0x55aaf296e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59b5800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf29635d0 0x55aaf29639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 0x55aaf296e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58748/0 (socket says 172.21.15.23:58748) 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59b4e00700 1 -- 172.21.15.23:0/2252501502 learned_addr learned my addr 172.21.15.23:0/2252501502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf29711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59ad600700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 0x7f59980a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 -- 172.21.15.23:0/2252501502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf29635d0 msgr2=0x55aaf29639b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf29635d0 0x55aaf29639b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 -- 172.21.15.23:0/2252501502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 msgr2=0x55aaf296e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 0x55aaf296e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.922+0000 7f59afe00700 1 -- 172.21.15.23:0/2252501502 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaf27373b0 con 0x55aaf296edf0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59ad600700 1 -- 172.21.15.23:0/4161001777 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f59a4014070 con 0x7f59b0100f60 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59b5800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 0x7f59980a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59afe00700 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf29711e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f599c0098e0 tx=0x7f599c005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a9ccdb54e996f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59a2000700 1 -- 172.21.15.23:0/2252501502 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c01d070 con 0x55aaf296edf0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59a2000700 1 -- 172.21.15.23:0/2252501502 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f599c01e040 con 0x55aaf296edf0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.923+0000 7f59a2000700 1 -- 172.21.15.23:0/2252501502 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c0047f0 con 0x55aaf296edf0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 -- 172.21.15.23:0/2252501502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 msgr2=0x55aaf29711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf29711e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f599c0098e0 tx=0x7f599c005ec0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59b5800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 0x7f59980a3750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59a80075c0 tx=0x7f59a8009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 -- 172.21.15.23:0/2252501502 shutdown_connections 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf29711e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf29635d0 0x55aaf29639b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 --2- 172.21.15.23:0/2252501502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf2963ef0 0x55aaf296e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 -- 172.21.15.23:0/2252501502 >> 172.21.15.23:0/2252501502 conn(0x55aaf28c36a0 msgr2=0x55aaf295fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 -- 172.21.15.23:0/2252501502 shutdown_connections 2024-03-30T04:07:56.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.924+0000 7f59cb383b80 1 -- 172.21.15.23:0/2252501502 wait complete. 2024-03-30T04:07:56.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.925+0000 7f59cb383b80 1 -- start start 2024-03-30T04:07:56.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.925+0000 7f59cb383b80 1 Processor -- start 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf296e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 0x55aaf296c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf2851d80 con 0x55aaf29635d0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf2851a20 con 0x55aaf2963ef0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf27396a0 con 0x55aaf296edf0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59cb383b80 10 client.0 _ll_register_callbacks cb 0x55aaf285b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf296e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34632/0 (socket says 172.21.15.23:34632) 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59afe00700 1 -- 172.21.15.23:0/1304495513 learned_addr learned my addr 172.21.15.23:0/1304495513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.926+0000 7f59b5800700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 0x55aaf296c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 -- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 msgr2=0x55aaf296c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 0x55aaf296c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 -- 172.21.15.23:0/1304495513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 msgr2=0x55aaf296e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf296e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 -- 172.21.15.23:0/1304495513 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b010e470 con 0x55aaf2963ef0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59b4e00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf296e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59b5800700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 0x55aaf296c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59afe00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f599c009ce0 tx=0x7f599c005af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f05c8e6257a158a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f5996a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c01d030 con 0x55aaf2963ef0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f599c00bc60 con 0x55aaf2963ef0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.927+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aaf2876380 con 0x55aaf2963ef0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f599c004500 con 0x55aaf2963ef0 2024-03-30T04:07:56.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 10 client.27197 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55aaf26e3b60 con 0x55aaf2963ef0 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aaf2876380 con 0x55aaf2963ef0 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f599c00f840 con 0x55aaf2963ef0 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 10 client.27197 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 10 client.27197 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 20 client.27197 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf297e410 0x55aaf2980800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 4 client.27197 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aaf28ae700 con 0x55aaf297e410 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.928+0000 7f59b4e00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf297e410 0x55aaf2980800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.929+0000 7f59b4e00700 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf297e410 0x55aaf2980800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=470d16683c41018a server_cookie=6ad83da20a0d6cab in_seq=0 out_seq=0 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.929+0000 7f5996a00700 10 client.27197 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.929+0000 7f5996a00700 1 -- 172.21.15.23:0/1304495513 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aaf28ae700 con 0x55aaf297e410 2024-03-30T04:07:56.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.929+0000 7f5996a00700 10 client.27197 handle_command_reply: tid=0 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 client.27197 shutdown 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 msgr2=0x55aaf296bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f599c009ce0 tx=0x7f599c005af0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 shutdown_connections 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf297e410 0x55aaf2980800 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf296edf0 0x55aaf296c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf296e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 --2- 172.21.15.23:0/1304495513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf2963ef0 0x55aaf296bb00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f5996a00700 0 client.27197 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 >> 172.21.15.23:0/1304495513 conn(0x55aaf28c36a0 msgr2=0x55aaf29656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 shutdown_connections 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 1 -- 172.21.15.23:0/1304495513 wait complete. 2024-03-30T04:07:56.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.930+0000 7f59cb383b80 20 client.27197 trim_cache size 0 max 16384 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.945+0000 7f59cb383b80 1 Processor -- start 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.945+0000 7f59cb383b80 1 -- start start 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf2975a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf296b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf2964850 con 0x55aaf29635d0 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf2975f50 con 0x55aaf296edf0 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59cb383b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf296a050 con 0x55aaf2963ef0 2024-03-30T04:07:56.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41438/0 (socket says 172.21.15.23:41438) 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 -- 172.21.15.23:0/4061169716 learned_addr learned my addr 172.21.15.23:0/4061169716 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59b4e00700 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf2975a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59b5800700 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf296b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 -- 172.21.15.23:0/4061169716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 msgr2=0x55aaf296b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf296b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 -- 172.21.15.23:0/4061169716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 msgr2=0x55aaf2975a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf2975a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 -- 172.21.15.23:0/4061169716 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aaf287bea0 con 0x55aaf2963ef0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.946+0000 7f59afe00700 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f599c0098e0 tx=0x7f599c009300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84a4ef8ebe457e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59a2a00700 1 -- 172.21.15.23:0/4061169716 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c01d040 con 0x55aaf2963ef0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59a2a00700 1 -- 172.21.15.23:0/4061169716 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f599c00f460 con 0x55aaf2963ef0 2024-03-30T04:07:56.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59a2a00700 1 -- 172.21.15.23:0/4061169716 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c02b800 con 0x55aaf2963ef0 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 -- 172.21.15.23:0/4061169716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 msgr2=0x55aaf296e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f599c0098e0 tx=0x7f599c009300 comp rx=0 tx=0).stop 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 -- 172.21.15.23:0/4061169716 shutdown_connections 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf296edf0 0x55aaf296b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf29635d0 0x55aaf2975a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 --2- 172.21.15.23:0/4061169716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf296e2f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.947+0000 7f59cb383b80 1 -- 172.21.15.23:0/4061169716 >> 172.21.15.23:0/4061169716 conn(0x55aaf28c36a0 msgr2=0x55aaf296a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.948+0000 7f59cb383b80 1 -- 172.21.15.23:0/4061169716 shutdown_connections 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.948+0000 7f59cb383b80 1 -- 172.21.15.23:0/4061169716 wait complete. 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.948+0000 7f59cb383b80 1 -- start start 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.948+0000 7f59cb383b80 1 Processor -- start 2024-03-30T04:07:56.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf2973f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf296edf0 0x55aaf29d1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf2974840 con 0x55aaf296edf0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf29d16b0 con 0x55aaf29635d0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aaf29d17f0 con 0x55aaf2963ef0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59cb383b80 10 client.0 _ll_register_callbacks cb 0x55aaf285b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34654/0 (socket says 172.21.15.23:34654) 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 -- 172.21.15.23:0/3558823717 learned_addr learned my addr 172.21.15.23:0/3558823717 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b5800700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf296edf0 0x55aaf29d1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59afe00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf2973f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 -- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 msgr2=0x55aaf2973f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf2973f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 -- 172.21.15.23:0/3558823717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf296edf0 msgr2=0x55aaf29d1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf296edf0 0x55aaf29d1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.949+0000 7f59b4e00700 1 -- 172.21.15.23:0/3558823717 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aaf26e3b60 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f59b4e00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f59a4005bf0 tx=0x7f59a40d0e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=526c1bdfa584c83f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4099e50 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f59a400cbd0 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f599c01e040 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59a4099c70 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f59cb383b80 10 client.27203 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55aaf273bcd0 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f599c01e040 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.950+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f59a40cd1b0 con 0x55aaf29635d0 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 10 client.27203 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:56.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 10 client.27203 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 20 client.27203 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf29d2e00 0x55aaf29d5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 4 client.27203 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55aaf2704cf0 con 0x55aaf29d2e00 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59afe00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf29d2e00 0x55aaf29d5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f59afe00700 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf29d2e00 0x55aaf29d5220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c754ace4f2f68626 server_cookie=ec09c6153b33c552 in_seq=0 out_seq=0 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.951+0000 7f5996a00700 10 client.27203 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f5996a00700 1 -- 172.21.15.23:0/3558823717 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55aaf2704cf0 con 0x55aaf29d2e00 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f5996a00700 10 client.27203 handle_command_reply: tid=0 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 client.27203 shutdown 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 msgr2=0x55aaf29739f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f59a4005bf0 tx=0x7f59a40d0e70 comp rx=0 tx=0).stop 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 shutdown_connections 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55aaf29d2e00 0x55aaf29d5220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aaf296edf0 0x55aaf29d1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aaf29635d0 0x55aaf29739f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 --2- 172.21.15.23:0/3558823717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aaf2963ef0 0x55aaf2973f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f5996a00700 0 client.27203 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 >> 172.21.15.23:0/3558823717 conn(0x55aaf28c36a0 msgr2=0x55aaf296b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 shutdown_connections 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 1 -- 172.21.15.23:0/3558823717 wait complete. 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.952+0000 7f59cb383b80 20 client.27203 trim_cache size 0 max 16384 2024-03-30T04:07:56.955 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 msgr2=0x7f59980a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 0x7f59980a3750 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f59a80075c0 tx=0x7f59a8009040 comp rx=0 tx=0).stop 2024-03-30T04:07:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 msgr2=0x7f59b00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:56.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f59a400eb30 tx=0x7f59a400ee40 comp rx=0 tx=0).stop 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 shutdown_connections 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f59980a1290 0x7f59980a3750 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59b0073b30 0x7f59b01a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59b0100f60 0x7f59b00730b0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 --2- 172.21.15.23:0/4161001777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b0106080 0x7f59b00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 >> 172.21.15.23:0/4161001777 conn(0x7f59b00fc7c0 msgr2=0x7f59b00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 shutdown_connections 2024-03-30T04:07:56.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:56.953+0000 7f59b7800700 1 -- 172.21.15.23:0/4161001777 wait complete. 2024-03-30T04:07:56.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:07:57.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 -- 172.21.15.23:0/781570959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 msgr2=0x7f54a8073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 --2- 172.21.15.23:0/781570959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a8073530 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f549c009990 tx=0x7f549c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 -- 172.21.15.23:0/781570959 shutdown_connections 2024-03-30T04:07:57.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 --2- 172.21.15.23:0/781570959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8111d00 0x7f54a81140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 --2- 172.21.15.23:0/781570959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a8073530 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 --2- 172.21.15.23:0/781570959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8073a70 0x7f54a81117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 -- 172.21.15.23:0/781570959 >> 172.21.15.23:0/781570959 conn(0x7f54a80fc7c0 msgr2=0x7f54a80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 -- 172.21.15.23:0/781570959 shutdown_connections 2024-03-30T04:07:57.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.192+0000 7f54afe00700 1 -- 172.21.15.23:0/781570959 wait complete. 2024-03-30T04:07:57.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 Processor -- start 2024-03-30T04:07:57.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 -- start start 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8111d00 0x7f54a819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a819dce0 con 0x7f54a8073a70 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a819de50 con 0x7f54a8073150 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.194+0000 7f54afe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a819e010 con 0x7f54a8111d00 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54aca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54aca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58786/0 (socket says 172.21.15.23:58786) 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34672/0 (socket says 172.21.15.23:34672) 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54aca00700 1 -- 172.21.15.23:0/3682937885 learned_addr learned my addr 172.21.15.23:0/3682937885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ade00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8111d00 0x7f54a819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 -- 172.21.15.23:0/3682937885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8111d00 msgr2=0x7f54a819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8111d00 0x7f54a819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 -- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 msgr2=0x7f54a81a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 -- 172.21.15.23:0/3682937885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5494009520 con 0x7f54a8073150 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54aca00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.195+0000 7f54ad400700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f549c0098e0 tx=0x7f549c005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.196+0000 7f54a6000700 1 -- 172.21.15.23:0/3682937885 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c01d070 con 0x7f54a8073150 2024-03-30T04:07:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.196+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c0095f0 con 0x7f54a8073150 2024-03-30T04:07:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.196+0000 7f54a6000700 1 -- 172.21.15.23:0/3682937885 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549c022470 con 0x7f54a8073150 2024-03-30T04:07:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.196+0000 7f54a6000700 1 -- 172.21.15.23:0/3682937885 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c017650 con 0x7f54a8073150 2024-03-30T04:07:57.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.196+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81aa360 con 0x7f54a8073150 2024-03-30T04:07:57.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.199+0000 7f54a6000700 1 -- 172.21.15.23:0/3682937885 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f549c0225e0 con 0x7f54a8073150 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.200+0000 7f54a6000700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 0x7f54900a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.200+0000 7f54c38bab80 1 Processor -- start 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.200+0000 7f54a6000700 1 -- 172.21.15.23:0/3682937885 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f549c0ce4a0 con 0x7f54a8073150 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 -- start start 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54aca00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 0x7f54900a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fa59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98efe990 con 0x564a98fa55d0 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98efc350 con 0x564a98fb0df0 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54c38bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98ef7c40 con 0x564a98fa5ef0 2024-03-30T04:07:57.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fa59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41480/0 (socket says 172.21.15.23:41480) 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54aca00700 1 -- 172.21.15.23:0/1635460004 learned_addr learned my addr 172.21.15.23:0/1635460004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.201+0000 7f54ade00700 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 0x7f54900a3750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5494009fd0 tx=0x7f5494009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 -- 172.21.15.23:0/1635460004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 msgr2=0x564a98fb31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 -- 172.21.15.23:0/1635460004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 msgr2=0x564a98fa59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fa59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 -- 172.21.15.23:0/1635460004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a98d793b0 con 0x564a98fa5ef0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54ad400700 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fa59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f54aca00700 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5494009930 tx=0x7f549401c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ddcb6a21343595a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f549a600700 1 -- 172.21.15.23:0/1635460004 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549401c600 con 0x564a98fa5ef0 2024-03-30T04:07:57.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f549a600700 1 -- 172.21.15.23:0/1635460004 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549401cc20 con 0x564a98fa5ef0 2024-03-30T04:07:57.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.202+0000 7f549a600700 1 -- 172.21.15.23:0/1635460004 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494024c10 con 0x564a98fa5ef0 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 -- 172.21.15.23:0/1635460004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 msgr2=0x564a98fb08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5494009930 tx=0x7f549401c000 comp rx=0 tx=0).stop 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 -- 172.21.15.23:0/1635460004 shutdown_connections 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fa59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1635460004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb08b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 -- 172.21.15.23:0/1635460004 >> 172.21.15.23:0/1635460004 conn(0x564a98f056a0 msgr2=0x564a98fa1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 -- 172.21.15.23:0/1635460004 shutdown_connections 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.203+0000 7f54c38bab80 1 -- 172.21.15.23:0/1635460004 wait complete. 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.204+0000 7f54c38bab80 1 -- start start 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.204+0000 7f54c38bab80 1 Processor -- start 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 0x564a98fada90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98e93d80 con 0x564a98fb0df0 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98e93a20 con 0x564a98fa5ef0 2024-03-30T04:07:57.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98d7b6a0 con 0x564a98fa55d0 2024-03-30T04:07:57.206 INFO:tasks.workunit.client.0.smithi023.stdout: 1 583249 11.52 MB/sec execute 154 sec latency 20.938 ms 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54c38bab80 10 client.0 _ll_register_callbacks cb 0x564a98e9d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 0x564a98fada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41482/0 (socket says 172.21.15.23:41482) 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 -- 172.21.15.23:0/3754445002 learned_addr learned my addr 172.21.15.23:0/3754445002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ade00700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 -- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 msgr2=0x564a98fada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 0x564a98fada90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 -- 172.21.15.23:0/3754445002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 msgr2=0x564a98fae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fae0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ad400700 1 -- 172.21.15.23:0/3754445002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a81aa360 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54aca00700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 0x564a98fada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.205+0000 7f54ade00700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f54ad400700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f549c009990 tx=0x7f549c097870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdddc578de2acfec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c09b880 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f549c0ce2d0 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a98eb8380 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f549c09bb20 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f54c38bab80 10 client.27211 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f5494017050 con 0x564a98fa55d0 2024-03-30T04:07:57.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a98eb8380 con 0x564a98fa55d0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.206+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f549c09a910 con 0x564a98fa55d0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 10 client.27211 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 10 client.27211 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 20 client.27211 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a98fc29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 4 client.27211 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a98ef0700 con 0x564a98fc05e0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54aca00700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a98fc29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f54aca00700 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a98fc29d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=225f143e16d83a7a server_cookie=940a2d9980491e44 in_seq=0 out_seq=0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.207+0000 7f548ea00700 10 client.27211 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f548ea00700 1 -- 172.21.15.23:0/3754445002 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564a98ef0700 con 0x564a98fc05e0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f548ea00700 10 client.27211 handle_command_reply: tid=0 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 client.27211 shutdown 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 msgr2=0x564a98fb0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f549c009990 tx=0x7f549c097870 comp rx=0 tx=0).stop 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 shutdown_connections 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a98fc29d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f548ea00700 0 client.27211 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.208+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa55d0 0x564a98fb0810 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.209+0000 7f54c38bab80 1 --2- 172.21.15.23:0/3754445002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa5ef0 0x564a98fada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.209+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 >> 172.21.15.23:0/3754445002 conn(0x564a98f056a0 msgr2=0x564a98fa76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.209+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 shutdown_connections 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.209+0000 7f54c38bab80 1 -- 172.21.15.23:0/3754445002 wait complete. 2024-03-30T04:07:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.209+0000 7f54c38bab80 20 client.27211 trim_cache size 0 max 16384 2024-03-30T04:07:57.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.226+0000 7f54c38bab80 1 Processor -- start 2024-03-30T04:07:57.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 -- start start 2024-03-30T04:07:57.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 0x564a98fae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98fa8570 con 0x564a98fb0df0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98fabfe0 con 0x564a98fa55d0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54c38bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a98fa1f20 con 0x564a98fa5ef0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54ad400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 0x564a98fae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41492/0 (socket says 172.21.15.23:41492) 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54aca00700 1 -- 172.21.15.23:0/4082559084 learned_addr learned my addr 172.21.15.23:0/4082559084 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.227+0000 7f54ade00700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 -- 172.21.15.23:0/4082559084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 msgr2=0x564a98fb0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 -- 172.21.15.23:0/4082559084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 msgr2=0x564a98fac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 -- 172.21.15.23:0/4082559084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a98d25b60 con 0x564a98fa55d0 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54aca00700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ade00700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f54ad400700 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 0x564a98fae040 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f549c097e90 tx=0x7f549c005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15b1154be31663df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f549b000700 1 -- 172.21.15.23:0/4082559084 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c0ce2d0 con 0x564a98fa55d0 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f549b000700 1 -- 172.21.15.23:0/4082559084 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549c09bb20 con 0x564a98fa55d0 2024-03-30T04:07:57.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.228+0000 7f549b000700 1 -- 172.21.15.23:0/4082559084 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549c09f2e0 con 0x564a98fa55d0 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- 172.21.15.23:0/4082559084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 msgr2=0x564a98fae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 0x564a98fae040 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f549c097e90 tx=0x7f549c005fb0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- 172.21.15.23:0/4082559084 shutdown_connections 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fb0df0 0x564a98fac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fa55d0 0x564a98fae040 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 --2- 172.21.15.23:0/4082559084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- 172.21.15.23:0/4082559084 >> 172.21.15.23:0/4082559084 conn(0x564a98f056a0 msgr2=0x564a98f9a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- 172.21.15.23:0/4082559084 shutdown_connections 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- 172.21.15.23:0/4082559084 wait complete. 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.229+0000 7f54c38bab80 1 -- start start 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 Processor -- start 2024-03-30T04:07:57.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a990134c0 con 0x564a98fa55d0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a99013630 con 0x564a98fb0df0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a990137f0 con 0x564a98fa5ef0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54c38bab80 10 client.0 _ll_register_callbacks cb 0x564a98e9d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58820/0 (socket says 172.21.15.23:58820) 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54ad400700 1 -- 172.21.15.23:0/1625306825 learned_addr learned my addr 172.21.15.23:0/1625306825 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.230+0000 7f54ade00700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 -- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 msgr2=0x564a98fb6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 -- 172.21.15.23:0/1625306825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 msgr2=0x564a98fb5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 -- 172.21.15.23:0/1625306825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a98d793b0 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54ade00700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54ad400700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54aca00700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb5fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f549400f770 tx=0x7f54940247c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cc00c45b5ccd462 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f548ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5494024a70 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f5494024bd0 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.231+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f549401cc20 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5494024d90 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 10 client.27217 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f5494017050 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f549401cc20 con 0x564a98fa5ef0 2024-03-30T04:07:57.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f549401c770 con 0x564a98fa5ef0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 10 client.27217 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 10 client.27217 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 20 client.27217 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a99016fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 4 client.27217 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.232+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564a98d46cf0 con 0x564a98fc05e0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f54ad400700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a99016fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f54ad400700 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a99016fa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2106aecc77619024 server_cookie=fe5d36b9b9f6d6d1 in_seq=0 out_seq=0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f548ea00700 10 client.27217 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f548ea00700 1 -- 172.21.15.23:0/1625306825 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564a98d46cf0 con 0x564a98fc05e0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f548ea00700 10 client.27217 handle_command_reply: tid=0 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.233+0000 7f54c38bab80 1 client.27217 shutdown 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 msgr2=0x564a98fb5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb5fc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f549400f770 tx=0x7f54940247c0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 shutdown_connections 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564a98fc05e0 0x564a99016fa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a98fb0df0 0x564a98fb6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f548ea00700 0 client.27217 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a98fa55d0 0x564a98fb5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 --2- 172.21.15.23:0/1625306825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a98fa5ef0 0x564a98fb5fc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 >> 172.21.15.23:0/1625306825 conn(0x564a98f056a0 msgr2=0x564a98fac320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 shutdown_connections 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 1 -- 172.21.15.23:0/1625306825 wait complete. 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.234+0000 7f54c38bab80 20 client.27217 trim_cache size 0 max 16384 2024-03-30T04:07:57.237 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:57.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 msgr2=0x7f54900a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 0x7f54900a3750 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5494009fd0 tx=0x7f5494009260 comp rx=0 tx=0).stop 2024-03-30T04:07:57.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 msgr2=0x7f54a81a3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f549c0098e0 tx=0x7f549c005740 comp rx=0 tx=0).stop 2024-03-30T04:07:57.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 shutdown_connections 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f54a8111d00 0x7f54a819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54900a1290 0x7f54900a3750 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f54a8073150 0x7f54a81a3550 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 --2- 172.21.15.23:0/3682937885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54a8073a70 0x7f54a81a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 >> 172.21.15.23:0/3682937885 conn(0x7f54a80fc7c0 msgr2=0x7f54a80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.235+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 shutdown_connections 2024-03-30T04:07:57.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.236+0000 7f54afe00700 1 -- 172.21.15.23:0/3682937885 wait complete. 2024-03-30T04:07:57.250 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:57 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/959444741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:57.253 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:07:57.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:57 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/959444741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 -- 172.21.15.23:0/1143570070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107fc0 msgr2=0x7f040810bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 --2- 172.21.15.23:0/1143570070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107fc0 0x7f040810bfb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009990 tx=0x7f03f4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 -- 172.21.15.23:0/1143570070 shutdown_connections 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 --2- 172.21.15.23:0/1143570070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 0x7f0408114260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 --2- 172.21.15.23:0/1143570070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0408107610 0x7f04081079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 --2- 172.21.15.23:0/1143570070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107fc0 0x7f040810bfb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 -- 172.21.15.23:0/1143570070 >> 172.21.15.23:0/1143570070 conn(0x7f0408076b90 msgr2=0x7f0408105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 -- 172.21.15.23:0/1143570070 shutdown_connections 2024-03-30T04:07:57.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.474+0000 7f0410200700 1 -- 172.21.15.23:0/1143570070 wait complete. 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.475+0000 7f0410200700 1 Processor -- start 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.475+0000 7f0410200700 1 -- start start 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 0x7f04081a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f040819d6e0 0x7f040819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040819e160 con 0x7f040810c6e0 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040819e2d0 con 0x7f0408107610 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f0410200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040819e490 con 0x7f040819d6e0 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 0x7f04081a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34716/0 (socket says 172.21.15.23:34716) 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 -- 172.21.15.23:0/3615795651 learned_addr learned my addr 172.21.15.23:0/3615795651 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040e200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f040819d6e0 0x7f040819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 -- 172.21.15.23:0/3615795651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f040819d6e0 msgr2=0x7f040819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f040819d6e0 0x7f040819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 -- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 msgr2=0x7f04081a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 0x7f04081a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.476+0000 7f040d800700 1 -- 172.21.15.23:0/3615795651 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f40095f0 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f040d800700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f040000ef10 tx=0x7f040000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f0406000700 1 -- 172.21.15.23:0/3615795651 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040000c9c0 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f0406000700 1 -- 172.21.15.23:0/3615795651 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f040000f040 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04081aa420 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f0406000700 1 -- 172.21.15.23:0/3615795651 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400014480 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.477+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04081aa950 con 0x7f0408107610 2024-03-30T04:07:57.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.480+0000 7f0406000700 1 -- 172.21.15.23:0/3615795651 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0400007690 con 0x7f0408107610 2024-03-30T04:07:57.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.481+0000 7f0406000700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 0x7f03ec0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 Processor -- start 2024-03-30T04:07:57.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0406000700 1 -- 172.21.15.23:0/3615795651 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f04000cd2b0 con 0x7f0408107610 2024-03-30T04:07:57.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f040ce00700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 0x7f03ec0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 -- start start 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 0x55d747f418b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747e8f990 con 0x55d747f36ef0 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747e8d350 con 0x55d747f365d0 2024-03-30T04:07:57.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.482+0000 7f0423c79b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747e88c40 con 0x55d747f41df0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 0x55d747f418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34720/0 (socket says 172.21.15.23:34720) 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 -- 172.21.15.23:0/29761144 learned_addr learned my addr 172.21.15.23:0/29761144 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040e200700 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 -- 172.21.15.23:0/29761144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 msgr2=0x55d747f441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 -- 172.21.15.23:0/29761144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 msgr2=0x55d747f418b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 0x55d747f418b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 -- 172.21.15.23:0/29761144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d747d0a3b0 con 0x55d747f365d0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040ce00700 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 0x55d747f418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040ce00700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 0x7f03ec0a37d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f040819edf0 tx=0x7f03f4005370 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f040d800700 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0400096a70 tx=0x7f0400096470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=279c0986cbfb404e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.483+0000 7f03fea00700 1 -- 172.21.15.23:0/29761144 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040009a000 con 0x55d747f365d0 2024-03-30T04:07:57.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f03fea00700 1 -- 172.21.15.23:0/29761144 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f040009de30 con 0x55d747f365d0 2024-03-30T04:07:57.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f03fea00700 1 -- 172.21.15.23:0/29761144 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04000994f0 con 0x55d747f365d0 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 -- 172.21.15.23:0/29761144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 msgr2=0x55d747f369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0400096a70 tx=0x7f0400096470 comp rx=0 tx=0).stop 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 -- 172.21.15.23:0/29761144 shutdown_connections 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f365d0 0x55d747f369b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 --2- 172.21.15.23:0/29761144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f36ef0 0x55d747f418b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 -- 172.21.15.23:0/29761144 >> 172.21.15.23:0/29761144 conn(0x55d747e966a0 msgr2=0x55d747f32a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.484+0000 7f0423c79b80 1 -- 172.21.15.23:0/29761144 shutdown_connections 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.485+0000 7f0423c79b80 1 -- 172.21.15.23:0/29761144 wait complete. 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.485+0000 7f0423c79b80 1 -- start start 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.485+0000 7f0423c79b80 1 Processor -- start 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f417c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 0x55d747f3f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747e24d80 con 0x55d747f365d0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747e24a20 con 0x55d747f41df0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747d0c6a0 con 0x55d747f36ef0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f0423c79b80 10 client.0 _ll_register_callbacks cb 0x55d747e2e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41532/0 (socket says 172.21.15.23:41532) 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040ce00700 1 -- 172.21.15.23:0/308020847 learned_addr learned my addr 172.21.15.23:0/308020847 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58864/0 (socket says 172.21.15.23:58864) 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.486+0000 7f040e200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 0x55d747f3f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 -- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 msgr2=0x55d747f3f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 0x55d747f3f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 -- 172.21.15.23:0/308020847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 msgr2=0x55d747f417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 -- 172.21.15.23:0/308020847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04081aa950 con 0x55d747f36ef0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040e200700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 0x55d747f3f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f040ce00700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03f4015000 tx=0x7f03f4009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dbdc72175c37d68 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f03f3400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f401d070 con 0x55d747f36ef0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d747e49380 con 0x55d747f36ef0 2024-03-30T04:07:57.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f03f40047d0 con 0x55d747f36ef0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03f4004990 con 0x55d747f36ef0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f0423c79b80 10 client.27229 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.487+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d747cb6b60 con 0x55d747f36ef0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d747e49380 con 0x55d747f36ef0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03f400fb00 con 0x55d747f36ef0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 10 client.27229 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 10 client.27229 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 20 client.27229 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747f51950 0x55d747f53d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 4 client.27229 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d747e81700 con 0x55d747f51950 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.488+0000 7f040d800700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747f51950 0x55d747f53d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.489+0000 7f040d800700 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747f51950 0x55d747f53d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67757b9086325ba3 server_cookie=71f776ee1365e7f1 in_seq=0 out_seq=0 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.489+0000 7f03f3400700 10 client.27229 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.489+0000 7f03f3400700 1 -- 172.21.15.23:0/308020847 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d747e81700 con 0x55d747f51950 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.489+0000 7f03f3400700 10 client.27229 handle_command_reply: tid=0 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.489+0000 7f0423c79b80 1 client.27229 shutdown 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 msgr2=0x55d747f3eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f03f4015000 tx=0x7f03f4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 shutdown_connections 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f41df0 0x55d747f3f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747f51950 0x55d747f53d70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 --2- 172.21.15.23:0/308020847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f36ef0 0x55d747f3eab0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 >> 172.21.15.23:0/308020847 conn(0x55d747e966a0 msgr2=0x55d747f384d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f03f3400700 0 client.27229 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:57.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 shutdown_connections 2024-03-30T04:07:57.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 1 -- 172.21.15.23:0/308020847 wait complete. 2024-03-30T04:07:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:57 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/959444741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:07:57.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.490+0000 7f0423c79b80 20 client.27229 trim_cache size 0 max 16384 2024-03-30T04:07:57.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 Processor -- start 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 -- start start 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f48a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f37850 con 0x55d747f365d0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f37570 con 0x55d747f36ef0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f0423c79b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f48fb0 con 0x55d747f41df0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f040ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f48a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58870/0 (socket says 172.21.15.23:58870) 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f040d800700 1 -- 172.21.15.23:0/1732139065 learned_addr learned my addr 172.21.15.23:0/1732139065 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.505+0000 7f040e200700 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 -- 172.21.15.23:0/1732139065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 msgr2=0x55d747f412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 -- 172.21.15.23:0/1732139065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 msgr2=0x55d747f53d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 -- 172.21.15.23:0/1732139065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d747e4eea0 con 0x55d747f36ef0 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040d800700 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f040ce00700 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f48a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000c00 tx=0x7f03f400b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bedc5ad66e30d678 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f03ff400700 1 -- 172.21.15.23:0/1732139065 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f401d040 con 0x55d747f36ef0 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f03ff400700 1 -- 172.21.15.23:0/1732139065 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03f40056c0 con 0x55d747f36ef0 2024-03-30T04:07:57.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.506+0000 7f03ff400700 1 -- 172.21.15.23:0/1732139065 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40189f0 con 0x55d747f36ef0 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 -- 172.21.15.23:0/1732139065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 msgr2=0x55d747f48a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f48a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f03f4000c00 tx=0x7f03f400b410 comp rx=0 tx=0).stop 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 -- 172.21.15.23:0/1732139065 shutdown_connections 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f53d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 --2- 172.21.15.23:0/1732139065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f48a70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 -- 172.21.15.23:0/1732139065 >> 172.21.15.23:0/1732139065 conn(0x55d747e966a0 msgr2=0x55d747f2b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 -- 172.21.15.23:0/1732139065 shutdown_connections 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.507+0000 7f0423c79b80 1 -- 172.21.15.23:0/1732139065 wait complete. 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 -- start start 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 Processor -- start 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f47aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f38f50 con 0x55d747f365d0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f39090 con 0x55d747f36ef0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.508+0000 7f0423c79b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d747f39250 con 0x55d747f41df0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f0423c79b80 10 client.0 _ll_register_callbacks cb 0x55d747e2e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34744/0 (socket says 172.21.15.23:34744) 2024-03-30T04:07:57.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f384d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58874/0 (socket says 172.21.15.23:58874) 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 -- 172.21.15.23:0/3688204579 learned_addr learned my addr 172.21.15.23:0/3688204579 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040e200700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f47aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 -- 172.21.15.23:0/3688204579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 msgr2=0x55d747f47aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f47aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 -- 172.21.15.23:0/3688204579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 msgr2=0x55d747f384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 -- 172.21.15.23:0/3688204579 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d747d0a3b0 con 0x55d747f36ef0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f040ce00700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005850 tx=0x7f03f4005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326e78209b6811ec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.509+0000 7f03f3400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f40185e0 con 0x55d747f36ef0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f040009de30 con 0x55d747f36ef0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f03f40049e0 con 0x55d747f36ef0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03f4004ba0 con 0x55d747f36ef0 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 10 client.27239 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d747cb6b60 con 0x55d747f36ef0 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f040009de30 con 0x55d747f36ef0 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03f4049030 con 0x55d747f36ef0 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 10 client.27239 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 10 client.27239 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 20 client.27239 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747fa5e20 0x55d747fa8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 4 client.27239 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.510+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d747cd7cf0 con 0x55d747fa5e20 2024-03-30T04:07:57.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f040d800700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747fa5e20 0x55d747fa8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f040d800700 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747fa5e20 0x55d747fa8240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=586fe41b521363c3 server_cookie=aab60abe00211138 in_seq=0 out_seq=0 2024-03-30T04:07:57.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f03f3400700 10 client.27239 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f03f3400700 1 -- 172.21.15.23:0/3688204579 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d747cd7cf0 con 0x55d747fa5e20 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f03f3400700 10 client.27239 handle_command_reply: tid=0 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f0423c79b80 1 client.27239 shutdown 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.511+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 msgr2=0x55d747f38a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f03f4005850 tx=0x7f03f4005280 comp rx=0 tx=0).stop 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 shutdown_connections 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d747fa5e20 0x55d747fa8240 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d747f41df0 0x55d747f47aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d747f365d0 0x55d747f384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f03f3400700 0 client.27239 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 --2- 172.21.15.23:0/3688204579 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d747f36ef0 0x55d747f38a10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 >> 172.21.15.23:0/3688204579 conn(0x55d747e966a0 msgr2=0x55d747f3d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 shutdown_connections 2024-03-30T04:07:57.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 1 -- 172.21.15.23:0/3688204579 wait complete. 2024-03-30T04:07:57.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0423c79b80 20 client.27239 trim_cache size 0 max 16384 2024-03-30T04:07:57.515 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 msgr2=0x7f03ec0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 0x7f03ec0a37d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f040819edf0 tx=0x7f03f4005370 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 msgr2=0x7f04081a35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.512+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f040000ef10 tx=0x7f040000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 shutdown_connections 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f040819d6e0 0x7f040819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03ec0a1310 0x7f03ec0a37d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0408107610 0x7f04081a35b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 --2- 172.21.15.23:0/3615795651 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f040810c6e0 0x7f04081a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 >> 172.21.15.23:0/3615795651 conn(0x7f0408076b90 msgr2=0x7f0408077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 shutdown_connections 2024-03-30T04:07:57.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.513+0000 7f0410200700 1 -- 172.21.15.23:0/3615795651 wait complete. 2024-03-30T04:07:57.530 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 -- 172.21.15.23:0/1733973888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 msgr2=0x7fba981035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 --2- 172.21.15.23:0/1733973888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba981035b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009990 tx=0x7fba8c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 -- 172.21.15.23:0/1733973888 shutdown_connections 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 --2- 172.21.15.23:0/1733973888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba981082f0 0x7fba9810fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 --2- 172.21.15.23:0/1733973888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba981035b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 --2- 172.21.15.23:0/1733973888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba98103b80 0x7fba98107bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 -- 172.21.15.23:0/1733973888 >> 172.21.15.23:0/1733973888 conn(0x7fba980fea10 msgr2=0x7fba98100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 -- 172.21.15.23:0/1733973888 shutdown_connections 2024-03-30T04:07:57.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.743+0000 7fbaa0600700 1 -- 172.21.15.23:0/1733973888 wait complete. 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.744+0000 7fbaa0600700 1 Processor -- start 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 -- start start 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba981082f0 0x7fba98199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba9819de20 0x7fba9819e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819e8a0 con 0x7fba981082f0 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819ea10 con 0x7fba9819de20 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fbaa0600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba9819ebd0 con 0x7fba981031d0 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fba9e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba9819de20 0x7fba9819e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fba9d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba981082f0 0x7fba98199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41568/0 (socket says 172.21.15.23:41568) 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.745+0000 7fba9dc00700 1 -- 172.21.15.23:0/2282967453 learned_addr learned my addr 172.21.15.23:0/2282967453 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 -- 172.21.15.23:0/2282967453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba9819de20 msgr2=0x7fba9819e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba9819de20 0x7fba9819e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 -- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba981082f0 msgr2=0x7fba98199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba981082f0 0x7fba98199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 -- 172.21.15.23:0/2282967453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0095f0 con 0x7fba981031d0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fba8c005910 tx=0x7fba8c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba96000700 1 -- 172.21.15.23:0/2282967453 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c01d070 con 0x7fba981031d0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba96000700 1 -- 172.21.15.23:0/2282967453 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba8c022470 con 0x7fba981031d0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fba96000700 1 -- 172.21.15.23:0/2282967453 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c017650 con 0x7fba981031d0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba981a60d0 con 0x7fba981031d0 2024-03-30T04:07:57.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.746+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba981a64a0 con 0x7fba981031d0 2024-03-30T04:07:57.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.749+0000 7fba96000700 1 -- 172.21.15.23:0/2282967453 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fba8c0225e0 con 0x7fba981031d0 2024-03-30T04:07:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 Processor -- start 2024-03-30T04:07:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fba96000700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 0x7fba800a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 -- start start 2024-03-30T04:07:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fba96000700 1 -- 172.21.15.23:0/2282967453 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fba8c0ce280 con 0x7fba981031d0 2024-03-30T04:07:57.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fba9e600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 0x7fba800a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b61ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b60f6990 con 0x5583b61a8df0 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b60f4350 con 0x5583b619def0 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.751+0000 7fbab4179b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b60efc40 con 0x5583b619d5d0 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41572/0 (socket says 172.21.15.23:41572) 2024-03-30T04:07:57.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 -- 172.21.15.23:0/2480620989 learned_addr learned my addr 172.21.15.23:0/2480620989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9d200700 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9e600700 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b61ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 -- 172.21.15.23:0/2480620989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 msgr2=0x5583b61a88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61a88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 -- 172.21.15.23:0/2480620989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 msgr2=0x5583b61ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b61ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 -- 172.21.15.23:0/2480620989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583b5f713b0 con 0x5583b619d5d0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9e600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 0x7fba800a37d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fba9000f590 tx=0x7fba90009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba9dc00700 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0a2f20 tx=0x7fba8c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=556d870c3dc5cd57 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba8aa00700 1 -- 172.21.15.23:0/2480620989 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c0aeea0 con 0x5583b619d5d0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba8aa00700 1 -- 172.21.15.23:0/2480620989 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba8c09b760 con 0x5583b619d5d0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.752+0000 7fba8aa00700 1 -- 172.21.15.23:0/2480620989 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c0228c0 con 0x5583b619d5d0 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 -- 172.21.15.23:0/2480620989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 msgr2=0x5583b619d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0a2f20 tx=0x7fba8c009310 comp rx=0 tx=0).stop 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 -- 172.21.15.23:0/2480620989 shutdown_connections 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b61ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b619d9b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 --2- 172.21.15.23:0/2480620989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61a88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 -- 172.21.15.23:0/2480620989 >> 172.21.15.23:0/2480620989 conn(0x5583b60fd6a0 msgr2=0x5583b6199a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 -- 172.21.15.23:0/2480620989 shutdown_connections 2024-03-30T04:07:57.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.753+0000 7fbab4179b80 1 -- 172.21.15.23:0/2480620989 wait complete. 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.754+0000 7fbab4179b80 1 -- start start 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.754+0000 7fbab4179b80 1 Processor -- start 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b61a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b608bd80 con 0x5583b619d5d0 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b608ba20 con 0x5583b61a8df0 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b5f736a0 con 0x5583b619def0 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fbab4179b80 10 client.0 _ll_register_callbacks cb 0x5583b6095470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fba9e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b61a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58910/0 (socket says 172.21.15.23:58910) 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.755+0000 7fba9dc00700 1 -- 172.21.15.23:0/3536481129 learned_addr learned my addr 172.21.15.23:0/3536481129 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9d200700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 -- 172.21.15.23:0/3536481129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 msgr2=0x5583b61a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 -- 172.21.15.23:0/3536481129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 msgr2=0x5583b61a87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 -- 172.21.15.23:0/3536481129 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba981a64a0 con 0x5583b61a8df0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9dc00700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba9e600700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b61a6180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fba9000fce0 tx=0x7fba90004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfecbfc54094e55c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba7f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.756+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000e040 con 0x5583b61a8df0 2024-03-30T04:07:57.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fba90019e30 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583b60b0380 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba90004bb0 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 10 client.27251 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5583b5f1db60 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583b60b0380 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fba900213f0 con 0x5583b61a8df0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 10 client.27251 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 10 client.27251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 20 client.27251 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b61b8670 0x5583b61baa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 4 client.27251 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.757+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583b60e8700 con 0x5583b61b8670 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.758+0000 7fba9dc00700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b61b8670 0x5583b61baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.758+0000 7fba9dc00700 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b61b8670 0x5583b61baa90 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559f6502d0c51a4b server_cookie=42e4507b2b2f7fc4 in_seq=0 out_seq=0 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.758+0000 7fba7f400700 10 client.27251 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fba7f400700 1 -- 172.21.15.23:0/3536481129 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5583b60e8700 con 0x5583b61b8670 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fba7f400700 10 client.27251 handle_command_reply: tid=0 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 client.27251 shutdown 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 msgr2=0x5583b61a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b61a6180 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fba9000fce0 tx=0x7fba90004b40 comp rx=0 tx=0).stop 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 shutdown_connections 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b61a6180 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61a87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b61b8670 0x5583b61baa90 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3536481129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fba7f400700 0 client.27251 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:57.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 >> 172.21.15.23:0/3536481129 conn(0x5583b60fd6a0 msgr2=0x5583b619f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 shutdown_connections 2024-03-30T04:07:57.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 1 -- 172.21.15.23:0/3536481129 wait complete. 2024-03-30T04:07:57.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.759+0000 7fbab4179b80 20 client.27251 trim_cache size 0 max 16384 2024-03-30T04:07:57.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 Processor -- start 2024-03-30T04:07:57.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 -- start start 2024-03-30T04:07:57.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b61a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b6192f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b61a6a60 con 0x5583b61a8df0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b61bae20 con 0x5583b619def0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.774+0000 7fbab4179b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b61afd20 con 0x5583b619d5d0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b61a60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34788/0 (socket says 172.21.15.23:34788) 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b61a60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41604/0 (socket says 172.21.15.23:41604) 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 -- 172.21.15.23:0/3442505038 learned_addr learned my addr 172.21.15.23:0/3442505038 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9e600700 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b6192f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 -- 172.21.15.23:0/3442505038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 msgr2=0x5583b61a60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b61a60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 -- 172.21.15.23:0/3442505038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 msgr2=0x5583b6192f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b6192f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 -- 172.21.15.23:0/3442505038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583b60b5ea0 con 0x5583b619def0 2024-03-30T04:07:57.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba9d200700 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fba84009e70 tx=0x7fba84009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86ca37e6711171ef server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.775+0000 7fba8b400700 1 -- 172.21.15.23:0/3442505038 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84017070 con 0x5583b619def0 2024-03-30T04:07:57.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fba8b400700 1 -- 172.21.15.23:0/3442505038 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba84018040 con 0x5583b619def0 2024-03-30T04:07:57.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fba8b400700 1 -- 172.21.15.23:0/3442505038 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba84003b50 con 0x5583b619def0 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 -- 172.21.15.23:0/3442505038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 msgr2=0x5583b61af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fba84009e70 tx=0x7fba84009260 comp rx=0 tx=0).stop 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 -- 172.21.15.23:0/3442505038 shutdown_connections 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b61a8df0 0x5583b6192f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619d5d0 0x5583b61a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 --2- 172.21.15.23:0/3442505038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b619def0 0x5583b61af7e0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 -- 172.21.15.23:0/3442505038 >> 172.21.15.23:0/3442505038 conn(0x5583b60fd6a0 msgr2=0x5583b61bf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 -- 172.21.15.23:0/3442505038 shutdown_connections 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.776+0000 7fbab4179b80 1 -- 172.21.15.23:0/3442505038 wait complete. 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.777+0000 7fbab4179b80 1 -- start start 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.777+0000 7fbab4179b80 1 Processor -- start 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b6213cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b6214260 con 0x5583b619d5d0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b62143d0 con 0x5583b61a8df0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583b6214590 con 0x5583b619def0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fbab4179b80 10 client.0 _ll_register_callbacks cb 0x5583b6095470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b6213cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41608/0 (socket says 172.21.15.23:41608) 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9d200700 1 -- 172.21.15.23:0/133004981 learned_addr learned my addr 172.21.15.23:0/133004981 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9dc00700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 -- 172.21.15.23:0/133004981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 msgr2=0x5583b61adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 -- 172.21.15.23:0/133004981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 msgr2=0x5583b61ad9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:57.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61ad9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9e600700 1 -- 172.21.15.23:0/133004981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba84009b20 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.778+0000 7fba9d200700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba9e600700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b6213cb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fba900218a0 tx=0x7fba90021f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f00811f783d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9000e040 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba8c09b760 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fba90004db0 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba90003a00 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fbab4179b80 10 client.27257 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5583b5f75cd0 con 0x5583b61a8df0 2024-03-30T04:07:57.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba8c09b760 con 0x5583b61a8df0 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.779+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fba90033070 con 0x5583b61a8df0 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 10 client.27257 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 10 client.27257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 20 client.27257 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b6215d30 0x5583b6218150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 4 client.27257 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5583b5f3ecf0 con 0x5583b6215d30 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fba9dc00700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b6215d30 0x5583b6218150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:57.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fba9dc00700 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b6215d30 0x5583b6218150 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=782c1f6250d23d7c server_cookie=75826f0eb68a881e in_seq=0 out_seq=0 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.780+0000 7fba7f400700 10 client.27257 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fba7f400700 1 -- 172.21.15.23:0/133004981 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5583b5f3ecf0 con 0x5583b6215d30 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fba7f400700 10 client.27257 handle_command_reply: tid=0 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 client.27257 shutdown 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 msgr2=0x5583b6213cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b6213cb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fba900218a0 tx=0x7fba90021f60 comp rx=0 tx=0).stop 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 shutdown_connections 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583b61a8df0 0x5583b6213cb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583b619d5d0 0x5583b61ad9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583b6215d30 0x5583b6218150 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 --2- 172.21.15.23:0/133004981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583b619def0 0x5583b61adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 >> 172.21.15.23:0/133004981 conn(0x5583b60fd6a0 msgr2=0x5583b6192f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fba7f400700 0 client.27257 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 shutdown_connections 2024-03-30T04:07:57.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 1 -- 172.21.15.23:0/133004981 wait complete. 2024-03-30T04:07:57.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.781+0000 7fbab4179b80 20 client.27257 trim_cache size 0 max 16384 2024-03-30T04:07:57.784 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:07:57.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 msgr2=0x7fba800a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 0x7fba800a37d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fba9000f590 tx=0x7fba90009920 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 msgr2=0x7fba98199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fba8c005910 tx=0x7fba8c004730 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 shutdown_connections 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fba800a1310 0x7fba800a37d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba9819de20 0x7fba9819e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba981031d0 0x7fba98199540 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 --2- 172.21.15.23:0/2282967453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fba981082f0 0x7fba98199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 >> 172.21.15.23:0/2282967453 conn(0x7fba980fea10 msgr2=0x7fba980ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 shutdown_connections 2024-03-30T04:07:57.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:57.782+0000 7fbaa0600700 1 -- 172.21.15.23:0/2282967453 wait complete. 2024-03-30T04:07:58.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 585422 11.52 MB/sec execute 155 sec latency 21.005 ms 2024-03-30T04:07:58.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:07:58 smithi154 ceph-mon[79161]: pgmap v183: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:58.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:07:58 smithi193 ceph-mon[79581]: pgmap v183: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:07:58 smithi023 ceph-mon[38254]: pgmap v183: 129 pgs: 129 active+clean; 357 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 187 op/s 2024-03-30T04:07:58.799 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:07:58.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:07:59.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 -- 172.21.15.23:0/1446519405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 msgr2=0x7f0bc4101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/1446519405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc4101350 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009990 tx=0x7f0bb8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 -- 172.21.15.23:0/1446519405 shutdown_connections 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/1446519405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4106090 0x7f0bc410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/1446519405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc4101350 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/1446519405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4101920 0x7f0bc4105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 -- 172.21.15.23:0/1446519405 >> 172.21.15.23:0/1446519405 conn(0x7f0bc40fc7f0 msgr2=0x7f0bc40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 -- 172.21.15.23:0/1446519405 shutdown_connections 2024-03-30T04:07:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.037+0000 7f0bcbc00700 1 -- 172.21.15.23:0/1446519405 wait complete. 2024-03-30T04:07:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.039+0000 7f0bcbc00700 1 Processor -- start 2024-03-30T04:07:59.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.039+0000 7f0bcbc00700 1 -- start start 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4106090 0x7f0bc40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4073b30 0x7f0bc41a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc41a59e0 con 0x7f0bc4073b30 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc41a5b50 con 0x7f0bc4106090 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bcbc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc41a5d10 con 0x7f0bc4100f70 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4106090 0x7f0bc40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52664/0 (socket says 172.21.15.23:52664) 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 -- 172.21.15.23:0/428230329 learned_addr learned my addr 172.21.15.23:0/428230329 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9c00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4073b30 0x7f0bc41a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 -- 172.21.15.23:0/428230329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4106090 msgr2=0x7f0bc40735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4106090 0x7f0bc40735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 -- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4073b30 msgr2=0x7f0bc41a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4073b30 0x7f0bc41a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.040+0000 7f0bc9200700 1 -- 172.21.15.23:0/428230329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80095f0 con 0x7f0bc4100f70 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bc9200700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8005910 tx=0x7f0bb8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bc1600700 1 -- 172.21.15.23:0/428230329 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb801d070 con 0x7f0bc4100f70 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bc1600700 1 -- 172.21.15.23:0/428230329 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb8022470 con 0x7f0bc4100f70 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc41a5f90 con 0x7f0bc4100f70 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bc1600700 1 -- 172.21.15.23:0/428230329 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8017650 con 0x7f0bc4100f70 2024-03-30T04:07:59.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.041+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc41a63b0 con 0x7f0bc4100f70 2024-03-30T04:07:59.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.045+0000 7f0bc1600700 1 -- 172.21.15.23:0/428230329 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0bb80225e0 con 0x7f0bc4100f70 2024-03-30T04:07:59.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.046+0000 7f0bc1600700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 0x7f0bac0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 Processor -- start 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bc1600700 1 -- 172.21.15.23:0/428230329 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0bb80ce270 con 0x7f0bc4100f70 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 0x7f0bac0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 -- start start 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08aba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08abd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08a08a10 con 0x557e08aaff70 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08a063d0 con 0x557e08abae70 2024-03-30T04:07:59.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.047+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08a01cc0 con 0x557e08aaf650 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52680/0 (socket says 172.21.15.23:52680) 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 -- 172.21.15.23:0/4089205546 learned_addr learned my addr 172.21.15.23:0/4089205546 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9c00700 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08abd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08aba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 0x7f0bac0a37d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40747e0 tx=0x7f0bb0009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 -- 172.21.15.23:0/4089205546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 msgr2=0x557e08abd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08abd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 -- 172.21.15.23:0/4089205546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 msgr2=0x557e08aba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08aba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 -- 172.21.15.23:0/4089205546 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e08883430 con 0x557e08aaf650 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bc9200700 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80a2f10 tx=0x7f0bb8009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ce676dc6972e930 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.048+0000 7f0bb6000700 1 -- 172.21.15.23:0/4089205546 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb809b750 con 0x557e08aaf650 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.049+0000 7f0bb6000700 1 -- 172.21.15.23:0/4089205546 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb80aee90 con 0x557e08aaf650 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.049+0000 7f0bb6000700 1 -- 172.21.15.23:0/4089205546 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80228c0 con 0x557e08aaf650 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.049+0000 7f0bdf693b80 1 -- 172.21.15.23:0/4089205546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 msgr2=0x557e08aafa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.049+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80a2f10 tx=0x7f0bb8009310 comp rx=0 tx=0).stop 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 -- 172.21.15.23:0/4089205546 shutdown_connections 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08abd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08aafa30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/4089205546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08aba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 -- 172.21.15.23:0/4089205546 >> 172.21.15.23:0/4089205546 conn(0x557e08a0f720 msgr2=0x557e08aabb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 -- 172.21.15.23:0/4089205546 shutdown_connections 2024-03-30T04:07:59.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 -- 172.21.15.23:0/4089205546 wait complete. 2024-03-30T04:07:59.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.050+0000 7f0bdf693b80 1 -- start start 2024-03-30T04:07:59.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.051+0000 7f0bdf693b80 1 Processor -- start 2024-03-30T04:07:59.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.051+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.051+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaff70 0x557e08ab7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56640/0 (socket says 172.21.15.23:56640) 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 -- 172.21.15.23:0/133582491 learned_addr learned my addr 172.21.15.23:0/133582491 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ab83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e0899de00 con 0x557e08abae70 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e0899daa0 con 0x557e08aaf650 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08885720 con 0x557e08aaff70 2024-03-30T04:07:59.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bdf693b80 10 client.0 _ll_register_callbacks cb 0x557e089a74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaff70 0x557e08ab7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 -- 172.21.15.23:0/133582491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaff70 msgr2=0x557e08ab7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaff70 0x557e08ab7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 -- 172.21.15.23:0/133582491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 msgr2=0x557e08ab83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ab83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.052+0000 7f0bc9200700 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc41a63b0 con 0x557e08aaf650 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0bc9200700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8097550 tx=0x7f0bb80cb5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a5903282a58ffe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0baaa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb809b050 con 0x557e08aaf650 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0bb809ee80 con 0x557e08aaf650 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e089c2400 con 0x557e08aaf650 2024-03-30T04:07:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bb80ce040 con 0x557e08aaf650 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0bdf693b80 10 client.27269 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557e0882fbe0 con 0x557e08aaf650 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.053+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb8080080 con 0x557e08aaf650 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0bb80a2fb0 con 0x557e08aaf650 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 10 client.27269 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 10 client.27269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 20 client.27269 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08aca270 0x557e08acc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 4 client.27269 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e089fa780 con 0x557e08aca270 2024-03-30T04:07:59.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08aca270 0x557e08acc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0baaa00700 10 client.27269 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.054+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08aca270 0x557e08acc690 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5531caf21b3fddf0 server_cookie=cc7cda45ffcc4c7a in_seq=0 out_seq=0 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.055+0000 7f0baaa00700 1 -- 172.21.15.23:0/133582491 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557e089fa780 con 0x557e08aca270 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.055+0000 7f0baaa00700 10 client.27269 handle_command_reply: tid=0 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.055+0000 7f0bdf693b80 1 client.27269 shutdown 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.055+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 msgr2=0x557e08aba8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.055+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8097550 tx=0x7f0bb80cb5a0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 shutdown_connections 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ab83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaf650 0x557e08aba8a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08aca270 0x557e08acc690 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/133582491 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaff70 0x557e08ab7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 >> 172.21.15.23:0/133582491 conn(0x557e08a0f720 msgr2=0x557e08ab1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0baaa00700 0 client.27269 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 shutdown_connections 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 1 -- 172.21.15.23:0/133582491 wait complete. 2024-03-30T04:07:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.056+0000 7f0bdf693b80 20 client.27269 trim_cache size 0 max 16384 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 Processor -- start 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 -- start start 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ab7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ac5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08ab0350 con 0x557e08abae70 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08abacc0 con 0x557e08aaff70 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08ac1da0 con 0x557e08aaf650 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bc3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bc9c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ac5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bc3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56652/0 (socket says 172.21.15.23:56652) 2024-03-30T04:07:59.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bc3e00700 1 -- 172.21.15.23:0/706094650 learned_addr learned my addr 172.21.15.23:0/706094650 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.078+0000 7f0bc9200700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ab7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 -- 172.21.15.23:0/706094650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 msgr2=0x557e08ac1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 -- 172.21.15.23:0/706094650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 msgr2=0x557e08ac5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ac5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 -- 172.21.15.23:0/706094650 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e08ad1280 con 0x557e08aaf650 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9c00700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ac5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:07:59.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bc9200700 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ab7550 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006010 tx=0x7f0bb8009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76f643ea67f2638e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bb6a00700 1 -- 172.21.15.23:0/706094650 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80ce0e0 con 0x557e08aaf650 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bb6a00700 1 -- 172.21.15.23:0/706094650 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e089c2400 con 0x557e08aaf650 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.079+0000 7f0bb6a00700 1 -- 172.21.15.23:0/706094650 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb809b050 con 0x557e08aaf650 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 -- 172.21.15.23:0/706094650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 msgr2=0x557e08ab7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ab7550 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8006010 tx=0x7f0bb8009310 comp rx=0 tx=0).stop 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 -- 172.21.15.23:0/706094650 shutdown_connections 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08abae70 0x557e08ac5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ab7550 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/706094650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08aaff70 0x557e08ac1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 -- 172.21.15.23:0/706094650 >> 172.21.15.23:0/706094650 conn(0x557e08a0f720 msgr2=0x557e08ab2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 -- 172.21.15.23:0/706094650 shutdown_connections 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.080+0000 7f0bdf693b80 1 -- 172.21.15.23:0/706094650 wait complete. 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 -- start start 2024-03-30T04:07:59.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 Processor -- start 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ac07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08ac0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08b26880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.081+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08b26dc0 con 0x557e08aaff70 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08b26f00 con 0x557e08abae70 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bdf693b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e08b27090 con 0x557e08aaf650 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bdf693b80 10 client.0 _ll_register_callbacks cb 0x557e089a74f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08ac0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08b26880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ac07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08b26880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56660/0 (socket says 172.21.15.23:56660) 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9c00700 1 -- 172.21.15.23:0/1482987852 learned_addr learned my addr 172.21.15.23:0/1482987852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 -- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 msgr2=0x557e08b26880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08b26880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 -- 172.21.15.23:0/1482987852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 msgr2=0x557e08ac0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08ac0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 -- 172.21.15.23:0/1482987852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb0017070 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0bc9200700 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ac07e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8097ac0 tx=0x7f0bb800b310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3d720de3204738b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.082+0000 7f0baaa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb809a740 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7f0bb80a3440 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bb80aee90 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bb80ce0e0 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 10 client.27271 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f0bbc014360 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bb80aee90 con 0x557e08aaf650 2024-03-30T04:07:59.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0bb80b3b70 con 0x557e08aaf650 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 10 client.27271 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 10 client.27271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 20 client.27271 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08b28420 0x557e08b2a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 4 client.27271 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.083+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557e08850d70 con 0x557e08b28420 2024-03-30T04:07:59.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.084+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08b28420 0x557e08b2a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.084+0000 7f0bc3e00700 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08b28420 0x557e08b2a840 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c63a342f15b8a8 server_cookie=7e9f7feb389246b4 in_seq=0 out_seq=0 2024-03-30T04:07:59.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.084+0000 7f0baaa00700 10 client.27271 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.085+0000 7f0baaa00700 1 -- 172.21.15.23:0/1482987852 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557e08850d70 con 0x557e08b28420 2024-03-30T04:07:59.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.085+0000 7f0baaa00700 10 client.27271 handle_command_reply: tid=0 2024-03-30T04:07:59.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.085+0000 7f0bdf693b80 1 client.27271 shutdown 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.085+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 msgr2=0x557e08ac07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.085+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ac07e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8097ac0 tx=0x7f0bb800b310 comp rx=0 tx=0).stop 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 shutdown_connections 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e08abae70 0x557e08b26880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e08b28420 0x557e08b2a840 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e08aaf650 0x557e08ac07e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 --2- 172.21.15.23:0/1482987852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e08aaff70 0x557e08ac0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0baaa00700 0 client.27271 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 >> 172.21.15.23:0/1482987852 conn(0x557e08a0f720 msgr2=0x557e08ab16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 shutdown_connections 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 1 -- 172.21.15.23:0/1482987852 wait complete. 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.086+0000 7f0bdf693b80 20 client.27271 trim_cache size 0 max 16384 2024-03-30T04:07:59.089 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:59.090 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:07:59.090 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "a62b6549-037d-4353-95b3-717e34f228c1", 2024-03-30T04:07:59.090 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:07:59.090 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:59.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 msgr2=0x7f0bac0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 0x7f0bac0a37d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40747e0 tx=0x7f0bb0009260 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 msgr2=0x7f0bc40730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8005910 tx=0x7f0bb8004730 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 shutdown_connections 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bc4073b30 0x7f0bc41a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bc4100f70 0x7f0bc40730b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bac0a1310 0x7f0bac0a37d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 --2- 172.21.15.23:0/428230329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bc4106090 0x7f0bc40735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 >> 172.21.15.23:0/428230329 conn(0x7f0bc40fc7f0 msgr2=0x7f0bc40fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 shutdown_connections 2024-03-30T04:07:59.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.087+0000 7f0bcbc00700 1 -- 172.21.15.23:0/428230329 wait complete. 2024-03-30T04:07:59.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:07:59.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 586939 11.49 MB/sec execute 156 sec latency 18.752 ms 2024-03-30T04:07:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 -- 172.21.15.23:0/2823310093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f8069260 msgr2=0x7fc5f8069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2823310093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f8069260 0x7fc5f8069640 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc5e80099c0 tx=0x7fc5e8009cd0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 -- 172.21.15.23:0/2823310093 shutdown_connections 2024-03-30T04:07:59.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2823310093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f81060b0 0x7fc5f810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2823310093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f8069260 0x7fc5f8069640 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2823310093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f8069c10 0x7fc5f8105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 -- 172.21.15.23:0/2823310093 >> 172.21.15.23:0/2823310093 conn(0x7fc5f8076d80 msgr2=0x7fc5f8077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 -- 172.21.15.23:0/2823310093 shutdown_connections 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.337+0000 7fc5fd600700 1 -- 172.21.15.23:0/2823310093 wait complete. 2024-03-30T04:07:59.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.338+0000 7fc5fd600700 1 Processor -- start 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.338+0000 7fc5fd600700 1 -- start start 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 0x7fc5f81993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f819dcc0 0x7fc5f819e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f819e740 con 0x7fc5f819dcc0 2024-03-30T04:07:59.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f819e8b0 con 0x7fc5f81060b0 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5fd600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f819ea70 con 0x7fc5f8069260 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 0x7fc5f81993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56676/0 (socket says 172.21.15.23:56676) 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 -- 172.21.15.23:0/2552320771 learned_addr learned my addr 172.21.15.23:0/2552320771 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 0x7fc5f81993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52728/0 (socket says 172.21.15.23:52728) 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f7400700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f819dcc0 0x7fc5f819e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 -- 172.21.15.23:0/2552320771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 msgr2=0x7fc5f81993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 0x7fc5f81993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 -- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f819dcc0 msgr2=0x7fc5f819e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f819dcc0 0x7fc5f819e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.339+0000 7fc5f6000700 1 -- 172.21.15.23:0/2552320771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e80095f0 con 0x7fc5f81060b0 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5f6000700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e970 tx=0x7fc5e400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5ef400700 1 -- 172.21.15.23:0/2552320771 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e4009790 con 0x7fc5f81060b0 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5ef400700 1 -- 172.21.15.23:0/2552320771 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e400c7e0 con 0x7fc5f81060b0 2024-03-30T04:07:59.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f81a5fd0 con 0x7fc5f81060b0 2024-03-30T04:07:59.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5ef400700 1 -- 172.21.15.23:0/2552320771 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40054e0 con 0x7fc5f81060b0 2024-03-30T04:07:59.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.340+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f81a6450 con 0x7fc5f81060b0 2024-03-30T04:07:59.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.343+0000 7fc5ef400700 1 -- 172.21.15.23:0/2552320771 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc5e400c950 con 0x7fc5f81060b0 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.344+0000 7fc5ef400700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 0x7fc5dc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 Processor -- start 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc5ef400700 1 -- 172.21.15.23:0/2552320771 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc5e4014070 con 0x7fc5f81060b0 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 0x7fc5dc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 -- start start 2024-03-30T04:07:59.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 0x557e363b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e362ff990 con 0x557e363a6ef0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e362fd350 con 0x557e363a65d0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.345+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e362f8c40 con 0x557e363b1df0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 0x557e363b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363a69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 0x557e363b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58584/0 (socket says 172.21.15.23:58584) 2024-03-30T04:07:59.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f6000700 1 -- 172.21.15.23:0/1650773147 learned_addr learned my addr 172.21.15.23:0/1650773147 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 0x7fc5dc0a3750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009990 tx=0x7fc5e8005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 -- 172.21.15.23:0/1650773147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 msgr2=0x557e363a69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363a69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 -- 172.21.15.23:0/1650773147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 msgr2=0x557e363b18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 0x557e363b18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 -- 172.21.15.23:0/1650773147 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e3617a3b0 con 0x557e363b1df0 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.346+0000 7fc5f7400700 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b41e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0003e80 tx=0x7fc5f0009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506027b1f01ce87f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.347+0000 7fc5e3400700 1 -- 172.21.15.23:0/1650773147 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0019070 con 0x557e363b1df0 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.347+0000 7fc5e3400700 1 -- 172.21.15.23:0/1650773147 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5f000f040 con 0x557e363b1df0 2024-03-30T04:07:59.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.347+0000 7fc5e3400700 1 -- 172.21.15.23:0/1650773147 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f00078a0 con 0x557e363b1df0 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1650773147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 msgr2=0x557e363b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b41e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0003e80 tx=0x7fc5f0009ba0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1650773147 shutdown_connections 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b41e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1650773147 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a6ef0 0x557e363b18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1650773147 >> 172.21.15.23:0/1650773147 conn(0x557e363066a0 msgr2=0x557e363a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1650773147 shutdown_connections 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1650773147 wait complete. 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.348+0000 7fc6110dbb80 1 -- start start 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 Processor -- start 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a65d0 0x557e363b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e36294d80 con 0x557e363b1df0 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e36294a20 con 0x557e363a6ef0 2024-03-30T04:07:59.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e3617c6a0 con 0x557e363a65d0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc6110dbb80 10 client.0 _ll_register_callbacks cb 0x557e3629e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc5f7400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56700/0 (socket says 172.21.15.23:56700) 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a65d0 0x557e363b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.349+0000 7fc5f6000700 1 -- 172.21.15.23:0/3981947138 learned_addr learned my addr 172.21.15.23:0/3981947138 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 -- 172.21.15.23:0/3981947138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a65d0 msgr2=0x557e363b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a65d0 0x557e363b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 -- 172.21.15.23:0/3981947138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 msgr2=0x557e363aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 -- 172.21.15.23:0/3981947138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f0003ae0 con 0x557e363a6ef0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5f6000700 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40964d0 tx=0x7fc5e409a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f709573b306f807 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5d7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e409de50 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc5e40a24a0 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e362b9380 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.350+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5e40b8250 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 10 client.27287 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557e36126b60 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e362b9380 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5e4099610 con 0x557e363a6ef0 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 10 client.27287 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 10 client.27287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:59.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 20 client.27287 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e363c1910 0x557e363c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 4 client.27287 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e362f1700 con 0x557e363c1910 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.351+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e363c1910 0x557e363c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.352+0000 7fc5d7e00700 10 client.27287 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.352+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e363c1910 0x557e363c3d30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d481e54ce5e619ba server_cookie=497603681fdc8603 in_seq=0 out_seq=0 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc5d7e00700 1 -- 172.21.15.23:0/3981947138 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557e362f1700 con 0x557e363c1910 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc5d7e00700 10 client.27287 handle_command_reply: tid=0 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 client.27287 shutdown 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 msgr2=0x557e363aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40964d0 tx=0x7fc5e409a7a0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 shutdown_connections 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e363c1910 0x557e363c3d30 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a65d0 0x557e363b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/3981947138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363aea00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 >> 172.21.15.23:0/3981947138 conn(0x557e363066a0 msgr2=0x557e363a8670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc5d7e00700 0 client.27287 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 shutdown_connections 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 1 -- 172.21.15.23:0/3981947138 wait complete. 2024-03-30T04:07:59.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.353+0000 7fc6110dbb80 20 client.27287 trim_cache size 0 max 16384 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.371+0000 7fc6110dbb80 1 Processor -- start 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.371+0000 7fc6110dbb80 1 -- start start 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.371+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.371+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 0x557e363ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.371+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363ad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e363aa460 con 0x557e363b1df0 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e363a7850 con 0x557e363a65d0 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e363c40c0 con 0x557e363a6ef0 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 0x557e363ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56714/0 (socket says 172.21.15.23:56714) 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 -- 172.21.15.23:0/1847741955 learned_addr learned my addr 172.21.15.23:0/1847741955 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6000700 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 0x557e363ad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52756/0 (socket says 172.21.15.23:52756) 2024-03-30T04:07:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f7400700 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 -- 172.21.15.23:0/1847741955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 msgr2=0x557e363ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 0x557e363ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 -- 172.21.15.23:0/1847741955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 msgr2=0x557e363ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 -- 172.21.15.23:0/1847741955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e362beea0 con 0x557e363a65d0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.372+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8005f50 tx=0x7fc5e8020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a936c784f1f17635 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc5e3e00700 1 -- 172.21.15.23:0/1847741955 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8034a80 con 0x557e363a65d0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc5e3e00700 1 -- 172.21.15.23:0/1847741955 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5e801e040 con 0x557e363a65d0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc5e3e00700 1 -- 172.21.15.23:0/1847741955 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e8003970 con 0x557e363a65d0 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1847741955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 msgr2=0x557e363b8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8005f50 tx=0x7fc5e8020000 comp rx=0 tx=0).stop 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1847741955 shutdown_connections 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363b1df0 0x557e363ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a65d0 0x557e363b8a70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/1847741955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363a6ef0 0x557e363ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.373+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1847741955 >> 172.21.15.23:0/1847741955 conn(0x557e363066a0 msgr2=0x557e363a9d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.374+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1847741955 shutdown_connections 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.374+0000 7fc6110dbb80 1 -- 172.21.15.23:0/1847741955 wait complete. 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.374+0000 7fc6110dbb80 1 -- start start 2024-03-30T04:07:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.374+0000 7fc6110dbb80 1 Processor -- start 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a65d0 0x557e363b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e36415750 con 0x557e363a65d0 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e36415890 con 0x557e363a6ef0 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e364159d0 con 0x557e363b1df0 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc6110dbb80 10 client.0 _ll_register_callbacks cb 0x557e3629e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc5f7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc5f6000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc5f7400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:52768/0 (socket says 172.21.15.23:52768) 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc5f7400700 1 -- 172.21.15.23:0/4143348608 learned_addr learned my addr 172.21.15.23:0/4143348608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.375+0000 7fc5f6a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a65d0 0x557e363b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 -- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 msgr2=0x557e363b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 -- 172.21.15.23:0/4143348608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a65d0 msgr2=0x557e363b71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a65d0 0x557e363b71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 -- 172.21.15.23:0/4143348608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f810e470 con 0x557e363b1df0 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5f7400700 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00093f0 tx=0x7fc5f0008cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=462af643c4710895 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5d7e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:07:59.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0025070 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9424+0+0 (secure 0 0 0) 0x7fc5f0015480 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5f000f040 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5f0019040 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc6110dbb80 10 client.27289 fetch_fsmap learned FSMap version 30 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.376+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557e3617ecd0 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5f000f040 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc5f0004780 con 0x557e363b1df0 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 10 client.27289 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 10 client.27289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 20 client.27289 populate_metadata read hostname 'smithi023' 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e36416fe0 0x557e36419400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:07:59.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 4 client.27289 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557e36147cf0 con 0x557e36416fe0 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.377+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e36416fe0 0x557e36419400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc5f6a00700 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e36416fe0 0x557e36419400 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26f99877fd2c25f7 server_cookie=f79c094e6be1ac50 in_seq=0 out_seq=0 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc5d7e00700 10 client.27289 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc5d7e00700 1 -- 172.21.15.23:0/4143348608 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557e36147cf0 con 0x557e36416fe0 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc5d7e00700 10 client.27289 handle_command_reply: tid=0 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 client.27289 shutdown 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 msgr2=0x557e363b7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00093f0 tx=0x7fc5f0008cd0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 shutdown_connections 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557e36416fe0 0x557e36419400 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557e363b1df0 0x557e363b7c30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc5d7e00700 0 client.27289 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557e363a65d0 0x557e363b71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 --2- 172.21.15.23:0/4143348608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557e363a6ef0 0x557e363b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 >> 172.21.15.23:0/4143348608 conn(0x557e363066a0 msgr2=0x557e363ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 shutdown_connections 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 1 -- 172.21.15.23:0/4143348608 wait complete. 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.378+0000 7fc6110dbb80 20 client.27289 trim_cache size 0 max 16384 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stdout: "a62b6549-037d-4353-95b3-717e34f228c1": { 2024-03-30T04:07:59.383 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:07:59.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "a62b6549-037d-4353-95b3-717e34f228c1", 2024-03-30T04:07:59.384 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:07:59.384 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:07:59.384 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:07:59.384 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 msgr2=0x7fc5dc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 0x7fc5dc0a3750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8009990 tx=0x7fc5e8005fd0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 msgr2=0x7fc5f8199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc5e400e970 tx=0x7fc5e400ec80 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 shutdown_connections 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc5f819dcc0 0x7fc5f819e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5f8069260 0x7fc5f81993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc5dc0a1290 0x7fc5dc0a3750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 --2- 172.21.15.23:0/2552320771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc5f81060b0 0x7fc5f8199920 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 >> 172.21.15.23:0/2552320771 conn(0x7fc5f8076d80 msgr2=0x7fc5f80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 shutdown_connections 2024-03-30T04:07:59.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:07:59.379+0000 7fc5fd600700 1 -- 172.21.15.23:0/2552320771 wait complete. 2024-03-30T04:07:59.399 INFO:tasks.cephfs.filesystem:scrub status for tag:a62b6549-037d-4353-95b3-717e34f228c1 - {'path': '/', 'tag': 'a62b6549-037d-4353-95b3-717e34f228c1', 'options': 'recursive,force'} 2024-03-30T04:08:00.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 589109 11.50 MB/sec execute 157 sec latency 18.369 ms 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: pgmap v184: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:00 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: pgmap v184: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.497 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:00 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: pgmap v184: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 280 op/s 2024-03-30T04:08:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:08:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:08:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:08:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:00 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:01.203 INFO:tasks.workunit.client.0.smithi023.stdout: 1 590314 11.47 MB/sec execute 158 sec latency 66.305 ms 2024-03-30T04:08:02.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 592380 11.46 MB/sec execute 159 sec latency 18.596 ms 2024-03-30T04:08:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:02 smithi154 ceph-mon[79161]: pgmap v185: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-30T04:08:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:02 smithi193 ceph-mon[79581]: pgmap v185: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-30T04:08:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:02 smithi023 ceph-mon[38254]: pgmap v185: 129 pgs: 129 active+clean; 359 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 191 op/s 2024-03-30T04:08:03.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 594048 11.45 MB/sec execute 160 sec latency 22.230 ms 2024-03-30T04:08:04.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 596337 11.45 MB/sec execute 161 sec latency 17.975 ms 2024-03-30T04:08:04.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:04 smithi154 ceph-mon[79161]: pgmap v186: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 300 op/s 2024-03-30T04:08:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:04 smithi193 ceph-mon[79581]: pgmap v186: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 300 op/s 2024-03-30T04:08:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:04 smithi023 ceph-mon[38254]: pgmap v186: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.7 MiB/s wr, 300 op/s 2024-03-30T04:08:05.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 597820 11.43 MB/sec execute 162 sec latency 29.231 ms 2024-03-30T04:08:06.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 599605 11.42 MB/sec execute 163 sec latency 108.545 ms 2024-03-30T04:08:06.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:06 smithi154 ceph-mon[79161]: pgmap v187: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 205 op/s 2024-03-30T04:08:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:06 smithi193 ceph-mon[79581]: pgmap v187: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 205 op/s 2024-03-30T04:08:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:06 smithi023 ceph-mon[38254]: pgmap v187: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.4 MiB/s wr, 205 op/s 2024-03-30T04:08:07.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 601200 11.40 MB/sec execute 164 sec latency 20.282 ms 2024-03-30T04:08:08.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 603602 11.41 MB/sec execute 165 sec latency 18.199 ms 2024-03-30T04:08:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:08 smithi154 ceph-mon[79161]: pgmap v188: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-03-30T04:08:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:08 smithi193 ceph-mon[79581]: pgmap v188: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-03-30T04:08:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:08 smithi023 ceph-mon[38254]: pgmap v188: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.3 MiB/s wr, 202 op/s 2024-03-30T04:08:09.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 604863 11.39 MB/sec execute 166 sec latency 21.140 ms 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:08:09.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:09 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:08:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:08:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:08:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:08:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:08:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:08:09.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:09 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:08:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:08:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:08:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:08:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:08:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:08:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:09 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:08:10.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 607165 11.39 MB/sec execute 167 sec latency 17.835 ms 2024-03-30T04:08:10.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:10 smithi154 ceph-mon[79161]: pgmap v189: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 293 op/s 2024-03-30T04:08:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:10 smithi154 ceph-mon[79161]: osdmap e90: 12 total, 12 up, 12 in 2024-03-30T04:08:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:10 smithi193 ceph-mon[79581]: pgmap v189: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 293 op/s 2024-03-30T04:08:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:10 smithi193 ceph-mon[79581]: osdmap e90: 12 total, 12 up, 12 in 2024-03-30T04:08:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:10 smithi023 ceph-mon[38254]: pgmap v189: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.8 MiB/s wr, 293 op/s 2024-03-30T04:08:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:10 smithi023 ceph-mon[38254]: osdmap e90: 12 total, 12 up, 12 in 2024-03-30T04:08:11.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 608228 11.36 MB/sec execute 168 sec latency 74.482 ms 2024-03-30T04:08:12.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 610641 11.37 MB/sec execute 169 sec latency 17.002 ms 2024-03-30T04:08:12.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:12 smithi154 ceph-mon[79161]: pgmap v191: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 240 op/s 2024-03-30T04:08:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:12 smithi193 ceph-mon[79581]: pgmap v191: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 240 op/s 2024-03-30T04:08:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:12 smithi023 ceph-mon[38254]: pgmap v191: 129 pgs: 129 active+clean; 362 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.9 MiB/s wr, 240 op/s 2024-03-30T04:08:13.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 612145 11.34 MB/sec execute 170 sec latency 20.761 ms 2024-03-30T04:08:14.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 614466 11.37 MB/sec execute 171 sec latency 18.455 ms 2024-03-30T04:08:14.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:14 smithi154 ceph-mon[79161]: pgmap v192: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:14 smithi193 ceph-mon[79581]: pgmap v192: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:14 smithi023 ceph-mon[38254]: pgmap v192: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:15.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 615822 11.34 MB/sec execute 172 sec latency 27.611 ms 2024-03-30T04:08:16.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 617773 11.33 MB/sec execute 173 sec latency 41.905 ms 2024-03-30T04:08:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:16 smithi154 ceph-mon[79161]: pgmap v193: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:16 smithi154 ceph-mon[79161]: osdmap e91: 12 total, 12 up, 12 in 2024-03-30T04:08:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:16 smithi193 ceph-mon[79581]: pgmap v193: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:16 smithi193 ceph-mon[79581]: osdmap e91: 12 total, 12 up, 12 in 2024-03-30T04:08:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:16 smithi023 ceph-mon[38254]: pgmap v193: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 211 op/s 2024-03-30T04:08:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:16 smithi023 ceph-mon[38254]: osdmap e91: 12 total, 12 up, 12 in 2024-03-30T04:08:17.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 619134 11.30 MB/sec execute 174 sec latency 19.899 ms 2024-03-30T04:08:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:17 smithi193 ceph-mon[79581]: pgmap v195: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-30T04:08:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:17 smithi023 ceph-mon[38254]: pgmap v195: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-30T04:08:17.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:17 smithi154 ceph-mon[79161]: pgmap v195: 129 pgs: 129 active+clean; 368 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-30T04:08:18.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 621502 11.32 MB/sec execute 175 sec latency 18.680 ms 2024-03-30T04:08:19.204 INFO:tasks.workunit.client.0.smithi023.stdout: 1 622979 11.30 MB/sec execute 176 sec latency 21.432 ms 2024-03-30T04:08:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:19 smithi193 ceph-mon[79581]: pgmap v196: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-30T04:08:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:19 smithi023 ceph-mon[38254]: pgmap v196: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-30T04:08:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:19 smithi154 ceph-mon[79161]: pgmap v196: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.0 MiB/s wr, 233 op/s 2024-03-30T04:08:20.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 625210 11.31 MB/sec execute 177 sec latency 19.536 ms 2024-03-30T04:08:21.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 626501 11.28 MB/sec execute 178 sec latency 82.614 ms 2024-03-30T04:08:22.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 628580 11.29 MB/sec execute 179 sec latency 55.800 ms 2024-03-30T04:08:22.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:22 smithi154 ceph-mon[79161]: pgmap v197: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-03-30T04:08:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:22 smithi193 ceph-mon[79581]: pgmap v197: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-03-30T04:08:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:22 smithi023 ceph-mon[38254]: pgmap v197: 129 pgs: 129 active+clean; 375 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-03-30T04:08:23.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 630042 11.26 MB/sec execute 180 sec latency 20.650 ms 2024-03-30T04:08:24.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 632296 11.27 MB/sec execute 181 sec latency 18.021 ms 2024-03-30T04:08:24.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:24 smithi154 ceph-mon[79161]: pgmap v198: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:24 smithi193 ceph-mon[79581]: pgmap v198: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:24 smithi023 ceph-mon[38254]: pgmap v198: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:25.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 633724 11.24 MB/sec execute 182 sec latency 21.173 ms 2024-03-30T04:08:26.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 635826 11.25 MB/sec execute 183 sec latency 51.330 ms 2024-03-30T04:08:26.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:26 smithi154 ceph-mon[79161]: pgmap v199: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:26 smithi193 ceph-mon[79581]: pgmap v199: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:26.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:26 smithi023 ceph-mon[38254]: pgmap v199: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:08:27.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 637144 11.22 MB/sec execute 184 sec latency 22.153 ms 2024-03-30T04:08:28.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 639387 11.23 MB/sec execute 185 sec latency 19.657 ms 2024-03-30T04:08:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:28 smithi154 ceph-mon[79161]: pgmap v200: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:08:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:28 smithi193 ceph-mon[79581]: pgmap v200: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:08:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:28 smithi023 ceph-mon[38254]: pgmap v200: 129 pgs: 129 active+clean; 383 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:08:29.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 640815 11.21 MB/sec execute 186 sec latency 20.359 ms 2024-03-30T04:08:29.400 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:08:29.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 -- 172.21.15.23:0/3652392174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073180 msgr2=0x7f7444073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 --2- 172.21.15.23:0/3652392174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073180 0x7f7444073560 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7434009990 tx=0x7f7434009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 -- 172.21.15.23:0/3652392174 shutdown_connections 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 --2- 172.21.15.23:0/3652392174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f744410d5a0 0x7f744410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 --2- 172.21.15.23:0/3652392174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073180 0x7f7444073560 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 --2- 172.21.15.23:0/3652392174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7444073aa0 0x7f744410d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 -- 172.21.15.23:0/3652392174 >> 172.21.15.23:0/3652392174 conn(0x7f74440fc7f0 msgr2=0x7f74440fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 -- 172.21.15.23:0/3652392174 shutdown_connections 2024-03-30T04:08:29.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.614+0000 7f744aa00700 1 -- 172.21.15.23:0/3652392174 wait complete. 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.615+0000 7f744aa00700 1 Processor -- start 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.615+0000 7f744aa00700 1 -- start start 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.615+0000 7f744aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.615+0000 7f744aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073aa0 0x7f7444199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f744aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744410d5a0 0x7f744419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f744aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744419e2f0 con 0x7f7444073aa0 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f744aa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744419e460 con 0x7f7444073180 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f744aa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744419e620 con 0x7f744410d5a0 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58702/0 (socket says 172.21.15.23:58702) 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 -- 172.21.15.23:0/2948592296 learned_addr learned my addr 172.21.15.23:0/2948592296 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7448a00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744410d5a0 0x7f744419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443400700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073aa0 0x7f7444199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 -- 172.21.15.23:0/2948592296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744410d5a0 msgr2=0x7f744419dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744410d5a0 0x7f744419dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 -- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073aa0 msgr2=0x7f7444199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073aa0 0x7f7444199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.616+0000 7f7443e00700 1 -- 172.21.15.23:0/2948592296 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340095f0 con 0x7f7444073180 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f7443e00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7434005770 tx=0x7f74340057a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f7440c00700 1 -- 172.21.15.23:0/2948592296 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743401d070 con 0x7f7444073180 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744419e8a0 con 0x7f7444073180 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f7440c00700 1 -- 172.21.15.23:0/2948592296 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7434021410 con 0x7f7444073180 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f7440c00700 1 -- 172.21.15.23:0/2948592296 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7434017650 con 0x7f7444073180 2024-03-30T04:08:29.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.617+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74441a6140 con 0x7f7444073180 2024-03-30T04:08:29.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.621+0000 7f7440c00700 1 -- 172.21.15.23:0/2948592296 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f743400fb00 con 0x7f7444073180 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.623+0000 7f7440c00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 0x7f74280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.623+0000 7f7440c00700 1 -- 172.21.15.23:0/2948592296 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74340ce380 con 0x7f7444073180 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.623+0000 7f745e5bdb80 1 Processor -- start 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443400700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 0x7f74280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 -- start start 2024-03-30T04:08:29.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff6f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff6f600 0x55f17ff719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17feb4990 con 0x55f17ff6f600 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17feb2350 con 0x55f17ff5bef0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17feadc40 con 0x55f17ff5b5d0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43570/0 (socket says 172.21.15.23:43570) 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443e00700 1 -- 172.21.15.23:0/2540727810 learned_addr learned my addr 172.21.15.23:0/2540727810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443400700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 0x7f74280a3750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f744419a920 tx=0x7f7430009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.624+0000 7f7443400700 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff6f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7448a00700 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff6f600 0x55f17ff719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 -- 172.21.15.23:0/2540727810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 msgr2=0x55f17ff6f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff6f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 -- 172.21.15.23:0/2540727810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff6f600 msgr2=0x55f17ff719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff6f600 0x55f17ff719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 -- 172.21.15.23:0/2540727810 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f17fd2f3b0 con 0x55f17ff5b5d0 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7443e00700 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f743409b0d0 tx=0x7f743409ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c700f72f04eabd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7439200700 1 -- 172.21.15.23:0/2540727810 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743409b140 con 0x55f17ff5b5d0 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7439200700 1 -- 172.21.15.23:0/2540727810 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743409fe90 con 0x55f17ff5b5d0 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.625+0000 7f7439200700 1 -- 172.21.15.23:0/2540727810 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743409b3a0 con 0x55f17ff5b5d0 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 -- 172.21.15.23:0/2540727810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 msgr2=0x55f17ff5b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f743409b0d0 tx=0x7f743409ac80 comp rx=0 tx=0).stop 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 -- 172.21.15.23:0/2540727810 shutdown_connections 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff6f600 0x55f17ff719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ff5b9b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/2540727810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff6f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 -- 172.21.15.23:0/2540727810 >> 172.21.15.23:0/2540727810 conn(0x55f17febb6a0 msgr2=0x55f17ff57a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 -- 172.21.15.23:0/2540727810 shutdown_connections 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.626+0000 7f745e5bdb80 1 -- 172.21.15.23:0/2540727810 wait complete. 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.627+0000 7f745e5bdb80 1 -- start start 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.627+0000 7f745e5bdb80 1 Processor -- start 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ff6ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff7aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17fe49d80 con 0x55f17ff5bef0 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17fe49a20 con 0x55f17ff5b5d0 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17fd316a0 con 0x55f17ff6f600 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f745e5bdb80 10 client.0 _ll_register_callbacks cb 0x55f17fe53470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58720/0 (socket says 172.21.15.23:58720) 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 -- 172.21.15.23:0/4094084494 learned_addr learned my addr 172.21.15.23:0/4094084494 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443400700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ff6ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7448a00700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff7aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 -- 172.21.15.23:0/4094084494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 msgr2=0x55f17ff7aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff7aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 -- 172.21.15.23:0/4094084494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 msgr2=0x55f17ff6ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ff6ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.628+0000 7f7443e00700 1 -- 172.21.15.23:0/4094084494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74441a6140 con 0x55f17ff5b5d0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f7443e00700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f74340a3000 tx=0x7f74340cb6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d63e063a2b2128a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743409ef70 con 0x55f17ff5b5d0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74340cbe40 con 0x55f17ff5b5d0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f17fe6e380 con 0x55f17ff5b5d0 2024-03-30T04:08:29.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74340a3530 con 0x55f17ff5b5d0 2024-03-30T04:08:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 10 client.27305 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:29.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f17fcdbb60 con 0x55f17ff5b5d0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f17fe6e380 con 0x55f17ff5b5d0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f743409b3a0 con 0x55f17ff5b5d0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 10 client.27305 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 10 client.27305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 20 client.27305 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ff7e9e0 0x55f17ff80dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 4 client.27305 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.629+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f17fea6700 con 0x55f17ff7e9e0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.630+0000 7f7443400700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ff7e9e0 0x55f17ff80dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.630+0000 7f7443400700 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ff7e9e0 0x55f17ff80dd0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=495c1a9c563ed6f0 server_cookie=9facaea2054a11ce in_seq=0 out_seq=0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.630+0000 7f742d600700 10 client.27305 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f742d600700 1 -- 172.21.15.23:0/4094084494 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f17fea6700 con 0x55f17ff7e9e0 2024-03-30T04:08:29.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f742d600700 10 client.27305 handle_command_reply: tid=0 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 client.27305 shutdown 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 msgr2=0x55f17ff6f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f74340a3000 tx=0x7f74340cb6e0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 shutdown_connections 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ff7e9e0 0x55f17ff80dd0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff7aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f742d600700 0 client.27305 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5b5d0 0x55f17ff6f9e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/4094084494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ff6ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 >> 172.21.15.23:0/4094084494 conn(0x55f17febb6a0 msgr2=0x55f17ff5d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 shutdown_connections 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.631+0000 7f745e5bdb80 1 -- 172.21.15.23:0/4094084494 wait complete. 2024-03-30T04:08:29.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.632+0000 7f745e5bdb80 20 client.27305 trim_cache size 0 max 16384 2024-03-30T04:08:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.653+0000 7f745e5bdb80 1 Processor -- start 2024-03-30T04:08:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.653+0000 7f745e5bdb80 1 -- start start 2024-03-30T04:08:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 0x55f17ff80dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ff5c640 con 0x55f17ff5b5d0 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ff57de0 con 0x55f17ff5bef0 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ff7b320 con 0x55f17ff6f600 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7448a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7443e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 0x55f17ff80dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7443400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7448a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43594/0 (socket says 172.21.15.23:43594) 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7443400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58734/0 (socket says 172.21.15.23:58734) 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7443e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 0x55f17ff80dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38434/0 (socket says 172.21.15.23:38434) 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.654+0000 7f7448a00700 1 -- 172.21.15.23:0/946173386 learned_addr learned my addr 172.21.15.23:0/946173386 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 -- 172.21.15.23:0/946173386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 msgr2=0x55f17ff74970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 -- 172.21.15.23:0/946173386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 msgr2=0x55f17ff80dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 0x55f17ff80dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 -- 172.21.15.23:0/946173386 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f17fe73ea0 con 0x55f17ff6f600 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7443400700 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7448a00700 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f743c007550 tx=0x7f743c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=816b5489554adae5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7439c00700 1 -- 172.21.15.23:0/946173386 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c00e040 con 0x55f17ff6f600 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7439c00700 1 -- 172.21.15.23:0/946173386 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743c00f040 con 0x55f17ff6f600 2024-03-30T04:08:29.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.655+0000 7f7439c00700 1 -- 172.21.15.23:0/946173386 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743c017650 con 0x55f17ff6f600 2024-03-30T04:08:29.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 -- 172.21.15.23:0/946173386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 msgr2=0x55f17ff6e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f743c007550 tx=0x7f743c00b040 comp rx=0 tx=0).stop 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 -- 172.21.15.23:0/946173386 shutdown_connections 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff6f600 0x55f17ff6e0d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5b5d0 0x55f17ff80dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/946173386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff5bef0 0x55f17ff74970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 -- 172.21.15.23:0/946173386 >> 172.21.15.23:0/946173386 conn(0x55f17febb6a0 msgr2=0x55f17ff69070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 -- 172.21.15.23:0/946173386 shutdown_connections 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.656+0000 7f745e5bdb80 1 -- 172.21.15.23:0/946173386 wait complete. 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 -- start start 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 Processor -- start 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ffcc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff6f600 0x55f17ffc6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ffc6e80 con 0x55f17ff5bef0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ffc6ff0 con 0x55f17ff6f600 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f17ffc71b0 con 0x55f17ff5b5d0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.657+0000 7f745e5bdb80 10 client.0 _ll_register_callbacks cb 0x55f17fe53470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43610/0 (socket says 172.21.15.23:43610) 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 -- 172.21.15.23:0/817005097 learned_addr learned my addr 172.21.15.23:0/817005097 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443400700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ffcc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7448a00700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff6f600 0x55f17ffc6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 -- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff6f600 msgr2=0x55f17ffc6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff6f600 0x55f17ffc6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 -- 172.21.15.23:0/817005097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 msgr2=0x55f17ffcc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ffcc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 -- 172.21.15.23:0/817005097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7430017070 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443400700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ffcc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f7443e00700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f743409b750 tx=0x7f74340996f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=874870312c5da31 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.658+0000 7f742d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74340b94a0 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74340cbe40 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f743409fe90 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74340a3530 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 10 client.27307 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f743c009280 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f743409fe90 con 0x55f17ff5b5d0 2024-03-30T04:08:29.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f743407c070 con 0x55f17ff5b5d0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 10 client.27307 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 10 client.27307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 20 client.27307 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ffd3ac0 0x55f17ffd5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 4 client.27307 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.659+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f17fcfccf0 con 0x55f17ffd3ac0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f7443400700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ffd3ac0 0x55f17ffd5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f7443400700 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ffd3ac0 0x55f17ffd5eb0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5cd3b27f29b48d3 server_cookie=b008362e961c36b2 in_seq=0 out_seq=0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f742d600700 10 client.27307 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f742d600700 1 -- 172.21.15.23:0/817005097 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f17fcfccf0 con 0x55f17ffd3ac0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f742d600700 10 client.27307 handle_command_reply: tid=0 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 client.27307 shutdown 2024-03-30T04:08:29.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 msgr2=0x55f17ffcc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f743409b750 tx=0x7f74340996f0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 shutdown_connections 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f17ff6f600 0x55f17ffc6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.660+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f17ffd3ac0 0x55f17ffd5eb0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f17ff5b5d0 0x55f17ffcc0d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 1 --2- 172.21.15.23:0/817005097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f17ff5bef0 0x55f17ffcc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f742d600700 0 client.27307 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 >> 172.21.15.23:0/817005097 conn(0x55f17febb6a0 msgr2=0x55f17ff69bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 shutdown_connections 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 1 -- 172.21.15.23:0/817005097 wait complete. 2024-03-30T04:08:29.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f745e5bdb80 20 client.27307 trim_cache size 0 max 16384 2024-03-30T04:08:29.666 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:08:29.666 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:08:29.666 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:08:29.666 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:08:29.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 msgr2=0x7f74280a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 0x7f74280a3750 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f744419a920 tx=0x7f7430009260 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 msgr2=0x7f74441993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.661+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7434005770 tx=0x7f74340057a0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 shutdown_connections 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744410d5a0 0x7f744419dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74280a1290 0x7f74280a3750 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7444073180 0x7f74441993d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 --2- 172.21.15.23:0/2948592296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7444073aa0 0x7f7444199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 >> 172.21.15.23:0/2948592296 conn(0x7f74440fc7f0 msgr2=0x7f74440fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 shutdown_connections 2024-03-30T04:08:29.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.662+0000 7f744aa00700 1 -- 172.21.15.23:0/2948592296 wait complete. 2024-03-30T04:08:29.680 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:08:29.681 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 -- 172.21.15.23:0/1936072322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c101db0 msgr2=0x7f175c102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 --2- 172.21.15.23:0/1936072322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c101db0 0x7f175c102190 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f174c009990 tx=0x7f174c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 -- 172.21.15.23:0/1936072322 shutdown_connections 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 --2- 172.21.15.23:0/1936072322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c10b1d0 0x7f175c10d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 --2- 172.21.15.23:0/1936072322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c101db0 0x7f175c102190 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 --2- 172.21.15.23:0/1936072322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 0x7f175c10abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 -- 172.21.15.23:0/1936072322 >> 172.21.15.23:0/1936072322 conn(0x7f175c0fb5f0 msgr2=0x7f175c0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 -- 172.21.15.23:0/1936072322 shutdown_connections 2024-03-30T04:08:29.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.896+0000 7f1762c00700 1 -- 172.21.15.23:0/1936072322 wait complete. 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.897+0000 7f1762c00700 1 Processor -- start 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.897+0000 7f1762c00700 1 -- start start 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c101db0 0x7f175c197100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 0x7f175c197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c19c020 con 0x7f175c101db0 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c19c190 con 0x7f175c1026d0 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1762c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175c19c350 con 0x7f175c10b1d0 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f175be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c101db0 0x7f175c197100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43626/0 (socket says 172.21.15.23:43626) 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 -- 172.21.15.23:0/791259391 learned_addr learned my addr 172.21.15.23:0/791259391 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f175b400700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 0x7f175c197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 -- 172.21.15.23:0/791259391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 msgr2=0x7f175c197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 0x7f175c197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 -- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c101db0 msgr2=0x7f175c197100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c101db0 0x7f175c197100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.898+0000 7f1760c00700 1 -- 172.21.15.23:0/791259391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1748009520 con 0x7f175c10b1d0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1760c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f175400ef30 tx=0x7f175400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1754018070 con 0x7f175c10b1d0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f175400f040 con 0x7f175c10b1d0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c0095f0 con 0x7f175c10b1d0 2024-03-30T04:08:29.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175400cd00 con 0x7f175c10b1d0 2024-03-30T04:08:29.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.899+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175c1a3c90 con 0x7f175c10b1d0 2024-03-30T04:08:29.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.901+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175c106300 con 0x7f175c10b1d0 2024-03-30T04:08:29.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.903+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1754004000 con 0x7f175c10b1d0 2024-03-30T04:08:29.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.905+0000 7f1758c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 0x7f17400a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:29.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.905+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f17540ccb10 con 0x7f175c10b1d0 2024-03-30T04:08:29.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.905+0000 7f175be00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 0x7f17400a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:29.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.906+0000 7f175be00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 0x7f17400a37d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f174c006010 tx=0x7f174c00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:29.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:29.909+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1754095c10 con 0x7f175c10b1d0 2024-03-30T04:08:30.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.192+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f175c0686d0 con 0x7f175c10b1d0 2024-03-30T04:08:30.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.193+0000 7f1758c00700 1 -- 172.21.15.23:0/791259391 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f1754099230 con 0x7f175c10b1d0 2024-03-30T04:08:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:08:30.192 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:08:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.195+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 msgr2=0x7f17400a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.195+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 0x7f17400a37d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f174c006010 tx=0x7f174c00b260 comp rx=0 tx=0).stop 2024-03-30T04:08:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 msgr2=0x7f175c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f175400ef30 tx=0x7f175400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 shutdown_connections 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f175c10b1d0 0x7f175c19b9e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f175c101db0 0x7f175c197100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f17400a1310 0x7f17400a37d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 --2- 172.21.15.23:0/791259391 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f175c1026d0 0x7f175c197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 >> 172.21.15.23:0/791259391 conn(0x7f175c0fb5f0 msgr2=0x7f175c0fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 shutdown_connections 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.196+0000 7f1762c00700 1 -- 172.21.15.23:0/791259391 wait complete. 2024-03-30T04:08:30.196 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:08:30.205 INFO:tasks.workunit.client.0.smithi023.stdout: 1 643059 11.22 MB/sec execute 187 sec latency 18.919 ms 2024-03-30T04:08:30.215 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 -- 172.21.15.23:0/650958195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9500731e0 msgr2=0x7fe9500735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 --2- 172.21.15.23:0/650958195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9500731e0 0x7fe9500735c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe940009990 tx=0x7fe940009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 -- 172.21.15.23:0/650958195 shutdown_connections 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 --2- 172.21.15.23:0/650958195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe95010d530 0x7fe95010f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 --2- 172.21.15.23:0/650958195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe9500731e0 0x7fe9500735c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 --2- 172.21.15.23:0/650958195 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe950073b00 0x7fe95010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 -- 172.21.15.23:0/650958195 >> 172.21.15.23:0/650958195 conn(0x7fe9500fc7c0 msgr2=0x7fe9500febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 -- 172.21.15.23:0/650958195 shutdown_connections 2024-03-30T04:08:30.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.424+0000 7fe956800700 1 -- 172.21.15.23:0/650958195 wait complete. 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.425+0000 7fe956800700 1 Processor -- start 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.425+0000 7fe956800700 1 -- start start 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9500731e0 0x7fe9501993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe950073b00 0x7fe950199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95019e310 con 0x7fe9500731e0 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95019e480 con 0x7fe95010d530 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe956800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe95019e640 con 0x7fe950073b00 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9500731e0 0x7fe9501993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe950073b00 0x7fe950199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58788/0 (socket says 172.21.15.23:58788) 2024-03-30T04:08:30.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 -- 172.21.15.23:0/766251639 learned_addr learned my addr 172.21.15.23:0/766251639 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 -- 172.21.15.23:0/766251639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe950073b00 msgr2=0x7fe950199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe950073b00 0x7fe950199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 -- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9500731e0 msgr2=0x7fe9501993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9500731e0 0x7fe9501993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.426+0000 7fe94fe00700 1 -- 172.21.15.23:0/766251639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9400095f0 con 0x7fe95010d530 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe94fe00700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe94800ef30 tx=0x7fe94800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe947e00700 1 -- 172.21.15.23:0/766251639 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948018070 con 0x7fe95010d530 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe947e00700 1 -- 172.21.15.23:0/766251639 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe94800f040 con 0x7fe95010d530 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95019e8c0 con 0x7fe95010d530 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe947e00700 1 -- 172.21.15.23:0/766251639 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94800cd00 con 0x7fe95010d530 2024-03-30T04:08:30.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.427+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9501a6160 con 0x7fe95010d530 2024-03-30T04:08:30.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.430+0000 7fe947e00700 1 -- 172.21.15.23:0/766251639 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe948003a00 con 0x7fe95010d530 2024-03-30T04:08:30.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:30 smithi154 ceph-mon[79161]: pgmap v201: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 259 op/s 2024-03-30T04:08:30.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe947e00700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 0x7fe9340a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 Processor -- start 2024-03-30T04:08:30.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe947e00700 1 -- 172.21.15.23:0/766251639 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe9480ccbd0 con 0x7fe95010d530 2024-03-30T04:08:30.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 -- start start 2024-03-30T04:08:30.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe94f400700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 0x7fe9340a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e4a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e3f1990 con 0x557a3e498ef0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e3ef350 con 0x557a3e4a3df0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.432+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e3eac40 con 0x557a3e4985d0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43656/0 (socket says 172.21.15.23:43656) 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 -- 172.21.15.23:0/2081231271 learned_addr learned my addr 172.21.15.23:0/2081231271 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94fe00700 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e4a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 0x7fe9340a3750 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b2e0 tx=0x7fe940005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 -- 172.21.15.23:0/2081231271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 msgr2=0x557a3e4a61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e4a61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 -- 172.21.15.23:0/2081231271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 msgr2=0x557a3e4a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 -- 172.21.15.23:0/2081231271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a3e26c3b0 con 0x557a3e4985d0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94fe00700 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e4a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.433+0000 7fe94f400700 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe940015000 tx=0x7fe940028000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83251f9fe7ec14ab server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.434+0000 7fe944c00700 1 -- 172.21.15.23:0/2081231271 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940004500 con 0x557a3e4985d0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.434+0000 7fe944c00700 1 -- 172.21.15.23:0/2081231271 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe94001e040 con 0x557a3e4985d0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.434+0000 7fe944c00700 1 -- 172.21.15.23:0/2081231271 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94000f9a0 con 0x557a3e4985d0 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.434+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2081231271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 msgr2=0x557a3e4989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.434+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe940015000 tx=0x7fe940028000 comp rx=0 tx=0).stop 2024-03-30T04:08:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2081231271 shutdown_connections 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e4a61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4989b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2081231271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2081231271 >> 172.21.15.23:0/2081231271 conn(0x557a3e3f86a0 msgr2=0x557a3e494a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2081231271 shutdown_connections 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2081231271 wait complete. 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 -- start start 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.435+0000 7fe96a2c0b80 1 Processor -- start 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e498ef0 0x557a3e4a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e4a3df0 0x557a3e4a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e386d80 con 0x557a3e4a3df0 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e386a20 con 0x557a3e4985d0 2024-03-30T04:08:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e26e6a0 con 0x557a3e498ef0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe96a2c0b80 10 client.0 _ll_register_callbacks cb 0x557a3e390470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe94f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe94f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58812/0 (socket says 172.21.15.23:58812) 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe94f400700 1 -- 172.21.15.23:0/2593615599 learned_addr learned my addr 172.21.15.23:0/2593615599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe94fe00700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e4a3df0 0x557a3e4a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.436+0000 7fe94ea00700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e498ef0 0x557a3e4a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 -- 172.21.15.23:0/2593615599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e498ef0 msgr2=0x557a3e4a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e498ef0 0x557a3e4a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 -- 172.21.15.23:0/2593615599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e4a3df0 msgr2=0x557a3e4a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e4a3df0 0x557a3e4a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 -- 172.21.15.23:0/2593615599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9501a6160 con 0x557a3e4985d0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe94f400700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe940031040 tx=0x7fe94000f460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aaa2968886cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe939600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940004f00 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe94000be40 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.437+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557a3e3ab380 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe94000b390 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 10 client.27329 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fe950110660 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557a3e3ab380 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe940038510 con 0x557a3e4985d0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 10 client.27329 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 10 client.27329 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 20 client.27329 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e4b3570 0x557a3e4b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 4 client.27329 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a3e3e3700 con 0x557a3e4b3570 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.438+0000 7fe94ea00700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e4b3570 0x557a3e4b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.439+0000 7fe94ea00700 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e4b3570 0x557a3e4b5990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a25ffc538de05e62 server_cookie=b2ff83357e0312c3 in_seq=0 out_seq=0 2024-03-30T04:08:30.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.439+0000 7fe939600700 10 client.27329 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.439+0000 7fe939600700 1 -- 172.21.15.23:0/2593615599 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557a3e3e3700 con 0x557a3e4b3570 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.439+0000 7fe939600700 10 client.27329 handle_command_reply: tid=0 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 client.27329 shutdown 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 msgr2=0x557a3e4a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe940031040 tx=0x7fe94000f460 comp rx=0 tx=0).stop 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 shutdown_connections 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e4a3df0 0x557a3e4a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a3850 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e4b3570 0x557a3e4b5990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/2593615599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e498ef0 0x557a3e4a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe939600700 0 client.27329 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 >> 172.21.15.23:0/2593615599 conn(0x557a3e3f86a0 msgr2=0x557a3e49a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 shutdown_connections 2024-03-30T04:08:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/2593615599 wait complete. 2024-03-30T04:08:30.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.440+0000 7fe96a2c0b80 20 client.27329 trim_cache size 0 max 16384 2024-03-30T04:08:30.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 Processor -- start 2024-03-30T04:08:30.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 -- start start 2024-03-30T04:08:30.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e48df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e490840 con 0x557a3e498ef0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e499830 con 0x557a3e4a3df0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e4957d0 con 0x557a3e4985d0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.455+0000 7fe94f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43666/0 (socket says 172.21.15.23:43666) 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 -- 172.21.15.23:0/44668860 learned_addr learned my addr 172.21.15.23:0/44668860 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e48df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94ea00700 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 -- 172.21.15.23:0/44668860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 msgr2=0x557a3e48df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e48df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 -- 172.21.15.23:0/44668860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 msgr2=0x557a3e4aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 -- 172.21.15.23:0/44668860 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a3e218b60 con 0x557a3e4985d0 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe94f400700 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe9400098e0 tx=0x7fe940009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd023360c2f2dca server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe945600700 1 -- 172.21.15.23:0/44668860 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9400058e0 con 0x557a3e4985d0 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe945600700 1 -- 172.21.15.23:0/44668860 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9400048d0 con 0x557a3e4985d0 2024-03-30T04:08:30.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.456+0000 7fe945600700 1 -- 172.21.15.23:0/44668860 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94000fad0 con 0x557a3e4985d0 2024-03-30T04:08:30.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/44668860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 msgr2=0x557a3e4a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe9400098e0 tx=0x7fe940009530 comp rx=0 tx=0).stop 2024-03-30T04:08:30.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/44668860 shutdown_connections 2024-03-30T04:08:30.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4a3df0 0x557a3e48df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4985d0 0x557a3e4a1240 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/44668860 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/44668860 >> 172.21.15.23:0/44668860 conn(0x557a3e3f86a0 msgr2=0x557a3e48d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/44668860 shutdown_connections 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.457+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/44668860 wait complete. 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 -- start start 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 Processor -- start 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4a3df0 0x557a3e500330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e500870 con 0x557a3e498ef0 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e5009e0 con 0x557a3e4985d0 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a3e500ba0 con 0x557a3e4a3df0 2024-03-30T04:08:30.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.458+0000 7fe96a2c0b80 10 client.0 _ll_register_callbacks cb 0x557a3e390470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58834/0 (socket says 172.21.15.23:58834) 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 -- 172.21.15.23:0/3857592769 learned_addr learned my addr 172.21.15.23:0/3857592769 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94ea00700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94fe00700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4a3df0 0x557a3e500330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 -- 172.21.15.23:0/3857592769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4a3df0 msgr2=0x557a3e500330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4a3df0 0x557a3e500330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 -- 172.21.15.23:0/3857592769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 msgr2=0x557a3e4a8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 -- 172.21.15.23:0/3857592769 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe950110660 con 0x557a3e4985d0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe94f400700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe940005660 tx=0x7fe940005380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7f5f08d27330d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.459+0000 7fe939600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe940004eb0 con 0x557a3e4985d0 2024-03-30T04:08:30.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe940038620 con 0x557a3e4985d0 2024-03-30T04:08:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe94001e040 con 0x557a3e4985d0 2024-03-30T04:08:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe94000f610 con 0x557a3e4985d0 2024-03-30T04:08:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 10 client.27335 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557a3e3b0ea0 con 0x557a3e4985d0 2024-03-30T04:08:30.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe940058020 con 0x557a3e4985d0 2024-03-30T04:08:30.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe940055070 con 0x557a3e4985d0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 10 client.27335 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 10 client.27335 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 20 client.27335 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e507ee0 0x557a3e50a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 4 client.27335 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.460+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557a3e239cf0 con 0x557a3e507ee0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe94ea00700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e507ee0 0x557a3e50a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe94ea00700 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e507ee0 0x557a3e50a2d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=141c309f3d0802e3 server_cookie=c1bdb2dc135f1b9 in_seq=0 out_seq=0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe939600700 10 client.27335 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe939600700 1 -- 172.21.15.23:0/3857592769 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557a3e239cf0 con 0x557a3e507ee0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe939600700 10 client.27335 handle_command_reply: tid=0 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.461+0000 7fe96a2c0b80 1 client.27335 shutdown 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 msgr2=0x557a3e4a8840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe940005660 tx=0x7fe940005380 comp rx=0 tx=0).stop 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 shutdown_connections 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x557a3e507ee0 0x557a3e50a2d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557a3e4a3df0 0x557a3e500330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557a3e4985d0 0x557a3e4a8840 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 --2- 172.21.15.23:0/3857592769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557a3e498ef0 0x557a3e4a8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe939600700 0 client.27335 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 >> 172.21.15.23:0/3857592769 conn(0x557a3e3f86a0 msgr2=0x557a3e48df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 shutdown_connections 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 1 -- 172.21.15.23:0/3857592769 wait complete. 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.462+0000 7fe96a2c0b80 20 client.27335 trim_cache size 0 max 16384 2024-03-30T04:08:30.465 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:08:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 msgr2=0x7fe9340a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 0x7fe9340a3750 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe94000b2e0 tx=0x7fe940005fd0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 msgr2=0x7fe95019dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe94800ef30 tx=0x7fe94800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 shutdown_connections 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe9340a1290 0x7fe9340a3750 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe95010d530 0x7fe95019dcd0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9500731e0 0x7fe9501993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 --2- 172.21.15.23:0/766251639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe950073b00 0x7fe950199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 >> 172.21.15.23:0/766251639 conn(0x7fe9500fc7c0 msgr2=0x7fe9500fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 shutdown_connections 2024-03-30T04:08:30.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.463+0000 7fe956800700 1 -- 172.21.15.23:0/766251639 wait complete. 2024-03-30T04:08:30.478 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:30 smithi023 ceph-mon[38254]: pgmap v201: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 259 op/s 2024-03-30T04:08:30.480 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:08:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:30 smithi193 ceph-mon[79581]: pgmap v201: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.5 MiB/s wr, 259 op/s 2024-03-30T04:08:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 -- 172.21.15.23:0/207050479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c100710 msgr2=0x7f378c100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 --2- 172.21.15.23:0/207050479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c100710 0x7f378c100af0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098f0 tx=0x7f377c009c00 comp rx=0 tx=0).stop 2024-03-30T04:08:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 -- 172.21.15.23:0/207050479 shutdown_connections 2024-03-30T04:08:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 --2- 172.21.15.23:0/207050479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c10d750 0x7f378c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 --2- 172.21.15.23:0/207050479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c100710 0x7f378c100af0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 --2- 172.21.15.23:0/207050479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c101030 0x7f378c10d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 -- 172.21.15.23:0/207050479 >> 172.21.15.23:0/207050479 conn(0x7f378c076d40 msgr2=0x7f378c077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.701+0000 7f3792800700 1 -- 172.21.15.23:0/207050479 shutdown_connections 2024-03-30T04:08:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.702+0000 7f3792800700 1 -- 172.21.15.23:0/207050479 wait complete. 2024-03-30T04:08:30.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 Processor -- start 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 -- start start 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c100710 0x7f378c199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c10d750 0x7f378c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c19e440 con 0x7f378c100710 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c19e5b0 con 0x7f378c10d750 2024-03-30T04:08:30.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f3792800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c19e770 con 0x7f378c101030 2024-03-30T04:08:30.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f378aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c100710 0x7f378c199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.703+0000 7f378aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43680/0 (socket says 172.21.15.23:43680) 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 -- 172.21.15.23:0/521741963 learned_addr learned my addr 172.21.15.23:0/521741963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378be00700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c10d750 0x7f378c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 -- 172.21.15.23:0/521741963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c10d750 msgr2=0x7f378c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c10d750 0x7f378c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 -- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c100710 msgr2=0x7f378c199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c100710 0x7f378c199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 -- 172.21.15.23:0/521741963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37780095f0 con 0x7f378c101030 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f378aa00700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f377800ec10 tx=0x7f377800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f3783e00700 1 -- 172.21.15.23:0/521741963 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377800cc90 con 0x7f378c101030 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3784009c40 con 0x7f378c101030 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f3783e00700 1 -- 172.21.15.23:0/521741963 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37780044e0 con 0x7f378c101030 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.704+0000 7f3783e00700 1 -- 172.21.15.23:0/521741963 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778010430 con 0x7f378c101030 2024-03-30T04:08:30.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.705+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c009520 con 0x7f378c101030 2024-03-30T04:08:30.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.708+0000 7f3783e00700 1 -- 172.21.15.23:0/521741963 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3778003fb0 con 0x7f378c101030 2024-03-30T04:08:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.709+0000 7f3783e00700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 0x7f37700a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.709+0000 7f3783e00700 1 -- 172.21.15.23:0/521741963 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3778014070 con 0x7f378c101030 2024-03-30T04:08:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f378b400700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 0x7f37700a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f37a62c5b80 1 Processor -- start 2024-03-30T04:08:30.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f37a62c5b80 1 -- start start 2024-03-30T04:08:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c9618c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c961978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.710+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c960e5990 con 0x555c9618c5d0 2024-03-30T04:08:30.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c960e3350 con 0x555c9618cef0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c960dec40 con 0x555c96197df0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378b400700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 0x7f37700a3750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098c0 tx=0x7f377c019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c9618c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43692/0 (socket says 172.21.15.23:43692) 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 -- 172.21.15.23:0/333249022 learned_addr learned my addr 172.21.15.23:0/333249022 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378aa00700 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c961978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 -- 172.21.15.23:0/333249022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 msgr2=0x555c961978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c961978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 -- 172.21.15.23:0/333249022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 msgr2=0x555c9618c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c9618c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.711+0000 7f378be00700 1 -- 172.21.15.23:0/333249022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c95f603b0 con 0x555c96197df0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f378be00700 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f378400e6c0 tx=0x7f3784009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a867dd97d4810d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f3780c00700 1 -- 172.21.15.23:0/333249022 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378400e040 con 0x555c96197df0 2024-03-30T04:08:30.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f3780c00700 1 -- 172.21.15.23:0/333249022 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f378400f040 con 0x555c96197df0 2024-03-30T04:08:30.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f3780c00700 1 -- 172.21.15.23:0/333249022 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37840046d0 con 0x555c96197df0 2024-03-30T04:08:30.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f37a62c5b80 1 -- 172.21.15.23:0/333249022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 msgr2=0x555c9619a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.712+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f378400e6c0 tx=0x7f3784009920 comp rx=0 tx=0).stop 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 -- 172.21.15.23:0/333249022 shutdown_connections 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c9619a1e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c9618c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/333249022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c961978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 -- 172.21.15.23:0/333249022 >> 172.21.15.23:0/333249022 conn(0x555c960ec6a0 msgr2=0x555c96188a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 -- 172.21.15.23:0/333249022 shutdown_connections 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 -- 172.21.15.23:0/333249022 wait complete. 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.713+0000 7f37a62c5b80 1 -- start start 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 Processor -- start 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c961977a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c96194b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c961950f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9607ad80 con 0x555c9618c5d0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9607aa20 con 0x555c9618cef0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c95f626a0 con 0x555c96197df0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.714+0000 7f37a62c5b80 10 client.0 _ll_register_callbacks cb 0x555c96084470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c961977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c961977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38552/0 (socket says 172.21.15.23:38552) 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c96194b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378b400700 1 -- 172.21.15.23:0/1622880741 learned_addr learned my addr 172.21.15.23:0/1622880741 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378be00700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c961950f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 -- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 msgr2=0x555c961950f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c961950f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 -- 172.21.15.23:0/1622880741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 msgr2=0x555c961977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c961977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 -- 172.21.15.23:0/1622880741 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378400e370 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378be00700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c961950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f378aa00700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c96194b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f37780ca960 tx=0x7f3778099ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60105653a0e7596b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.715+0000 7f3775600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377809aa20 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c9609f380 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f377809a320 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f377809e150 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 10 client.27347 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555c95f0cb60 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f377807d020 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3778099810 con 0x555c9618cef0 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 10 client.27347 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 10 client.27347 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:08:30.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 20 client.27347 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:30.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961a7a00 0x555c961a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 4 client.27347 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:30.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.716+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c960d7700 con 0x555c961a7a00 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.717+0000 7f378b400700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961a7a00 0x555c961a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.717+0000 7f378b400700 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961a7a00 0x555c961a9e20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15c59ca2ea47081a server_cookie=64ae43bf470c01ba in_seq=0 out_seq=0 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.717+0000 7f3775600700 10 client.27347 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f3775600700 1 -- 172.21.15.23:0/1622880741 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555c960d7700 con 0x555c961a7a00 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f3775600700 10 client.27347 handle_command_reply: tid=0 2024-03-30T04:08:30.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 client.27347 shutdown 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 msgr2=0x555c96194b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c96194b30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f37780ca960 tx=0x7f3778099ee0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 shutdown_connections 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961a7a00 0x555c961a9e20 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c96197df0 0x555c961950f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c961977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f3775600700 0 client.27347 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/1622880741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c96194b30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.718+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 >> 172.21.15.23:0/1622880741 conn(0x555c960ec6a0 msgr2=0x555c9618e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.719+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 shutdown_connections 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.719+0000 7f37a62c5b80 1 -- 172.21.15.23:0/1622880741 wait complete. 2024-03-30T04:08:30.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.719+0000 7f37a62c5b80 20 client.27347 trim_cache size 0 max 16384 2024-03-30T04:08:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 Processor -- start 2024-03-30T04:08:30.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 -- start start 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c96194af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 0x555c96187f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c96197df0 0x555c9619eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c961aa1b0 con 0x555c9618c5d0 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.734+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c961956f0 con 0x555c96197df0 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9619efe0 con 0x555c9618cef0 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c96194af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 0x555c96187f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c96194af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38556/0 (socket says 172.21.15.23:38556) 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378b400700 1 -- 172.21.15.23:0/3455182575 learned_addr learned my addr 172.21.15.23:0/3455182575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378be00700 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c96197df0 0x555c9619eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 -- 172.21.15.23:0/3455182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c96197df0 msgr2=0x555c9619eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c96197df0 0x555c9619eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 -- 172.21.15.23:0/3455182575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 msgr2=0x555c96194af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c96194af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 -- 172.21.15.23:0/3455182575 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c960a4ea0 con 0x555c9618cef0 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.735+0000 7f378aa00700 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 0x555c96187f10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f377809a020 tx=0x7f3778004470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29367c2ccb4b57ef server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.736+0000 7f3781600700 1 -- 172.21.15.23:0/3455182575 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778009a10 con 0x555c9618cef0 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.736+0000 7f3781600700 1 -- 172.21.15.23:0/3455182575 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f377807d070 con 0x555c9618cef0 2024-03-30T04:08:30.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.736+0000 7f3781600700 1 -- 172.21.15.23:0/3455182575 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37780a27a0 con 0x555c9618cef0 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3455182575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 msgr2=0x555c96187f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 0x555c96187f10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f377809a020 tx=0x7f3778004470 comp rx=0 tx=0).stop 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3455182575 shutdown_connections 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c96197df0 0x555c9619eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c9618c5d0 0x555c96194af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3455182575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618cef0 0x555c96187f10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3455182575 >> 172.21.15.23:0/3455182575 conn(0x555c960ec6a0 msgr2=0x555c961943c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3455182575 shutdown_connections 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3455182575 wait complete. 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 -- start start 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.737+0000 7f37a62c5b80 1 Processor -- start 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 0x555c9619e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c9619e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96197df0 0x555c9619ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9619d050 con 0x555c96197df0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9619d1c0 con 0x555c9618cef0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c9619d380 con 0x555c9618c5d0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f37a62c5b80 10 client.0 _ll_register_callbacks cb 0x555c96084470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f378b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 0x555c9619e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f378aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c9619e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f378b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 0x555c9619e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43706/0 (socket says 172.21.15.23:43706) 2024-03-30T04:08:30.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f378b400700 1 -- 172.21.15.23:0/3202885386 learned_addr learned my addr 172.21.15.23:0/3202885386 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.738+0000 7f378be00700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96197df0 0x555c9619ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 -- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 msgr2=0x555c9619e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 0x555c9619e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 -- 172.21.15.23:0/3202885386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96197df0 msgr2=0x555c9619ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96197df0 0x555c9619ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 -- 172.21.15.23:0/3202885386 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c95f0cb60 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f378aa00700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c9619e750 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x555c9618e270 tx=0x7f3778000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4466e2b2ebc02bdb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f3775600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3778009a10 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f378400f040 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f377807e080 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3778099c80 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f37a62c5b80 10 client.27353 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555c95f64cd0 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.739+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37780ab080 con 0x555c9618cef0 2024-03-30T04:08:30.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3778096110 con 0x555c9618cef0 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 10 client.27353 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 10 client.27353 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 20 client.27353 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961fc570 0x555c961fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 4 client.27353 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:08:30.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555c95f2dcf0 con 0x555c961fc570 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.740+0000 7f378b400700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961fc570 0x555c961fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f378b400700 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961fc570 0x555c961fe960 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e576a5b28b19fa9b server_cookie=38a31cef56f5a920 in_seq=0 out_seq=0 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f3775600700 10 client.27353 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f3775600700 1 -- 172.21.15.23:0/3202885386 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555c95f2dcf0 con 0x555c961fc570 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f3775600700 10 client.27353 handle_command_reply: tid=0 2024-03-30T04:08:30.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 client.27353 shutdown 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 msgr2=0x555c9619e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c9619e750 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x555c9618e270 tx=0x7f3778000bc0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 shutdown_connections 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x555c961fc570 0x555c961fe960 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555c96197df0 0x555c9619ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555c9618c5d0 0x555c9619e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 --2- 172.21.15.23:0/3202885386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555c9618cef0 0x555c9619e750 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f3775600700 0 client.27353 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 >> 172.21.15.23:0/3202885386 conn(0x555c960ec6a0 msgr2=0x555c9618fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 shutdown_connections 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 1 -- 172.21.15.23:0/3202885386 wait complete. 2024-03-30T04:08:30.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.741+0000 7f37a62c5b80 20 client.27353 trim_cache size 0 max 16384 2024-03-30T04:08:30.745 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:08:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 msgr2=0x7f37700a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 0x7f37700a3750 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098c0 tx=0x7f377c019040 comp rx=0 tx=0).stop 2024-03-30T04:08:30.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 msgr2=0x7f378c199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f377800ec10 tx=0x7f377800ef20 comp rx=0 tx=0).stop 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 shutdown_connections 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f37700a1290 0x7f37700a3750 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f378c10d750 0x7f378c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f378c100710 0x7f378c199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 --2- 172.21.15.23:0/521741963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f378c101030 0x7f378c199a60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 >> 172.21.15.23:0/521741963 conn(0x7f378c076d40 msgr2=0x7f378c10eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 shutdown_connections 2024-03-30T04:08:30.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:30.742+0000 7f3792800700 1 -- 172.21.15.23:0/521741963 wait complete. 2024-03-30T04:08:30.759 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 -- 172.21.15.23:0/1588909819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 msgr2=0x7f027410db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 --2- 172.21.15.23:0/1588909819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f027410db50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f026c009fc0 tx=0x7f026c00b330 comp rx=0 tx=0).stop 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 -- 172.21.15.23:0/1588909819 shutdown_connections 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 --2- 172.21.15.23:0/1588909819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f027410db50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 --2- 172.21.15.23:0/1588909819 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f02741060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 --2- 172.21.15.23:0/1588909819 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02740682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.006+0000 7f027a000700 1 -- 172.21.15.23:0/1588909819 >> 172.21.15.23:0/1588909819 conn(0x7f0274076d40 msgr2=0x7f0274077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.007+0000 7f027a000700 1 -- 172.21.15.23:0/1588909819 shutdown_connections 2024-03-30T04:08:31.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.007+0000 7f027a000700 1 -- 172.21.15.23:0/1588909819 wait complete. 2024-03-30T04:08:31.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.008+0000 7f027a000700 1 Processor -- start 2024-03-30T04:08:31.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 -- start start 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02741a3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f027419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419de50 con 0x7f0274067e30 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419dfc0 con 0x7f0274105ce0 2024-03-30T04:08:31.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.009+0000 7f027a000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027419e180 con 0x7f02740687f0 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.010+0000 7f0273e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f027419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.010+0000 7f0272a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.010+0000 7f0272a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43730/0 (socket says 172.21.15.23:43730) 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.010+0000 7f0272a00700 1 -- 172.21.15.23:0/2366234155 learned_addr learned my addr 172.21.15.23:0/2366234155 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.010+0000 7f0273400700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02741a3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 -- 172.21.15.23:0/2366234155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 msgr2=0x7f027419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f027419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 -- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 msgr2=0x7f02741a3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02741a3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 -- 172.21.15.23:0/2366234155 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c009c40 con 0x7f02740687f0 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0273e00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f027419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:31.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0273400700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02741a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f0272a00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f026000e9a0 tx=0x7f026000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f026be00700 1 -- 172.21.15.23:0/2366234155 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260009790 con 0x7f02740687f0 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.011+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027419e460 con 0x7f02740687f0 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.012+0000 7f026be00700 1 -- 172.21.15.23:0/2366234155 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f026000cbd0 con 0x7f02740687f0 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.012+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02741aa650 con 0x7f02740687f0 2024-03-30T04:08:31.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.012+0000 7f026be00700 1 -- 172.21.15.23:0/2366234155 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260018680 con 0x7f02740687f0 2024-03-30T04:08:31.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.017+0000 7f026be00700 1 -- 172.21.15.23:0/2366234155 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f02600187e0 con 0x7f02740687f0 2024-03-30T04:08:31.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.019+0000 7f028dbedb80 1 Processor -- start 2024-03-30T04:08:31.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.019+0000 7f026be00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 0x7f02580a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 -- start start 2024-03-30T04:08:31.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f026be00700 1 -- 172.21.15.23:0/2366234155 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0260014070 con 0x7f02740687f0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f0273e00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 0x7f02580a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87a46990 con 0x55eb87aedef0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87a44350 con 0x55eb87af8df0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f028dbedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87a3fc40 con 0x55eb87aed5d0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f0272a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f0273e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f0272a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38582/0 (socket says 172.21.15.23:38582) 2024-03-30T04:08:31.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.020+0000 7f0272a00700 1 -- 172.21.15.23:0/1693867108 learned_addr learned my addr 172.21.15.23:0/1693867108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0273400700 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 -- 172.21.15.23:0/1693867108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 msgr2=0x55eb87aed9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aed9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 -- 172.21.15.23:0/1693867108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 msgr2=0x55eb87afb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 -- 172.21.15.23:0/1693867108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb878c13b0 con 0x55eb87aedef0 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0273e00700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 0x7f02580a37d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f026c007970 tx=0x7f026c003ba0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f0272a00700 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f026000cf90 tx=0x7f02600a2450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cdecf995ca4f4bb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.021+0000 7f025fe00700 1 -- 172.21.15.23:0/1693867108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260010630 con 0x55eb87aedef0 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f025fe00700 1 -- 172.21.15.23:0/1693867108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0260018a90 con 0x55eb87aedef0 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f025fe00700 1 -- 172.21.15.23:0/1693867108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02600990b0 con 0x55eb87aedef0 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f028dbedb80 1 -- 172.21.15.23:0/1693867108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 msgr2=0x55eb87af88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f026000cf90 tx=0x7f02600a2450 comp rx=0 tx=0).stop 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f028dbedb80 1 -- 172.21.15.23:0/1693867108 shutdown_connections 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.022+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aed9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1693867108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af88b0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 -- 172.21.15.23:0/1693867108 >> 172.21.15.23:0/1693867108 conn(0x55eb87a4d6a0 msgr2=0x55eb87ae9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 -- 172.21.15.23:0/1693867108 shutdown_connections 2024-03-30T04:08:31.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 -- 172.21.15.23:0/1693867108 wait complete. 2024-03-30T04:08:31.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.023+0000 7f028dbedb80 1 -- start start 2024-03-30T04:08:31.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 Processor -- start 2024-03-30T04:08:31.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 0x55eb87af6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb879dbd80 con 0x55eb87aedef0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb879dba20 con 0x55eb87aed5d0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb878c36a0 con 0x55eb87af8df0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.024+0000 7f028dbedb80 10 client.0 _ll_register_callbacks cb 0x55eb879e5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58886/0 (socket says 172.21.15.23:58886) 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 -- 172.21.15.23:0/2829169343 learned_addr learned my addr 172.21.15.23:0/2829169343 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0272a00700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273e00700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 0x55eb87af6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 -- 172.21.15.23:0/2829169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 msgr2=0x55eb87af6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 0x55eb87af6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 -- 172.21.15.23:0/2829169343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 msgr2=0x55eb87af5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 -- 172.21.15.23:0/2829169343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02741aa650 con 0x55eb87aed5d0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273e00700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 0x55eb87af6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f0273400700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f026400b860 tx=0x7f026400bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e494a37022b62ae server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.025+0000 7f025cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264012040 con 0x55eb87aed5d0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0264018880 con 0x55eb87aed5d0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb87a00380 con 0x55eb87aed5d0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0264018a40 con 0x55eb87aed5d0 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f028dbedb80 10 client.27365 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.026+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55eb8786db60 con 0x55eb87aed5d0 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb87a00380 con 0x55eb87aed5d0 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f02640173d0 con 0x55eb87aed5d0 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 10 client.27365 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 10 client.27365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 20 client.27365 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b08af0 0x55eb87b0af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 4 client.27365 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:31.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb87a38700 con 0x55eb87b08af0 2024-03-30T04:08:31.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.027+0000 7f0272a00700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b08af0 0x55eb87b0af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.028+0000 7f0272a00700 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b08af0 0x55eb87b0af10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ec83d51dd42db96a server_cookie=f0e37c1e0a4903e6 in_seq=0 out_seq=0 2024-03-30T04:08:31.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.028+0000 7f025cc00700 10 client.27365 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.029+0000 7f025cc00700 1 -- 172.21.15.23:0/2829169343 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eb87a38700 con 0x55eb87b08af0 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.029+0000 7f025cc00700 10 client.27365 handle_command_reply: tid=0 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.029+0000 7f028dbedb80 1 client.27365 shutdown 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 msgr2=0x55eb87af87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f026400b860 tx=0x7f026400bb70 comp rx=0 tx=0).stop 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 shutdown_connections 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b08af0 0x55eb87b0af10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87af8df0 0x55eb87af6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f025cc00700 0 client.27365 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87aed5d0 0x55eb87af87d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 --2- 172.21.15.23:0/2829169343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 >> 172.21.15.23:0/2829169343 conn(0x55eb87a4d6a0 msgr2=0x55eb87aef670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 shutdown_connections 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 1 -- 172.21.15.23:0/2829169343 wait complete. 2024-03-30T04:08:31.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.030+0000 7f028dbedb80 20 client.27365 trim_cache size 0 max 16384 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 Processor -- start 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 -- start start 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87af4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87b0b2e0 con 0x55eb87aedef0 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87af6e10 con 0x55eb87af8df0 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f028dbedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87af4ee0 con 0x55eb87aed5d0 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f0273400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.045+0000 7f0272a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43744/0 (socket says 172.21.15.23:43744) 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 -- 172.21.15.23:0/388420111 learned_addr learned my addr 172.21.15.23:0/388420111 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273e00700 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87af4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 -- 172.21.15.23:0/388420111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 msgr2=0x55eb87af4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87af4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 -- 172.21.15.23:0/388420111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 msgr2=0x55eb87af4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 -- 172.21.15.23:0/388420111 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb87a05ea0 con 0x55eb87aed5d0 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0272a00700 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0273400700 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f026400d440 tx=0x7f026400d8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3576bf70b84ae3ce server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0269200700 1 -- 172.21.15.23:0/388420111 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264012070 con 0x55eb87aed5d0 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0269200700 1 -- 172.21.15.23:0/388420111 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02640058e0 con 0x55eb87aed5d0 2024-03-30T04:08:31.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.046+0000 7f0269200700 1 -- 172.21.15.23:0/388420111 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264017ce0 con 0x55eb87aed5d0 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 -- 172.21.15.23:0/388420111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 msgr2=0x55eb87affad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f026400d440 tx=0x7f026400d8c0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 -- 172.21.15.23:0/388420111 shutdown_connections 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87af4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87affad0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 --2- 172.21.15.23:0/388420111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87af4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 -- 172.21.15.23:0/388420111 >> 172.21.15.23:0/388420111 conn(0x55eb87a4d6a0 msgr2=0x55eb87af0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 -- 172.21.15.23:0/388420111 shutdown_connections 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.047+0000 7f028dbedb80 1 -- 172.21.15.23:0/388420111 wait complete. 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 -- start start 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 Processor -- start 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87affc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87afe4f0 con 0x55eb87aedef0 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87afe660 con 0x55eb87af8df0 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb87afe820 con 0x55eb87aed5d0 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.048+0000 7f028dbedb80 10 client.0 _ll_register_callbacks cb 0x55eb879e5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0272a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87affc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58902/0 (socket says 172.21.15.23:58902) 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43760/0 (socket says 172.21.15.23:43760) 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273e00700 1 -- 172.21.15.23:0/1161168171 learned_addr learned my addr 172.21.15.23:0/1161168171 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 -- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 msgr2=0x55eb87afde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 -- 172.21.15.23:0/1161168171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 msgr2=0x55eb87affc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87affc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 -- 172.21.15.23:0/1161168171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb878c13b0 con 0x55eb87aed5d0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f0273400700 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f026400d440 tx=0x7f0264017b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=642d7be0bb8b5f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.049+0000 7f025cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264012070 con 0x55eb87aed5d0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0260018a90 con 0x55eb87aed5d0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0264017c90 con 0x55eb87aed5d0 2024-03-30T04:08:31.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0264017e50 con 0x55eb87aed5d0 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 10 client.27367 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55eb8786db60 con 0x55eb87aed5d0 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0260018a90 con 0x55eb87aed5d0 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0264020ad0 con 0x55eb87aed5d0 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 10 client.27367 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 10 client.27367 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 20 client.27367 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b5d8f0 0x55eb87b5fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.050+0000 7f028dbedb80 4 client.27367 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.051+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55eb8788ecf0 con 0x55eb87b5d8f0 2024-03-30T04:08:31.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.051+0000 7f0272a00700 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b5d8f0 0x55eb87b5fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.051+0000 7f0272a00700 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b5d8f0 0x55eb87b5fce0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=38feb998d29baefd server_cookie=cba0dbb2acef3a75 in_seq=0 out_seq=0 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.051+0000 7f025cc00700 10 client.27367 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.052+0000 7f025cc00700 1 -- 172.21.15.23:0/1161168171 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55eb8788ecf0 con 0x55eb87b5d8f0 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.052+0000 7f025cc00700 10 client.27367 handle_command_reply: tid=0 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.052+0000 7f028dbedb80 1 client.27367 shutdown 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 msgr2=0x55eb87aff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f026400d440 tx=0x7f0264017b80 comp rx=0 tx=0).stop 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 shutdown_connections 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55eb87b5d8f0 0x55eb87b5fce0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55eb87af8df0 0x55eb87afde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eb87aed5d0 0x55eb87aff720 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 --2- 172.21.15.23:0/1161168171 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb87aedef0 0x55eb87affc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 >> 172.21.15.23:0/1161168171 conn(0x55eb87a4d6a0 msgr2=0x55eb87af0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f025cc00700 0 client.27367 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:08:31.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 shutdown_connections 2024-03-30T04:08:31.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 1 -- 172.21.15.23:0/1161168171 wait complete. 2024-03-30T04:08:31.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.053+0000 7f028dbedb80 20 client.27367 trim_cache size 0 max 16384 2024-03-30T04:08:31.056 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:08:31.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 msgr2=0x7f02580a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 0x7f02580a37d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f026c007970 tx=0x7f026c003ba0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 msgr2=0x7f02741a3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f026000e9a0 tx=0x7f026000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 shutdown_connections 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f02580a1310 0x7f02580a37d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0274105ce0 0x7f027419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0274067e30 0x7f02741a3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 --2- 172.21.15.23:0/2366234155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f02740687f0 0x7f02741a3ba0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 >> 172.21.15.23:0/2366234155 conn(0x7f0274076d40 msgr2=0x7f02740fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 shutdown_connections 2024-03-30T04:08:31.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.054+0000 7f027a000700 1 -- 172.21.15.23:0/2366234155 wait complete. 2024-03-30T04:08:31.073 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:08:31.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 644438 11.19 MB/sec execute 188 sec latency 32.264 ms 2024-03-30T04:08:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 -- 172.21.15.23:0/1197493645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac100800 msgr2=0x7f14ac104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 --2- 172.21.15.23:0/1197493645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac100800 0x7f14ac104830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1498009990 tx=0x7f1498009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 -- 172.21.15.23:0/1197493645 shutdown_connections 2024-03-30T04:08:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 --2- 172.21.15.23:0/1197493645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac104f60 0x7f14ac10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 --2- 172.21.15.23:0/1197493645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac0ffe50 0x7f14ac100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 --2- 172.21.15.23:0/1197493645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac100800 0x7f14ac104830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 -- 172.21.15.23:0/1197493645 >> 172.21.15.23:0/1197493645 conn(0x7f14ac076d40 msgr2=0x7f14ac077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 -- 172.21.15.23:0/1197493645 shutdown_connections 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.328+0000 7f14b2600700 1 -- 172.21.15.23:0/1197493645 wait complete. 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 Processor -- start 2024-03-30T04:08:31.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 -- start start 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac0ffe50 0x7f14ac198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac104f60 0x7f14ac198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1a3790 con 0x7f14ac104f60 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1a3900 con 0x7f14ac198ad0 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.329+0000 7f14b2600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac1a3ac0 con 0x7f14ac0ffe50 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58906/0 (socket says 172.21.15.23:58906) 2024-03-30T04:08:31.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 -- 172.21.15.23:0/2555320374 learned_addr learned my addr 172.21.15.23:0/2555320374 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14ab400700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac0ffe50 0x7f14ac198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14aaa00700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac104f60 0x7f14ac198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 -- 172.21.15.23:0/2555320374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac0ffe50 msgr2=0x7f14ac198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac0ffe50 0x7f14ac198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 -- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac104f60 msgr2=0x7f14ac198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac104f60 0x7f14ac198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 -- 172.21.15.23:0/2555320374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980095f0 con 0x7f14ac198ad0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.330+0000 7f14abe00700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14a400ef30 tx=0x7f14a400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.331+0000 7f14a3e00700 1 -- 172.21.15.23:0/2555320374 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a4018070 con 0x7f14ac198ad0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.331+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ac1a3d40 con 0x7f14ac198ad0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.331+0000 7f14a3e00700 1 -- 172.21.15.23:0/2555320374 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f14a400f040 con 0x7f14ac198ad0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.331+0000 7f14a3e00700 1 -- 172.21.15.23:0/2555320374 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a400ccb0 con 0x7f14ac198ad0 2024-03-30T04:08:31.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.331+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac1a42a0 con 0x7f14ac198ad0 2024-03-30T04:08:31.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.334+0000 7f14a3e00700 1 -- 172.21.15.23:0/2555320374 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f14a4003a00 con 0x7f14ac198ad0 2024-03-30T04:08:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.335+0000 7f14a3e00700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 0x7f14900a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.335+0000 7f14c607db80 1 Processor -- start 2024-03-30T04:08:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.335+0000 7f14a3e00700 1 -- 172.21.15.23:0/2555320374 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f14a40cca60 con 0x7f14ac198ad0 2024-03-30T04:08:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.335+0000 7f14ab400700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 0x7f14900a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 -- start start 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e13d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e148df0 0x55c12e14b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e096990 con 0x55c12e148df0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e094350 con 0x55c12e13def0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14c607db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e08fc40 con 0x55c12e13d5d0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14ab400700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 0x7f14900a37d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f149c009c60 tx=0x7f149c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14ab400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e13d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58910/0 (socket says 172.21.15.23:58910) 2024-03-30T04:08:31.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14aaa00700 1 -- 172.21.15.23:0/2269918952 learned_addr learned my addr 172.21.15.23:0/2269918952 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.336+0000 7f14abe00700 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e148df0 0x55c12e14b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 -- 172.21.15.23:0/2269918952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 msgr2=0x55c12e13d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e13d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 -- 172.21.15.23:0/2269918952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e148df0 msgr2=0x55c12e14b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e148df0 0x55c12e14b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 -- 172.21.15.23:0/2269918952 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c12df113b0 con 0x55c12e13def0 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14aaa00700 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1498009990 tx=0x7f1498005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7095f7e34961f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14a0c00700 1 -- 172.21.15.23:0/2269918952 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149801d070 con 0x55c12e13def0 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14a0c00700 1 -- 172.21.15.23:0/2269918952 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f149801e040 con 0x55c12e13def0 2024-03-30T04:08:31.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.337+0000 7f14a0c00700 1 -- 172.21.15.23:0/2269918952 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14980047f0 con 0x55c12e13def0 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 -- 172.21.15.23:0/2269918952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 msgr2=0x55c12e1488b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f1498009990 tx=0x7f1498005ec0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 -- 172.21.15.23:0/2269918952 shutdown_connections 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e148df0 0x55c12e14b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e13d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 --2- 172.21.15.23:0/2269918952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e1488b0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 -- 172.21.15.23:0/2269918952 >> 172.21.15.23:0/2269918952 conn(0x55c12e09d6a0 msgr2=0x55c12e139a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 -- 172.21.15.23:0/2269918952 shutdown_connections 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.338+0000 7f14c607db80 1 -- 172.21.15.23:0/2269918952 wait complete. 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.339+0000 7f14c607db80 1 -- start start 2024-03-30T04:08:31.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.339+0000 7f14c607db80 1 Processor -- start 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13d5d0 0x55c12e148800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 0x55c12e146120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e02bd80 con 0x55c12e13d5d0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e02ba20 con 0x55c12e13def0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12df136a0 con 0x55c12e148df0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14c607db80 10 client.0 _ll_register_callbacks cb 0x55c12e035470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14ab400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13d5d0 0x55c12e148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58922/0 (socket says 172.21.15.23:58922) 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 -- 172.21.15.23:0/1082541466 learned_addr learned my addr 172.21.15.23:0/1082541466 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14abe00700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 0x55c12e146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 -- 172.21.15.23:0/1082541466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 msgr2=0x55c12e146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 0x55c12e146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 -- 172.21.15.23:0/1082541466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13d5d0 msgr2=0x55c12e148800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13d5d0 0x55c12e148800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14aaa00700 1 -- 172.21.15.23:0/1082541466 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ac10d350 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.340+0000 7f14abe00700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 0x55c12e146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f14aaa00700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f149800be80 tx=0x7f149800ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a6ab32edca5397 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f1495600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149801d030 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f14980058e0 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c12e050380 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1498005bc0 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f14c607db80 10 client.27383 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.341+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c12debdb60 con 0x55c12e13def0 2024-03-30T04:08:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c12e050380 con 0x55c12e13def0 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f149800f8a0 con 0x55c12e13def0 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 10 client.27383 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 10 client.27383 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 20 client.27383 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e158490 0x55c12e15a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 4 client.27383 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c12e088700 con 0x55c12e158490 2024-03-30T04:08:31.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.342+0000 7f14ab400700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e158490 0x55c12e15a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.343+0000 7f14ab400700 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e158490 0x55c12e15a880 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c48ce2fde8e4afe0 server_cookie=f534f5467ad5c789 in_seq=0 out_seq=0 2024-03-30T04:08:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.343+0000 7f1495600700 10 client.27383 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:08:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f1495600700 1 -- 172.21.15.23:0/1082541466 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c12e088700 con 0x55c12e158490 2024-03-30T04:08:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f1495600700 10 client.27383 handle_command_reply: tid=0 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 client.27383 shutdown 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 msgr2=0x55c12e145af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f149800be80 tx=0x7f149800ba30 comp rx=0 tx=0).stop 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 shutdown_connections 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e158490 0x55c12e15a880 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e148df0 0x55c12e146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13d5d0 0x55c12e148800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f1495600700 0 client.27383 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 --2- 172.21.15.23:0/1082541466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e13def0 0x55c12e145af0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 >> 172.21.15.23:0/1082541466 conn(0x55c12e09d6a0 msgr2=0x55c12e13f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.344+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 shutdown_connections 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.345+0000 7f14c607db80 1 -- 172.21.15.23:0/1082541466 wait complete. 2024-03-30T04:08:31.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.345+0000 7f14c607db80 20 client.27383 trim_cache size 0 max 16384 2024-03-30T04:08:31.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 Processor -- start 2024-03-30T04:08:31.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 -- start start 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e145380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e146720 con 0x55c12e13def0 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e146a00 con 0x55c12e148df0 2024-03-30T04:08:31.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14c607db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e14ff80 con 0x55c12e13d5d0 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14ab400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.359+0000 7f14abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e145380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43800/0 (socket says 172.21.15.23:43800) 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38648/0 (socket says 172.21.15.23:38648) 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e145380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58936/0 (socket says 172.21.15.23:58936) 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 -- 172.21.15.23:0/3640420654 learned_addr learned my addr 172.21.15.23:0/3640420654 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 -- 172.21.15.23:0/3640420654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 msgr2=0x55c12e145380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e145380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 -- 172.21.15.23:0/3640420654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 msgr2=0x55c12e14fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 -- 172.21.15.23:0/3640420654 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c12e055ea0 con 0x55c12e13d5d0 2024-03-30T04:08:31.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14ab400700 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55c12e15b150 tx=0x7f149c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=256f2ee6b7242bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14a1600700 1 -- 172.21.15.23:0/3640420654 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c023040 con 0x55c12e13d5d0 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14a1600700 1 -- 172.21.15.23:0/3640420654 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f149c024040 con 0x55c12e13d5d0 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.360+0000 7f14a1600700 1 -- 172.21.15.23:0/3640420654 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149c004a90 con 0x55c12e13d5d0 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 -- 172.21.15.23:0/3640420654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 msgr2=0x55c12e146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55c12e15b150 tx=0x7f149c014000 comp rx=0 tx=0).stop 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 -- 172.21.15.23:0/3640420654 shutdown_connections 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e145380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e146120 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 --2- 172.21.15.23:0/3640420654 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 -- 172.21.15.23:0/3640420654 >> 172.21.15.23:0/3640420654 conn(0x55c12e09d6a0 msgr2=0x55c12e1325a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 -- 172.21.15.23:0/3640420654 shutdown_connections 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.361+0000 7f14c607db80 1 -- 172.21.15.23:0/3640420654 wait complete. 2024-03-30T04:08:31.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 -- start start 2024-03-30T04:08:31.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 Processor -- start 2024-03-30T04:08:31.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e14f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e14da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e14e130 con 0x55c12e13def0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e14e2a0 con 0x55c12e148df0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c12e14e460 con 0x55c12e13d5d0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.362+0000 7f14c607db80 10 client.0 _ll_register_callbacks cb 0x55c12e035470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14abe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e14da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14ab400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e14f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38650/0 (socket says 172.21.15.23:38650) 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 -- 172.21.15.23:0/265986713 learned_addr learned my addr 172.21.15.23:0/265986713 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 -- 172.21.15.23:0/265986713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 msgr2=0x55c12e14f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e14f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 -- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 msgr2=0x55c12e14da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e14da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 -- 172.21.15.23:0/265986713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149c017070 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14abe00700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f14aaa00700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f149800fc30 tx=0x7f1498005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8161a6e2dea2753a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.363+0000 7f1495600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149801d040 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1498000e40 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f149801e040 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1498005bc0 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 10 client.17388 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c12df15cd0 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f149801e040 con 0x55c12e13def0 2024-03-30T04:08:31.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1498004ba0 con 0x55c12e13def0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 10 client.17388 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 10 client.17388 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 20 client.17388 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e1ad0a0 0x55c12e1af490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 4 client.17388 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.364+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c12dedecf0 con 0x55c12e1ad0a0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f14ab400700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e1ad0a0 0x55c12e1af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f14ab400700 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e1ad0a0 0x55c12e1af490 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=206d1b4fb484380a server_cookie=14e923b958afab1 in_seq=0 out_seq=0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f1495600700 10 client.17388 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f1495600700 1 -- 172.21.15.23:0/265986713 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c12dedecf0 con 0x55c12e1ad0a0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f1495600700 10 client.17388 handle_command_reply: tid=0 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f14c607db80 1 client.17388 shutdown 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 msgr2=0x55c12e14f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.365+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f149800fc30 tx=0x7f1498005410 comp rx=0 tx=0).stop 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 shutdown_connections 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c12e148df0 0x55c12e14da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c12e1ad0a0 0x55c12e1af490 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c12e13d5d0 0x55c12e14f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 --2- 172.21.15.23:0/265986713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c12e13def0 0x55c12e14f9a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f1495600700 0 client.17388 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 >> 172.21.15.23:0/265986713 conn(0x55c12e09d6a0 msgr2=0x55c12e13f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 shutdown_connections 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 1 -- 172.21.15.23:0/265986713 wait complete. 2024-03-30T04:08:31.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14c607db80 20 client.17388 trim_cache size 0 max 16384 2024-03-30T04:08:31.371 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:08:31.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 msgr2=0x7f14900a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 0x7f14900a37d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f149c009c60 tx=0x7f149c009260 comp rx=0 tx=0).stop 2024-03-30T04:08:31.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 msgr2=0x7f14ac1a3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.366+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f14a400ef30 tx=0x7f14a400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 shutdown_connections 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f14900a1310 0x7f14900a37d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14ac198ad0 0x7f14ac1a3080 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14ac0ffe50 0x7f14ac198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 --2- 172.21.15.23:0/2555320374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f14ac104f60 0x7f14ac198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 >> 172.21.15.23:0/2555320374 conn(0x7f14ac076d40 msgr2=0x7f14ac0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 shutdown_connections 2024-03-30T04:08:31.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.367+0000 7f14b2600700 1 -- 172.21.15.23:0/2555320374 wait complete. 2024-03-30T04:08:31.383 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:31 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/791259391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:08:31.385 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:08:31.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:31 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/791259391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:08:31.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:31 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/791259391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 -- 172.21.15.23:0/105381798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 msgr2=0x7f9814073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 --2- 172.21.15.23:0/105381798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814073540 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9804009990 tx=0x7f9804009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 -- 172.21.15.23:0/105381798 shutdown_connections 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 --2- 172.21.15.23:0/105381798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981410f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 --2- 172.21.15.23:0/105381798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814073540 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 --2- 172.21.15.23:0/105381798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f981410cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 -- 172.21.15.23:0/105381798 >> 172.21.15.23:0/105381798 conn(0x7f98140fc670 msgr2=0x7f98140fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 -- 172.21.15.23:0/105381798 shutdown_connections 2024-03-30T04:08:31.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.608+0000 7f9813200700 1 -- 172.21.15.23:0/105381798 wait complete. 2024-03-30T04:08:31.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.609+0000 7f9813200700 1 Processor -- start 2024-03-30T04:08:31.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 -- start start 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f98141997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981419e180 con 0x7f9814073a80 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981419e2f0 con 0x7f981410d400 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9813200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981419e4b0 con 0x7f9814073160 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9811e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9811e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43816/0 (socket says 172.21.15.23:43816) 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9811e00700 1 -- 172.21.15.23:0/1831389954 learned_addr learned my addr 172.21.15.23:0/1831389954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.610+0000 7f9811400700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f98141997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9812800700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 -- 172.21.15.23:0/1831389954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 msgr2=0x7f981419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 -- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 msgr2=0x7f98141997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f98141997a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 -- 172.21.15.23:0/1831389954 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9800009520 con 0x7f9814073160 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811400700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f98141997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9812800700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:31.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.611+0000 7f9811e00700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9804005740 tx=0x7f9804005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.612+0000 7f980aa00700 1 -- 172.21.15.23:0/1831389954 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980401d070 con 0x7f9814073160 2024-03-30T04:08:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.612+0000 7f980aa00700 1 -- 172.21.15.23:0/1831389954 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9804022470 con 0x7f9814073160 2024-03-30T04:08:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.612+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98040095f0 con 0x7f9814073160 2024-03-30T04:08:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.612+0000 7f980aa00700 1 -- 172.21.15.23:0/1831389954 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804017650 con 0x7f9814073160 2024-03-30T04:08:31.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.612+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98141a5df0 con 0x7f9814073160 2024-03-30T04:08:31.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.616+0000 7f980aa00700 1 -- 172.21.15.23:0/1831389954 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f98040225e0 con 0x7f9814073160 2024-03-30T04:08:31.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.618+0000 7f982c81cb80 1 Processor -- start 2024-03-30T04:08:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f980aa00700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 0x7f97f80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 -- start start 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f980aa00700 1 -- 172.21.15.23:0/1831389954 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f98040ce340 con 0x7f9814073160 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b635916990 con 0x55b6359be0c0 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b635914350 con 0x55b6359bd7a0 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f982c81cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b63590fc40 con 0x55b6359c8fc0 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.619+0000 7f9812800700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 0x7f97f80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58976/0 (socket says 172.21.15.23:58976) 2024-03-30T04:08:31.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 -- 172.21.15.23:0/3897665676 learned_addr learned my addr 172.21.15.23:0/3897665676 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9812800700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811400700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 -- 172.21.15.23:0/3897665676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 msgr2=0x55b6359cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 -- 172.21.15.23:0/3897665676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 msgr2=0x55b6359c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 -- 172.21.15.23:0/3897665676 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6357913b0 con 0x55b6359bd7a0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811400700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9812800700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.620+0000 7f9811e00700 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f980409b090 tx=0x7f980409aed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7888f0b3eac373ea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f9812800700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 0x7f97f80a37d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f980c004090 tx=0x7f980c004450 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f97ff000700 1 -- 172.21.15.23:0/3897665676 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980409b800 con 0x55b6359bd7a0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f97ff000700 1 -- 172.21.15.23:0/3897665676 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f980409b100 con 0x55b6359bd7a0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f97ff000700 1 -- 172.21.15.23:0/3897665676 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98040a1070 con 0x55b6359bd7a0 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f982c81cb80 1 -- 172.21.15.23:0/3897665676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 msgr2=0x55b6359bdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.621+0000 7f982c81cb80 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f980409b090 tx=0x7f980409aed0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 -- 172.21.15.23:0/3897665676 shutdown_connections 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359bd7a0 0x55b6359bdb80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 --2- 172.21.15.23:0/3897665676 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 -- 172.21.15.23:0/3897665676 >> 172.21.15.23:0/3897665676 conn(0x55b63591d6a0 msgr2=0x55b6359b9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 -- 172.21.15.23:0/3897665676 shutdown_connections 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 -- 172.21.15.23:0/3897665676 wait complete. 2024-03-30T04:08:31.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.622+0000 7f982c81cb80 1 -- start start 2024-03-30T04:08:31.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 Processor -- start 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359c8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 0x55b6359c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6358abd80 con 0x55b6359bd7a0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6358aba20 con 0x55b6359c8fc0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6357936a0 con 0x55b6359be0c0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f982c81cb80 10 client.0 _ll_register_callbacks cb 0x55b6358b5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9811400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 0x55b6359c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9812800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9811e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9811400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 0x55b6359c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43830/0 (socket says 172.21.15.23:43830) 2024-03-30T04:08:31.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9812800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58980/0 (socket says 172.21.15.23:58980) 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.623+0000 7f9811400700 1 -- 172.21.15.23:0/229956359 learned_addr learned my addr 172.21.15.23:0/229956359 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 -- 172.21.15.23:0/229956359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 msgr2=0x55b6359c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 0x55b6359c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 -- 172.21.15.23:0/229956359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 msgr2=0x55b6359c8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359c8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 -- 172.21.15.23:0/229956359 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98141a5df0 con 0x55b6359c8fc0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f9812800700 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f980c004850 tx=0x7f980c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d6af1779a95f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f97f3400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c024600 con 0x55b6359c8fc0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f980c024760 con 0x55b6359c8fc0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.624+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b6358d0380 con 0x55b6359c8fc0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f980c018e90 con 0x55b6359c8fc0 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 10 client.27401 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b63573db60 con 0x55b6359c8fc0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b6358d0380 con 0x55b6359c8fc0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f980c017910 con 0x55b6359c8fc0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 10 client.27401 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 10 client.27401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 20 client.27401 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b6359d8840 0x55b6359dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 4 client.27401 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.625+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b635908700 con 0x55b6359d8840 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.626+0000 7f9811e00700 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b6359d8840 0x55b6359dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.626+0000 7f9811e00700 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b6359d8840 0x55b6359dac60 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca2e0a5600d759e server_cookie=ae7c20e2f4b86ac9 in_seq=0 out_seq=0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.626+0000 7f97f3400700 10 client.27401 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f97f3400700 1 -- 172.21.15.23:0/229956359 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b635908700 con 0x55b6359d8840 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f97f3400700 10 client.27401 handle_command_reply: tid=0 2024-03-30T04:08:31.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 client.27401 shutdown 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 msgr2=0x55b6359c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f980c004850 tx=0x7f980c004b40 comp rx=0 tx=0).stop 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 shutdown_connections 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359c61b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359c8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b6359d8840 0x55b6359dac60 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 --2- 172.21.15.23:0/229956359 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359be0c0 0x55b6359c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 >> 172.21.15.23:0/229956359 conn(0x55b63591d6a0 msgr2=0x55b6359bf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f97f3400700 0 client.27401 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 shutdown_connections 2024-03-30T04:08:31.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.627+0000 7f982c81cb80 1 -- 172.21.15.23:0/229956359 wait complete. 2024-03-30T04:08:31.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.628+0000 7f982c81cb80 20 client.27401 trim_cache size 0 max 16384 2024-03-30T04:08:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.643+0000 7f982c81cb80 1 Processor -- start 2024-03-30T04:08:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.643+0000 7f982c81cb80 1 -- start start 2024-03-30T04:08:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359be0c0 0x55b6359c84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359c0780 con 0x55b6359bd7a0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359c7260 con 0x55b6359be0c0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f982c81cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359d5340 con 0x55b6359c8fc0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9811e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43842/0 (socket says 172.21.15.23:43842) 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9811400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359be0c0 0x55b6359c84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 -- 172.21.15.23:0/331823205 learned_addr learned my addr 172.21.15.23:0/331823205 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 -- 172.21.15.23:0/331823205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359be0c0 msgr2=0x55b6359c84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359be0c0 0x55b6359c84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 -- 172.21.15.23:0/331823205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 msgr2=0x55b6359cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.644+0000 7f9812800700 1 -- 172.21.15.23:0/331823205 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6358d5ea0 con 0x55b6359c8fc0 2024-03-30T04:08:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f9812800700 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f980c017fd0 tx=0x7f980c009980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c425f1859d927ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f97ffa00700 1 -- 172.21.15.23:0/331823205 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c006000 con 0x55b6359c8fc0 2024-03-30T04:08:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f97ffa00700 1 -- 172.21.15.23:0/331823205 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f980c018e90 con 0x55b6359c8fc0 2024-03-30T04:08:31.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f97ffa00700 1 -- 172.21.15.23:0/331823205 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c008510 con 0x55b6359c8fc0 2024-03-30T04:08:31.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f982c81cb80 1 -- 172.21.15.23:0/331823205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 msgr2=0x55b6359d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.645+0000 7f982c81cb80 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f980c017fd0 tx=0x7f980c009980 comp rx=0 tx=0).stop 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 -- 172.21.15.23:0/331823205 shutdown_connections 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359c8fc0 0x55b6359d4100 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359bd7a0 0x55b6359cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 --2- 172.21.15.23:0/331823205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359be0c0 0x55b6359c84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 -- 172.21.15.23:0/331823205 >> 172.21.15.23:0/331823205 conn(0x55b63591d6a0 msgr2=0x55b6359c44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 -- 172.21.15.23:0/331823205 shutdown_connections 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 -- 172.21.15.23:0/331823205 wait complete. 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 -- start start 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.646+0000 7f982c81cb80 1 Processor -- start 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359bd7a0 0x55b6359cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359cdd00 con 0x55b6359be0c0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359cde70 con 0x55b6359c8fc0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6359ce030 con 0x55b6359bd7a0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f982c81cb80 10 client.0 _ll_register_callbacks cb 0x55b6358b5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9811400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:58998/0 (socket says 172.21.15.23:58998) 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 -- 172.21.15.23:0/2560190563 learned_addr learned my addr 172.21.15.23:0/2560190563 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9811e00700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359bd7a0 0x55b6359cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 -- 172.21.15.23:0/2560190563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359bd7a0 msgr2=0x55b6359cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359bd7a0 0x55b6359cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 -- 172.21.15.23:0/2560190563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 msgr2=0x55b6359cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.647+0000 7f9812800700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f9812800700 1 -- 172.21.15.23:0/2560190563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b63573db60 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f9812800700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f980c000f80 tx=0x7f980c0173b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c786730e79c10ad8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f97f3400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980c006000 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f980c017930 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f980409b100 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f980c009020 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f982c81cb80 10 client.27407 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.648+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b635795cd0 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f980409b100 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f980c02c070 con 0x55b6359c8fc0 2024-03-30T04:08:31.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 10 client.27407 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 10 client.27407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 20 client.27407 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b635a2d2a0 0x55b635a2f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 4 client.27407 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b63575ecf0 con 0x55b635a2d2a0 2024-03-30T04:08:31.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.649+0000 7f9811e00700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b635a2d2a0 0x55b635a2f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:31.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f9811e00700 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b635a2d2a0 0x55b635a2f690 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1e44c2a5efad983 server_cookie=f64b918b5d564399 in_seq=0 out_seq=0 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f97f3400700 10 client.27407 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f97f3400700 1 -- 172.21.15.23:0/2560190563 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b63575ecf0 con 0x55b635a2d2a0 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f97f3400700 10 client.27407 handle_command_reply: tid=0 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 client.27407 shutdown 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 msgr2=0x55b6359cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f980c000f80 tx=0x7f980c0173b0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 shutdown_connections 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b6359c8fc0 0x55b6359cd5f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b635a2d2a0 0x55b635a2f690 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b6359bd7a0 0x55b6359cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 --2- 172.21.15.23:0/2560190563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b6359be0c0 0x55b6359cf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f97f3400700 0 client.27407 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:31.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 >> 172.21.15.23:0/2560190563 conn(0x55b63591d6a0 msgr2=0x55b6359d4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 shutdown_connections 2024-03-30T04:08:31.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 1 -- 172.21.15.23:0/2560190563 wait complete. 2024-03-30T04:08:31.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.650+0000 7f982c81cb80 20 client.27407 trim_cache size 0 max 16384 2024-03-30T04:08:31.655 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 msgr2=0x7f97f80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 0x7f97f80a37d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f980c004090 tx=0x7f980c004450 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 msgr2=0x7f9814199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9804005740 tx=0x7f9804005770 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 shutdown_connections 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97f80a1310 0x7f97f80a37d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f981410d400 0x7f981419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9814073160 0x7f9814199260 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 --2- 172.21.15.23:0/1831389954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9814073a80 0x7f98141997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 >> 172.21.15.23:0/1831389954 conn(0x7f98140fc670 msgr2=0x7f98140fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 shutdown_connections 2024-03-30T04:08:31.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:31.651+0000 7f9813200700 1 -- 172.21.15.23:0/1831389954 wait complete. 2024-03-30T04:08:32.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 646491 11.20 MB/sec execute 189 sec latency 48.060 ms 2024-03-30T04:08:32.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:32 smithi154 ceph-mon[79161]: pgmap v202: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:08:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:32 smithi193 ceph-mon[79581]: pgmap v202: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:08:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:32 smithi023 ceph-mon[38254]: pgmap v202: 129 pgs: 129 active+clean; 382 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:08:32.672 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:08:32.672 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 -- 172.21.15.23:0/2426368100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1964100f60 msgr2=0x7f1964101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 --2- 172.21.15.23:0/2426368100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1964100f60 0x7f1964101340 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1954009990 tx=0x7f1954009ca0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 -- 172.21.15.23:0/2426368100 shutdown_connections 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 --2- 172.21.15.23:0/2426368100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964106080 0x7f196410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 --2- 172.21.15.23:0/2426368100 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1964100f60 0x7f1964101340 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 --2- 172.21.15.23:0/2426368100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964101910 0x7f1964105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 -- 172.21.15.23:0/2426368100 >> 172.21.15.23:0/2426368100 conn(0x7f19640fc7c0 msgr2=0x7f19640febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 -- 172.21.15.23:0/2426368100 shutdown_connections 2024-03-30T04:08:32.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.913+0000 7f196aa00700 1 -- 172.21.15.23:0/2426368100 wait complete. 2024-03-30T04:08:32.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.914+0000 7f196aa00700 1 Processor -- start 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.914+0000 7f196aa00700 1 -- start start 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.914+0000 7f196aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f196aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f196aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 0x7f196419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f196aa00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419c5f0 con 0x7f1964100f60 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f196aa00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419c760 con 0x7f196419bb70 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f196aa00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196419c920 con 0x7f1964106080 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 0x7f196419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43858/0 (socket says 172.21.15.23:43858) 2024-03-30T04:08:32.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963400700 1 -- 172.21.15.23:0/828447852 learned_addr learned my addr 172.21.15.23:0/828447852 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38724/0 (socket says 172.21.15.23:38724) 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 -- 172.21.15.23:0/828447852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 msgr2=0x7f19641977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 -- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 msgr2=0x7f1964197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1968a00700 1 -- 172.21.15.23:0/828447852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540095f0 con 0x7f196419bb70 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963400700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.915+0000 7f1963e00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f1968a00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 0x7f196419bff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f195c007f10 tx=0x7f195c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f1960c00700 1 -- 172.21.15.23:0/828447852 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c009ea0 con 0x7f196419bb70 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f1960c00700 1 -- 172.21.15.23:0/828447852 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f195c00f040 con 0x7f196419bb70 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f1960c00700 1 -- 172.21.15.23:0/828447852 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c004350 con 0x7f196419bb70 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1964073040 con 0x7f196419bb70 2024-03-30T04:08:32.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.916+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964073500 con 0x7f196419bb70 2024-03-30T04:08:32.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.919+0000 7f1960c00700 1 -- 172.21.15.23:0/828447852 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f195c00db20 con 0x7f196419bb70 2024-03-30T04:08:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.920+0000 7f1960c00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 0x7f19480a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.921+0000 7f1960c00700 1 -- 172.21.15.23:0/828447852 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f195c0cc980 con 0x7f196419bb70 2024-03-30T04:08:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.921+0000 7f1963e00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 0x7f19480a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.921+0000 7f197e59cb80 1 Processor -- start 2024-03-30T04:08:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.921+0000 7f197e59cb80 1 -- start start 2024-03-30T04:08:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72ef4f70 0x55ae72eff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 0x55ae72f02260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72e4da10 con 0x55ae72ef4f70 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72e4b3d0 con 0x55ae72effe70 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f197e59cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72e46cc0 con 0x55ae72ef4650 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 0x7f19480a3750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f195400b2e0 tx=0x7f195401e040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43870/0 (socket says 172.21.15.23:43870) 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 -- 172.21.15.23:0/2636704311 learned_addr learned my addr 172.21.15.23:0/2636704311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963400700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72ef4f70 0x55ae72eff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1968a00700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 0x55ae72f02260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 -- 172.21.15.23:0/2636704311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 msgr2=0x55ae72f02260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 0x55ae72f02260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 -- 172.21.15.23:0/2636704311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72ef4f70 msgr2=0x55ae72eff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72ef4f70 0x55ae72eff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1963e00700 1 -- 172.21.15.23:0/2636704311 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae72cc8430 con 0x55ae72ef4650 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.922+0000 7f1968a00700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 0x55ae72f02260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.923+0000 7f1963e00700 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1954015000 tx=0x7f1954024000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7092bfdda29b8a3e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.923+0000 7f1959200700 1 -- 172.21.15.23:0/2636704311 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954005040 con 0x55ae72ef4650 2024-03-30T04:08:32.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.923+0000 7f1959200700 1 -- 172.21.15.23:0/2636704311 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1954022040 con 0x55ae72ef4650 2024-03-30T04:08:32.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.923+0000 7f1959200700 1 -- 172.21.15.23:0/2636704311 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954004610 con 0x55ae72ef4650 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 -- 172.21.15.23:0/2636704311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 msgr2=0x55ae72ef4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1954015000 tx=0x7f1954024000 comp rx=0 tx=0).stop 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 -- 172.21.15.23:0/2636704311 shutdown_connections 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72effe70 0x55ae72f02260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72ef4a30 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 --2- 172.21.15.23:0/2636704311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72ef4f70 0x55ae72eff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 -- 172.21.15.23:0/2636704311 >> 172.21.15.23:0/2636704311 conn(0x55ae72e54720 msgr2=0x55ae72ef0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 -- 172.21.15.23:0/2636704311 shutdown_connections 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.924+0000 7f197e59cb80 1 -- 172.21.15.23:0/2636704311 wait complete. 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.925+0000 7f197e59cb80 1 -- start start 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.925+0000 7f197e59cb80 1 Processor -- start 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72efcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72efd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72de2e00 con 0x55ae72effe70 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72de2aa0 con 0x55ae72ef4f70 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72cca720 con 0x55ae72ef4650 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f197e59cb80 10 client.0 _ll_register_callbacks cb 0x55ae72dec4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43886/0 (socket says 172.21.15.23:43886) 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 -- 172.21.15.23:0/1911550093 learned_addr learned my addr 172.21.15.23:0/1911550093 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:32.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1968a00700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72efd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963400700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72efcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 -- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 msgr2=0x55ae72efcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72efcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 -- 172.21.15.23:0/1911550093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 msgr2=0x55ae72efd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72efd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.926+0000 7f1963e00700 1 -- 172.21.15.23:0/1911550093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964073500 con 0x55ae72ef4650 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f1963e00700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1954009350 tx=0x7f195400be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f0c843c614b1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f194d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:08:32.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954005450 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f19540055b0 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae72e07400 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1954004610 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f197e59cb80 10 client.27412 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.927+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f196410e470 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ae72e07400 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f195400f660 con 0x55ae72ef4650 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 10 client.27412 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 10 client.27412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 20 client.27412 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f0f4f0 0x55ae72f11910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 4 client.27412 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae72e3f780 con 0x55ae72f0f4f0 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.928+0000 7f1963400700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f0f4f0 0x55ae72f11910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.929+0000 7f1963400700 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f0f4f0 0x55ae72f11910 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98a6bdc0734ec345 server_cookie=b49ce9b300f4ed32 in_seq=0 out_seq=0 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.929+0000 7f194d600700 10 client.27412 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f194d600700 1 -- 172.21.15.23:0/1911550093 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ae72e3f780 con 0x55ae72f0f4f0 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f194d600700 10 client.27412 handle_command_reply: tid=0 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 client.27412 shutdown 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 msgr2=0x55ae72eff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1954009350 tx=0x7f195400be60 comp rx=0 tx=0).stop 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 shutdown_connections 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72efd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f0f4f0 0x55ae72f11910 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72eff7c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1911550093 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72efcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f194d600700 0 client.27412 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 >> 172.21.15.23:0/1911550093 conn(0x55ae72e54720 msgr2=0x55ae72ef6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 shutdown_connections 2024-03-30T04:08:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 1 -- 172.21.15.23:0/1911550093 wait complete. 2024-03-30T04:08:32.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.930+0000 7f197e59cb80 20 client.27412 trim_cache size 0 max 16384 2024-03-30T04:08:32.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 Processor -- start 2024-03-30T04:08:32.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 -- start start 2024-03-30T04:08:32.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72ef05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f0b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72e07400 con 0x55ae72effe70 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72ef18c0 con 0x55ae72ef4f70 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.950+0000 7f197e59cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72effb70 con 0x55ae72ef4650 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43892/0 (socket says 172.21.15.23:43892) 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 -- 172.21.15.23:0/718840562 learned_addr learned my addr 172.21.15.23:0/718840562 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1968a00700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f0b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963400700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72ef05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 -- 172.21.15.23:0/718840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 msgr2=0x55ae72ef05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72ef05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 -- 172.21.15.23:0/718840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 msgr2=0x55ae72f0b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f0b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 -- 172.21.15.23:0/718840562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae72c74be0 con 0x55ae72ef4650 2024-03-30T04:08:32.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963400700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72ef05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.951+0000 7f1963e00700 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f195400bc00 tx=0x7f195400bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30ab238824a79288 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.952+0000 7f1959c00700 1 -- 172.21.15.23:0/718840562 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400f460 con 0x55ae72ef4650 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.952+0000 7f1959c00700 1 -- 172.21.15.23:0/718840562 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1954005710 con 0x55ae72ef4650 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.952+0000 7f1959c00700 1 -- 172.21.15.23:0/718840562 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954004cd0 con 0x55ae72ef4650 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- 172.21.15.23:0/718840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 msgr2=0x55ae72f06a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f195400bc00 tx=0x7f195400bed0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- 172.21.15.23:0/718840562 shutdown_connections 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f0b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4650 0x55ae72f06a90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 --2- 172.21.15.23:0/718840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4f70 0x55ae72ef05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- 172.21.15.23:0/718840562 >> 172.21.15.23:0/718840562 conn(0x55ae72e54720 msgr2=0x55ae72eff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- 172.21.15.23:0/718840562 shutdown_connections 2024-03-30T04:08:32.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- 172.21.15.23:0/718840562 wait complete. 2024-03-30T04:08:32.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.953+0000 7f197e59cb80 1 -- start start 2024-03-30T04:08:32.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 Processor -- start 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 0x55ae72f05b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72f5d800 con 0x55ae72effe70 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72f5d970 con 0x55ae72ef4650 2024-03-30T04:08:32.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae72f5db30 con 0x55ae72ef4f70 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f197e59cb80 10 client.0 _ll_register_callbacks cb 0x55ae72dec4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1968a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1963400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1963400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43906/0 (socket says 172.21.15.23:43906) 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1963e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 0x55ae72f05b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1968a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38754/0 (socket says 172.21.15.23:38754) 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.954+0000 7f1963400700 1 -- 172.21.15.23:0/1982635999 learned_addr learned my addr 172.21.15.23:0/1982635999 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963400700 1 -- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 msgr2=0x55ae72f05b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963400700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 0x55ae72f05b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963400700 1 -- 172.21.15.23:0/1982635999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 msgr2=0x55ae72f5d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963400700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963400700 1 -- 172.21.15.23:0/1982635999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196410e470 con 0x55ae72ef4f70 2024-03-30T04:08:32.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1968a00700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.955+0000 7f1963e00700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 0x55ae72f05b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.956+0000 7f1963400700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1950009690 tx=0x7f1950013aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3e1cf582c6350da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:32.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.956+0000 7f194d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.956+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950017040 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.956+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1954022040 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.957+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1950021c00 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.957+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1950017040 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.958+0000 7f197e59cb80 10 client.27418 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.958+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ae72e0cf20 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.958+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1954022040 con 0x55ae72ef4f70 2024-03-30T04:08:32.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.958+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1950027030 con 0x55ae72ef4f70 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 10 client.27418 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 10 client.27418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 20 client.27418 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f654c0 0x55ae72f678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 4 client.27418 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55ae72c95d70 con 0x55ae72f654c0 2024-03-30T04:08:32.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.959+0000 7f1963e00700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f654c0 0x55ae72f678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.960+0000 7f1963e00700 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f654c0 0x55ae72f678e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dd4780058d4587d server_cookie=1b01031852e0cbef in_seq=0 out_seq=0 2024-03-30T04:08:32.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.961+0000 7f194d600700 10 client.27418 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.962+0000 7f194d600700 1 -- 172.21.15.23:0/1982635999 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ae72c95d70 con 0x55ae72f654c0 2024-03-30T04:08:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.962+0000 7f194d600700 10 client.27418 handle_command_reply: tid=0 2024-03-30T04:08:32.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.962+0000 7f197e59cb80 1 client.27418 shutdown 2024-03-30T04:08:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 msgr2=0x55ae72f06080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1950009690 tx=0x7f1950013aa0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 shutdown_connections 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ae72effe70 0x55ae72f5d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ae72ef4650 0x55ae72f05b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55ae72f654c0 0x55ae72f678e0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 --2- 172.21.15.23:0/1982635999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ae72ef4f70 0x55ae72f06080 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 >> 172.21.15.23:0/1982635999 conn(0x55ae72e54720 msgr2=0x55ae72ef0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f194d600700 0 client.27418 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 shutdown_connections 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.963+0000 7f197e59cb80 1 -- 172.21.15.23:0/1982635999 wait complete. 2024-03-30T04:08:32.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.964+0000 7f197e59cb80 20 client.27418 trim_cache size 0 max 16384 2024-03-30T04:08:32.964 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:08:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:08:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "d7baf556-6965-4f89-b936-8fcd5b389a8c", 2024-03-30T04:08:32.964 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:08:32.964 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 msgr2=0x7f19480a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 0x7f19480a3750 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f195400b2e0 tx=0x7f195401e040 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 msgr2=0x7f196419bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 0x7f196419bff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f195c007f10 tx=0x7f195c00d3b0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 shutdown_connections 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f19480a1290 0x7f19480a3750 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f196419bb70 0x7f196419bff0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1964100f60 0x7f1964197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 --2- 172.21.15.23:0/828447852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1964106080 0x7f19641977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:32.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 >> 172.21.15.23:0/828447852 conn(0x7f19640fc7c0 msgr2=0x7f19640fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.965+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 shutdown_connections 2024-03-30T04:08:32.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:32.966+0000 7f196aa00700 1 -- 172.21.15.23:0/828447852 wait complete. 2024-03-30T04:08:32.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:08:33.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 647880 11.17 MB/sec execute 190 sec latency 24.944 ms 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 -- 172.21.15.23:0/2402644128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94103170 msgr2=0x7f5e94103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 --2- 172.21.15.23:0/2402644128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94103170 0x7f5e94103550 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5e840098f0 tx=0x7f5e84009c00 comp rx=0 tx=0).stop 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 -- 172.21.15.23:0/2402644128 shutdown_connections 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 --2- 172.21.15.23:0/2402644128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94108290 0x7f5e9410fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 --2- 172.21.15.23:0/2402644128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94103170 0x7f5e94103550 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 --2- 172.21.15.23:0/2402644128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e94103b20 0x7f5e94107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 -- 172.21.15.23:0/2402644128 >> 172.21.15.23:0/2402644128 conn(0x7f5e940fe990 msgr2=0x7f5e94100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 -- 172.21.15.23:0/2402644128 shutdown_connections 2024-03-30T04:08:33.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.226+0000 7f5e99400700 1 -- 172.21.15.23:0/2402644128 wait complete. 2024-03-30T04:08:33.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.227+0000 7f5e99400700 1 Processor -- start 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.227+0000 7f5e99400700 1 -- start start 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 0x7f5e94199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e9419dd90 0x7f5e9419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9419e810 con 0x7f5e94103170 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9419e980 con 0x7f5e9419dd90 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e99400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9419eb40 con 0x7f5e94108290 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e92800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 0x7f5e94199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43928/0 (socket says 172.21.15.23:43928) 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 -- 172.21.15.23:0/4131356373 learned_addr learned my addr 172.21.15.23:0/4131356373 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e93200700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e9419dd90 0x7f5e9419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 -- 172.21.15.23:0/4131356373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e9419dd90 msgr2=0x7f5e9419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e9419dd90 0x7f5e9419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 -- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 msgr2=0x7f5e94199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 0x7f5e94199500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.228+0000 7f5e91e00700 1 -- 172.21.15.23:0/4131356373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e84009520 con 0x7f5e94108290 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e92800700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 0x7f5e94199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e91e00700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000e9f0 tx=0x7f5e8000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e8b400700 1 -- 172.21.15.23:0/4131356373 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80009690 con 0x7f5e94108290 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e8b400700 1 -- 172.21.15.23:0/4131356373 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e8000ce80 con 0x7f5e94108290 2024-03-30T04:08:33.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e8b400700 1 -- 172.21.15.23:0/4131356373 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80018420 con 0x7f5e94108290 2024-03-30T04:08:33.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e941a60a0 con 0x7f5e94108290 2024-03-30T04:08:33.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.229+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e941a6580 con 0x7f5e94108290 2024-03-30T04:08:33.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.234+0000 7f5e8b400700 1 -- 172.21.15.23:0/4131356373 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f5e8000c9c0 con 0x7f5e94108290 2024-03-30T04:08:33.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 Processor -- start 2024-03-30T04:08:33.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 -- start start 2024-03-30T04:08:33.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 0x5583f35149b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f351fdf0 0x5583f35221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f346d990 con 0x5583f35145d0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f346b350 con 0x5583f351fdf0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.235+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3466c40 con 0x5583f3514ef0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43930/0 (socket says 172.21.15.23:43930) 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e93200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 0x5583f35149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 -- 172.21.15.23:0/3143756065 learned_addr learned my addr 172.21.15.23:0/3143756065 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e8b400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 0x7f5e780a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e91e00700 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f351fdf0 0x5583f35221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 -- 172.21.15.23:0/3143756065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f351fdf0 msgr2=0x5583f35221e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f351fdf0 0x5583f35221e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 -- 172.21.15.23:0/3143756065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 msgr2=0x5583f35149b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 0x5583f35149b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e8b400700 1 -- 172.21.15.23:0/4131356373 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e80014070 con 0x7f5e94108290 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 -- 172.21.15.23:0/3143756065 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583f32e83b0 con 0x5583f3514ef0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e93200700 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 0x5583f35149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e93200700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 0x7f5e780a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.236+0000 7f5e92800700 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009230 tx=0x7f5e84005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c53042fb323d284f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5e7f400700 1 -- 172.21.15.23:0/3143756065 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8401c050 con 0x5583f3514ef0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5e7f400700 1 -- 172.21.15.23:0/3143756065 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e8401d040 con 0x5583f3514ef0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5e7f400700 1 -- 172.21.15.23:0/3143756065 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e840052a0 con 0x5583f3514ef0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 -- 172.21.15.23:0/3143756065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 msgr2=0x5583f351f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5e93200700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 0x7f5e780a37d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009ec0 tx=0x7f5e8c009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009230 tx=0x7f5e84005dc0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 -- 172.21.15.23:0/3143756065 shutdown_connections 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f351fdf0 0x5583f35221e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f35145d0 0x5583f35149b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/3143756065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f8b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 -- 172.21.15.23:0/3143756065 >> 172.21.15.23:0/3143756065 conn(0x5583f34746a0 msgr2=0x5583f3510a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.237+0000 7f5eacf09b80 1 -- 172.21.15.23:0/3143756065 shutdown_connections 2024-03-30T04:08:33.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.238+0000 7f5eacf09b80 1 -- 172.21.15.23:0/3143756065 wait complete. 2024-03-30T04:08:33.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.238+0000 7f5eacf09b80 1 -- start start 2024-03-30T04:08:33.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.238+0000 7f5eacf09b80 1 Processor -- start 2024-03-30T04:08:33.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f3514ef0 0x5583f351cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f351fdf0 0x5583f351d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3402d80 con 0x5583f3514ef0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3402a20 con 0x5583f35145d0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f32ea6a0 con 0x5583f351fdf0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5eacf09b80 10 client.0 _ll_register_callbacks cb 0x5583f340c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:59068/0 (socket says 172.21.15.23:59068) 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 -- 172.21.15.23:0/707920039 learned_addr learned my addr 172.21.15.23:0/707920039 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f3514ef0 0x5583f351cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e93200700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f351fdf0 0x5583f351d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 -- 172.21.15.23:0/707920039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f351fdf0 msgr2=0x5583f351d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f351fdf0 0x5583f351d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 -- 172.21.15.23:0/707920039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f3514ef0 msgr2=0x5583f351cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.239+0000 7f5e92800700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f3514ef0 0x5583f351cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e92800700 1 -- 172.21.15.23:0/707920039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e94110680 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e92800700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5e84006010 tx=0x7f5e8400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3ef75f0ec94de5d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e73e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e840058e0 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e840050c0 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5583f3427380 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e8401c020 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5eacf09b80 10 client.27437 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.240+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5583f3294b60 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5583f3427380 con 0x5583f35145d0 2024-03-30T04:08:33.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e84009c70 con 0x5583f35145d0 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 10 client.27437 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 10 client.27437 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 20 client.27437 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f352f480 0x5583f3531870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 4 client.27437 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5583f345f700 con 0x5583f352f480 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.241+0000 7f5e91e00700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f352f480 0x5583f3531870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.242+0000 7f5e91e00700 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f352f480 0x5583f3531870 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970319c13d15dce1 server_cookie=7d36ee69cde2f153 in_seq=0 out_seq=0 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.242+0000 7f5e73e00700 10 client.27437 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5e73e00700 1 -- 172.21.15.23:0/707920039 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5583f345f700 con 0x5583f352f480 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5e73e00700 10 client.27437 handle_command_reply: tid=0 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 client.27437 shutdown 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 msgr2=0x5583f351f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5e84006010 tx=0x7f5e8400f790 comp rx=0 tx=0).stop 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 shutdown_connections 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f352f480 0x5583f3531870 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f351fdf0 0x5583f351d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f820 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/707920039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f3514ef0 0x5583f351cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5e73e00700 0 client.27437 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 >> 172.21.15.23:0/707920039 conn(0x5583f34746a0 msgr2=0x5583f35166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 shutdown_connections 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 1 -- 172.21.15.23:0/707920039 wait complete. 2024-03-30T04:08:33.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.243+0000 7f5eacf09b80 20 client.27437 trim_cache size 0 max 16384 2024-03-30T04:08:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 Processor -- start 2024-03-30T04:08:33.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 -- start start 2024-03-30T04:08:33.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f351c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f35179d0 con 0x5583f351fdf0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3511660 con 0x5583f35145d0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.264+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f352bdb0 con 0x5583f3514ef0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:59084/0 (socket says 172.21.15.23:59084) 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 -- 172.21.15.23:0/813657677 learned_addr learned my addr 172.21.15.23:0/813657677 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e91e00700 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e93200700 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f351c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 -- 172.21.15.23:0/813657677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 msgr2=0x5583f351f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 -- 172.21.15.23:0/813657677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 msgr2=0x5583f351c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f351c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 -- 172.21.15.23:0/813657677 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5583f342cea0 con 0x5583f35145d0 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e92800700 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5e84001eb0 tx=0x7f5e84009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2326853e5f51e546 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.265+0000 7f5e7fe00700 1 -- 172.21.15.23:0/813657677 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8401c070 con 0x5583f35145d0 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5e7fe00700 1 -- 172.21.15.23:0/813657677 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e840058e0 con 0x5583f35145d0 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5e7fe00700 1 -- 172.21.15.23:0/813657677 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84005500 con 0x5583f35145d0 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 -- 172.21.15.23:0/813657677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 msgr2=0x5583f351f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5e84001eb0 tx=0x7f5e84009440 comp rx=0 tx=0).stop 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 -- 172.21.15.23:0/813657677 shutdown_connections 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f351c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f351f350 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/813657677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f351f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 -- 172.21.15.23:0/813657677 >> 172.21.15.23:0/813657677 conn(0x5583f34746a0 msgr2=0x5583f3509f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 -- 172.21.15.23:0/813657677 shutdown_connections 2024-03-30T04:08:33.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.266+0000 7f5eacf09b80 1 -- 172.21.15.23:0/813657677 wait complete. 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.267+0000 7f5eacf09b80 1 -- start start 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.267+0000 7f5eacf09b80 1 Processor -- start 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f3524c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f35251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3525710 con 0x5583f351fdf0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3525880 con 0x5583f35145d0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583f3525a40 con 0x5583f3514ef0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5eacf09b80 10 client.0 _ll_register_callbacks cb 0x5583f340c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5e91e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f3524c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5e92800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:59090/0 (socket says 172.21.15.23:59090) 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5e92800700 1 -- 172.21.15.23:0/1447380698 learned_addr learned my addr 172.21.15.23:0/1447380698 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:08:33.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.268+0000 7f5e93200700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f35251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 -- 172.21.15.23:0/1447380698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 msgr2=0x5583f3524c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f3524c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 -- 172.21.15.23:0/1447380698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 msgr2=0x5583f35251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f35251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 -- 172.21.15.23:0/1447380698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5583f3294b60 con 0x5583f35145d0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e91e00700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f3524c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e92800700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009810 tx=0x7f5e84005fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebc0e39fda72f50 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e73e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8401c040 con 0x5583f35145d0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e84000e40 con 0x5583f35145d0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e8401d040 con 0x5583f35145d0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e84005160 con 0x5583f35145d0 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5eacf09b80 10 client.27443 fetch_fsmap learned FSMap version 30 2024-03-30T04:08:33.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.269+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5583f32eccd0 con 0x5583f35145d0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e8401d040 con 0x5583f35145d0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e8404a020 con 0x5583f35145d0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 10 client.27443 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 10 client.27443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 20 client.27443 populate_metadata read hostname 'smithi023' 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f3584290 0x5583f35866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 4 client.27443 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.270+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5583f32b5cf0 con 0x5583f3584290 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5e91e00700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f3584290 0x5583f35866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5e91e00700 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f3584290 0x5583f35866b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6fc1d6047eb5639 server_cookie=d79bdd941574366e in_seq=0 out_seq=0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5e73e00700 10 client.27443 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5e73e00700 1 -- 172.21.15.23:0/1447380698 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5583f32b5cf0 con 0x5583f3584290 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5e73e00700 10 client.27443 handle_command_reply: tid=0 2024-03-30T04:08:33.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5eacf09b80 1 client.27443 shutdown 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 msgr2=0x5583f35267a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f5e84009810 tx=0x7f5e84005fc0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.271+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 shutdown_connections 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5583f351fdf0 0x5583f35251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5583f35145d0 0x5583f35267a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5583f3584290 0x5583f35866b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 --2- 172.21.15.23:0/1447380698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5583f3514ef0 0x5583f3524c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5e73e00700 0 client.27443 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 >> 172.21.15.23:0/1447380698 conn(0x5583f34746a0 msgr2=0x5583f35164d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 shutdown_connections 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 1 -- 172.21.15.23:0/1447380698 wait complete. 2024-03-30T04:08:33.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.272+0000 7f5eacf09b80 20 client.27443 trim_cache size 0 max 16384 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "d7baf556-6965-4f89-b936-8fcd5b389a8c": { 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "d7baf556-6965-4f89-b936-8fcd5b389a8c", 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:08:33.280 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 msgr2=0x7f5e780a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 0x7f5e780a37d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009ec0 tx=0x7f5e8c009040 comp rx=0 tx=0).stop 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 msgr2=0x7f5e94199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000e9f0 tx=0x7f5e8000ed00 comp rx=0 tx=0).stop 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 shutdown_connections 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e780a1310 0x7f5e780a37d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e9419dd90 0x7f5e9419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e94103170 0x7f5e94199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 --2- 172.21.15.23:0/4131356373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e94108290 0x7f5e94199a40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:08:33.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 >> 172.21.15.23:0/4131356373 conn(0x7f5e940fe990 msgr2=0x7f5e940ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:08:33.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 shutdown_connections 2024-03-30T04:08:33.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:08:33.273+0000 7f5e99400700 1 -- 172.21.15.23:0/4131356373 wait complete. 2024-03-30T04:08:33.291 INFO:tasks.cephfs.filesystem:scrub status for tag:d7baf556-6965-4f89-b936-8fcd5b389a8c - {'path': '/', 'tag': 'd7baf556-6965-4f89-b936-8fcd5b389a8c', 'options': 'recursive,force'} 2024-03-30T04:08:34.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 650143 11.18 MB/sec execute 191 sec latency 17.564 ms 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: pgmap v203: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:34 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: pgmap v203: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:08:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:08:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:08:34.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:08:34.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:34 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: pgmap v203: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:08:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:08:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:08:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:08:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:34.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:34 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:08:35.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 651583 11.16 MB/sec execute 192 sec latency 27.235 ms 2024-03-30T04:08:36.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 653829 11.17 MB/sec execute 193 sec latency 19.545 ms 2024-03-30T04:08:36.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:36 smithi154 ceph-mon[79161]: pgmap v204: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:36 smithi193 ceph-mon[79581]: pgmap v204: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:36 smithi023 ceph-mon[38254]: pgmap v204: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:37.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 655049 11.14 MB/sec execute 194 sec latency 49.733 ms 2024-03-30T04:08:38.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 657310 11.15 MB/sec execute 195 sec latency 19.152 ms 2024-03-30T04:08:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:38 smithi154 ceph-mon[79161]: pgmap v205: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:38 smithi193 ceph-mon[79581]: pgmap v205: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:38 smithi023 ceph-mon[38254]: pgmap v205: 129 pgs: 129 active+clean; 384 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:39.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 658809 11.12 MB/sec execute 196 sec latency 31.149 ms 2024-03-30T04:08:39.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:39 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:08:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:39 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:08:39.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:39 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:08:39.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:39 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:08:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:39 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:08:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:39 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:08:40.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 660946 11.14 MB/sec execute 197 sec latency 18.648 ms 2024-03-30T04:08:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:40 smithi154 ceph-mon[79161]: pgmap v206: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-30T04:08:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:40 smithi193 ceph-mon[79581]: pgmap v206: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-30T04:08:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:40 smithi023 ceph-mon[38254]: pgmap v206: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 254 op/s 2024-03-30T04:08:41.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 662556 11.11 MB/sec execute 198 sec latency 35.962 ms 2024-03-30T04:08:42.207 INFO:tasks.workunit.client.0.smithi023.stdout: 1 664412 11.12 MB/sec execute 199 sec latency 85.864 ms 2024-03-30T04:08:42.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:42 smithi154 ceph-mon[79161]: pgmap v207: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:08:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:42 smithi193 ceph-mon[79581]: pgmap v207: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:08:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:42 smithi023 ceph-mon[38254]: pgmap v207: 129 pgs: 129 active+clean; 392 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:08:43.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 665849 11.09 MB/sec execute 200 sec latency 31.686 ms 2024-03-30T04:08:44.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 668001 11.10 MB/sec execute 201 sec latency 40.838 ms 2024-03-30T04:08:44.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:44 smithi154 ceph-mon[79161]: pgmap v208: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-30T04:08:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:44 smithi193 ceph-mon[79581]: pgmap v208: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-30T04:08:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:44 smithi023 ceph-mon[38254]: pgmap v208: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 252 op/s 2024-03-30T04:08:45.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 669602 11.08 MB/sec execute 202 sec latency 27.998 ms 2024-03-30T04:08:46.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 671742 11.09 MB/sec execute 203 sec latency 15.876 ms 2024-03-30T04:08:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:46 smithi154 ceph-mon[79161]: pgmap v209: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:46 smithi193 ceph-mon[79581]: pgmap v209: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:46 smithi023 ceph-mon[38254]: pgmap v209: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:47.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 673025 11.06 MB/sec execute 204 sec latency 50.616 ms 2024-03-30T04:08:48.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 675245 11.08 MB/sec execute 205 sec latency 28.471 ms 2024-03-30T04:08:48.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:48 smithi154 ceph-mon[79161]: pgmap v210: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:48 smithi193 ceph-mon[79581]: pgmap v210: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:48 smithi023 ceph-mon[38254]: pgmap v210: 129 pgs: 129 active+clean; 389 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 171 op/s 2024-03-30T04:08:49.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 676966 11.05 MB/sec execute 206 sec latency 37.376 ms 2024-03-30T04:08:50.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 678912 11.06 MB/sec execute 207 sec latency 18.810 ms 2024-03-30T04:08:50.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:50 smithi154 ceph-mon[79161]: pgmap v211: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 251 op/s 2024-03-30T04:08:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:50 smithi193 ceph-mon[79581]: pgmap v211: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 251 op/s 2024-03-30T04:08:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:50 smithi023 ceph-mon[38254]: pgmap v211: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 251 op/s 2024-03-30T04:08:51.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 680750 11.04 MB/sec execute 208 sec latency 38.598 ms 2024-03-30T04:08:52.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 682418 11.05 MB/sec execute 209 sec latency 60.130 ms 2024-03-30T04:08:52.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:52 smithi154 ceph-mon[79161]: pgmap v212: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-03-30T04:08:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:52 smithi193 ceph-mon[79581]: pgmap v212: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-03-30T04:08:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:52 smithi023 ceph-mon[38254]: pgmap v212: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 168 op/s 2024-03-30T04:08:53.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 684275 11.03 MB/sec execute 210 sec latency 21.363 ms 2024-03-30T04:08:54.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 686110 11.03 MB/sec execute 211 sec latency 21.412 ms 2024-03-30T04:08:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:54 smithi154 ceph-mon[79161]: pgmap v213: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-03-30T04:08:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:54 smithi193 ceph-mon[79581]: pgmap v213: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-03-30T04:08:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:54 smithi023 ceph-mon[38254]: pgmap v213: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 254 op/s 2024-03-30T04:08:55.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 688007 11.02 MB/sec execute 212 sec latency 19.543 ms 2024-03-30T04:08:56.208 INFO:tasks.workunit.client.0.smithi023.stdout: 1 689671 11.02 MB/sec execute 213 sec latency 48.785 ms 2024-03-30T04:08:56.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:56 smithi154 ceph-mon[79161]: pgmap v214: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:56 smithi193 ceph-mon[79581]: pgmap v214: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:56 smithi023 ceph-mon[38254]: pgmap v214: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:57.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 691040 10.99 MB/sec execute 214 sec latency 133.143 ms 2024-03-30T04:08:58.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 693056 11.00 MB/sec execute 215 sec latency 18.115 ms 2024-03-30T04:08:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:08:58 smithi154 ceph-mon[79161]: pgmap v215: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:08:58 smithi193 ceph-mon[79581]: pgmap v215: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:08:58 smithi023 ceph-mon[38254]: pgmap v215: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-03-30T04:08:59.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 694916 10.99 MB/sec execute 216 sec latency 28.191 ms 2024-03-30T04:09:00.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 696751 10.99 MB/sec execute 217 sec latency 28.821 ms 2024-03-30T04:09:00.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:00 smithi154 ceph-mon[79161]: pgmap v216: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-30T04:09:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:00 smithi193 ceph-mon[79581]: pgmap v216: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-30T04:09:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:00 smithi023 ceph-mon[38254]: pgmap v216: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 249 op/s 2024-03-30T04:09:01.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 697074 10.95 MB/sec execute 218 sec latency 77.230 ms 2024-03-30T04:09:02.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 698375 10.92 MB/sec execute 219 sec latency 72.493 ms 2024-03-30T04:09:02.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:02 smithi154 ceph-mon[79161]: pgmap v217: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:09:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:02 smithi193 ceph-mon[79581]: pgmap v217: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:09:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:02 smithi023 ceph-mon[38254]: pgmap v217: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-03-30T04:09:03.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 700140 10.93 MB/sec execute 220 sec latency 20.115 ms 2024-03-30T04:09:03.292 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:09:03.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.512+0000 7f74fba00700 1 -- 172.21.15.23:0/4204103479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4067e30 msgr2=0x7f74f40682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.512+0000 7f74fba00700 1 --2- 172.21.15.23:0/4204103479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4067e30 0x7f74f40682b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f74e00099c0 tx=0x7f74e0009cd0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 -- 172.21.15.23:0/4204103479 shutdown_connections 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 --2- 172.21.15.23:0/4204103479 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 0x7f74f410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 --2- 172.21.15.23:0/4204103479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4105ce0 0x7f74f41060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 --2- 172.21.15.23:0/4204103479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4067e30 0x7f74f40682b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 -- 172.21.15.23:0/4204103479 >> 172.21.15.23:0/4204103479 conn(0x7f74f4076d40 msgr2=0x7f74f4077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 -- 172.21.15.23:0/4204103479 shutdown_connections 2024-03-30T04:09:03.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.513+0000 7f74fba00700 1 -- 172.21.15.23:0/4204103479 wait complete. 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 Processor -- start 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 -- start start 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4067e30 0x7f74f4199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 0x7f74f4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f419e360 con 0x7f74f40687f0 2024-03-30T04:09:03.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f419e4d0 con 0x7f74f4105ce0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.514+0000 7f74fba00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f419e690 con 0x7f74f4067e30 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 0x7f74f4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56450/0 (socket says 172.21.15.23:56450) 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 -- 172.21.15.23:0/3798112218 learned_addr learned my addr 172.21.15.23:0/3798112218 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9000700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4067e30 0x7f74f4199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 -- 172.21.15.23:0/3798112218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4067e30 msgr2=0x7f74f4199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4067e30 0x7f74f4199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 -- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 msgr2=0x7f74f4199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 0x7f74f4199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 -- 172.21.15.23:0/3798112218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8009520 con 0x7f74f4105ce0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.515+0000 7f74f9a00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00ef30 tx=0x7f74ec00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.516+0000 7f74f1600700 1 -- 172.21.15.23:0/3798112218 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec018070 con 0x7f74f4105ce0 2024-03-30T04:09:03.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.516+0000 7f74f1600700 1 -- 172.21.15.23:0/3798112218 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74ec00f040 con 0x7f74f4105ce0 2024-03-30T04:09:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.516+0000 7f74f1600700 1 -- 172.21.15.23:0/3798112218 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec00cd00 con 0x7f74f4105ce0 2024-03-30T04:09:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.516+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e00095f0 con 0x7f74f4105ce0 2024-03-30T04:09:03.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.516+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f41a5fd0 con 0x7f74f4105ce0 2024-03-30T04:09:03.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.519+0000 7f74f1600700 1 -- 172.21.15.23:0/3798112218 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f74ec003a00 con 0x7f74f4105ce0 2024-03-30T04:09:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.521+0000 7f750f453b80 1 Processor -- start 2024-03-30T04:09:03.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.521+0000 7f750f453b80 1 -- start start 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e4928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e4951e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f1600700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 0x7f74dc0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e3e0990 con 0x55872e4875d0 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e3de350 con 0x55872e487ef0 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f750f453b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e3d9c40 con 0x55872e492df0 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f1600700 1 -- 172.21.15.23:0/3798112218 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74ec0ccbb0 con 0x7f74f4105ce0 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47348/0 (socket says 172.21.15.23:47348) 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f3e00700 1 -- 172.21.15.23:0/1897945302 learned_addr learned my addr 172.21.15.23:0/1897945302 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.522+0000 7f74f3e00700 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e4951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f9000700 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e4928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f9a00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 0x7f74dc0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 -- 172.21.15.23:0/1897945302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 msgr2=0x55872e4951e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e4951e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 -- 172.21.15.23:0/1897945302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 msgr2=0x55872e4928b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e4928b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 -- 172.21.15.23:0/1897945302 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55872e25b3b0 con 0x55872e4875d0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.523+0000 7f74f3e00700 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f74e00058e0 tx=0x7f74e001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad0d195fb9fb7e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f74f9a00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 0x7f74dc0a37d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0c9f20 tx=0x7f74ec009ed0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f74e6000700 1 -- 172.21.15.23:0/1897945302 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0005540 con 0x55872e4875d0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f74e6000700 1 -- 172.21.15.23:0/1897945302 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74e00056a0 con 0x55872e4875d0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f74e6000700 1 -- 172.21.15.23:0/1897945302 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0003b40 con 0x55872e4875d0 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f750f453b80 1 -- 172.21.15.23:0/1897945302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 msgr2=0x55872e4879b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.524+0000 7f750f453b80 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f74e00058e0 tx=0x7f74e001a040 comp rx=0 tx=0).stop 2024-03-30T04:09:03.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 -- 172.21.15.23:0/1897945302 shutdown_connections 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e4951e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4879b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 --2- 172.21.15.23:0/1897945302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e4928b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 -- 172.21.15.23:0/1897945302 >> 172.21.15.23:0/1897945302 conn(0x55872e3e76a0 msgr2=0x55872e483a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 -- 172.21.15.23:0/1897945302 shutdown_connections 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 -- 172.21.15.23:0/1897945302 wait complete. 2024-03-30T04:09:03.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.525+0000 7f750f453b80 1 -- start start 2024-03-30T04:09:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 Processor -- start 2024-03-30T04:09:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e492820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e4902a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e375d80 con 0x55872e4875d0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e375a20 con 0x55872e492df0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e25d6a0 con 0x55872e487ef0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f750f453b80 10 client.0 _ll_register_callbacks cb 0x55872e37f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f74f9000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f74f9000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44766/0 (socket says 172.21.15.23:44766) 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.526+0000 7f74f9000700 1 -- 172.21.15.23:0/197954644 learned_addr learned my addr 172.21.15.23:0/197954644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f3e00700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e492820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f3e00700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e4902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 -- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 msgr2=0x55872e4902a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e4902a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 -- 172.21.15.23:0/197954644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 msgr2=0x55872e492820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e492820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 -- 172.21.15.23:0/197954644 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e0020040 con 0x55872e487ef0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74f9000700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74e8012810 tx=0x7f74e8012b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5218fa4685d5da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.527+0000 7f74daa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e8017070 con 0x55872e487ef0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74e8004830 con 0x55872e487ef0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55872e39a380 con 0x55872e487ef0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74e80122e0 con 0x55872e487ef0 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 10 client.27448 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:03.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55872e207b60 con 0x55872e487ef0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55872e39a380 con 0x55872e487ef0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f74e801c770 con 0x55872e487ef0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 10 client.27448 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 10 client.27448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 20 client.27448 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4a29a0 0x55872e4a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 4 client.27448 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.528+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55872e3d2700 con 0x55872e4a29a0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.529+0000 7f74f3e00700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4a29a0 0x55872e4a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.529+0000 7f74daa00700 10 client.27448 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.529+0000 7f74f3e00700 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4a29a0 0x55872e4a4dc0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=739bb0f696910995 server_cookie=e9b6732b6bb04ef1 in_seq=0 out_seq=0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f74daa00700 1 -- 172.21.15.23:0/197954644 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55872e3d2700 con 0x55872e4a29a0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f74daa00700 10 client.27448 handle_command_reply: tid=0 2024-03-30T04:09:03.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 client.27448 shutdown 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 msgr2=0x55872e48fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f74e8012810 tx=0x7f74e8012b20 comp rx=0 tx=0).stop 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 shutdown_connections 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e4902a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e492820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4a29a0 0x55872e4a4dc0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 --2- 172.21.15.23:0/197954644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e487ef0 0x55872e48fb60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f74daa00700 0 client.27448 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.530+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 >> 172.21.15.23:0/197954644 conn(0x55872e3e76a0 msgr2=0x55872e4896c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.531+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 shutdown_connections 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.531+0000 7f750f453b80 1 -- 172.21.15.23:0/197954644 wait complete. 2024-03-30T04:09:03.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.531+0000 7f750f453b80 20 client.27448 trim_cache size 0 max 16384 2024-03-30T04:09:03.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.548+0000 7f750f453b80 1 Processor -- start 2024-03-30T04:09:03.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.548+0000 7f750f453b80 1 -- start start 2024-03-30T04:09:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e487ef0 0x55872e48f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e48f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e490ab0 con 0x55872e487ef0 2024-03-30T04:09:03.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e483fa0 con 0x55872e492df0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f750f453b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e499f50 con 0x55872e4875d0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44782/0 (socket says 172.21.15.23:44782) 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 -- 172.21.15.23:0/3729650629 learned_addr learned my addr 172.21.15.23:0/3729650629 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f9000700 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e487ef0 0x55872e48f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e48f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 -- 172.21.15.23:0/3729650629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 msgr2=0x55872e48f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e48f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 -- 172.21.15.23:0/3729650629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e487ef0 msgr2=0x55872e48f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e487ef0 0x55872e48f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.549+0000 7f74f3e00700 1 -- 172.21.15.23:0/3729650629 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55872e39fea0 con 0x55872e4875d0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f74f3e00700 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f74e00093f0 tx=0x7f74e0005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c820474e5dd4e450 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f74e6a00700 1 -- 172.21.15.23:0/3729650629 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e002c040 con 0x55872e4875d0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f74e6a00700 1 -- 172.21.15.23:0/3729650629 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74e0037560 con 0x55872e4875d0 2024-03-30T04:09:03.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f74e6a00700 1 -- 172.21.15.23:0/3729650629 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0003790 con 0x55872e4875d0 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f750f453b80 1 -- 172.21.15.23:0/3729650629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 msgr2=0x55872e499a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.550+0000 7f750f453b80 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f74e00093f0 tx=0x7f74e0005810 comp rx=0 tx=0).stop 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 -- 172.21.15.23:0/3729650629 shutdown_connections 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e492df0 0x55872e48f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e4875d0 0x55872e499a10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 --2- 172.21.15.23:0/3729650629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e487ef0 0x55872e48f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 -- 172.21.15.23:0/3729650629 >> 172.21.15.23:0/3729650629 conn(0x55872e3e76a0 msgr2=0x55872e47cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 -- 172.21.15.23:0/3729650629 shutdown_connections 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 -- 172.21.15.23:0/3729650629 wait complete. 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 -- start start 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.551+0000 7f750f453b80 1 Processor -- start 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e497e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e498360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e4989e0 con 0x55872e4875d0 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e498b50 con 0x55872e487ef0 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55872e4f67c0 con 0x55872e492df0 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f750f453b80 10 client.0 _ll_register_callbacks cb 0x55872e37f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:03.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f74f3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47372/0 (socket says 172.21.15.23:47372) 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f74f3e00700 1 -- 172.21.15.23:0/4152454639 learned_addr learned my addr 172.21.15.23:0/4152454639 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.552+0000 7f74f9000700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e497e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e498360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 -- 172.21.15.23:0/4152454639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 msgr2=0x55872e498360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e498360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 -- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 msgr2=0x55872e497e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e497e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 -- 172.21.15.23:0/4152454639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55872e25b3b0 con 0x55872e4875d0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f9000700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e497e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74f3e00700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f74e0005050 tx=0x7f74e000bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b82e033eca1fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74daa00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e002c070 con 0x55872e4875d0 2024-03-30T04:09:03.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f74e0000e40 con 0x55872e4875d0 2024-03-30T04:09:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.553+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74e00056a0 con 0x55872e4875d0 2024-03-30T04:09:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74e0037a20 con 0x55872e4875d0 2024-03-30T04:09:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 10 client.17457 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55872e25fcd0 con 0x55872e4875d0 2024-03-30T04:09:03.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74e00056a0 con 0x55872e4875d0 2024-03-30T04:09:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f74e000f460 con 0x55872e4875d0 2024-03-30T04:09:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 10 client.17457 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:03.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 20 client.17457 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.554+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4f7960 0x55872e4f9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.555+0000 7f750f453b80 4 client.17457 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.555+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55872e228cf0 con 0x55872e4f7960 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.555+0000 7f74f9000700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4f7960 0x55872e4f9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.555+0000 7f74f9000700 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4f7960 0x55872e4f9d50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40210739703f89da server_cookie=59abc55e51a7cb65 in_seq=0 out_seq=0 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.555+0000 7f74daa00700 10 client.17457 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f74daa00700 1 -- 172.21.15.23:0/4152454639 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55872e228cf0 con 0x55872e4f7960 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f74daa00700 10 client.17457 handle_command_reply: tid=0 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 client.17457 shutdown 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 msgr2=0x55872e4998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f74e0005050 tx=0x7f74e000bc70 comp rx=0 tx=0).stop 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 shutdown_connections 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55872e4f7960 0x55872e4f9d50 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872e492df0 0x55872e498360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55872e4875d0 0x55872e4998c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 --2- 172.21.15.23:0/4152454639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55872e487ef0 0x55872e497e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f74daa00700 0 client.17457 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 >> 172.21.15.23:0/4152454639 conn(0x55872e3e76a0 msgr2=0x55872e48f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 shutdown_connections 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 1 -- 172.21.15.23:0/4152454639 wait complete. 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.556+0000 7f750f453b80 20 client.17457 trim_cache size 0 max 16384 2024-03-30T04:09:03.558 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:09:03.559 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:09:03.559 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 msgr2=0x7f74dc0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 0x7f74dc0a37d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0c9f20 tx=0x7f74ec009ed0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 msgr2=0x7f74f419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00ef30 tx=0x7f74ec00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 shutdown_connections 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f74dc0a1310 0x7f74dc0a37d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74f4105ce0 0x7f74f419dd20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74f4067e30 0x7f74f4199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 --2- 172.21.15.23:0/3798112218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74f40687f0 0x7f74f4199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 >> 172.21.15.23:0/3798112218 conn(0x7f74f4076d40 msgr2=0x7f74f40fe860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 shutdown_connections 2024-03-30T04:09:03.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.557+0000 7f74fba00700 1 -- 172.21.15.23:0/3798112218 wait complete. 2024-03-30T04:09:03.574 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:09:03.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:09:03.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 -- 172.21.15.23:0/768751640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 msgr2=0x7fe694101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 --2- 172.21.15.23:0/768751640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694101240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe684009990 tx=0x7fe684009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 -- 172.21.15.23:0/768751640 shutdown_connections 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 --2- 172.21.15.23:0/768751640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe694105f80 0x7fe69410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 --2- 172.21.15.23:0/768751640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694101240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 --2- 172.21.15.23:0/768751640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694101810 0x7fe694105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.780+0000 7fe699200700 1 -- 172.21.15.23:0/768751640 >> 172.21.15.23:0/768751640 conn(0x7fe6940fc700 msgr2=0x7fe6940feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.781+0000 7fe699200700 1 -- 172.21.15.23:0/768751640 shutdown_connections 2024-03-30T04:09:03.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.781+0000 7fe699200700 1 -- 172.21.15.23:0/768751640 wait complete. 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 Processor -- start 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 -- start start 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 0x7fe694197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe69419b9b0 0x7fe69419be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69419c430 con 0x7fe69419b9b0 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69419c5a0 con 0x7fe694100e60 2024-03-30T04:09:03.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.782+0000 7fe699200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe69419c760 con 0x7fe694105f80 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56508/0 (socket says 172.21.15.23:56508) 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 -- 172.21.15.23:0/159143510 learned_addr learned my addr 172.21.15.23:0/159143510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691400700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 0x7fe694197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe692800700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe69419b9b0 0x7fe69419be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 -- 172.21.15.23:0/159143510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 msgr2=0x7fe694197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 0x7fe694197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 -- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe69419b9b0 msgr2=0x7fe69419be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe69419b9b0 0x7fe69419be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 -- 172.21.15.23:0/159143510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6840095f0 con 0x7fe694100e60 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691400700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 0x7fe694197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.783+0000 7fe691e00700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe684005910 tx=0x7fe684004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.784+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68401d070 con 0x7fe694100e60 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.784+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe684022470 con 0x7fe694100e60 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.784+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684017650 con 0x7fe694100e60 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.784+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe694072fe0 con 0x7fe694100e60 2024-03-30T04:09:03.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.784+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6940734f0 con 0x7fe694100e60 2024-03-30T04:09:03.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.785+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6941091a0 con 0x7fe694100e60 2024-03-30T04:09:03.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.787+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe6840225e0 con 0x7fe694100e60 2024-03-30T04:09:03.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.788+0000 7fe68aa00700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 0x7fe6780a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:03.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.788+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe6840ce280 con 0x7fe694100e60 2024-03-30T04:09:03.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.788+0000 7fe691400700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 0x7fe6780a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:03.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.789+0000 7fe691400700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 0x7fe6780a7be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe694198620 tx=0x7fe680009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:03.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:03.792+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe684097440 con 0x7fe694100e60 2024-03-30T04:09:04.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.062+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe694102330 con 0x7fe694100e60 2024-03-30T04:09:04.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.063+0000 7fe68aa00700 1 -- 172.21.15.23:0/159143510 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fe68409aa60 con 0x7fe694100e60 2024-03-30T04:09:04.061 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:09:04.062 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 msgr2=0x7fe6780a7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 0x7fe6780a7be0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe694198620 tx=0x7fe680009260 comp rx=0 tx=0).stop 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 msgr2=0x7fe694197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe684005910 tx=0x7fe684004730 comp rx=0 tx=0).stop 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 shutdown_connections 2024-03-30T04:09:04.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe69419b9b0 0x7fe69419be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe6780a5720 0x7fe6780a7be0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe694100e60 0x7fe694197120 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.065+0000 7fe699200700 1 --2- 172.21.15.23:0/159143510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe694105f80 0x7fe694197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.066+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 >> 172.21.15.23:0/159143510 conn(0x7fe6940fc700 msgr2=0x7fe6940fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.066+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 shutdown_connections 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.066+0000 7fe699200700 1 -- 172.21.15.23:0/159143510 wait complete. 2024-03-30T04:09:04.065 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:09:04.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:09:04.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 702062 10.91 MB/sec execute 221 sec latency 20.874 ms 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 -- 172.21.15.23:0/2109517813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c073aa0 msgr2=0x7fbe0c10d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 --2- 172.21.15.23:0/2109517813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c10d060 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009990 tx=0x7fbdf8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 -- 172.21.15.23:0/2109517813 shutdown_connections 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 --2- 172.21.15.23:0/2109517813 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 --2- 172.21.15.23:0/2109517813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 0x7fbe0c073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 --2- 172.21.15.23:0/2109517813 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c10d060 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 -- 172.21.15.23:0/2109517813 >> 172.21.15.23:0/2109517813 conn(0x7fbe0c0fc7f0 msgr2=0x7fbe0c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 -- 172.21.15.23:0/2109517813 shutdown_connections 2024-03-30T04:09:04.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.296+0000 7fbe13400700 1 -- 172.21.15.23:0/2109517813 wait complete. 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.297+0000 7fbe13400700 1 Processor -- start 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.297+0000 7fbe13400700 1 -- start start 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 0x7fbe0c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c19e350 con 0x7fbe0c073180 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c19e4c0 con 0x7fbe0c073aa0 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe13400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c19e680 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 0x7fbe0c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44826/0 (socket says 172.21.15.23:44826) 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 -- 172.21.15.23:0/3529027281 learned_addr learned my addr 172.21.15.23:0/3529027281 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 -- 172.21.15.23:0/3529027281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c073aa0 msgr2=0x7fbe0c199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 -- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 msgr2=0x7fbe0c199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 0x7fbe0c199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.298+0000 7fbe11400700 1 -- 172.21.15.23:0/3529027281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf80095f0 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe11400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400ef90 tx=0x7fbe0400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe09600700 1 -- 172.21.15.23:0/3529027281 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04018070 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe09600700 1 -- 172.21.15.23:0/3529027281 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe0400f040 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c19e900 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe09600700 1 -- 172.21.15.23:0/3529027281 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0400cda0 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.299+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c1a61a0 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.303+0000 7fbe09600700 1 -- 172.21.15.23:0/3529027281 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fbe04003a00 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.304+0000 7fbe26f05b80 1 Processor -- start 2024-03-30T04:09:04.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.304+0000 7fbe09600700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 0x7fbdf40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 -- start start 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe09600700 1 -- 172.21.15.23:0/3529027281 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fbe040ccb40 con 0x7fbe0c10d5a0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe0be00700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 0x7fbdf40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c80602d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c80602ddf0 0x55c8060301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805f7b990 con 0x55c806022ef0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805f79350 con 0x55c8060225d0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805f74c40 con 0x55c80602ddf0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56544/0 (socket says 172.21.15.23:56544) 2024-03-30T04:09:04.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe10a00700 1 -- 172.21.15.23:0/3724035599 learned_addr learned my addr 172.21.15.23:0/3724035599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe11400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c80602ddf0 0x55c8060301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.305+0000 7fbe0be00700 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c80602d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 -- 172.21.15.23:0/3724035599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c80602ddf0 msgr2=0x55c8060301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c80602ddf0 0x55c8060301e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 -- 172.21.15.23:0/3724035599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 msgr2=0x55c80602d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c80602d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 -- 172.21.15.23:0/3724035599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c805df63b0 con 0x55c8060225d0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe0be00700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 0x7fbdf40a37d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c19a980 tx=0x7fbdf8005320 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbe10a00700 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009d80 tx=0x7fbe00011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39f81229b3017ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbdfdc00700 1 -- 172.21.15.23:0/3724035599 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00004230 con 0x55c8060225d0 2024-03-30T04:09:04.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbdfdc00700 1 -- 172.21.15.23:0/3724035599 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe00004390 con 0x55c8060225d0 2024-03-30T04:09:04.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.306+0000 7fbdfdc00700 1 -- 172.21.15.23:0/3724035599 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe000198c0 con 0x55c8060225d0 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 -- 172.21.15.23:0/3724035599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 msgr2=0x55c8060229b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009d80 tx=0x7fbe00011720 comp rx=0 tx=0).stop 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 -- 172.21.15.23:0/3724035599 shutdown_connections 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c80602ddf0 0x55c8060301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c8060225d0 0x55c8060229b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/3724035599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c80602d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 -- 172.21.15.23:0/3724035599 >> 172.21.15.23:0/3724035599 conn(0x55c805f826a0 msgr2=0x55c80601ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 -- 172.21.15.23:0/3724035599 shutdown_connections 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.307+0000 7fbe26f05b80 1 -- 172.21.15.23:0/3724035599 wait complete. 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.308+0000 7fbe26f05b80 1 -- start start 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.308+0000 7fbe26f05b80 1 Processor -- start 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8060225d0 0x55c80602d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c80602b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805f10d80 con 0x55c8060225d0 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805f10a20 con 0x55c80602ddf0 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c805df86a0 con 0x55c806022ef0 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe26f05b80 10 client.0 _ll_register_callbacks cb 0x55c805f1a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.312 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe0be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8060225d0 0x55c80602d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe0be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44848/0 (socket says 172.21.15.23:44848) 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe0be00700 1 -- 172.21.15.23:0/2870133416 learned_addr learned my addr 172.21.15.23:0/2870133416 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe11400700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c80602b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.309+0000 7fbe0be00700 1 -- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 msgr2=0x55c80602b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c80602b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe0be00700 1 -- 172.21.15.23:0/2870133416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8060225d0 msgr2=0x55c80602d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8060225d0 0x55c80602d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe0be00700 1 -- 172.21.15.23:0/2870133416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe000099e0 con 0x55c806022ef0 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe11400700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c80602b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8015000 tx=0x7fbdf8009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a22ab382ba365ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbdf2000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf801d070 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fbdf8004710 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.310+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c805f35380 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdf80048d0 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 10 client.27472 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c805da2b60 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c805f35380 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbdf800f9e0 con 0x55c806022ef0 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 10 client.27472 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 10 client.27472 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:09:04.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 20 client.27472 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c80603d9e0 0x55c80603fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 4 client.27472 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.311+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c805f6d700 con 0x55c80603d9e0 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.312+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c80603d9e0 0x55c80603fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.312+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c80603d9e0 0x55c80603fe00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=36fa2e45e9ffeded server_cookie=71e7f68aa037e09e in_seq=0 out_seq=0 2024-03-30T04:09:04.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.312+0000 7fbdf2000700 10 client.27472 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbdf2000700 1 -- 172.21.15.23:0/2870133416 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c805f6d700 con 0x55c80603d9e0 2024-03-30T04:09:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbdf2000700 10 client.27472 handle_command_reply: tid=0 2024-03-30T04:09:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 client.27472 shutdown 2024-03-30T04:09:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 msgr2=0x55c80602ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8015000 tx=0x7fbdf8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 shutdown_connections 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c80602b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c8060225d0 0x55c80602d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c80603d9e0 0x55c80603fe00 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2870133416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c806022ef0 0x55c80602ab10 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 >> 172.21.15.23:0/2870133416 conn(0x55c805f826a0 msgr2=0x55c8060246c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.313+0000 7fbdf2000700 0 client.27472 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.314+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 shutdown_connections 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.314+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2870133416 wait complete. 2024-03-30T04:09:04.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.314+0000 7fbe26f05b80 20 client.27472 trim_cache size 0 max 16384 2024-03-30T04:09:04.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.327+0000 7fbe26f05b80 1 Processor -- start 2024-03-30T04:09:04.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 -- start start 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c806034a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c8060175a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c806034fb0 con 0x55c806022ef0 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c80602b700 con 0x55c80602ddf0 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c806017ae0 con 0x55c8060225d0 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe0be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c806034a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44856/0 (socket says 172.21.15.23:44856) 2024-03-30T04:09:04.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe10a00700 1 -- 172.21.15.23:0/2588429697 learned_addr learned my addr 172.21.15.23:0/2588429697 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.328+0000 7fbe11400700 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c8060175a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 -- 172.21.15.23:0/2588429697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 msgr2=0x55c8060175a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c8060175a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 -- 172.21.15.23:0/2588429697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 msgr2=0x55c806034a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c806034a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 -- 172.21.15.23:0/2588429697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c805f3aea0 con 0x55c8060225d0 2024-03-30T04:09:04.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.329+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbe00004640 tx=0x7fbe00011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64904946e735f146 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.330+0000 7fbdfe600700 1 -- 172.21.15.23:0/2588429697 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00029040 con 0x55c8060225d0 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.330+0000 7fbdfe600700 1 -- 172.21.15.23:0/2588429697 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe00009ed0 con 0x55c8060225d0 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.330+0000 7fbdfe600700 1 -- 172.21.15.23:0/2588429697 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00005040 con 0x55c8060225d0 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.330+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2588429697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 msgr2=0x55c80603fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.330+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fbe00004640 tx=0x7fbe00011720 comp rx=0 tx=0).stop 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2588429697 shutdown_connections 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c80602ddf0 0x55c8060175a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c80603fdd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2588429697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c806022ef0 0x55c806034a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2588429697 >> 172.21.15.23:0/2588429697 conn(0x55c805f826a0 msgr2=0x55c806044720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2588429697 shutdown_connections 2024-03-30T04:09:04.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2588429697 wait complete. 2024-03-30T04:09:04.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 -- start start 2024-03-30T04:09:04.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.331+0000 7fbe26f05b80 1 Processor -- start 2024-03-30T04:09:04.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 0x55c806032c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c80602ddf0 0x55c8060331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c806033790 con 0x55c80602ddf0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c806033900 con 0x55c806022ef0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c806033ac0 con 0x55c8060225d0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe26f05b80 10 client.0 _ll_register_callbacks cb 0x55c805f1a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe0be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 0x55c806032c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44860/0 (socket says 172.21.15.23:44860) 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 -- 172.21.15.23:0/2439615848 learned_addr learned my addr 172.21.15.23:0/2439615848 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe11400700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c80602ddf0 0x55c8060331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 -- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 msgr2=0x55c806032c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 0x55c806032c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 -- 172.21.15.23:0/2439615848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c80602ddf0 msgr2=0x55c8060331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c80602ddf0 0x55c8060331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe10a00700 1 -- 172.21.15.23:0/2439615848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c805df63b0 con 0x55c8060225d0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.332+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 0x55c806032c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbe10a00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbe00000c00 tx=0x7fbe00003f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=324e2aa70a6c802c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbdf2000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00029040 con 0x55c8060225d0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fbe00003a90 con 0x55c8060225d0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe00004390 con 0x55c8060225d0 2024-03-30T04:09:04.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe00005030 con 0x55c8060225d0 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbe26f05b80 10 client.27478 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c805dfacd0 con 0x55c8060225d0 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.333+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe00004390 con 0x55c8060225d0 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbe00011440 con 0x55c8060225d0 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 10 client.27478 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 10 client.27478 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 20 client.27478 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c806092960 0x55c806094d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 4 client.27478 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c805dc3cf0 con 0x55c806092960 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c806092960 0x55c806094d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbe0be00700 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c806092960 0x55c806094d80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8df6f73f69eb5405 server_cookie=1c2325e19e8bf7d8 in_seq=0 out_seq=0 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.334+0000 7fbdf2000700 10 client.27478 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbdf2000700 1 -- 172.21.15.23:0/2439615848 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c805dc3cf0 con 0x55c806092960 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbdf2000700 10 client.27478 handle_command_reply: tid=0 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 client.27478 shutdown 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 msgr2=0x55c8060347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbe00000c00 tx=0x7fbe00003f40 comp rx=0 tx=0).stop 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 shutdown_connections 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c80602ddf0 0x55c8060331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c806092960 0x55c806094d80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8060225d0 0x55c8060347d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 --2- 172.21.15.23:0/2439615848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c806022ef0 0x55c806032c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 >> 172.21.15.23:0/2439615848 conn(0x55c805f826a0 msgr2=0x55c80602a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbdf2000700 0 client.27478 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:04.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 shutdown_connections 2024-03-30T04:09:04.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 1 -- 172.21.15.23:0/2439615848 wait complete. 2024-03-30T04:09:04.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.335+0000 7fbe26f05b80 20 client.27478 trim_cache size 0 max 16384 2024-03-30T04:09:04.339 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 msgr2=0x7fbdf40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 0x7fbdf40a37d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c19a980 tx=0x7fbdf8005320 comp rx=0 tx=0).stop 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 msgr2=0x7fbe0c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400ef90 tx=0x7fbe0400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 shutdown_connections 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10d5a0 0x7fbe0c19dd10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbe0c073180 0x7fbe0c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fbdf40a1310 0x7fbdf40a37d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 --2- 172.21.15.23:0/3529027281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c073aa0 0x7fbe0c199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 >> 172.21.15.23:0/3529027281 conn(0x7fbe0c0fc7f0 msgr2=0x7fbe0c0fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 shutdown_connections 2024-03-30T04:09:04.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.336+0000 7fbe13400700 1 -- 172.21.15.23:0/3529027281 wait complete. 2024-03-30T04:09:04.350 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:04 smithi023 ceph-mon[38254]: pgmap v218: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 278 op/s 2024-03-30T04:09:04.351 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:04 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/159143510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:04.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:09:04.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:04 smithi154 ceph-mon[79161]: pgmap v218: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 278 op/s 2024-03-30T04:09:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:04 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/159143510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:04 smithi193 ceph-mon[79581]: pgmap v218: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 278 op/s 2024-03-30T04:09:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:04 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/159143510' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 -- 172.21.15.23:0/2013485806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 msgr2=0x7f448c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 --2- 172.21.15.23:0/2013485806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c0735c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f447c009990 tx=0x7f447c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 -- 172.21.15.23:0/2013485806 shutdown_connections 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 --2- 172.21.15.23:0/2013485806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c10f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 --2- 172.21.15.23:0/2013485806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c0735c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 --2- 172.21.15.23:0/2013485806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 -- 172.21.15.23:0/2013485806 >> 172.21.15.23:0/2013485806 conn(0x7f448c0fc7c0 msgr2=0x7f448c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 -- 172.21.15.23:0/2013485806 shutdown_connections 2024-03-30T04:09:04.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.571+0000 7f4491400700 1 -- 172.21.15.23:0/2013485806 wait complete. 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.572+0000 7f4491400700 1 Processor -- start 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.572+0000 7f4491400700 1 -- start start 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c19e310 con 0x7f448c073b00 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c19e480 con 0x7f448c10d530 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4491400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f448c19e640 con 0x7f448c0731e0 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f448a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4489e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f448a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44868/0 (socket says 172.21.15.23:44868) 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f448a800700 1 -- 172.21.15.23:0/4103712841 learned_addr learned my addr 172.21.15.23:0/4103712841 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f448b200700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4489e00700 1 -- 172.21.15.23:0/4103712841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 msgr2=0x7f448c1993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.573+0000 7f4489e00700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4489e00700 1 -- 172.21.15.23:0/4103712841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 msgr2=0x7f448c19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4489e00700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4489e00700 1 -- 172.21.15.23:0/4103712841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4478009520 con 0x7f448c073b00 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f448a800700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f448b200700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4489e00700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c199930 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f447800e9f0 tx=0x7f447800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4483400700 1 -- 172.21.15.23:0/4103712841 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447800cb70 con 0x7f448c073b00 2024-03-30T04:09:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4483400700 1 -- 172.21.15.23:0/4103712841 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f447800ee90 con 0x7f448c073b00 2024-03-30T04:09:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c0095f0 con 0x7f448c073b00 2024-03-30T04:09:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4483400700 1 -- 172.21.15.23:0/4103712841 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4478003cf0 con 0x7f448c073b00 2024-03-30T04:09:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.574+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c1a5f80 con 0x7f448c073b00 2024-03-30T04:09:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.578+0000 7f4483400700 1 -- 172.21.15.23:0/4103712841 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f447800ccd0 con 0x7f448c073b00 2024-03-30T04:09:04.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.579+0000 7f4483400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 0x7f44700a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.579+0000 7f4483400700 1 -- 172.21.15.23:0/4103712841 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4478014070 con 0x7f448c073b00 2024-03-30T04:09:04.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.579+0000 7f448a800700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 0x7f44700a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.579+0000 7f44a4fffb80 1 Processor -- start 2024-03-30T04:09:04.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 -- start start 2024-03-30T04:09:04.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f448a800700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 0x7f44700a3750 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c00b280 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a599b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566239b2990 con 0x556623a595d0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566239b0350 con 0x556623a64df0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566239abc40 con 0x556623a59ef0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f4489e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f448b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f448a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f448b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56590/0 (socket says 172.21.15.23:56590) 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f4489e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44880/0 (socket says 172.21.15.23:44880) 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.580+0000 7f448b200700 1 -- 172.21.15.23:0/728811526 learned_addr learned my addr 172.21.15.23:0/728811526 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 -- 172.21.15.23:0/728811526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 msgr2=0x556623a671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a671e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 -- 172.21.15.23:0/728811526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 msgr2=0x556623a599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 -- 172.21.15.23:0/728811526 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55662382d3b0 con 0x556623a59ef0 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4489e00700 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f44780ca6b0 tx=0x7f44780999d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c70be4a41f736011 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4477400700 1 -- 172.21.15.23:0/728811526 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780cd160 con 0x556623a59ef0 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4477400700 1 -- 172.21.15.23:0/728811526 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f447809dea0 con 0x556623a59ef0 2024-03-30T04:09:04.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.581+0000 7f4477400700 1 -- 172.21.15.23:0/728811526 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447809a530 con 0x556623a59ef0 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 -- 172.21.15.23:0/728811526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 msgr2=0x556623a648b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f44780ca6b0 tx=0x7f44780999d0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 -- 172.21.15.23:0/728811526 shutdown_connections 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/728811526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a648b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 -- 172.21.15.23:0/728811526 >> 172.21.15.23:0/728811526 conn(0x5566239b96a0 msgr2=0x556623a55a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 -- 172.21.15.23:0/728811526 shutdown_connections 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.582+0000 7f44a4fffb80 1 -- 172.21.15.23:0/728811526 wait complete. 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.583+0000 7f44a4fffb80 1 -- start start 2024-03-30T04:09:04.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.583+0000 7f44a4fffb80 1 Processor -- start 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a595d0 0x556623a64890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 0x556623a61b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623947d80 con 0x556623a59ef0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623947a20 con 0x556623a64df0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55662382f6a0 con 0x556623a595d0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f44a4fffb80 10 client.0 _ll_register_callbacks cb 0x556623951470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f448b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f448a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a595d0 0x556623a64890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f448b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56592/0 (socket says 172.21.15.23:56592) 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.584+0000 7f448b200700 1 -- 172.21.15.23:0/946455604 learned_addr learned my addr 172.21.15.23:0/946455604 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f4489e00700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 0x556623a61b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 -- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a595d0 msgr2=0x556623a64890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a595d0 0x556623a64890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 -- 172.21.15.23:0/946455604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 msgr2=0x556623a61b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 0x556623a61b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 -- 172.21.15.23:0/946455604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c1a5f80 con 0x556623a64df0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f4489e00700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 0x556623a61b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f448b200700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4484004770 tx=0x7f4484003f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83858e515c70e8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f446be00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448400eb30 con 0x556623a64df0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55662396c380 con 0x556623a64df0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.585+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f448400ec90 con 0x556623a64df0 2024-03-30T04:09:04.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f448400ee50 con 0x556623a64df0 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 10 client.27497 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5566237d9b60 con 0x556623a64df0 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55662396c380 con 0x556623a64df0 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f448401c020 con 0x556623a64df0 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 10 client.27497 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 10 client.27497 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 20 client.27497 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623a74a50 0x556623a76e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 4 client.27497 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566239a4700 con 0x556623a74a50 2024-03-30T04:09:04.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.586+0000 7f448a800700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623a74a50 0x556623a76e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.587+0000 7f448a800700 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623a74a50 0x556623a76e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5f5fd10f19aec569 server_cookie=246c393ee8943039 in_seq=0 out_seq=0 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.587+0000 7f446be00700 10 client.27497 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f446be00700 1 -- 172.21.15.23:0/946455604 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5566239a4700 con 0x556623a74a50 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f446be00700 10 client.27497 handle_command_reply: tid=0 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 client.27497 shutdown 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 msgr2=0x556623a622c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4484004770 tx=0x7f4484003f80 comp rx=0 tx=0).stop 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 shutdown_connections 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623a74a50 0x556623a76e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a64df0 0x556623a622c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a595d0 0x556623a64890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f446be00700 0 client.27497 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/946455604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a59ef0 0x556623a61b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 >> 172.21.15.23:0/946455604 conn(0x5566239b96a0 msgr2=0x556623a5b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 shutdown_connections 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 1 -- 172.21.15.23:0/946455604 wait complete. 2024-03-30T04:09:04.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.588+0000 7f44a4fffb80 20 client.27497 trim_cache size 0 max 16384 2024-03-30T04:09:04.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.605+0000 7f44a4fffb80 1 Processor -- start 2024-03-30T04:09:04.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.605+0000 7f44a4fffb80 1 -- start start 2024-03-30T04:09:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a64df0 0x556623a4ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a59d70 con 0x556623a595d0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a63180 con 0x556623a59ef0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a5b4d0 con 0x556623a64df0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f448a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56594/0 (socket says 172.21.15.23:56594) 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f448a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47488/0 (socket says 172.21.15.23:47488) 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 -- 172.21.15.23:0/1676853142 learned_addr learned my addr 172.21.15.23:0/1676853142 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f448b200700 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a64df0 0x556623a4ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 -- 172.21.15.23:0/1676853142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a64df0 msgr2=0x556623a4ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a64df0 0x556623a4ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 -- 172.21.15.23:0/1676853142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 msgr2=0x556623a6ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f4489e00700 1 -- 172.21.15.23:0/1676853142 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556623971ea0 con 0x556623a59ef0 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.606+0000 7f448a800700 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.607+0000 7f4489e00700 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f44780983c0 tx=0x7f4478099c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e58dd0cbecda7c92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.607+0000 7f4477e00700 1 -- 172.21.15.23:0/1676853142 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780b8070 con 0x556623a59ef0 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.607+0000 7f4477e00700 1 -- 172.21.15.23:0/1676853142 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44780cd160 con 0x556623a59ef0 2024-03-30T04:09:04.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.607+0000 7f4477e00700 1 -- 172.21.15.23:0/1676853142 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44780b8430 con 0x556623a59ef0 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- 172.21.15.23:0/1676853142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 msgr2=0x556623a4e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f44780983c0 tx=0x7f4478099c60 comp rx=0 tx=0).stop 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- 172.21.15.23:0/1676853142 shutdown_connections 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a64df0 0x556623a4ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a595d0 0x556623a6ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/1676853142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a59ef0 0x556623a4e5a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- 172.21.15.23:0/1676853142 >> 172.21.15.23:0/1676853142 conn(0x5566239b96a0 msgr2=0x556623a7b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- 172.21.15.23:0/1676853142 shutdown_connections 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- 172.21.15.23:0/1676853142 wait complete. 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 -- start start 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.608+0000 7f44a4fffb80 1 Processor -- start 2024-03-30T04:09:04.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a6b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 0x556623a69890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a69f20 con 0x556623a64df0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a6a090 con 0x556623a595d0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556623a6a250 con 0x556623a59ef0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f44a4fffb80 10 client.0 _ll_register_callbacks cb 0x556623951470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f448a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f448a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56600/0 (socket says 172.21.15.23:56600) 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f448a800700 1 -- 172.21.15.23:0/2364447948 learned_addr learned my addr 172.21.15.23:0/2364447948 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f448b200700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 0x556623a69890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.609+0000 7f4489e00700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a6b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 -- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 msgr2=0x556623a6b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a6b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 -- 172.21.15.23:0/2364447948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 msgr2=0x556623a69890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 0x556623a69890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 -- 172.21.15.23:0/2364447948 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c110660 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448b200700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 0x556623a69890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f448a800700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f447c00bbd0 tx=0x7f447c01a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2dc01a5a872de95 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f446be00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c02b040 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f447c004dc0 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f447809dea0 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f447c02d040 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f44a4fffb80 10 client.27503 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.610+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55662382d3b0 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f447809dea0 con 0x556623a595d0 2024-03-30T04:09:04.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f447c00f880 con 0x556623a595d0 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 10 client.27503 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 10 client.27503 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 20 client.27503 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623ac9a00 0x556623acbe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 4 client.27503 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5566237facf0 con 0x556623ac9a00 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.611+0000 7f4489e00700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623ac9a00 0x556623acbe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f446be00700 10 client.27503 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f4489e00700 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623ac9a00 0x556623acbe20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1bc9db5b5399991b server_cookie=ee1d1fa53ba05b13 in_seq=0 out_seq=0 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f446be00700 1 -- 172.21.15.23:0/2364447948 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5566237facf0 con 0x556623ac9a00 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f446be00700 10 client.27503 handle_command_reply: tid=0 2024-03-30T04:09:04.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 client.27503 shutdown 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 msgr2=0x556623a6b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f447c00bbd0 tx=0x7f447c01a050 comp rx=0 tx=0).stop 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 shutdown_connections 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556623ac9a00 0x556623acbe20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556623a64df0 0x556623a69890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f446be00700 0 client.27503 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556623a595d0 0x556623a6b3a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 --2- 172.21.15.23:0/2364447948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556623a59ef0 0x556623a6b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 >> 172.21.15.23:0/2364447948 conn(0x5566239b96a0 msgr2=0x556623a4e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 shutdown_connections 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 1 -- 172.21.15.23:0/2364447948 wait complete. 2024-03-30T04:09:04.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.612+0000 7f44a4fffb80 20 client.27503 trim_cache size 0 max 16384 2024-03-30T04:09:04.616 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:04.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 msgr2=0x7f44700a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 0x7f44700a3750 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f447c006010 tx=0x7f447c00b280 comp rx=0 tx=0).stop 2024-03-30T04:09:04.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 msgr2=0x7f448c199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c199930 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f447800e9f0 tx=0x7f447800ed00 comp rx=0 tx=0).stop 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 shutdown_connections 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44700a1290 0x7f44700a3750 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f448c10d530 0x7f448c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f448c0731e0 0x7f448c1993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 --2- 172.21.15.23:0/4103712841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f448c073b00 0x7f448c199930 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 >> 172.21.15.23:0/4103712841 conn(0x7f448c0fc7c0 msgr2=0x7f448c0fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 shutdown_connections 2024-03-30T04:09:04.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.613+0000 7f4491400700 1 -- 172.21.15.23:0/4103712841 wait complete. 2024-03-30T04:09:04.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:09:04.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 -- 172.21.15.23:0/456605537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb190105c40 msgr2=0x7fb1900692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 --2- 172.21.15.23:0/456605537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb190105c40 0x7fb1900692b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 -- 172.21.15.23:0/456605537 shutdown_connections 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 --2- 172.21.15.23:0/456605537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190103740 0x7fb190103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 --2- 172.21.15.23:0/456605537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb190105c40 0x7fb1900692b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 --2- 172.21.15.23:0/456605537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1900697f0 0x7fb190069c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 -- 172.21.15.23:0/456605537 >> 172.21.15.23:0/456605537 conn(0x7fb190076d50 msgr2=0x7fb190077160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 -- 172.21.15.23:0/456605537 shutdown_connections 2024-03-30T04:09:04.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.836+0000 7fb195e00700 1 -- 172.21.15.23:0/456605537 wait complete. 2024-03-30T04:09:04.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 Processor -- start 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 -- start start 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1900697f0 0x7fb19019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190105c40 0x7fb1901a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901a1600 con 0x7fb1900697f0 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901a1770 con 0x7fb190103740 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.837+0000 7fb195e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1901a1930 con 0x7fb190105c40 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1900697f0 0x7fb19019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56616/0 (socket says 172.21.15.23:56616) 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 -- 172.21.15.23:0/263914056 learned_addr learned my addr 172.21.15.23:0/263914056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18f400700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190105c40 0x7fb1901a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 -- 172.21.15.23:0/263914056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190105c40 msgr2=0x7fb1901a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190105c40 0x7fb1901a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 -- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1900697f0 msgr2=0x7fb19019d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1900697f0 0x7fb19019d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 -- 172.21.15.23:0/263914056 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800095f0 con 0x7fb190103740 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.838+0000 7fb18e000700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00e9a0 tx=0x7fb17c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.839+0000 7fb194a00700 1 -- 172.21.15.23:0/263914056 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c009790 con 0x7fb190103740 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.839+0000 7fb194a00700 1 -- 172.21.15.23:0/263914056 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb17c010430 con 0x7fb190103740 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.839+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1901a1c10 con 0x7fb190103740 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.839+0000 7fb194a00700 1 -- 172.21.15.23:0/263914056 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17c018680 con 0x7fb190103740 2024-03-30T04:09:04.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.839+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1901aa570 con 0x7fb190103740 2024-03-30T04:09:04.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.842+0000 7fb194a00700 1 -- 172.21.15.23:0/263914056 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb17c0105a0 con 0x7fb190103740 2024-03-30T04:09:04.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.843+0000 7fb1a9a1ab80 1 Processor -- start 2024-03-30T04:09:04.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.843+0000 7fb1a9a1ab80 1 -- start start 2024-03-30T04:09:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb194a00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 0x7fb1740a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d001e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1c4b990 con 0x55d9c1cf2ef0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1c49350 con 0x55d9c1cfddf0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1c44c40 con 0x55d9c1cf25d0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb194a00700 1 -- 172.21.15.23:0/263914056 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb17c014070 con 0x7fb190103740 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44932/0 (socket says 172.21.15.23:44932) 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 -- 172.21.15.23:0/1908666324 learned_addr learned my addr 172.21.15.23:0/1908666324 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18e000700 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18ea00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 0x7fb1740a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 -- 172.21.15.23:0/1908666324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 msgr2=0x55d9c1d001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 -- 172.21.15.23:0/1908666324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf2ef0 msgr2=0x55d9c1cfd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.844+0000 7fb18f400700 1 -- 172.21.15.23:0/1908666324 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9c1ac63b0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb18f400700 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb188008cd0 tx=0x7fb188009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6cf59caf2e59e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb18ea00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 0x7fb1740a37d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb18000b2e0 tx=0x7fb180005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb17be00700 1 -- 172.21.15.23:0/1908666324 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18800e050 con 0x55d9c1cf25d0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb17be00700 1 -- 172.21.15.23:0/1908666324 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb18800f040 con 0x55d9c1cf25d0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb17be00700 1 -- 172.21.15.23:0/1908666324 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188018ee0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1908666324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 msgr2=0x55d9c1cf29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.845+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb188008cd0 tx=0x7fb188009920 comp rx=0 tx=0).stop 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1908666324 shutdown_connections 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf29b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1908666324 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1908666324 >> 172.21.15.23:0/1908666324 conn(0x55d9c1c526a0 msgr2=0x55d9c1ceea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1908666324 shutdown_connections 2024-03-30T04:09:04.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1908666324 wait complete. 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.846+0000 7fb1a9a1ab80 1 -- start start 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 Processor -- start 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cfd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1be0d80 con 0x55d9c1cfddf0 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1be0a20 con 0x55d9c1cf25d0 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1ac86a0 con 0x55d9c1cf2ef0 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb1a9a1ab80 10 client.0 _ll_register_callbacks cb 0x55d9c1bea470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44940/0 (socket says 172.21.15.23:44940) 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18e000700 1 -- 172.21.15.23:0/2523717951 learned_addr learned my addr 172.21.15.23:0/2523717951 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47522/0 (socket says 172.21.15.23:47522) 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.847+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cfd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 -- 172.21.15.23:0/2523717951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 msgr2=0x55d9c1cfaba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 -- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cf25d0 msgr2=0x55d9c1cfd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cfd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 -- 172.21.15.23:0/2523717951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1880089b0 con 0x55d9c1cfddf0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18e000700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb18f400700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb18801d040 tx=0x7fb1880085a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5995881f6c2bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb178c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:09:04.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18800e040 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb1880094e0 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.848+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9c1c05380 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb188004cd0 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 10 client.17511 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d9c1a72b60 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9c1c05380 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb188012050 con 0x55d9c1cfddf0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 10 client.17511 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 10 client.17511 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 20 client.17511 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d0d8e0 0x55d9c1d0fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 4 client.17511 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.849+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9c1c3d700 con 0x55d9c1d0d8e0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.850+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d0d8e0 0x55d9c1d0fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.850+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d0d8e0 0x55d9c1d0fcd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a729abb424eb76b0 server_cookie=a902e044d39a9bda in_seq=0 out_seq=0 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.850+0000 7fb178c00700 10 client.17511 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb178c00700 1 -- 172.21.15.23:0/2523717951 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d9c1c3d700 con 0x55d9c1d0d8e0 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb178c00700 10 client.17511 handle_command_reply: tid=0 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 client.17511 shutdown 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 msgr2=0x55d9c1cfb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fb18801d040 tx=0x7fb1880085a0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 shutdown_connections 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d0d8e0 0x55d9c1d0fcd0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cfddf0 0x55d9c1cfb2e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cfd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2523717951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1cfaba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 >> 172.21.15.23:0/2523717951 conn(0x55d9c1c526a0 msgr2=0x55d9c1cf46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.851+0000 7fb178c00700 0 client.17511 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.852+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 shutdown_connections 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.852+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2523717951 wait complete. 2024-03-30T04:09:04.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.852+0000 7fb1a9a1ab80 20 client.17511 trim_cache size 0 max 16384 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 Processor -- start 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 -- start start 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d04aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1ce7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1cf3650 con 0x55d9c1cf25d0 2024-03-30T04:09:04.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1cfbaf0 con 0x55d9c1cfddf0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.866+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1cef740 con 0x55d9c1cf2ef0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d04aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47528/0 (socket says 172.21.15.23:47528) 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 -- 172.21.15.23:0/2063972618 learned_addr learned my addr 172.21.15.23:0/2063972618 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1ce7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 -- 172.21.15.23:0/2063972618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 msgr2=0x55d9c1d04aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d04aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 -- 172.21.15.23:0/2063972618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 msgr2=0x55d9c1ce7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1ce7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 -- 172.21.15.23:0/2063972618 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9c1c0aea0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb18ea00700 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fb2e8fbb6f0af03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.867+0000 7fb185200700 1 -- 172.21.15.23:0/2063972618 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180005720 con 0x55d9c1cf25d0 2024-03-30T04:09:04.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb185200700 1 -- 172.21.15.23:0/2063972618 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb18001e040 con 0x55d9c1cf25d0 2024-03-30T04:09:04.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb185200700 1 -- 172.21.15.23:0/2063972618 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb18000f6c0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2063972618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 msgr2=0x55d9c1cf5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180020000 comp rx=0 tx=0).stop 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2063972618 shutdown_connections 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1ce7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1cf5570 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/2063972618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d04aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2063972618 >> 172.21.15.23:0/2063972618 conn(0x55d9c1c526a0 msgr2=0x55d9c1ce75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2063972618 shutdown_connections 2024-03-30T04:09:04.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.868+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/2063972618 wait complete. 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.869+0000 7fb1a9a1ab80 1 -- start start 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.869+0000 7fb1a9a1ab80 1 Processor -- start 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d02fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d03500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1d04dc0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1d606e0 con 0x55d9c1cfddf0 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9c1d60820 con 0x55d9c1cf2ef0 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb1a9a1ab80 10 client.0 _ll_register_callbacks cb 0x55d9c1bea470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47538/0 (socket says 172.21.15.23:47538) 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 -- 172.21.15.23:0/1629511181 learned_addr learned my addr 172.21.15.23:0/1629511181 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:04.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18f400700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d03500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18e000700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d02fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 -- 172.21.15.23:0/1629511181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 msgr2=0x55d9c1d02fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d02fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 -- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 msgr2=0x55d9c1d03500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d03500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18ea00700 1 -- 172.21.15.23:0/1629511181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb190114b40 con 0x55d9c1cf25d0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.870+0000 7fb18f400700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d03500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb18ea00700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb18000bcd0 tx=0x7fb180035f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c78e340777885a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb178c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb180035360 con 0x55d9c1cf25d0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb18800f040 con 0x55d9c1cf25d0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb18000be70 con 0x55d9c1cf25d0 2024-03-30T04:09:04.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1800354c0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb1a9a1ab80 10 client.17517 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d9c1acacd0 con 0x55d9c1cf25d0 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.871+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb18800f040 con 0x55d9c1cf25d0 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb18003e030 con 0x55d9c1cf25d0 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 10 client.17517 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 10 client.17517 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 20 client.17517 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d61b30 0x55d9c1d63f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 4 client.17517 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d9c1a93cf0 con 0x55d9c1d61b30 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb18e000700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d61b30 0x55d9c1d63f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:04.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb18e000700 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d61b30 0x55d9c1d63f20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7fcbda21418dfd84 server_cookie=b5a901fd7ee2134e in_seq=0 out_seq=0 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.872+0000 7fb178c00700 10 client.17517 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb178c00700 1 -- 172.21.15.23:0/1629511181 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d9c1a93cf0 con 0x55d9c1d61b30 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb178c00700 10 client.17517 handle_command_reply: tid=0 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 client.17517 shutdown 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 msgr2=0x55d9c1d02a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb18000bcd0 tx=0x7fb180035f50 comp rx=0 tx=0).stop 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 shutdown_connections 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d9c1d61b30 0x55d9c1d63f20 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d9c1cfddf0 0x55d9c1d03500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d9c1cf25d0 0x55d9c1d02a80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 --2- 172.21.15.23:0/1629511181 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9c1cf2ef0 0x55d9c1d02fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb178c00700 0 client.17517 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 >> 172.21.15.23:0/1629511181 conn(0x55d9c1c526a0 msgr2=0x55d9c1ce7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 shutdown_connections 2024-03-30T04:09:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 1 -- 172.21.15.23:0/1629511181 wait complete. 2024-03-30T04:09:04.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.873+0000 7fb1a9a1ab80 20 client.17517 trim_cache size 0 max 16384 2024-03-30T04:09:04.877 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 msgr2=0x7fb1740a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 0x7fb1740a37d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb18000b2e0 tx=0x7fb180005fd0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 msgr2=0x7fb19019dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00e9a0 tx=0x7fb17c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 shutdown_connections 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb190105c40 0x7fb1901a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1900697f0 0x7fb19019d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1740a1310 0x7fb1740a37d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 --2- 172.21.15.23:0/263914056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb190103740 0x7fb19019dd10 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 >> 172.21.15.23:0/263914056 conn(0x7fb190076d50 msgr2=0x7fb190102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 shutdown_connections 2024-03-30T04:09:04.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:04.874+0000 7fb195e00700 1 -- 172.21.15.23:0/263914056 wait complete. 2024-03-30T04:09:04.891 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 -- 172.21.15.23:0/421943374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a4103b80 msgr2=0x7fd3a4107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/421943374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a4103b80 0x7fd3a4107bc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009990 tx=0x7fd38c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 -- 172.21.15.23:0/421943374 shutdown_connections 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/421943374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41082f0 0x7fd3a410fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/421943374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a41031d0 0x7fd3a41035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/421943374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a4103b80 0x7fd3a4107bc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 -- 172.21.15.23:0/421943374 >> 172.21.15.23:0/421943374 conn(0x7fd3a40fea10 msgr2=0x7fd3a4100e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 -- 172.21.15.23:0/421943374 shutdown_connections 2024-03-30T04:09:05.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.120+0000 7fd3a8e00700 1 -- 172.21.15.23:0/421943374 wait complete. 2024-03-30T04:09:05.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.121+0000 7fd3a8e00700 1 Processor -- start 2024-03-30T04:09:05.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 -- start start 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 0x7fd3a40752c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a4075d40 0x7fd3a41a9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a40794c0 con 0x7fd3a4075d40 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a41aa050 con 0x7fd3a41031d0 2024-03-30T04:09:05.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.122+0000 7fd3a8e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a41aa210 con 0x7fd3a41082f0 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44970/0 (socket says 172.21.15.23:44970) 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 -- 172.21.15.23:0/3228261036 learned_addr learned my addr 172.21.15.23:0/3228261036 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 0x7fd3a40752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a4075d40 0x7fd3a41a9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 -- 172.21.15.23:0/3228261036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 msgr2=0x7fd3a40752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 0x7fd3a40752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 -- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a4075d40 msgr2=0x7fd3a41a9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a4075d40 0x7fd3a41a9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 -- 172.21.15.23:0/3228261036 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd38c0095f0 con 0x7fd3a41082f0 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 0x7fd3a40752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.123+0000 7fd399400700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd38c000c00 tx=0x7fd38c009510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.124+0000 7fd39b400700 1 -- 172.21.15.23:0/3228261036 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c01d070 con 0x7fd3a41082f0 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.124+0000 7fd39b400700 1 -- 172.21.15.23:0/3228261036 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c021410 con 0x7fd3a41082f0 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.124+0000 7fd39b400700 1 -- 172.21.15.23:0/3228261036 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c017650 con 0x7fd3a41082f0 2024-03-30T04:09:05.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.124+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a41aa490 con 0x7fd3a41082f0 2024-03-30T04:09:05.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.124+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a41aa860 con 0x7fd3a41082f0 2024-03-30T04:09:05.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.129+0000 7fd39b400700 1 -- 172.21.15.23:0/3228261036 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd38c00fb00 con 0x7fd3a41082f0 2024-03-30T04:09:05.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.130+0000 7fd3bc8b5b80 1 Processor -- start 2024-03-30T04:09:05.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.130+0000 7fd3bc8b5b80 1 -- start start 2024-03-30T04:09:05.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd39b400700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 0x7fd3880a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd39b400700 1 -- 172.21.15.23:0/3228261036 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd38c0ce340 con 0x7fd3a41082f0 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2e01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb22b990 con 0x55dacb2d25d0 2024-03-30T04:09:05.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb229350 con 0x55dacb2dddf0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb224c40 con 0x55dacb2d2ef0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44984/0 (socket says 172.21.15.23:44984) 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd399400700 1 -- 172.21.15.23:0/1335956780 learned_addr learned my addr 172.21.15.23:0/1335956780 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.131+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 0x7fd3880a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 -- 172.21.15.23:0/1335956780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 msgr2=0x55dacb2e01e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2e01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 -- 172.21.15.23:0/1335956780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 msgr2=0x55dacb2d29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2d29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 -- 172.21.15.23:0/1335956780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dacb0a63b0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2d29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:09:05.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd399400700 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0a2fc0 tx=0x7fd38c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc2f5fb523af5fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd393400700 1 -- 172.21.15.23:0/1335956780 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c09b800 con 0x55dacb2d2ef0 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd393400700 1 -- 172.21.15.23:0/1335956780 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c09b100 con 0x55dacb2d2ef0 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 0x7fd3880a37d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009380 tx=0x7fd39c00b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.132+0000 7fd393400700 1 -- 172.21.15.23:0/1335956780 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c021670 con 0x55dacb2d2ef0 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1335956780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 msgr2=0x55dacb2dd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd38c0a2fc0 tx=0x7fd38c009310 comp rx=0 tx=0).stop 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1335956780 shutdown_connections 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2e01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2d29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1335956780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dd8b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1335956780 >> 172.21.15.23:0/1335956780 conn(0x55dacb2326a0 msgr2=0x55dacb2cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.133+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1335956780 shutdown_connections 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.134+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1335956780 wait complete. 2024-03-30T04:09:05.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.134+0000 7fd3bc8b5b80 1 -- start start 2024-03-30T04:09:05.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.134+0000 7fd3bc8b5b80 1 Processor -- start 2024-03-30T04:09:05.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb1c0d80 con 0x55dacb2d25d0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb1c0a20 con 0x55dacb2d2ef0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb0a86a0 con 0x55dacb2dddf0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3bc8b5b80 10 client.0 _ll_register_callbacks cb 0x55dacb1ca470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3a2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3a2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44996/0 (socket says 172.21.15.23:44996) 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.135+0000 7fd3a2800700 1 -- 172.21.15.23:0/3606330460 learned_addr learned my addr 172.21.15.23:0/3606330460 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd399400700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 -- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 msgr2=0x55dacb2dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 -- 172.21.15.23:0/3606330460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 msgr2=0x55dacb2dd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2dd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 -- 172.21.15.23:0/3606330460 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a41aa860 con 0x55dacb2dddf0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd399400700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.136+0000 7fd3a2800700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd39c015b00 tx=0x7fd39c004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e411ca5307b0b112 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd387e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c00e050 con 0x55dacb2dddf0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd39c016c90 con 0x55dacb2dddf0 2024-03-30T04:09:05.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dacb1e5380 con 0x55dacb2dddf0 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd39c016e50 con 0x55dacb2dddf0 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd3bc8b5b80 10 client.27526 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.137+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55dacb052b60 con 0x55dacb2dddf0 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dacb1e5380 con 0x55dacb2dddf0 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd39c02a440 con 0x55dacb2dddf0 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 10 client.27526 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 10 client.27526 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 20 client.27526 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb2ed670 0x55dacb2efa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 4 client.27526 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dacb21d700 con 0x55dacb2ed670 2024-03-30T04:09:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.138+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb2ed670 0x55dacb2efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.139+0000 7fd387e00700 10 client.27526 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.139+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb2ed670 0x55dacb2efa90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f40a46cbcd36a76e server_cookie=2fbb12a0869c3827 in_seq=0 out_seq=0 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.139+0000 7fd387e00700 1 -- 172.21.15.23:0/3606330460 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dacb21d700 con 0x55dacb2ed670 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.139+0000 7fd387e00700 10 client.27526 handle_command_reply: tid=0 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 client.27526 shutdown 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 msgr2=0x55dacb2db2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd39c015b00 tx=0x7fd39c004ad0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 shutdown_connections 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb2ed670 0x55dacb2efa90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2db2a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2dd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/3606330460 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d2ef0 0x55dacb2dab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 >> 172.21.15.23:0/3606330460 conn(0x55dacb2326a0 msgr2=0x55dacb2d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd387e00700 0 client.27526 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 shutdown_connections 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/3606330460 wait complete. 2024-03-30T04:09:05.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.140+0000 7fd3bc8b5b80 20 client.27526 trim_cache size 0 max 16384 2024-03-30T04:09:05.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.154+0000 7fd3bc8b5b80 1 Processor -- start 2024-03-30T04:09:05.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.154+0000 7fd3bc8b5b80 1 -- start start 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2dd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2dc0c0 con 0x55dacb2d25d0 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2e4d20 con 0x55dacb2dddf0 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2d9150 con 0x55dacb2d2ef0 2024-03-30T04:09:05.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45008/0 (socket says 172.21.15.23:45008) 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 -- 172.21.15.23:0/172608803 learned_addr learned my addr 172.21.15.23:0/172608803 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3a1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47576/0 (socket says 172.21.15.23:47576) 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd3a2800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2dd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 -- 172.21.15.23:0/172608803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 msgr2=0x55dacb2dd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2dd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.155+0000 7fd399400700 1 -- 172.21.15.23:0/172608803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 msgr2=0x55dacb2e47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.156+0000 7fd399400700 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.156+0000 7fd399400700 1 -- 172.21.15.23:0/172608803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dacb1eaea0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.156+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:05.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.156+0000 7fd399400700 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd38c09ae00 tx=0x7fd38c096e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80d8a3499a301f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.157+0000 7fd393e00700 1 -- 172.21.15.23:0/172608803 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c00b2b0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.157+0000 7fd393e00700 1 -- 172.21.15.23:0/172608803 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c0a34f0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.157+0000 7fd393e00700 1 -- 172.21.15.23:0/172608803 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c0a10e0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/172608803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 msgr2=0x55dacb2c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd38c09ae00 tx=0x7fd38c096e40 comp rx=0 tx=0).stop 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/172608803 shutdown_connections 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2dddf0 0x55dacb2dd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d25d0 0x55dacb2e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/172608803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2d2ef0 0x55dacb2c7f70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/172608803 >> 172.21.15.23:0/172608803 conn(0x55dacb2326a0 msgr2=0x55dacb2f43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.158+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/172608803 shutdown_connections 2024-03-30T04:09:05.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.159+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/172608803 wait complete. 2024-03-30T04:09:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.159+0000 7fd3bc8b5b80 1 -- start start 2024-03-30T04:09:05.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 Processor -- start 2024-03-30T04:09:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d25d0 0x55dacb2e43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2e2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2e32a0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2e3410 con 0x55dacb2d25d0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dacb2e35d0 con 0x55dacb2dddf0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.160+0000 7fd3bc8b5b80 10 client.0 _ll_register_callbacks cb 0x55dacb1ca470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47588/0 (socket says 172.21.15.23:47588) 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd3a1e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d25d0 0x55dacb2e43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 -- 172.21.15.23:0/1392558958 learned_addr learned my addr 172.21.15.23:0/1392558958 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd3a2800700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2e2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 -- 172.21.15.23:0/1392558958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 msgr2=0x55dacb2e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2e2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 -- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d25d0 msgr2=0x55dacb2e43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d25d0 0x55dacb2e43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 -- 172.21.15.23:0/1392558958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39400e040 con 0x55dacb2d2ef0 2024-03-30T04:09:05.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd399400700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009ce0 tx=0x7fd38c09ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537042ca90bcb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.161+0000 7fd387e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd38c099240 con 0x55dacb2d2ef0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd38c09b100 con 0x55dacb2d2ef0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd38c09ef30 con 0x55dacb2d2ef0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd38c0217f0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 10 client.17529 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55dacb0a63b0 con 0x55dacb2d2ef0 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd38c09b100 con 0x55dacb2d2ef0 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd38c09b360 con 0x55dacb2d2ef0 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 10 client.17529 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 10 client.17529 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.162+0000 7fd3bc8b5b80 20 client.17529 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb342420 0x55dacb344810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd3bc8b5b80 4 client.17529 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dacb073cf0 con 0x55dacb342420 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb342420 0x55dacb344810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd3a1e00700 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb342420 0x55dacb344810 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d5d37c9107e3a0bc server_cookie=2b9a98c6b6de6ce7 in_seq=0 out_seq=0 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.163+0000 7fd387e00700 10 client.17529 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd387e00700 1 -- 172.21.15.23:0/1392558958 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dacb073cf0 con 0x55dacb342420 2024-03-30T04:09:05.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd387e00700 10 client.17529 handle_command_reply: tid=0 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 client.17529 shutdown 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 msgr2=0x55dacb2e2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009ce0 tx=0x7fd38c09ad40 comp rx=0 tx=0).stop 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 shutdown_connections 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55dacb342420 0x55dacb344810 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dacb2dddf0 0x55dacb2e2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55dacb2d25d0 0x55dacb2e43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 --2- 172.21.15.23:0/1392558958 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55dacb2d2ef0 0x55dacb2e2820 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd387e00700 0 client.17529 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 >> 172.21.15.23:0/1392558958 conn(0x55dacb2326a0 msgr2=0x55dacb2c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 shutdown_connections 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.164+0000 7fd3bc8b5b80 1 -- 172.21.15.23:0/1392558958 wait complete. 2024-03-30T04:09:05.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3bc8b5b80 20 client.17529 trim_cache size 0 max 16384 2024-03-30T04:09:05.166 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 msgr2=0x7fd3880a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 0x7fd3880a37d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009380 tx=0x7fd39c00b040 comp rx=0 tx=0).stop 2024-03-30T04:09:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 msgr2=0x7fd3a4075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd38c000c00 tx=0x7fd38c009510 comp rx=0 tx=0).stop 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 shutdown_connections 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3a4075d40 0x7fd3a41a9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd3880a1310 0x7fd3880a37d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3a41031d0 0x7fd3a40752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 --2- 172.21.15.23:0/3228261036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd3a41082f0 0x7fd3a4075800 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.165+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 >> 172.21.15.23:0/3228261036 conn(0x7fd3a40fea10 msgr2=0x7fd3a40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.166+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 shutdown_connections 2024-03-30T04:09:05.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.166+0000 7fd3a8e00700 1 -- 172.21.15.23:0/3228261036 wait complete. 2024-03-30T04:09:05.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:09:05.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 703931 10.92 MB/sec execute 222 sec latency 22.794 ms 2024-03-30T04:09:05.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 -- 172.21.15.23:0/2524237264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 msgr2=0x7f8f9410d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/2524237264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f9410d210 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009990 tx=0x7f8f80009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 -- 172.21.15.23:0/2524237264 shutdown_connections 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/2524237264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f9410d750 0x7f8f9410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/2524237264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f94100710 0x7f8f94100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/2524237264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f9410d210 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.398+0000 7f8f9ba00700 1 -- 172.21.15.23:0/2524237264 >> 172.21.15.23:0/2524237264 conn(0x7f8f94076d40 msgr2=0x7f8f94077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.399+0000 7f8f9ba00700 1 -- 172.21.15.23:0/2524237264 shutdown_connections 2024-03-30T04:09:05.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.399+0000 7f8f9ba00700 1 -- 172.21.15.23:0/2524237264 wait complete. 2024-03-30T04:09:05.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 Processor -- start 2024-03-30T04:09:05.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 -- start start 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f94199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 0x7f8f9419dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9419e3e0 con 0x7f8f9410d750 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9419e550 con 0x7f8f94101030 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f9ba00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9419e710 con 0x7f8f94100710 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f99000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f99000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45042/0 (socket says 172.21.15.23:45042) 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f99000700 1 -- 172.21.15.23:0/4020501484 learned_addr learned my addr 172.21.15.23:0/4020501484 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.400+0000 7f8f93e00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f94199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99a00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 0x7f8f9419dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 -- 172.21.15.23:0/4020501484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 msgr2=0x7f8f94199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f94199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 -- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 msgr2=0x7f8f9419dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 0x7f8f9419dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 -- 172.21.15.23:0/4020501484 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800095f0 con 0x7f8f94100710 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99a00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 0x7f8f9419dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f99000700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e9a0 tx=0x7f8f8800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f91600700 1 -- 172.21.15.23:0/4020501484 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88009790 con 0x7f8f94100710 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f91600700 1 -- 172.21.15.23:0/4020501484 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f8800cbd0 con 0x7f8f94100710 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9419e9f0 con 0x7f8f94100710 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.401+0000 7f8f91600700 1 -- 172.21.15.23:0/4020501484 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88018680 con 0x7f8f94100710 2024-03-30T04:09:05.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.402+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f941a6230 con 0x7f8f94100710 2024-03-30T04:09:05.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.404+0000 7f8f91600700 1 -- 172.21.15.23:0/4020501484 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8f8800cd40 con 0x7f8f94100710 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8faf4cfb80 1 Processor -- start 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8faf4cfb80 1 -- start start 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8f91600700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 0x7f8f7c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8f91600700 1 -- 172.21.15.23:0/4020501484 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8f88014070 con 0x7f8f94100710 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8f99a00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 0x7f8f7c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5dfe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.406+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 0x55a4c5e0c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5d57990 con 0x55a4c5e09df0 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5d55350 con 0x55a4c5dfeef0 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5d50c40 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5dfe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5dfe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45054/0 (socket says 172.21.15.23:45054) 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99000700 1 -- 172.21.15.23:0/1396594530 learned_addr learned my addr 172.21.15.23:0/1396594530 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99a00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 0x55a4c5e0c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 -- 172.21.15.23:0/1396594530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 msgr2=0x55a4c5dfe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5dfe9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 -- 172.21.15.23:0/1396594530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 msgr2=0x55a4c5e0c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 0x55a4c5e0c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f93e00700 1 -- 172.21.15.23:0/1396594530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4c5bd23b0 con 0x55a4c5dfeef0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99a00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 0x55a4c5e0c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.407+0000 7f8f99a00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 0x7f8f7c0a3750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009fd0 tx=0x7f8f8c007b00 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8f93e00700 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e098b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8f80006010 tx=0x7f8f80005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ead756c413ae865 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8f86000700 1 -- 172.21.15.23:0/1396594530 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8001d070 con 0x55a4c5dfeef0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8f86000700 1 -- 172.21.15.23:0/1396594530 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f8001e040 con 0x55a4c5dfeef0 2024-03-30T04:09:05.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8f86000700 1 -- 172.21.15.23:0/1396594530 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f80004760 con 0x55a4c5dfeef0 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1396594530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 msgr2=0x55a4c5e098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.408+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e098b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8f80006010 tx=0x7f8f80005ec0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1396594530 shutdown_connections 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5e09df0 0x55a4c5e0c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5dfe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1396594530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e098b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1396594530 >> 172.21.15.23:0/1396594530 conn(0x55a4c5d5e6a0 msgr2=0x55a4c5dfaa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1396594530 shutdown_connections 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1396594530 wait complete. 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 -- start start 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.409+0000 7f8faf4cfb80 1 Processor -- start 2024-03-30T04:09:05.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e06ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e07050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5cecd80 con 0x55a4c5dfeef0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5ceca20 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5bd46a0 con 0x55a4c5e09df0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8f99000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8faf4cfb80 10 client.0 _ll_register_callbacks cb 0x55a4c5cf6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8f99000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56734/0 (socket says 172.21.15.23:56734) 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8f99000700 1 -- 172.21.15.23:0/559242918 learned_addr learned my addr 172.21.15.23:0/559242918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8f99a00700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e07050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.410+0000 7f8f93e00700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e06ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 -- 172.21.15.23:0/559242918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 msgr2=0x55a4c5e07050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e07050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 -- 172.21.15.23:0/559242918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 msgr2=0x55a4c5e06ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e06ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 -- 172.21.15.23:0/559242918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94110740 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f99000700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e970 tx=0x7f8f8809a820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb0b77c0efb348d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f7aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8809de90 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8f880a24e0 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.411+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4c5d11380 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f88099550 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 10 client.27548 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a4c5b7eb60 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f88063070 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8f880b8080 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 10 client.27548 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 10 client.27548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 20 client.27548 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e19910 0x55a4c5e1bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 4 client.27548 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4c5d49700 con 0x55a4c5e19910 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.412+0000 7f8f93e00700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e19910 0x55a4c5e1bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.413+0000 7f8f93e00700 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e19910 0x55a4c5e1bd00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107718c43419c732 server_cookie=f6f42578d1c38f6d in_seq=0 out_seq=0 2024-03-30T04:09:05.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.413+0000 7f8f7aa00700 10 client.27548 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8f7aa00700 1 -- 172.21.15.23:0/559242918 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a4c5d49700 con 0x55a4c5e19910 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8f7aa00700 10 client.27548 handle_command_reply: tid=0 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 client.27548 shutdown 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 msgr2=0x55a4c5e09770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e970 tx=0x7f8f8809a820 comp rx=0 tx=0).stop 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 shutdown_connections 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e19910 0x55a4c5e1bd00 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e07050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e09770 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8f7aa00700 0 client.27548 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/559242918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e06ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.414+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 >> 172.21.15.23:0/559242918 conn(0x55a4c5d5e6a0 msgr2=0x55a4c5e006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.415+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 shutdown_connections 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.415+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/559242918 wait complete. 2024-03-30T04:09:05.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.415+0000 7f8faf4cfb80 20 client.27548 trim_cache size 0 max 16384 2024-03-30T04:09:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 Processor -- start 2024-03-30T04:09:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 -- start start 2024-03-30T04:09:05.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e07050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e10a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 0x55a4c5df3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5dff640 con 0x55a4c5dfeef0 2024-03-30T04:09:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5e08060 con 0x55a4c5e09df0 2024-03-30T04:09:05.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5dff850 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e10a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f99a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 0x55a4c5df3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f99000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e07050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e10a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47628/0 (socket says 172.21.15.23:47628) 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f99a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 0x55a4c5df3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56738/0 (socket says 172.21.15.23:56738) 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.429+0000 7f8f93e00700 1 -- 172.21.15.23:0/3714008342 learned_addr learned my addr 172.21.15.23:0/3714008342 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 -- 172.21.15.23:0/3714008342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 msgr2=0x55a4c5df3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 0x55a4c5df3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 -- 172.21.15.23:0/3714008342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 msgr2=0x55a4c5e10a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e10a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 -- 172.21.15.23:0/3714008342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4c5d16ea0 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f99000700 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e07050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8f88098fc0 tx=0x7f8f8800eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663773aaec39a186 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f86a00700 1 -- 172.21.15.23:0/3714008342 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880a24e0 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f86a00700 1 -- 172.21.15.23:0/3714008342 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f88063070 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.430+0000 7f8f86a00700 1 -- 172.21.15.23:0/3714008342 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88099550 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/3714008342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 msgr2=0x55a4c5e07050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e07050 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8f88098fc0 tx=0x7f8f8800eef0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/3714008342 shutdown_connections 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5e09df0 0x55a4c5df3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e07050 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/3714008342 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e10a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/3714008342 >> 172.21.15.23:0/3714008342 conn(0x55a4c5d5e6a0 msgr2=0x55a4c5df35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/3714008342 shutdown_connections 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/3714008342 wait complete. 2024-03-30T04:09:05.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.431+0000 7f8faf4cfb80 1 -- start start 2024-03-30T04:09:05.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 Processor -- start 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e0e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e6c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5e6cc70 con 0x55a4c5dfeef0 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5e6cdb0 con 0x55a4c5dfe5d0 2024-03-30T04:09:05.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4c5e6cef0 con 0x55a4c5e09df0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8faf4cfb80 10 client.0 _ll_register_callbacks cb 0x55a4c5cf6470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8f93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8f99a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e6c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8f93e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47644/0 (socket says 172.21.15.23:47644) 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8f93e00700 1 -- 172.21.15.23:0/1819151828 learned_addr learned my addr 172.21.15.23:0/1819151828 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.432+0000 7f8f99000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e0e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 -- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 msgr2=0x55a4c5e0e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e0e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 -- 172.21.15.23:0/1819151828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 msgr2=0x55a4c5e0ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 -- 172.21.15.23:0/1819151828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4c5b7eb60 con 0x55a4c5e09df0 2024-03-30T04:09:05.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f93e00700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f99a00700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e6c730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8f94101d80 tx=0x7f8f8c004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=676af93822756483 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f7aa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c00e050 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f8001e040 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8f8c029870 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f8c029a30 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8faf4cfb80 10 client.27550 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.433+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a4c5bd6cd0 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f8001e040 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8f8c017cf0 con 0x55a4c5e09df0 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 10 client.27550 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 10 client.27550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 20 client.27550 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:05.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e6e710 0x55a4c5e70b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 4 client.27550 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a4c5b9fcf0 con 0x55a4c5e6e710 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.434+0000 7f8f99000700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e6e710 0x55a4c5e70b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8f99000700 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e6e710 0x55a4c5e70b30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66ad888a7b0a091a server_cookie=d62bc0dc4c5a016a in_seq=0 out_seq=0 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8f7aa00700 10 client.27550 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8f7aa00700 1 -- 172.21.15.23:0/1819151828 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a4c5b9fcf0 con 0x55a4c5e6e710 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8f7aa00700 10 client.27550 handle_command_reply: tid=0 2024-03-30T04:09:05.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 client.27550 shutdown 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 msgr2=0x55a4c5e6c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e6c730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8f94101d80 tx=0x7f8f8c004b40 comp rx=0 tx=0).stop 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 shutdown_connections 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a4c5e6e710 0x55a4c5e70b30 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a4c5e09df0 0x55a4c5e6c730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a4c5dfe5d0 0x55a4c5e0e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8f7aa00700 0 client.27550 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 --2- 172.21.15.23:0/1819151828 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a4c5dfeef0 0x55a4c5e0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 >> 172.21.15.23:0/1819151828 conn(0x55a4c5d5e6a0 msgr2=0x55a4c5e05340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 shutdown_connections 2024-03-30T04:09:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 1 -- 172.21.15.23:0/1819151828 wait complete. 2024-03-30T04:09:05.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.435+0000 7f8faf4cfb80 20 client.27550 trim_cache size 0 max 16384 2024-03-30T04:09:05.440 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 msgr2=0x7f8f7c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 0x7f8f7c0a3750 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c009fd0 tx=0x7f8f8c007b00 comp rx=0 tx=0).stop 2024-03-30T04:09:05.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 msgr2=0x7f8f94199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8f8800e9a0 tx=0x7f8f8800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 shutdown_connections 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f9410d750 0x7f8f9419dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8f94100710 0x7f8f94199510 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8f7c0a1290 0x7f8f7c0a3750 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 --2- 172.21.15.23:0/4020501484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f94101030 0x7f8f94199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 >> 172.21.15.23:0/4020501484 conn(0x7f8f94076d40 msgr2=0x7f8f9410ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 shutdown_connections 2024-03-30T04:09:05.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:05.436+0000 7f8f9ba00700 1 -- 172.21.15.23:0/4020501484 wait complete. 2024-03-30T04:09:06.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 706129 10.92 MB/sec execute 223 sec latency 19.155 ms 2024-03-30T04:09:06.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:06 smithi154 ceph-mon[79161]: pgmap v219: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-03-30T04:09:06.454 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:09:06.454 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:09:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:06 smithi193 ceph-mon[79581]: pgmap v219: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-03-30T04:09:06.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:06 smithi023 ceph-mon[38254]: pgmap v219: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 195 op/s 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 -- 172.21.15.23:0/1304639965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 msgr2=0x7fa6c0101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1304639965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c0101350 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009990 tx=0x7fa6b0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 -- 172.21.15.23:0/1304639965 shutdown_connections 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1304639965 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1304639965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c0101350 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1304639965 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c0101920 0x7fa6c0105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 -- 172.21.15.23:0/1304639965 >> 172.21.15.23:0/1304639965 conn(0x7fa6c00fc7f0 msgr2=0x7fa6c00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 -- 172.21.15.23:0/1304639965 shutdown_connections 2024-03-30T04:09:06.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.665+0000 7fa6c5800700 1 -- 172.21.15.23:0/1304639965 wait complete. 2024-03-30T04:09:06.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.666+0000 7fa6c5800700 1 Processor -- start 2024-03-30T04:09:06.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 -- start start 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c0199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c019dcc0 0x7fa6c019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c019e740 con 0x7fa6c0106090 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c019e8b0 con 0x7fa6c0100f70 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6c5800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c019ea70 con 0x7fa6c019dcc0 2024-03-30T04:09:06.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.667+0000 7fa6be000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c0199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56770/0 (socket says 172.21.15.23:56770) 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6bea00700 1 -- 172.21.15.23:0/1221295305 learned_addr learned my addr 172.21.15.23:0/1221295305 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6bf400700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c019dcc0 0x7fa6c019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 -- 172.21.15.23:0/1221295305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c019dcc0 msgr2=0x7fa6c019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c019dcc0 0x7fa6c019e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 -- 172.21.15.23:0/1221295305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 msgr2=0x7fa6c01993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 -- 172.21.15.23:0/1221295305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00095f0 con 0x7fa6c0106090 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6bea00700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.668+0000 7fa6be000700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c0199920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00e9c0 tx=0x7fa6ac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.669+0000 7fa6b7400700 1 -- 172.21.15.23:0/1221295305 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00c9c0 con 0x7fa6c0106090 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.669+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c01a5fd0 con 0x7fa6c0106090 2024-03-30T04:09:06.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.669+0000 7fa6b7400700 1 -- 172.21.15.23:0/1221295305 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6ac010430 con 0x7fa6c0106090 2024-03-30T04:09:06.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.669+0000 7fa6b7400700 1 -- 172.21.15.23:0/1221295305 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac018680 con 0x7fa6c0106090 2024-03-30T04:09:06.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.669+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c01a6450 con 0x7fa6c0106090 2024-03-30T04:09:06.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.672+0000 7fa6b7400700 1 -- 172.21.15.23:0/1221295305 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa6ac00cb20 con 0x7fa6c0106090 2024-03-30T04:09:06.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.673+0000 7fa6d93f1b80 1 Processor -- start 2024-03-30T04:09:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.673+0000 7fa6d93f1b80 1 -- start start 2024-03-30T04:09:06.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.673+0000 7fa6b7400700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 0x7fa6a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6b7400700 1 -- 172.21.15.23:0/1221295305 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6ac014070 con 0x7fa6c0106090 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c4c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c4f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2b9aa10 con 0x5648e2c41f70 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2b983d0 con 0x5648e2c4ce70 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2b93cc0 con 0x5648e2c41650 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bf400700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 0x7fa6a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6be000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c4c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45100/0 (socket says 172.21.15.23:45100) 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 -- 172.21.15.23:0/733250163 learned_addr learned my addr 172.21.15.23:0/733250163 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 -- 172.21.15.23:0/733250163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 msgr2=0x5648e2c4f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bf400700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c4f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c4f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 -- 172.21.15.23:0/733250163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 msgr2=0x5648e2c4c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c4c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bea00700 1 -- 172.21.15.23:0/733250163 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648e2a15430 con 0x5648e2c41650 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6be000700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c4c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.674+0000 7fa6bf400700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c4f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.675+0000 7fa6bea00700 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b660 tx=0x7fa6b000b690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14fa8270c129ded6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.675+0000 7fa6bf400700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 0x7fa6a40a3750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800f660 tx=0x7fa6b8009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.675+0000 7fa6abe00700 1 -- 172.21.15.23:0/733250163 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b001d070 con 0x5648e2c41650 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.675+0000 7fa6abe00700 1 -- 172.21.15.23:0/733250163 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6b001e040 con 0x5648e2c41650 2024-03-30T04:09:06.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.675+0000 7fa6abe00700 1 -- 172.21.15.23:0/733250163 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0004da0 con 0x5648e2c41650 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/733250163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 msgr2=0x5648e2c41a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b660 tx=0x7fa6b000b690 comp rx=0 tx=0).stop 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/733250163 shutdown_connections 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c4f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c41a30 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/733250163 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c4c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/733250163 >> 172.21.15.23:0/733250163 conn(0x5648e2ba1720 msgr2=0x5648e2c3db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/733250163 shutdown_connections 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.676+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/733250163 wait complete. 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.677+0000 7fa6d93f1b80 1 -- start start 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.677+0000 7fa6d93f1b80 1 Processor -- start 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c49b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c4ce70 0x5648e2c4a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2b2fe00 con 0x5648e2c41f70 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2b2faa0 con 0x5648e2c41650 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2a17720 con 0x5648e2c4ce70 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6d93f1b80 10 client.0 _ll_register_callbacks cb 0x5648e2b394f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:06.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56792/0 (socket says 172.21.15.23:56792) 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 -- 172.21.15.23:0/4096476052 learned_addr learned my addr 172.21.15.23:0/4096476052 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6be000700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c49b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bf400700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c4ce70 0x5648e2c4a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 -- 172.21.15.23:0/4096476052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c4ce70 msgr2=0x5648e2c4a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c4ce70 0x5648e2c4a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 -- 172.21.15.23:0/4096476052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 msgr2=0x5648e2c49b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c49b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6bea00700 1 -- 172.21.15.23:0/4096476052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c01a6450 con 0x5648e2c41650 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.678+0000 7fa6be000700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c49b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6bea00700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b630 tx=0x7fa6b00229a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b40b7bc8ebac68f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b001d030 con 0x5648e2c41650 2024-03-30T04:09:06.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648e2b54400 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6b0022e10 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6b0005940 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6d93f1b80 10 client.27563 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fa6c010e480 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.679+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5648e2b54400 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa6b002c400 con 0x5648e2c41650 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 10 client.27563 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 10 client.27563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 20 client.27563 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2c5c4b0 0x5648e2c5e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 4 client.27563 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648e2b8c780 con 0x5648e2c5c4b0 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6be000700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2c5c4b0 0x5648e2c5e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6be000700 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2c5c4b0 0x5648e2c5e8a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e32f999a87603155 server_cookie=c5d134733c535db in_seq=0 out_seq=0 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.680+0000 7fa6a8c00700 10 client.27563 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6a8c00700 1 -- 172.21.15.23:0/4096476052 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5648e2b8c780 con 0x5648e2c5c4b0 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6a8c00700 10 client.27563 handle_command_reply: tid=0 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 client.27563 shutdown 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 msgr2=0x5648e2c4c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa6b000b630 tx=0x7fa6b00229a0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 shutdown_connections 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2c5c4b0 0x5648e2c5e8a0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c4ce70 0x5648e2c4a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c41650 0x5648e2c4c850 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/4096476052 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c49b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6a8c00700 0 client.27563 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.681+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 >> 172.21.15.23:0/4096476052 conn(0x5648e2ba1720 msgr2=0x5648e2c436a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.682+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 shutdown_connections 2024-03-30T04:09:06.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.682+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/4096476052 wait complete. 2024-03-30T04:09:06.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.682+0000 7fa6d93f1b80 20 client.27563 trim_cache size 0 max 16384 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 Processor -- start 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 -- start start 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c48410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c48950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2c428d0 con 0x5648e2c41f70 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2c48e90 con 0x5648e2c4ce70 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.700+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2c4b110 con 0x5648e2c41650 2024-03-30T04:09:06.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c48950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45124/0 (socket says 172.21.15.23:45124) 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c48950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56808/0 (socket says 172.21.15.23:56808) 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6be000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c48410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bf400700 1 -- 172.21.15.23:0/1817759945 learned_addr learned my addr 172.21.15.23:0/1817759945 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 -- 172.21.15.23:0/1817759945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 msgr2=0x5648e2c48950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c48950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 -- 172.21.15.23:0/1817759945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 msgr2=0x5648e2c48410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c48410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 -- 172.21.15.23:0/1817759945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648e2c634a0 con 0x5648e2c41650 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6be000700 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c48410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6bea00700 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0004730 tx=0x7fa6b0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2159ec2d27d9f184 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6b4c00700 1 -- 172.21.15.23:0/1817759945 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b001d070 con 0x5648e2c41650 2024-03-30T04:09:06.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6b4c00700 1 -- 172.21.15.23:0/1817759945 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6b0005bc0 con 0x5648e2c41650 2024-03-30T04:09:06.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.701+0000 7fa6b4c00700 1 -- 172.21.15.23:0/1817759945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b000f9e0 con 0x5648e2c41650 2024-03-30T04:09:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/1817759945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 msgr2=0x5648e2c53a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0004730 tx=0x7fa6b0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/1817759945 shutdown_connections 2024-03-30T04:09:06.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c48950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41650 0x5648e2c53a90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/1817759945 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41f70 0x5648e2c48410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/1817759945 >> 172.21.15.23:0/1817759945 conn(0x5648e2ba1720 msgr2=0x5648e2c44e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/1817759945 shutdown_connections 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.702+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/1817759945 wait complete. 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 -- start start 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 Processor -- start 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 0x5648e2c523a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 0x5648e2c528e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c52e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2cb0290 con 0x5648e2c41650 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2cb03d0 con 0x5648e2c4ce70 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648e2cb0590 con 0x5648e2c41f70 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.703+0000 7fa6d93f1b80 10 client.0 _ll_register_callbacks cb 0x5648e2b394f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:06.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 0x5648e2c523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 0x5648e2c528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6bea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 0x5648e2c523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47688/0 (socket says 172.21.15.23:47688) 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6bea00700 1 -- 172.21.15.23:0/959444174 learned_addr learned my addr 172.21.15.23:0/959444174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6bf400700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c52e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 -- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 msgr2=0x5648e2c52e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c52e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 -- 172.21.15.23:0/959444174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 msgr2=0x5648e2c523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 0x5648e2c523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 -- 172.21.15.23:0/959444174 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c01a5fd0 con 0x5648e2c41f70 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6be000700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 0x5648e2c528e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0ca510 tx=0x7fa6ac099b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fdd3053e5ad63ce server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6a8c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:06.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.704+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac09dd00 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6ac0a2350 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648e29e2d70 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6ac0b82b0 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 10 client.27568 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fa6c010e480 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5648e29e2d70 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa6ac099450 con 0x5648e2c41f70 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 10 client.27568 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 10 client.27568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 20 client.27568 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2cb1d60 0x5648e2cb4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 4 client.27568 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5648e2a17d60 con 0x5648e2cb1d60 2024-03-30T04:09:06.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.705+0000 7fa6bea00700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2cb1d60 0x5648e2cb4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.706+0000 7fa6a8c00700 10 client.27568 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.706+0000 7fa6bea00700 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2cb1d60 0x5648e2cb4150 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79d0efdf21267a4e server_cookie=654d6dd1e795a336 in_seq=0 out_seq=0 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6a8c00700 1 -- 172.21.15.23:0/959444174 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5648e2a17d60 con 0x5648e2cb1d60 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6a8c00700 10 client.27568 handle_command_reply: tid=0 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 client.27568 shutdown 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 msgr2=0x5648e2c528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 0x5648e2c528e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac0ca510 tx=0x7fa6ac099b30 comp rx=0 tx=0).stop 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 shutdown_connections 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648e2c4ce70 0x5648e2c52e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648e2c41650 0x5648e2c523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648e2cb1d60 0x5648e2cb4150 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 --2- 172.21.15.23:0/959444174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648e2c41f70 0x5648e2c528e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 >> 172.21.15.23:0/959444174 conn(0x5648e2ba1720 msgr2=0x5648e2c482a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.707+0000 7fa6a8c00700 0 client.27568 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 shutdown_connections 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6d93f1b80 1 -- 172.21.15.23:0/959444174 wait complete. 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6d93f1b80 20 client.27568 trim_cache size 0 max 16384 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "a13c4472-d554-43d7-8450-ad14346a508c", 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:09:06.711 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 msgr2=0x7fa6a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 0x7fa6a40a3750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa6b800f660 tx=0x7fa6b8009920 comp rx=0 tx=0).stop 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 msgr2=0x7fa6c0199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.708+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c0199920 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00e9c0 tx=0x7fa6ac00ecd0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 shutdown_connections 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c019dcc0 0x7fa6c019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6a40a1290 0x7fa6a40a3750 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c0100f70 0x7fa6c01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 --2- 172.21.15.23:0/1221295305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6c0106090 0x7fa6c0199920 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 >> 172.21.15.23:0/1221295305 conn(0x7fa6c00fc7f0 msgr2=0x7fa6c00fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 shutdown_connections 2024-03-30T04:09:06.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.709+0000 7fa6c5800700 1 -- 172.21.15.23:0/1221295305 wait complete. 2024-03-30T04:09:06.725 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:09:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 -- 172.21.15.23:0/679683633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918101760 msgr2=0x7f79181057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 --2- 172.21.15.23:0/679683633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918101760 0x7f79181057a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7904009990 tx=0x7f7904009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 -- 172.21.15.23:0/679683633 shutdown_connections 2024-03-30T04:09:06.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 --2- 172.21.15.23:0/679683633 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7918105ed0 0x7f791810da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 --2- 172.21.15.23:0/679683633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 --2- 172.21.15.23:0/679683633 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918101760 0x7f79181057a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 -- 172.21.15.23:0/679683633 >> 172.21.15.23:0/679683633 conn(0x7f79180fc670 msgr2=0x7f79180fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 -- 172.21.15.23:0/679683633 shutdown_connections 2024-03-30T04:09:06.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.936+0000 7f7917e00700 1 -- 172.21.15.23:0/679683633 wait complete. 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.937+0000 7f7917e00700 1 Processor -- start 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.937+0000 7f7917e00700 1 -- start start 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918105ed0 0x7f79181975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f791819b8e0 0x7f791819bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791819c360 con 0x7f791819b8e0 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791819c4d0 con 0x7f7918105ed0 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791819c690 con 0x7f7918100db0 2024-03-30T04:09:06.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918105ed0 0x7f79181975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45152/0 (socket says 172.21.15.23:45152) 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 -- 172.21.15.23:0/1986811525 learned_addr learned my addr 172.21.15.23:0/1986811525 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7917400700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f791819b8e0 0x7f791819bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 -- 172.21.15.23:0/1986811525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918105ed0 msgr2=0x7f79181975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918105ed0 0x7f79181975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 -- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f791819b8e0 msgr2=0x7f791819bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f791819b8e0 0x7f791819bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.938+0000 7f7916a00700 1 -- 172.21.15.23:0/1986811525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040095f0 con 0x7f7918100db0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f7916a00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f790800e9a0 tx=0x7f790800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f790f400700 1 -- 172.21.15.23:0/1986811525 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908009790 con 0x7f7918100db0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f790f400700 1 -- 172.21.15.23:0/1986811525 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7908010430 con 0x7f7918100db0 2024-03-30T04:09:06.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79181a3bf0 con 0x7f7918100db0 2024-03-30T04:09:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f790f400700 1 -- 172.21.15.23:0/1986811525 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7908018680 con 0x7f7918100db0 2024-03-30T04:09:06.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.939+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79181a40d0 con 0x7f7918100db0 2024-03-30T04:09:06.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.944+0000 7f790f400700 1 -- 172.21.15.23:0/1986811525 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f79080105a0 con 0x7f7918100db0 2024-03-30T04:09:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.945+0000 7f79311c6b80 1 Processor -- start 2024-03-30T04:09:06.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.945+0000 7f79311c6b80 1 -- start start 2024-03-30T04:09:06.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.945+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ec5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.945+0000 7f790f400700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 0x7f78fc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f79311c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8e1e990 con 0x55a9a8ec57a0 2024-03-30T04:09:06.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f79311c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8e1c350 con 0x55a9a8ed0fc0 2024-03-30T04:09:06.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f79311c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8e17c40 con 0x55a9a8ec60c0 2024-03-30T04:09:06.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f790f400700 1 -- 172.21.15.23:0/1986811525 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7908014070 con 0x7f7918100db0 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916a00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 0x7f78fc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7917400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ec5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7917400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45158/0 (socket says 172.21.15.23:45158) 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 -- 172.21.15.23:0/532944685 learned_addr learned my addr 172.21.15.23:0/532944685 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 -- 172.21.15.23:0/532944685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ed0fc0 msgr2=0x55a9a8ed33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 -- 172.21.15.23:0/532944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 msgr2=0x55a9a8ec5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ec5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7916000700 1 -- 172.21.15.23:0/532944685 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9a8c993b0 con 0x55a9a8ec60c0 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.946+0000 7f7917400700 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ec5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:06.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.947+0000 7f7916a00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 0x7f78fc0a3750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79080964b0 tx=0x7f790800c480 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.947+0000 7f7916000700 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7904009990 tx=0x7f7904005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0673dde32832de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.947+0000 7f7903400700 1 -- 172.21.15.23:0/532944685 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790401d070 con 0x55a9a8ec60c0 2024-03-30T04:09:06.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.947+0000 7f7903400700 1 -- 172.21.15.23:0/532944685 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f790401e040 con 0x55a9a8ec60c0 2024-03-30T04:09:06.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.947+0000 7f7903400700 1 -- 172.21.15.23:0/532944685 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904004880 con 0x55a9a8ec60c0 2024-03-30T04:09:06.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 -- 172.21.15.23:0/532944685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 msgr2=0x55a9a8ed0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7904009990 tx=0x7f7904005ec0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 -- 172.21.15.23:0/532944685 shutdown_connections 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ec5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 --2- 172.21.15.23:0/532944685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed0a80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 -- 172.21.15.23:0/532944685 >> 172.21.15.23:0/532944685 conn(0x55a9a8e256a0 msgr2=0x55a9a8ec1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 -- 172.21.15.23:0/532944685 shutdown_connections 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.948+0000 7f79311c6b80 1 -- 172.21.15.23:0/532944685 wait complete. 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.949+0000 7f79311c6b80 1 -- start start 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.949+0000 7f79311c6b80 1 Processor -- start 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8db3d80 con 0x55a9a8ec60c0 2024-03-30T04:09:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8db3a20 con 0x55a9a8ec57a0 2024-03-30T04:09:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8c9b6a0 con 0x55a9a8ed0fc0 2024-03-30T04:09:06.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.950+0000 7f79311c6b80 10 client.0 _ll_register_callbacks cb 0x55a9a8dbd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56856/0 (socket says 172.21.15.23:56856) 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 -- 172.21.15.23:0/753614362 learned_addr learned my addr 172.21.15.23:0/753614362 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7916000700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 -- 172.21.15.23:0/753614362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 msgr2=0x55a9a8ece0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 -- 172.21.15.23:0/753614362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec60c0 msgr2=0x55a9a8ecdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.951+0000 7f7917400700 1 -- 172.21.15.23:0/753614362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791810e2c0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f7917400700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7910017d60 tx=0x7f7910009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bdcc8c5a88fc9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f78f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791000e070 con 0x55a9a8ec57a0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7910004db0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9a8dd8380 con 0x55a9a8ec57a0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79100170d0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f79311c6b80 10 client.27581 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:06.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.952+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a9a8c45b60 con 0x55a9a8ec57a0 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a9a8dd8380 con 0x55a9a8ec57a0 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7910015640 con 0x55a9a8ec57a0 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 10 client.27581 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 10 client.27581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 20 client.27581 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:06.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8ee0a30 0x55a9a8ee2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 4 client.27581 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9a8e10700 con 0x55a9a8ee0a30 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.953+0000 7f7916000700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8ee0a30 0x55a9a8ee2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.954+0000 7f7916000700 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8ee0a30 0x55a9a8ee2e50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbaabb051d10adc8 server_cookie=3383f890ec58c50e in_seq=0 out_seq=0 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.954+0000 7f78f7e00700 10 client.27581 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.954+0000 7f78f7e00700 1 -- 172.21.15.23:0/753614362 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a9a8e10700 con 0x55a9a8ee0a30 2024-03-30T04:09:06.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.954+0000 7f78f7e00700 10 client.27581 handle_command_reply: tid=0 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.954+0000 7f79311c6b80 1 client.27581 shutdown 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 msgr2=0x55a9a8ed0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7910017d60 tx=0x7f7910009920 comp rx=0 tx=0).stop 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 shutdown_connections 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8ee0a30 0x55a9a8ee2e50 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed0890 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 --2- 172.21.15.23:0/753614362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f78f7e00700 0 client.27581 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 >> 172.21.15.23:0/753614362 conn(0x55a9a8e256a0 msgr2=0x55a9a8ec7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 shutdown_connections 2024-03-30T04:09:06.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 1 -- 172.21.15.23:0/753614362 wait complete. 2024-03-30T04:09:06.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.955+0000 7f79311c6b80 20 client.27581 trim_cache size 0 max 16384 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.976+0000 7f79311c6b80 1 Processor -- start 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.976+0000 7f79311c6b80 1 -- start start 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ec2a10 con 0x55a9a8ec57a0 2024-03-30T04:09:06.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ec7310 con 0x55a9a8ec60c0 2024-03-30T04:09:06.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f79311c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ec6a20 con 0x55a9a8ed0fc0 2024-03-30T04:09:06.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7916000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7916000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56860/0 (socket says 172.21.15.23:56860) 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7916000700 1 -- 172.21.15.23:0/593097352 learned_addr learned my addr 172.21.15.23:0/593097352 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 -- 172.21.15.23:0/593097352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 msgr2=0x55a9a8ece080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 -- 172.21.15.23:0/593097352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 msgr2=0x55a9a8ecdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.977+0000 7f7917400700 1 -- 172.21.15.23:0/593097352 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9a8dddea0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.978+0000 7f7917400700 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7c40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7910009010 tx=0x7f7910017cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb7894dae5261b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.978+0000 7f7903e00700 1 -- 172.21.15.23:0/593097352 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791000e070 con 0x55a9a8ec57a0 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.978+0000 7f7903e00700 1 -- 172.21.15.23:0/593097352 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f791000f040 con 0x55a9a8ec57a0 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.978+0000 7f7903e00700 1 -- 172.21.15.23:0/593097352 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7910004ea0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- 172.21.15.23:0/593097352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 msgr2=0x55a9a8ed7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7c40 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f7910009010 tx=0x7f7910017cb0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- 172.21.15.23:0/593097352 shutdown_connections 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ece080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7c40 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 --2- 172.21.15.23:0/593097352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ecdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- 172.21.15.23:0/593097352 >> 172.21.15.23:0/593097352 conn(0x55a9a8e256a0 msgr2=0x55a9a8ecd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- 172.21.15.23:0/593097352 shutdown_connections 2024-03-30T04:09:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- 172.21.15.23:0/593097352 wait complete. 2024-03-30T04:09:06.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.979+0000 7f79311c6b80 1 -- start start 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 Processor -- start 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ed6640 con 0x55a9a8ed0fc0 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ed67b0 con 0x55a9a8ec57a0 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a8ed6970 con 0x55a9a8ec60c0 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f79311c6b80 10 client.0 _ll_register_callbacks cb 0x55a9a8dbd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:06.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f7916000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f7917400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f7916000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:45204/0 (socket says 172.21.15.23:45204) 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f7916000700 1 -- 172.21.15.23:0/3036199405 learned_addr learned my addr 172.21.15.23:0/3036199405 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.980+0000 7f7917400700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 -- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 msgr2=0x55a9a8ed7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 -- 172.21.15.23:0/3036199405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ed0fc0 msgr2=0x55a9a8ed5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 -- 172.21.15.23:0/3036199405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a9a8c45b60 con 0x55a9a8ec60c0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7917400700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f7916000700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7904009350 tx=0x7f790401a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38b62e4e1e5c4baa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f78f7e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904032040 con 0x55a9a8ec60c0 2024-03-30T04:09:06.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7904005320 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f790401e040 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.981+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f790401d040 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 10 client.27586 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a9a8c9dcd0 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f790401e040 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7904034b70 con 0x55a9a8ec60c0 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 10 client.27586 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 10 client.27586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 20 client.27586 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8f357d0 0x55a9a8f37bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 4 client.27586 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:09:06.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a9a8c66cf0 con 0x55a9a8f357d0 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.982+0000 7f7917400700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8f357d0 0x55a9a8f37bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f78f7e00700 10 client.27586 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f7917400700 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8f357d0 0x55a9a8f37bf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d34b99f743a79145 server_cookie=6d2ab26a67a5ca82 in_seq=0 out_seq=0 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f78f7e00700 1 -- 172.21.15.23:0/3036199405 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a9a8c66cf0 con 0x55a9a8f357d0 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f78f7e00700 10 client.27586 handle_command_reply: tid=0 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 client.27586 shutdown 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 msgr2=0x55a9a8ed7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7904009350 tx=0x7f790401a040 comp rx=0 tx=0).stop 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 shutdown_connections 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a9a8ed0fc0 0x55a9a8ed5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a9a8ec57a0 0x55a9a8ed7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a9a8f357d0 0x55a9a8f37bf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 --2- 172.21.15.23:0/3036199405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a9a8ec60c0 0x55a9a8ed7db0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 >> 172.21.15.23:0/3036199405 conn(0x55a9a8e256a0 msgr2=0x55a9a8ecc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f78f7e00700 0 client.27586 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 shutdown_connections 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.983+0000 7f79311c6b80 1 -- 172.21.15.23:0/3036199405 wait complete. 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f79311c6b80 20 client.27586 trim_cache size 0 max 16384 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "a13c4472-d554-43d7-8450-ad14346a508c": { 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "a13c4472-d554-43d7-8450-ad14346a508c", 2024-03-30T04:09:06.988 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:09:06.989 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:09:06.989 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:09:06.989 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 msgr2=0x7f78fc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 0x7f78fc0a3750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f79080964b0 tx=0x7f790800c480 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 msgr2=0x7f7918197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f790800e9a0 tx=0x7f790800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 shutdown_connections 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f791819b8e0 0x7f791819bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7918100db0 0x7f7918197070 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f78fc0a1290 0x7f78fc0a3750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 --2- 172.21.15.23:0/1986811525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7918105ed0 0x7f79181975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 >> 172.21.15.23:0/1986811525 conn(0x7f79180fc670 msgr2=0x7f79180fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 shutdown_connections 2024-03-30T04:09:06.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:06.984+0000 7f7917e00700 1 -- 172.21.15.23:0/1986811525 wait complete. 2024-03-30T04:09:07.003 INFO:tasks.cephfs.filesystem:scrub status for tag:a13c4472-d554-43d7-8450-ad14346a508c - {'path': '/', 'tag': 'a13c4472-d554-43d7-8450-ad14346a508c', 'options': 'recursive,force'} 2024-03-30T04:09:07.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 707474 10.91 MB/sec execute 224 sec latency 95.520 ms 2024-03-30T04:09:08.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 709467 10.90 MB/sec execute 225 sec latency 19.012 ms 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: pgmap v220: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:09:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:08 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: pgmap v220: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:09:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:09:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:09:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:09:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:08 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: pgmap v220: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:09:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:09:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:09:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:09:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:08 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:09.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 711033 10.89 MB/sec execute 226 sec latency 22.732 ms 2024-03-30T04:09:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:09 smithi193 ceph-mon[79581]: pgmap v221: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:09:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:09 smithi193 ceph-mon[79581]: osdmap e92: 12 total, 12 up, 12 in 2024-03-30T04:09:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:09 smithi023 ceph-mon[38254]: pgmap v221: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:09:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:09 smithi023 ceph-mon[38254]: osdmap e92: 12 total, 12 up, 12 in 2024-03-30T04:09:09.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:09 smithi154 ceph-mon[79161]: pgmap v221: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 276 op/s 2024-03-30T04:09:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:09 smithi154 ceph-mon[79161]: osdmap e92: 12 total, 12 up, 12 in 2024-03-30T04:09:10.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 713216 10.90 MB/sec execute 227 sec latency 18.221 ms 2024-03-30T04:09:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:10 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:09:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:10 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:09:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:10 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:09:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:10 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:09:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:10 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:09:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:10 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:09:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:10 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:09:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:10 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:09:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:10 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:09:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:10 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:09:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:10 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:09:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:10 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:09:11.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 714754 10.88 MB/sec execute 228 sec latency 21.958 ms 2024-03-30T04:09:12.209 INFO:tasks.workunit.client.0.smithi023.stdout: 1 716932 10.89 MB/sec execute 229 sec latency 36.731 ms 2024-03-30T04:09:12.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:12 smithi154 ceph-mon[79161]: pgmap v223: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 234 op/s 2024-03-30T04:09:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:12 smithi193 ceph-mon[79581]: pgmap v223: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 234 op/s 2024-03-30T04:09:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:12 smithi023 ceph-mon[38254]: pgmap v223: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 234 op/s 2024-03-30T04:09:13.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 718221 10.87 MB/sec execute 230 sec latency 51.554 ms 2024-03-30T04:09:14.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 720293 10.88 MB/sec execute 231 sec latency 19.058 ms 2024-03-30T04:09:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:14 smithi154 ceph-mon[79161]: pgmap v224: 129 pgs: 129 active+clean; 416 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-03-30T04:09:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:14 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:09:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:14 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:09:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:14 smithi193 ceph-mon[79581]: pgmap v224: 129 pgs: 129 active+clean; 416 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-03-30T04:09:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:14 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:09:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:14 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:09:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:14 smithi023 ceph-mon[38254]: pgmap v224: 129 pgs: 129 active+clean; 416 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 191 op/s 2024-03-30T04:09:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:14 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:09:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:14 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:09:15.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 721780 10.86 MB/sec execute 232 sec latency 20.811 ms 2024-03-30T04:09:16.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 724150 10.87 MB/sec execute 233 sec latency 20.678 ms 2024-03-30T04:09:16.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:16 smithi154 ceph-mon[79161]: pgmap v225: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-30T04:09:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:16 smithi154 ceph-mon[79161]: osdmap e93: 12 total, 12 up, 12 in 2024-03-30T04:09:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:16 smithi193 ceph-mon[79581]: pgmap v225: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-30T04:09:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:16 smithi193 ceph-mon[79581]: osdmap e93: 12 total, 12 up, 12 in 2024-03-30T04:09:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:16 smithi023 ceph-mon[38254]: pgmap v225: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 194 op/s 2024-03-30T04:09:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:16 smithi023 ceph-mon[38254]: osdmap e93: 12 total, 12 up, 12 in 2024-03-30T04:09:17.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 725685 10.86 MB/sec execute 234 sec latency 21.077 ms 2024-03-30T04:09:18.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 727578 10.86 MB/sec execute 235 sec latency 119.119 ms 2024-03-30T04:09:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:18 smithi154 ceph-mon[79161]: pgmap v227: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-03-30T04:09:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:18 smithi193 ceph-mon[79581]: pgmap v227: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-03-30T04:09:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:18 smithi023 ceph-mon[38254]: pgmap v227: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 118 op/s 2024-03-30T04:09:19.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 729054 10.84 MB/sec execute 236 sec latency 25.545 ms 2024-03-30T04:09:20.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 731446 10.86 MB/sec execute 237 sec latency 20.928 ms 2024-03-30T04:09:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:20 smithi154 ceph-mon[79161]: pgmap v228: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-03-30T04:09:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:20 smithi193 ceph-mon[79581]: pgmap v228: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-03-30T04:09:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:20 smithi023 ceph-mon[38254]: pgmap v228: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-03-30T04:09:21.210 INFO:tasks.workunit.client.0.smithi023.stdout: 1 732880 10.85 MB/sec execute 238 sec latency 23.106 ms 2024-03-30T04:09:22.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 735044 10.85 MB/sec execute 239 sec latency 19.974 ms 2024-03-30T04:09:22.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:22 smithi154 ceph-mon[79161]: pgmap v229: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:09:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:22 smithi193 ceph-mon[79581]: pgmap v229: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:09:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:22 smithi023 ceph-mon[38254]: pgmap v229: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 201 op/s 2024-03-30T04:09:23.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 736027 10.82 MB/sec execute 240 sec latency 64.876 ms 2024-03-30T04:09:24.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 738394 10.83 MB/sec execute 241 sec latency 17.747 ms 2024-03-30T04:09:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:24 smithi154 ceph-mon[79161]: pgmap v230: 129 pgs: 129 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-03-30T04:09:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:24 smithi193 ceph-mon[79581]: pgmap v230: 129 pgs: 129 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-03-30T04:09:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:24 smithi023 ceph-mon[38254]: pgmap v230: 129 pgs: 129 active+clean; 420 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.2 MiB/s wr, 189 op/s 2024-03-30T04:09:25.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 739741 10.81 MB/sec execute 242 sec latency 21.389 ms 2024-03-30T04:09:26.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 741947 10.82 MB/sec execute 243 sec latency 60.514 ms 2024-03-30T04:09:26.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:26 smithi154 ceph-mon[79161]: pgmap v231: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 208 op/s 2024-03-30T04:09:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:26 smithi193 ceph-mon[79581]: pgmap v231: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 208 op/s 2024-03-30T04:09:26.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:26 smithi023 ceph-mon[38254]: pgmap v231: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 208 op/s 2024-03-30T04:09:27.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 743283 10.80 MB/sec execute 244 sec latency 23.887 ms 2024-03-30T04:09:28.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 745048 10.80 MB/sec execute 245 sec latency 40.208 ms 2024-03-30T04:09:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:28 smithi154 ceph-mon[79161]: pgmap v232: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 206 op/s 2024-03-30T04:09:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:28 smithi193 ceph-mon[79581]: pgmap v232: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 206 op/s 2024-03-30T04:09:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:28 smithi023 ceph-mon[38254]: pgmap v232: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 206 op/s 2024-03-30T04:09:29.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 746655 10.79 MB/sec execute 246 sec latency 22.020 ms 2024-03-30T04:09:30.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 748971 10.80 MB/sec execute 247 sec latency 19.670 ms 2024-03-30T04:09:30.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:30 smithi154 ceph-mon[79161]: pgmap v233: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-30T04:09:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:30 smithi193 ceph-mon[79581]: pgmap v233: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-30T04:09:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:30 smithi023 ceph-mon[38254]: pgmap v233: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 258 op/s 2024-03-30T04:09:31.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 750441 10.78 MB/sec execute 248 sec latency 23.259 ms 2024-03-30T04:09:32.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 752773 10.80 MB/sec execute 249 sec latency 18.290 ms 2024-03-30T04:09:32.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:32 smithi154 ceph-mon[79161]: pgmap v234: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-30T04:09:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:32 smithi193 ceph-mon[79581]: pgmap v234: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-30T04:09:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:32 smithi023 ceph-mon[38254]: pgmap v234: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-03-30T04:09:33.211 INFO:tasks.workunit.client.0.smithi023.stdout: 1 753947 10.77 MB/sec execute 250 sec latency 57.937 ms 2024-03-30T04:09:34.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 756125 10.78 MB/sec execute 251 sec latency 21.095 ms 2024-03-30T04:09:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:34 smithi154 ceph-mon[79161]: pgmap v235: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:09:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:34 smithi193 ceph-mon[79581]: pgmap v235: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:09:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:34 smithi023 ceph-mon[38254]: pgmap v235: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 249 op/s 2024-03-30T04:09:35.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 757472 10.76 MB/sec execute 252 sec latency 21.229 ms 2024-03-30T04:09:36.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 759677 10.77 MB/sec execute 253 sec latency 20.164 ms 2024-03-30T04:09:36.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:36 smithi154 ceph-mon[79161]: pgmap v236: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-03-30T04:09:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:36 smithi193 ceph-mon[79581]: pgmap v236: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-03-30T04:09:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:36 smithi023 ceph-mon[38254]: pgmap v236: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-03-30T04:09:37.005 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:09:37.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 761098 10.75 MB/sec execute 254 sec latency 21.693 ms 2024-03-30T04:09:37.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 -- 172.21.15.23:0/826851793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073aa0 msgr2=0x7f3b3410d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 --2- 172.21.15.23:0/826851793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073aa0 0x7f3b3410d060 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009990 tx=0x7f3b20009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 -- 172.21.15.23:0/826851793 shutdown_connections 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 --2- 172.21.15.23:0/826851793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 0x7f3b3410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 --2- 172.21.15.23:0/826851793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073180 0x7f3b34073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 --2- 172.21.15.23:0/826851793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073aa0 0x7f3b3410d060 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 -- 172.21.15.23:0/826851793 >> 172.21.15.23:0/826851793 conn(0x7f3b340fc7f0 msgr2=0x7f3b340fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 -- 172.21.15.23:0/826851793 shutdown_connections 2024-03-30T04:09:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.238+0000 7f3b3b600700 1 -- 172.21.15.23:0/826851793 wait complete. 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 Processor -- start 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 -- start start 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 0x7f3b34199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 0x7f3b3419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419e370 con 0x7f3b34073aa0 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419e4e0 con 0x7f3b34073180 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.239+0000 7f3b3b600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3419e6a0 con 0x7f3b3410d5a0 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 0x7f3b34199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56854/0 (socket says 172.21.15.23:56854) 2024-03-30T04:09:37.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 -- 172.21.15.23:0/1887276089 learned_addr learned my addr 172.21.15.23:0/1887276089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b39600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 0x7f3b3419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 -- 172.21.15.23:0/1887276089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 msgr2=0x7f3b3419dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 0x7f3b3419dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 -- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 msgr2=0x7f3b34199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 0x7f3b34199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 -- 172.21.15.23:0/1887276089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b28009520 con 0x7f3b34073180 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b33e00700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 0x7f3b34199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.240+0000 7f3b38c00700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800eb30 tx=0x7f3b2800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.241+0000 7f3b31600700 1 -- 172.21.15.23:0/1887276089 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28009880 con 0x7f3b34073180 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.241+0000 7f3b31600700 1 -- 172.21.15.23:0/1887276089 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b280044e0 con 0x7f3b34073180 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.241+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b200095f0 con 0x7f3b34073180 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.241+0000 7f3b31600700 1 -- 172.21.15.23:0/1887276089 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28003d80 con 0x7f3b34073180 2024-03-30T04:09:37.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.241+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3419eb70 con 0x7f3b34073180 2024-03-30T04:09:37.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.244+0000 7f3b31600700 1 -- 172.21.15.23:0/1887276089 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3b28003f20 con 0x7f3b34073180 2024-03-30T04:09:37.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.245+0000 7f3b4f07cb80 1 Processor -- start 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.245+0000 7f3b4f07cb80 1 -- start start 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b31600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 0x7f3b1c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b31600700 1 -- 172.21.15.23:0/1887276089 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3b28014070 con 0x7f3b34073180 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 0x55b93da3c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b39600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 0x7f3b1c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 0x55b93da4a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d995990 con 0x55b93da3c5d0 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d993350 con 0x55b93da47df0 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d98ec40 con 0x55b93da3cef0 2024-03-30T04:09:37.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b39600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 0x55b93da4a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33626/0 (socket says 172.21.15.23:33626) 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b33e00700 1 -- 172.21.15.23:0/724696153 learned_addr learned my addr 172.21.15.23:0/724696153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.246+0000 7f3b38c00700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 0x55b93da3c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 -- 172.21.15.23:0/724696153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 msgr2=0x55b93da4a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 0x55b93da4a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 -- 172.21.15.23:0/724696153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 msgr2=0x55b93da3c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 0x55b93da3c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 -- 172.21.15.23:0/724696153 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b93d8103b0 con 0x55b93da3cef0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b39600700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 0x55b93da4a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b38c00700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 0x55b93da3c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b39600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 0x7f3b1c0a37d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406a3d0 tx=0x7f3b2c00f930 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b33e00700 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3b20015000 tx=0x7f3b20005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f8f16cd580a2f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b25c00700 1 -- 172.21.15.23:0/724696153 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2001d070 con 0x55b93da3cef0 2024-03-30T04:09:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b25c00700 1 -- 172.21.15.23:0/724696153 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b2001e040 con 0x55b93da3cef0 2024-03-30T04:09:37.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.247+0000 7f3b25c00700 1 -- 172.21.15.23:0/724696153 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20004cf0 con 0x55b93da3cef0 2024-03-30T04:09:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/724696153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 msgr2=0x55b93da478b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3b20015000 tx=0x7f3b20005ec0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/724696153 shutdown_connections 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da47df0 0x55b93da4a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3c5d0 0x55b93da3c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/724696153 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da478b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/724696153 >> 172.21.15.23:0/724696153 conn(0x55b93d99c6a0 msgr2=0x55b93da38a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/724696153 shutdown_connections 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.248+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/724696153 wait complete. 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 -- start start 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 Processor -- start 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da44a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da44f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d92ad80 con 0x55b93da47df0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d92aa20 con 0x55b93da3c5d0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93d8126a0 con 0x55b93da3cef0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.249+0000 7f3b4f07cb80 10 client.0 _ll_register_callbacks cb 0x55b93d934470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b39600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da44f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56876/0 (socket says 172.21.15.23:56876) 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 -- 172.21.15.23:0/4062671505 learned_addr learned my addr 172.21.15.23:0/4062671505 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b33e00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da44a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 -- 172.21.15.23:0/4062671505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 msgr2=0x55b93da44a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da44a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 -- 172.21.15.23:0/4062671505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 msgr2=0x55b93da44f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da44f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 -- 172.21.15.23:0/4062671505 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b341106e0 con 0x55b93da3c5d0 2024-03-30T04:09:37.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b39600700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da44f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b38c00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3b28099fa0 tx=0x7f3b28099cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a53253f0819540c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.250+0000 7f3b1a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2809a2a0 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3b2809e0d0 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b93d94f380 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b280a2720 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 10 client.27596 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b93d7bcb60 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b93d94f380 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3b280b1030 con 0x55b93da3c5d0 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 10 client.27596 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 10 client.27596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 20 client.27596 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:37.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93da57800 0x55b93da59c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 4 client.27596 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.251+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b93d987700 con 0x55b93da57800 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.252+0000 7f3b33e00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93da57800 0x55b93da59c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.252+0000 7f3b33e00700 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93da57800 0x55b93da59c20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90da5b399bd0b53 server_cookie=5a42bfd0bf74c899 in_seq=0 out_seq=0 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.252+0000 7f3b1a000700 10 client.27596 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b1a000700 1 -- 172.21.15.23:0/4062671505 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b93d987700 con 0x55b93da57800 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b1a000700 10 client.27596 handle_command_reply: tid=0 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 client.27596 shutdown 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 msgr2=0x55b93da47690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3b28099fa0 tx=0x7f3b28099cf0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 shutdown_connections 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da44f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da47690 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93da57800 0x55b93da59c20 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/4062671505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3cef0 0x55b93da44a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b1a000700 0 client.27596 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.253+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 >> 172.21.15.23:0/4062671505 conn(0x55b93d99c6a0 msgr2=0x55b93da3e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.254+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 shutdown_connections 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.254+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/4062671505 wait complete. 2024-03-30T04:09:37.257 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.254+0000 7f3b4f07cb80 20 client.27596 trim_cache size 0 max 16384 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.273+0000 7f3b4f07cb80 1 Processor -- start 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.273+0000 7f3b4f07cb80 1 -- start start 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 0x55b93da4ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 0x55b93da3fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93da4ef80 con 0x55b93da3cef0 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93da4a430 con 0x55b93da3c5d0 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93da44f90 con 0x55b93da47df0 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 0x55b93da4ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56886/0 (socket says 172.21.15.23:56886) 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b38c00700 1 -- 172.21.15.23:0/2082741182 learned_addr learned my addr 172.21.15.23:0/2082741182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b39600700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 0x55b93da3fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.274+0000 7f3b38c00700 1 -- 172.21.15.23:0/2082741182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 msgr2=0x55b93da3fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b38c00700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 0x55b93da3fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b38c00700 1 -- 172.21.15.23:0/2082741182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 msgr2=0x55b93da4ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b38c00700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 0x55b93da4ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b38c00700 1 -- 172.21.15.23:0/2082741182 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b93d954ea0 con 0x55b93da3c5d0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b39600700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 0x55b93da3fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b33e00700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 0x55b93da4ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b38c00700 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3b2809a230 tx=0x7f3b28098850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46681b119ff31660 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b26600700 1 -- 172.21.15.23:0/2082741182 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2800cbd0 con 0x55b93da3c5d0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b26600700 1 -- 172.21.15.23:0/2082741182 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b2809e0d0 con 0x55b93da3c5d0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.275+0000 7f3b26600700 1 -- 172.21.15.23:0/2082741182 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2809a740 con 0x55b93da3c5d0 2024-03-30T04:09:37.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/2082741182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 msgr2=0x55b93da59bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3b2809a230 tx=0x7f3b28098850 comp rx=0 tx=0).stop 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/2082741182 shutdown_connections 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da47df0 0x55b93da3fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3c5d0 0x55b93da59bf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/2082741182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da3cef0 0x55b93da4ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/2082741182 >> 172.21.15.23:0/2082741182 conn(0x55b93d99c6a0 msgr2=0x55b93da315a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/2082741182 shutdown_connections 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/2082741182 wait complete. 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.276+0000 7f3b4f07cb80 1 -- start start 2024-03-30T04:09:37.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 Processor -- start 2024-03-30T04:09:37.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da4cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93daaa6f0 con 0x55b93da47df0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93daaa860 con 0x55b93da3cef0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b93daaaa20 con 0x55b93da3c5d0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b4f07cb80 10 client.0 _ll_register_callbacks cb 0x55b93d934470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.277+0000 7f3b38c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b33e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56896/0 (socket says 172.21.15.23:56896) 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b33e00700 1 -- 172.21.15.23:0/3306898274 learned_addr learned my addr 172.21.15.23:0/3306898274 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33654/0 (socket says 172.21.15.23:33654) 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b39600700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da4cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 -- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 msgr2=0x55b93da4c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 -- 172.21.15.23:0/3306898274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 msgr2=0x55b93da4cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da4cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 -- 172.21.15.23:0/3306898274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b93d8103b0 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b33e00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b38c00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3b28099fa0 tx=0x7f3b28003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc434d595c0cfd44 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b1a000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2800cbd0 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3b2809a2a0 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.278+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b2001e040 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b280a2720 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 10 client.27604 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3b2c009340 con 0x55b93da3c5d0 2024-03-30T04:09:37.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3b2001e040 con 0x55b93da3c5d0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3b280b1050 con 0x55b93da3c5d0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 10 client.27604 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 10 client.27604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 20 client.27604 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93daabce0 0x55b93daae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 4 client.27604 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b93d7ddcf0 con 0x55b93daabce0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.279+0000 7f3b33e00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93daabce0 0x55b93daae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b33e00700 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93daabce0 0x55b93daae0d0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e800c8a9f6ecbf41 server_cookie=df7ef3039110c0e6 in_seq=0 out_seq=0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b1a000700 10 client.27604 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b1a000700 1 -- 172.21.15.23:0/3306898274 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b93d7ddcf0 con 0x55b93daabce0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b1a000700 10 client.27604 handle_command_reply: tid=0 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 client.27604 shutdown 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 msgr2=0x55b93da4c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3b28099fa0 tx=0x7f3b28003820 comp rx=0 tx=0).stop 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 shutdown_connections 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b93da47df0 0x55b93da4cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55b93daabce0 0x55b93daae0d0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b93da3c5d0 0x55b93da4c0a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 --2- 172.21.15.23:0/3306898274 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b93da3cef0 0x55b93da4c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b1a000700 0 client.27604 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.280+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 >> 172.21.15.23:0/3306898274 conn(0x55b93d99c6a0 msgr2=0x55b93da3fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 shutdown_connections 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b4f07cb80 1 -- 172.21.15.23:0/3306898274 wait complete. 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b4f07cb80 20 client.27604 trim_cache size 0 max 16384 2024-03-30T04:09:37.284 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:37.285 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:09:37.285 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:09:37.285 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:37.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 msgr2=0x7f3b1c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 0x7f3b1c0a37d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3b3406a3d0 tx=0x7f3b2c00f930 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 msgr2=0x7f3b34199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800eb30 tx=0x7f3b2800ee40 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 shutdown_connections 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b3410d5a0 0x7f3b3419dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3b1c0a1310 0x7f3b1c0a37d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b34073180 0x7f3b34199450 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 --2- 172.21.15.23:0/1887276089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3b34073aa0 0x7f3b34199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.281+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 >> 172.21.15.23:0/1887276089 conn(0x7f3b340fc7f0 msgr2=0x7f3b340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.282+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 shutdown_connections 2024-03-30T04:09:37.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.282+0000 7f3b3b600700 1 -- 172.21.15.23:0/1887276089 wait complete. 2024-03-30T04:09:37.299 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:09:37.299 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 -- 172.21.15.23:0/2265434938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884101810 msgr2=0x7fa884105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 --2- 172.21.15.23:0/2265434938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884101810 0x7fa884105850 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa870009990 tx=0x7fa870009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 -- 172.21.15.23:0/2265434938 shutdown_connections 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 --2- 172.21.15.23:0/2265434938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884105f80 0x7fa88410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 --2- 172.21.15.23:0/2265434938 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884100e60 0x7fa884101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 --2- 172.21.15.23:0/2265434938 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa884101810 0x7fa884105850 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 -- 172.21.15.23:0/2265434938 >> 172.21.15.23:0/2265434938 conn(0x7fa8840fc700 msgr2=0x7fa8840feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 -- 172.21.15.23:0/2265434938 shutdown_connections 2024-03-30T04:09:37.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.507+0000 7fa88b400700 1 -- 172.21.15.23:0/2265434938 wait complete. 2024-03-30T04:09:37.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.508+0000 7fa88b400700 1 Processor -- start 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.508+0000 7fa88b400700 1 -- start start 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884100e60 0x7fa884197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884105f80 0x7fa8841975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419c3d0 con 0x7fa884105f80 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419c540 con 0x7fa88419b950 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa88b400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88419c700 con 0x7fa884100e60 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa889400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa883e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884105f80 0x7fa8841975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa889400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56928/0 (socket says 172.21.15.23:56928) 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa889400700 1 -- 172.21.15.23:0/1992256184 learned_addr learned my addr 172.21.15.23:0/1992256184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.509+0000 7fa888a00700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884100e60 0x7fa884197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 -- 172.21.15.23:0/1992256184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884100e60 msgr2=0x7fa884197070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884100e60 0x7fa884197070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 -- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884105f80 msgr2=0x7fa8841975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884105f80 0x7fa8841975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 -- 172.21.15.23:0/1992256184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8700095f0 con 0x7fa88419b950 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa889400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00ef30 tx=0x7fa87c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c018070 con 0x7fa88419b950 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa87c00f040 con 0x7fa88419b950 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8841a3c00 con 0x7fa88419b950 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c00cd00 con 0x7fa88419b950 2024-03-30T04:09:37.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.510+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8841a4130 con 0x7fa88419b950 2024-03-30T04:09:37.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.511+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8841091a0 con 0x7fa88419b950 2024-03-30T04:09:37.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.513+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa87c003a00 con 0x7fa88419b950 2024-03-30T04:09:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.515+0000 7fa881600700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 0x7fa86c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.515+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa87c0ccb50 con 0x7fa88419b950 2024-03-30T04:09:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.515+0000 7fa888a00700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 0x7fa86c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:37.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.515+0000 7fa888a00700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 0x7fa86c0a37d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa878009c60 tx=0x7fa878009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:37.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.520+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa87c095bf0 con 0x7fa88419b950 2024-03-30T04:09:37.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.839+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa884102330 con 0x7fa88419b950 2024-03-30T04:09:37.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.839+0000 7fa881600700 1 -- 172.21.15.23:0/1992256184 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fa87c099210 con 0x7fa88419b950 2024-03-30T04:09:37.838 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:09:37.838 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 msgr2=0x7fa86c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 0x7fa86c0a37d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa878009c60 tx=0x7fa878009260 comp rx=0 tx=0).stop 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 msgr2=0x7fa88419bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00ef30 tx=0x7fa87c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 shutdown_connections 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa86c0a1310 0x7fa86c0a37d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa88419b950 0x7fa88419bdd0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa884100e60 0x7fa884197070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 --2- 172.21.15.23:0/1992256184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa884105f80 0x7fa8841975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 >> 172.21.15.23:0/1992256184 conn(0x7fa8840fc700 msgr2=0x7fa8840fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 shutdown_connections 2024-03-30T04:09:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:37.842+0000 7fa88b400700 1 -- 172.21.15.23:0/1992256184 wait complete. 2024-03-30T04:09:37.842 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:09:37.861 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 -- 172.21.15.23:0/2856848584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c073180 msgr2=0x7f3f6c073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 --2- 172.21.15.23:0/2856848584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c073180 0x7f3f6c073560 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009990 tx=0x7f3f5c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 -- 172.21.15.23:0/2856848584 shutdown_connections 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 --2- 172.21.15.23:0/2856848584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c10d580 0x7f3f6c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 --2- 172.21.15.23:0/2856848584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c073180 0x7f3f6c073560 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 --2- 172.21.15.23:0/2856848584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c10d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.075+0000 7f3f71400700 1 -- 172.21.15.23:0/2856848584 >> 172.21.15.23:0/2856848584 conn(0x7f3f6c0fc7f0 msgr2=0x7f3f6c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.076+0000 7f3f71400700 1 -- 172.21.15.23:0/2856848584 shutdown_connections 2024-03-30T04:09:38.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.076+0000 7f3f71400700 1 -- 172.21.15.23:0/2856848584 wait complete. 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 Processor -- start 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 -- start start 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c10d580 0x7f3f6c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c19e340 con 0x7f3f6c073aa0 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c19e4b0 con 0x7f3f6c10d580 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f71400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c19e670 con 0x7f3f6c073180 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.077+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c10d580 0x7f3f6c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33464/0 (socket says 172.21.15.23:33464) 2024-03-30T04:09:38.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51492/0 (socket says 172.21.15.23:51492) 2024-03-30T04:09:38.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f69e00700 1 -- 172.21.15.23:0/1044766248 learned_addr learned my addr 172.21.15.23:0/1044766248 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 -- 172.21.15.23:0/1044766248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c10d580 msgr2=0x7f3f6c19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c10d580 0x7f3f6c19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 -- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 msgr2=0x7f3f6c199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 -- 172.21.15.23:0/1044766248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c0095f0 con 0x7f3f6c073180 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f69e00700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.078+0000 7f3f6a800700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c000c00 tx=0x7f3f5c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.079+0000 7f3f63400700 1 -- 172.21.15.23:0/1044766248 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c01d070 con 0x7f3f6c073180 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.079+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c19e8f0 con 0x7f3f6c073180 2024-03-30T04:09:38.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.079+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c1a6190 con 0x7f3f6c073180 2024-03-30T04:09:38.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.079+0000 7f3f63400700 1 -- 172.21.15.23:0/1044766248 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f5c021410 con 0x7f3f6c073180 2024-03-30T04:09:38.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.079+0000 7f3f63400700 1 -- 172.21.15.23:0/1044766248 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c017650 con 0x7f3f6c073180 2024-03-30T04:09:38.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.082+0000 7f3f63400700 1 -- 172.21.15.23:0/1044766248 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3f5c00fb10 con 0x7f3f6c073180 2024-03-30T04:09:38.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.084+0000 7f3f63400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 0x7f3f500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.084+0000 7f3f63400700 1 -- 172.21.15.23:0/1044766248 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f5c0ce120 con 0x7f3f6c073180 2024-03-30T04:09:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.084+0000 7f3f84e78b80 1 Processor -- start 2024-03-30T04:09:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.084+0000 7f3f69e00700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 0x7f3f500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.084+0000 7f3f84e78b80 1 -- start start 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da2598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da25c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da1a7990 con 0x5646da24e5d0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da1a5350 con 0x5646da259df0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da1a0c40 con 0x5646da24eef0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f69e00700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 0x7f3f500a3750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c19a970 tx=0x7f3f58009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51502/0 (socket says 172.21.15.23:51502) 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6a800700 1 -- 172.21.15.23:0/3703661900 learned_addr learned my addr 172.21.15.23:0/3703661900 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da25c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f69e00700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da2598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 -- 172.21.15.23:0/3703661900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 msgr2=0x5646da2598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da2598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 -- 172.21.15.23:0/3703661900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 msgr2=0x5646da24e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6b200700 1 -- 172.21.15.23:0/3703661900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646da0223b0 con 0x5646da259df0 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.085+0000 7f3f6a800700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.086+0000 7f3f6b200700 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da25c1e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f64011550 tx=0x7f3f64009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a9c1d997471be0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.086+0000 7f3f57400700 1 -- 172.21.15.23:0/3703661900 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400e040 con 0x5646da259df0 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.086+0000 7f3f57400700 1 -- 172.21.15.23:0/3703661900 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f6400f040 con 0x5646da259df0 2024-03-30T04:09:38.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.086+0000 7f3f57400700 1 -- 172.21.15.23:0/3703661900 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64004430 con 0x5646da259df0 2024-03-30T04:09:38.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 -- 172.21.15.23:0/3703661900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 msgr2=0x5646da25c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da25c1e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f64011550 tx=0x7f3f64009920 comp rx=0 tx=0).stop 2024-03-30T04:09:38.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 -- 172.21.15.23:0/3703661900 shutdown_connections 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da25c1e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da24e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/3703661900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da2598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 -- 172.21.15.23:0/3703661900 >> 172.21.15.23:0/3703661900 conn(0x5646da1ae6a0 msgr2=0x5646da24aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 -- 172.21.15.23:0/3703661900 shutdown_connections 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.087+0000 7f3f84e78b80 1 -- 172.21.15.23:0/3703661900 wait complete. 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 -- start start 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 Processor -- start 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 0x5646da256a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 0x5646da257040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da13cd80 con 0x5646da24e5d0 2024-03-30T04:09:38.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da13ca20 con 0x5646da24eef0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.088+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da0246a0 con 0x5646da259df0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f84e78b80 10 client.0 _ll_register_callbacks cb 0x5646da146470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 0x5646da256a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51508/0 (socket says 172.21.15.23:51508) 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6a800700 1 -- 172.21.15.23:0/4116488011 learned_addr learned my addr 172.21.15.23:0/4116488011 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6b200700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 0x5646da257040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 -- 172.21.15.23:0/4116488011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 msgr2=0x5646da257040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 0x5646da257040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 -- 172.21.15.23:0/4116488011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 msgr2=0x5646da259790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 -- 172.21.15.23:0/4116488011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c1a6190 con 0x5646da24eef0 2024-03-30T04:09:38.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f6b200700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 0x5646da257040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.089+0000 7f3f69e00700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 0x5646da256a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f58011c70 tx=0x7f3f580047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8af2f61afe2d4ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f4be00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5801e320 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5646da161380 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f5800e840 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f5800ea00 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f84e78b80 10 client.27620 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3f64011230 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.090+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5646da161380 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f5801d410 con 0x5646da24eef0 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 10 client.27620 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 10 client.27620 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:09:38.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 20 client.27620 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da269a90 0x5646da26beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 4 client.27620 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5646da199700 con 0x5646da269a90 2024-03-30T04:09:38.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da269a90 0x5646da26beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da269a90 0x5646da26beb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ba41203f2a569d19 server_cookie=116b87a70f7ffecb in_seq=0 out_seq=0 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.091+0000 7f3f4be00700 10 client.27620 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f4be00700 1 -- 172.21.15.23:0/4116488011 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5646da199700 con 0x5646da269a90 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f4be00700 10 client.27620 handle_command_reply: tid=0 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f84e78b80 1 client.27620 shutdown 2024-03-30T04:09:38.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 msgr2=0x5646da256a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 0x5646da256a80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f58011c70 tx=0x7f3f580047c0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.092+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 shutdown_connections 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da269a90 0x5646da26beb0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da259df0 0x5646da257040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da259790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4116488011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da24eef0 0x5646da256a80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 >> 172.21.15.23:0/4116488011 conn(0x5646da1ae6a0 msgr2=0x5646da2506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f4be00700 0 client.27620 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 shutdown_connections 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4116488011 wait complete. 2024-03-30T04:09:38.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.093+0000 7f3f84e78b80 20 client.27620 trim_cache size 0 max 16384 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.109+0000 7f3f84e78b80 1 Processor -- start 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.109+0000 7f3f84e78b80 1 -- start start 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.109+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 0x5646da26be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.109+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24eef0 0x5646da260b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.109+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2592c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da2610d0 con 0x5646da24eef0 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da24afd0 con 0x5646da259df0 2024-03-30T04:09:38.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da267120 con 0x5646da24e5d0 2024-03-30T04:09:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 0x5646da26be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35032/0 (socket says 172.21.15.23:35032) 2024-03-30T04:09:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6b200700 1 -- 172.21.15.23:0/4029354683 learned_addr learned my addr 172.21.15.23:0/4029354683 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24eef0 0x5646da260b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 -- 172.21.15.23:0/4029354683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 msgr2=0x5646da2592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 -- 172.21.15.23:0/4029354683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24eef0 msgr2=0x5646da260b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24eef0 0x5646da260b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.110+0000 7f3f6a800700 1 -- 172.21.15.23:0/4029354683 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646d9fceb60 con 0x5646da24e5d0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f6a800700 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 0x5646da26be80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c005f50 tx=0x7f3f5c023040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b7e82df8fafafa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f57e00700 1 -- 172.21.15.23:0/4029354683 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c09b580 con 0x5646da24e5d0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f57e00700 1 -- 172.21.15.23:0/4029354683 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f5c09be90 con 0x5646da24e5d0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f57e00700 1 -- 172.21.15.23:0/4029354683 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c09a370 con 0x5646da24e5d0 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4029354683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 msgr2=0x5646da26be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.111+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 0x5646da26be80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c005f50 tx=0x7f3f5c023040 comp rx=0 tx=0).stop 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4029354683 shutdown_connections 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24e5d0 0x5646da26be80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/4029354683 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24eef0 0x5646da260b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4029354683 >> 172.21.15.23:0/4029354683 conn(0x5646da1ae6a0 msgr2=0x5646da2435a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4029354683 shutdown_connections 2024-03-30T04:09:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 -- 172.21.15.23:0/4029354683 wait complete. 2024-03-30T04:09:38.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 -- start start 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.112+0000 7f3f84e78b80 1 Processor -- start 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da25e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2bc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da25f4b0 con 0x5646da24e5d0 2024-03-30T04:09:38.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da2bce00 con 0x5646da259df0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646da2bcf40 con 0x5646da24eef0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f84e78b80 10 client.0 _ll_register_callbacks cb 0x5646da146470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da25e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f6b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2bc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f6a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da25e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51526/0 (socket says 172.21.15.23:51526) 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f6a800700 1 -- 172.21.15.23:0/2474333121 learned_addr learned my addr 172.21.15.23:0/2474333121 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33522/0 (socket says 172.21.15.23:33522) 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 -- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 msgr2=0x5646da2bc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2bc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 -- 172.21.15.23:0/2474333121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 msgr2=0x5646da25e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da25e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.113+0000 7f3f69e00700 1 -- 172.21.15.23:0/2474333121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c19e8f0 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f69e00700 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800efd0 tx=0x7f3f58005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4ccc7b48d66e4ea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f4be00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58004830 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f58004990 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f6400f040 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f58004b50 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f84e78b80 10 client.27628 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.114+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5646da166ea0 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f6400f040 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f5800e890 con 0x5646da24eef0 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 10 client.27628 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 10 client.27628 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 20 client.27628 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da2be6d0 0x5646da2c0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 4 client.27628 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5646d9fefcf0 con 0x5646da2be6d0 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.115+0000 7f3f6a800700 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da2be6d0 0x5646da2c0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f4be00700 10 client.27628 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f6a800700 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da2be6d0 0x5646da2c0af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8934a6d51a9990da server_cookie=9930abf63013416f in_seq=0 out_seq=0 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f4be00700 1 -- 172.21.15.23:0/2474333121 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5646d9fefcf0 con 0x5646da2be6d0 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f4be00700 10 client.27628 handle_command_reply: tid=0 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 client.27628 shutdown 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 msgr2=0x5646da25eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800efd0 tx=0x7f3f58005b40 comp rx=0 tx=0).stop 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 shutdown_connections 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5646da259df0 0x5646da2bc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5646da24e5d0 0x5646da25e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5646da2be6d0 0x5646da2c0af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 --2- 172.21.15.23:0/2474333121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646da24eef0 0x5646da25eba0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 >> 172.21.15.23:0/2474333121 conn(0x5646da1ae6a0 msgr2=0x5646da251d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f4be00700 0 client.27628 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 shutdown_connections 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.116+0000 7f3f84e78b80 1 -- 172.21.15.23:0/2474333121 wait complete. 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f84e78b80 20 client.27628 trim_cache size 0 max 16384 2024-03-30T04:09:38.119 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 msgr2=0x7f3f500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 0x7f3f500a3750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c19a970 tx=0x7f3f58009260 comp rx=0 tx=0).stop 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 msgr2=0x7f3f6c199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c000c00 tx=0x7f3f5c004730 comp rx=0 tx=0).stop 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 shutdown_connections 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f500a1290 0x7f3f500a3750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f6c10d580 0x7f3f6c19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f6c073180 0x7f3f6c199420 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 --2- 172.21.15.23:0/1044766248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f6c073aa0 0x7f3f6c199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.117+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 >> 172.21.15.23:0/1044766248 conn(0x7f3f6c0fc7f0 msgr2=0x7f3f6c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.118+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 shutdown_connections 2024-03-30T04:09:38.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.118+0000 7f3f71400700 1 -- 172.21.15.23:0/1044766248 wait complete. 2024-03-30T04:09:38.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:09:38.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 763076 10.76 MB/sec execute 255 sec latency 71.183 ms 2024-03-30T04:09:38.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:38 smithi154 ceph-mon[79161]: pgmap v237: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:09:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:38 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1992256184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 -- 172.21.15.23:0/4129636811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38067e30 msgr2=0x7f5a380682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 --2- 172.21.15.23:0/4129636811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38067e30 0x7f5a380682b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5a280099c0 tx=0x7f5a28009cd0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 -- 172.21.15.23:0/4129636811 shutdown_connections 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 --2- 172.21.15.23:0/4129636811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a380687f0 0x7f5a3810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 --2- 172.21.15.23:0/4129636811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a38105ce0 0x7f5a381060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 --2- 172.21.15.23:0/4129636811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38067e30 0x7f5a380682b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 -- 172.21.15.23:0/4129636811 >> 172.21.15.23:0/4129636811 conn(0x7f5a38076d40 msgr2=0x7f5a38077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 -- 172.21.15.23:0/4129636811 shutdown_connections 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.435+0000 7f5a40800700 1 -- 172.21.15.23:0/4129636811 wait complete. 2024-03-30T04:09:38.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.436+0000 7f5a40800700 1 Processor -- start 2024-03-30T04:09:38.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.436+0000 7f5a40800700 1 -- start start 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38067e30 0x7f5a38199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a380687f0 0x7f5a38199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3819e360 con 0x7f5a380687f0 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3819e4d0 con 0x7f5a38105ce0 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a40800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3819e690 con 0x7f5a38067e30 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a380687f0 0x7f5a38199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35054/0 (socket says 172.21.15.23:35054) 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 -- 172.21.15.23:0/218406070 learned_addr learned my addr 172.21.15.23:0/218406070 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3de00700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38067e30 0x7f5a38199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 -- 172.21.15.23:0/218406070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38067e30 msgr2=0x7f5a38199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38067e30 0x7f5a38199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 -- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a380687f0 msgr2=0x7f5a38199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a380687f0 0x7f5a38199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.437+0000 7f5a3e800700 1 -- 172.21.15.23:0/218406070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a280095f0 con 0x7f5a38105ce0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a3e800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400ef30 tx=0x7f5a3400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a32a00700 1 -- 172.21.15.23:0/218406070 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a34018070 con 0x7f5a38105ce0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a32a00700 1 -- 172.21.15.23:0/218406070 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a3400f040 con 0x7f5a38105ce0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3819e910 con 0x7f5a38105ce0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a32a00700 1 -- 172.21.15.23:0/218406070 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400cd00 con 0x7f5a38105ce0 2024-03-30T04:09:38.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.438+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a381a61b0 con 0x7f5a38105ce0 2024-03-30T04:09:38.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.441+0000 7f5a32a00700 1 -- 172.21.15.23:0/218406070 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f5a34003a00 con 0x7f5a38105ce0 2024-03-30T04:09:38.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.443+0000 7f5a54295b80 1 Processor -- start 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.443+0000 7f5a54295b80 1 -- start start 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a32a00700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 0x7f5a200a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a32a00700 1 -- 172.21.15.23:0/218406070 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5a340ccba0 con 0x7f5a38105ce0 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3d400700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 0x7f5a200a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511888b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562351188df0 0x56235118b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623510d6990 con 0x56235117def0 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623510d4350 con 0x562351188df0 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a54295b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623510cfc40 con 0x56235117d5d0 2024-03-30T04:09:38.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33534/0 (socket says 172.21.15.23:33534) 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3de00700 1 -- 172.21.15.23:0/2449918537 learned_addr learned my addr 172.21.15.23:0/2449918537 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3e800700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562351188df0 0x56235118b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.444+0000 7f5a3d400700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511888b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 -- 172.21.15.23:0/2449918537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562351188df0 msgr2=0x56235118b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562351188df0 0x56235118b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 -- 172.21.15.23:0/2449918537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 msgr2=0x5623511888b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511888b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 -- 172.21.15.23:0/2449918537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562350f513b0 con 0x56235117d5d0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3d400700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009d80 tx=0x7f5a2c011720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a63e64bdfba24c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a3d400700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 0x7f5a200a37d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009990 tx=0x7f5a2800b7d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a27000700 1 -- 172.21.15.23:0/2449918537 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c004230 con 0x56235117d5d0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a27000700 1 -- 172.21.15.23:0/2449918537 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a2c004390 con 0x56235117d5d0 2024-03-30T04:09:38.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.445+0000 7f5a27000700 1 -- 172.21.15.23:0/2449918537 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c0198c0 con 0x56235117d5d0 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 -- 172.21.15.23:0/2449918537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 msgr2=0x56235117d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009d80 tx=0x7f5a2c011720 comp rx=0 tx=0).stop 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 -- 172.21.15.23:0/2449918537 shutdown_connections 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562351188df0 0x56235118b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56235117d5d0 0x56235117d9b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2449918537 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511888b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 -- 172.21.15.23:0/2449918537 >> 172.21.15.23:0/2449918537 conn(0x5623510dd6a0 msgr2=0x562351179a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 -- 172.21.15.23:0/2449918537 shutdown_connections 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.446+0000 7f5a54295b80 1 -- 172.21.15.23:0/2449918537 wait complete. 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.447+0000 7f5a54295b80 1 -- start start 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.447+0000 7f5a54295b80 1 Processor -- start 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x562351185f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56235106bd80 con 0x56235117def0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56235106ba20 con 0x56235117d5d0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562350f536a0 con 0x562351188df0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a54295b80 10 client.0 _ll_register_callbacks cb 0x562351075470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x562351185f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35070/0 (socket says 172.21.15.23:35070) 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3de00700 1 -- 172.21.15.23:0/2651398142 learned_addr learned my addr 172.21.15.23:0/2651398142 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 -- 172.21.15.23:0/2651398142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 msgr2=0x562351188750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 -- 172.21.15.23:0/2651398142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 msgr2=0x5623511859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511859d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3e800700 1 -- 172.21.15.23:0/2651398142 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c0099e0 con 0x562351188df0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.448+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a3e800700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x562351185f50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5a340c9eb0 tx=0x7f5a34099350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=234dcfcf85d5e7fe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a1b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3409a000 con 0x562351188df0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5a3400ce60 con 0x562351188df0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562351090380 con 0x562351188df0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a340b2200 con 0x562351188df0 2024-03-30T04:09:38.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a54295b80 10 client.27640 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562350efdb60 con 0x562351188df0 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.449+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562351090380 con 0x562351188df0 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5a34098df0 con 0x562351188df0 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 10 client.27640 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 10 client.27640 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 20 client.27640 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x562351198710 0x56235119ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 4 client.27640 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623510c8700 con 0x562351198710 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x562351198710 0x56235119ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a1b400700 10 client.27640 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:38.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.450+0000 7f5a3de00700 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x562351198710 0x56235119ab00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4b3f9da9ce12a347 server_cookie=2ec36c7484fdaa71 in_seq=0 out_seq=0 2024-03-30T04:09:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a1b400700 1 -- 172.21.15.23:0/2651398142 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5623510c8700 con 0x562351198710 2024-03-30T04:09:38.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a1b400700 10 client.27640 handle_command_reply: tid=0 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 client.27640 shutdown 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 msgr2=0x562351185f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x562351185f50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5a340c9eb0 tx=0x7f5a34099350 comp rx=0 tx=0).stop 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 shutdown_connections 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x562351198710 0x56235119ab00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x562351185f50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a1b400700 0 client.27640 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351188750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 --2- 172.21.15.23:0/2651398142 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x5623511859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.451+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 >> 172.21.15.23:0/2651398142 conn(0x5623510dd6a0 msgr2=0x56235117f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.452+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 shutdown_connections 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.452+0000 7f5a54295b80 1 -- 172.21.15.23:0/2651398142 wait complete. 2024-03-30T04:09:38.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.452+0000 7f5a54295b80 20 client.27640 trim_cache size 0 max 16384 2024-03-30T04:09:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 Processor -- start 2024-03-30T04:09:38.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 -- start start 2024-03-30T04:09:38.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351175480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x5623511725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562351186a80 con 0x56235117def0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56235117a7d0 con 0x56235117d5d0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a54295b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56235118ffe0 con 0x562351188df0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351175480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a3e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x5623511725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a3d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351175480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35086/0 (socket says 172.21.15.23:35086) 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.467+0000 7f5a3de00700 1 -- 172.21.15.23:0/1663758562 learned_addr learned my addr 172.21.15.23:0/1663758562 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 -- 172.21.15.23:0/1663758562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 msgr2=0x562351175480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351175480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 -- 172.21.15.23:0/1663758562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 msgr2=0x56235118faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 -- 172.21.15.23:0/1663758562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56235117d440 con 0x562351188df0 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a3e800700 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x5623511725a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5a34095e00 tx=0x7f5a34095d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5fd1569bd214d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a27a00700 1 -- 172.21.15.23:0/1663758562 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340b2200 con 0x562351188df0 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a27a00700 1 -- 172.21.15.23:0/1663758562 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a34004ad0 con 0x562351188df0 2024-03-30T04:09:38.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.468+0000 7f5a27a00700 1 -- 172.21.15.23:0/1663758562 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a340a1cf0 con 0x562351188df0 2024-03-30T04:09:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 -- 172.21.15.23:0/1663758562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 msgr2=0x5623511725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x5623511725a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f5a34095e00 tx=0x7f5a34095d70 comp rx=0 tx=0).stop 2024-03-30T04:09:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 -- 172.21.15.23:0/1663758562 shutdown_connections 2024-03-30T04:09:38.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x5623511725a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x562351175480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1663758562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 -- 172.21.15.23:0/1663758562 >> 172.21.15.23:0/1663758562 conn(0x5623510dd6a0 msgr2=0x562351185230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 -- 172.21.15.23:0/1663758562 shutdown_connections 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.469+0000 7f5a54295b80 1 -- 172.21.15.23:0/1663758562 wait complete. 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.470+0000 7f5a54295b80 1 -- start start 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.470+0000 7f5a54295b80 1 Processor -- start 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x56235118de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623511eb5c0 con 0x56235117def0 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623511eb730 con 0x56235117d5d0 2024-03-30T04:09:38.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623511eb8f0 con 0x562351188df0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a54295b80 10 client.0 _ll_register_callbacks cb 0x562351075470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35088/0 (socket says 172.21.15.23:35088) 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 -- 172.21.15.23:0/1586350350 learned_addr learned my addr 172.21.15.23:0/1586350350 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3e800700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x56235118de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3d400700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 -- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 msgr2=0x56235118de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x56235118de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 -- 172.21.15.23:0/1586350350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 msgr2=0x56235118d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3de00700 1 -- 172.21.15.23:0/1586350350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562350efdb60 con 0x56235117d5d0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.471+0000 7f5a3e800700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x56235118de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a3de00700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c005c50 tx=0x7f5a2c009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1980f4df5c9582eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a1b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:38.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c029040 con 0x56235117d5d0 2024-03-30T04:09:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a2c004390 con 0x56235117d5d0 2024-03-30T04:09:38.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5a2c009260 con 0x56235117d5d0 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a2c003680 con 0x56235117d5d0 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a54295b80 10 client.27644 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.472+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562351095ea0 con 0x56235117d5d0 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a2c004390 con 0x56235117d5d0 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5a2c030020 con 0x56235117d5d0 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 10 client.27644 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 10 client.27644 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 20 client.27644 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5623511ed010 0x5623511ef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 4 client.27644 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:38.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562350f1ecf0 con 0x5623511ed010 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.473+0000 7f5a3d400700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5623511ed010 0x5623511ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a3d400700 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5623511ed010 0x5623511ef430 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6736b016d220efc server_cookie=6bda0d79c1eec87e in_seq=0 out_seq=0 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a1b400700 10 client.27644 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a1b400700 1 -- 172.21.15.23:0/1586350350 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562350f1ecf0 con 0x5623511ed010 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a1b400700 10 client.27644 handle_command_reply: tid=0 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 client.27644 shutdown 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 msgr2=0x56235118d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c005c50 tx=0x7f5a2c009fc0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 shutdown_connections 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5623511ed010 0x5623511ef430 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562351188df0 0x56235118de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56235117d5d0 0x56235118d3d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a1b400700 0 client.27644 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:09:38.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 --2- 172.21.15.23:0/1586350350 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56235117def0 0x56235118d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 >> 172.21.15.23:0/1586350350 conn(0x5623510dd6a0 msgr2=0x5623511794c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 shutdown_connections 2024-03-30T04:09:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 1 -- 172.21.15.23:0/1586350350 wait complete. 2024-03-30T04:09:38.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.474+0000 7f5a54295b80 20 client.27644 trim_cache size 0 max 16384 2024-03-30T04:09:38.477 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 msgr2=0x7f5a200a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 0x7f5a200a37d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009990 tx=0x7f5a2800b7d0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 msgr2=0x7f5a3819dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400ef30 tx=0x7f5a3400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 shutdown_connections 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5a200a1310 0x7f5a200a37d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a38105ce0 0x7f5a3819dd20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5a38067e30 0x7f5a38199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 --2- 172.21.15.23:0/218406070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5a380687f0 0x7f5a38199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 >> 172.21.15.23:0/218406070 conn(0x7f5a38076d40 msgr2=0x7f5a38104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 shutdown_connections 2024-03-30T04:09:38.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.475+0000 7f5a40800700 1 -- 172.21.15.23:0/218406070 wait complete. 2024-03-30T04:09:38.490 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:38 smithi023 ceph-mon[38254]: pgmap v237: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:09:38.490 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:38 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1992256184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:38.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:09:38.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:38 smithi193 ceph-mon[79581]: pgmap v237: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:09:38.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:38 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1992256184' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 -- 172.21.15.23:0/1836753267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c100f70 msgr2=0x7fda9c101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 --2- 172.21.15.23:0/1836753267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c100f70 0x7fda9c101350 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009990 tx=0x7fda8c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 -- 172.21.15.23:0/1836753267 shutdown_connections 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 --2- 172.21.15.23:0/1836753267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c106090 0x7fda9c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 --2- 172.21.15.23:0/1836753267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c100f70 0x7fda9c101350 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 --2- 172.21.15.23:0/1836753267 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c101920 0x7fda9c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 -- 172.21.15.23:0/1836753267 >> 172.21.15.23:0/1836753267 conn(0x7fda9c0fc7f0 msgr2=0x7fda9c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 -- 172.21.15.23:0/1836753267 shutdown_connections 2024-03-30T04:09:38.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.707+0000 7fdaa2400700 1 -- 172.21.15.23:0/1836753267 wait complete. 2024-03-30T04:09:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 Processor -- start 2024-03-30T04:09:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 -- start start 2024-03-30T04:09:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c100f70 0x7fda9c199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 0x7fda9c19e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c19e770 con 0x7fda9c19dcf0 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c19e8e0 con 0x7fda9c106090 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fdaa2400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c19eaa0 con 0x7fda9c100f70 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.709+0000 7fda9be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 0x7fda9c19e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35104/0 (socket says 172.21.15.23:35104) 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 -- 172.21.15.23:0/3421165325 learned_addr learned my addr 172.21.15.23:0/3421165325 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9b400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c100f70 0x7fda9c199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 0x7fda9c19e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51574/0 (socket says 172.21.15.23:51574) 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 -- 172.21.15.23:0/3421165325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c100f70 msgr2=0x7fda9c199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c100f70 0x7fda9c199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 -- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 msgr2=0x7fda9c19e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 0x7fda9c19e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 -- 172.21.15.23:0/3421165325 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda88009520 con 0x7fda9c106090 2024-03-30T04:09:38.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda9aa00700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fda8800ea30 tx=0x7fda8800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.710+0000 7fda93e00700 1 -- 172.21.15.23:0/3421165325 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8800cbd0 con 0x7fda9c106090 2024-03-30T04:09:38.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.711+0000 7fda93e00700 1 -- 172.21.15.23:0/3421165325 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda8800eec0 con 0x7fda9c106090 2024-03-30T04:09:38.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.711+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda94009c40 con 0x7fda9c106090 2024-03-30T04:09:38.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.711+0000 7fda93e00700 1 -- 172.21.15.23:0/3421165325 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88003d20 con 0x7fda9c106090 2024-03-30T04:09:38.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.711+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c0095f0 con 0x7fda9c106090 2024-03-30T04:09:38.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.714+0000 7fda93e00700 1 -- 172.21.15.23:0/3421165325 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fda88003ee0 con 0x7fda9c106090 2024-03-30T04:09:38.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.715+0000 7fda93e00700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 0x7fda800a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.715+0000 7fda93e00700 1 -- 172.21.15.23:0/3421165325 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fda88014070 con 0x7fda9c106090 2024-03-30T04:09:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.715+0000 7fdab5e77b80 1 Processor -- start 2024-03-30T04:09:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.715+0000 7fda9b400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 0x7fda800a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 -- start start 2024-03-30T04:09:38.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c577498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c57749df0 0x564c5774c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c57697990 con 0x564c5773eef0 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c57695350 con 0x564c5773e5d0 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c57690c40 con 0x564c57749df0 2024-03-30T04:09:38.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fda9b400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 0x7fda800a3750 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda8c006010 tx=0x7fda8c00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fda9b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fda9b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35110/0 (socket says 172.21.15.23:35110) 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fda9b400700 1 -- 172.21.15.23:0/4202267452 learned_addr learned my addr 172.21.15.23:0/4202267452 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.716+0000 7fda9aa00700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c577498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9be00700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c57749df0 0x564c5774c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 -- 172.21.15.23:0/4202267452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c57749df0 msgr2=0x564c5774c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c57749df0 0x564c5774c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 -- 172.21.15.23:0/4202267452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 msgr2=0x564c5773e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 -- 172.21.15.23:0/4202267452 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c575123b0 con 0x564c5773eef0 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9b400700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda9aa00700 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c577498b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca750 tx=0x7fda88005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1094f5b90873be server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda90c00700 1 -- 172.21.15.23:0/4202267452 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880caea0 con 0x564c5773eef0 2024-03-30T04:09:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda90c00700 1 -- 172.21.15.23:0/4202267452 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda8809df40 con 0x564c5773eef0 2024-03-30T04:09:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.717+0000 7fda90c00700 1 -- 172.21.15.23:0/4202267452 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880ca9d0 con 0x564c5773eef0 2024-03-30T04:09:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 -- 172.21.15.23:0/4202267452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 msgr2=0x564c577498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c577498b0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca750 tx=0x7fda88005760 comp rx=0 tx=0).stop 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 -- 172.21.15.23:0/4202267452 shutdown_connections 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c57749df0 0x564c5774c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773e5d0 0x564c5773e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/4202267452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c577498b0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 -- 172.21.15.23:0/4202267452 >> 172.21.15.23:0/4202267452 conn(0x564c5769e6a0 msgr2=0x564c5773aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 -- 172.21.15.23:0/4202267452 shutdown_connections 2024-03-30T04:09:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.718+0000 7fdab5e77b80 1 -- 172.21.15.23:0/4202267452 wait complete. 2024-03-30T04:09:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.719+0000 7fdab5e77b80 1 -- start start 2024-03-30T04:09:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.719+0000 7fdab5e77b80 1 Processor -- start 2024-03-30T04:09:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 0x564c57746b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57747280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c5762cd80 con 0x564c5773e5d0 2024-03-30T04:09:38.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c5762ca20 con 0x564c57749df0 2024-03-30T04:09:38.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c575146a0 con 0x564c5773eef0 2024-03-30T04:09:38.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fdab5e77b80 10 client.0 _ll_register_callbacks cb 0x564c57636470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 0x564c57746b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51604/0 (socket says 172.21.15.23:51604) 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9b400700 1 -- 172.21.15.23:0/1993887401 learned_addr learned my addr 172.21.15.23:0/1993887401 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9be00700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57747280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 -- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 msgr2=0x564c57747280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57747280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 -- 172.21.15.23:0/1993887401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 msgr2=0x564c57749870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9aa00700 1 -- 172.21.15.23:0/1993887401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c10e480 con 0x564c5773eef0 2024-03-30T04:09:38.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.720+0000 7fda9b400700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 0x564c57746b90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca750 tx=0x7fda880ca6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f16466ec27b44df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880caea0 con 0x564c5773eef0 2024-03-30T04:09:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fda880ca9d0 con 0x564c5773eef0 2024-03-30T04:09:38.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c57651380 con 0x564c5773eef0 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda880cab90 con 0x564c5773eef0 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fdab5e77b80 10 client.27658 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fda8c0095f0 con 0x564c5773eef0 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564c57651380 con 0x564c5773eef0 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.721+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fda88099600 con 0x564c5773eef0 2024-03-30T04:09:38.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 10 client.27658 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 10 client.27658 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 20 client.27658 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c57759660 0x564c5775ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 4 client.27658 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c57689700 con 0x564c57759660 2024-03-30T04:09:38.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fda9b400700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c57759660 0x564c5775ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fda9b400700 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c57759660 0x564c5775ba50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce745a0137b4f23 server_cookie=d1f1a3b0d6a04c01 in_seq=0 out_seq=0 2024-03-30T04:09:38.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.722+0000 7fda85600700 10 client.27658 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:38.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fda85600700 1 -- 172.21.15.23:0/1993887401 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564c57689700 con 0x564c57759660 2024-03-30T04:09:38.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fda85600700 10 client.27658 handle_command_reply: tid=0 2024-03-30T04:09:38.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 client.27658 shutdown 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 msgr2=0x564c57746b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 0x564c57746b90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda880ca750 tx=0x7fda880ca6c0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 shutdown_connections 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c57759660 0x564c5775ba50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57747280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773e5d0 0x564c57749870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fda85600700 0 client.27658 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1993887401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773eef0 0x564c57746b90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.724+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 >> 172.21.15.23:0/1993887401 conn(0x564c5769e6a0 msgr2=0x564c57740670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.725+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 shutdown_connections 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.725+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1993887401 wait complete. 2024-03-30T04:09:38.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.725+0000 7fdab5e77b80 20 client.27658 trim_cache size 0 max 16384 2024-03-30T04:09:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.741+0000 7fdab5e77b80 1 Processor -- start 2024-03-30T04:09:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.741+0000 7fdab5e77b80 1 -- start start 2024-03-30T04:09:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c57755a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57733f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c57747860 con 0x564c5773eef0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c5773f850 con 0x564c57749df0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c57747f80 con 0x564c5773e5d0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51612/0 (socket says 172.21.15.23:51612) 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 -- 172.21.15.23:0/2459076295 learned_addr learned my addr 172.21.15.23:0/2459076295 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9be00700 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57733f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9b400700 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c57755a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 -- 172.21.15.23:0/2459076295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 msgr2=0x564c57755a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c57755a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 -- 172.21.15.23:0/2459076295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 msgr2=0x564c57733f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57733f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.742+0000 7fda9aa00700 1 -- 172.21.15.23:0/2459076295 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c574beb60 con 0x564c5773eef0 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.743+0000 7fda9aa00700 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fda8800c5b0 tx=0x7fda880045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642386bda0399a3d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.743+0000 7fda91600700 1 -- 172.21.15.23:0/2459076295 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880caea0 con 0x564c5773eef0 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.743+0000 7fda91600700 1 -- 172.21.15.23:0/2459076295 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda88099600 con 0x564c5773eef0 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.743+0000 7fda91600700 1 -- 172.21.15.23:0/2459076295 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880a2150 con 0x564c5773eef0 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- 172.21.15.23:0/2459076295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 msgr2=0x564c57750a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fda8800c5b0 tx=0x7fda880045e0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- 172.21.15.23:0/2459076295 shutdown_connections 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c57749df0 0x564c57733f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c57755a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/2459076295 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c5773eef0 0x564c57750a70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- 172.21.15.23:0/2459076295 >> 172.21.15.23:0/2459076295 conn(0x564c5769e6a0 msgr2=0x564c577335a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- 172.21.15.23:0/2459076295 shutdown_connections 2024-03-30T04:09:38.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- 172.21.15.23:0/2459076295 wait complete. 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 -- start start 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.744+0000 7fdab5e77b80 1 Processor -- start 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c5774e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 0x564c5774ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c57749df0 0x564c577ac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c5774f4a0 con 0x564c57749df0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c577ac970 con 0x564c5773eef0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c577acb00 con 0x564c5773e5d0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fdab5e77b80 10 client.0 _ll_register_callbacks cb 0x564c57636470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fda9b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c5774e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fda9b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c5774e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33608/0 (socket says 172.21.15.23:33608) 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fda9b400700 1 -- 172.21.15.23:0/1876868573 learned_addr learned my addr 172.21.15.23:0/1876868573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 0x564c5774ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.745+0000 7fda9be00700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c57749df0 0x564c577ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 -- 172.21.15.23:0/1876868573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 msgr2=0x564c5774e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c5774e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 -- 172.21.15.23:0/1876868573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c57749df0 msgr2=0x564c577ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c57749df0 0x564c577ac430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 -- 172.21.15.23:0/1876868573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c0095f0 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda9aa00700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 0x564c5774ed70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fda88009fd0 tx=0x7fda88003780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b78066abc144c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda85600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880caea0 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fda880a2150 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda8809df40 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda8809a580 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fdab5e77b80 10 client.27662 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.746+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564c57656ea0 con 0x564c5773eef0 2024-03-30T04:09:38.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda8809df40 con 0x564c5773eef0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fda88099070 con 0x564c5773eef0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 10 client.27662 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 10 client.27662 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 20 client.27662 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c577addf0 0x564c577b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 4 client.27662 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564c574dfcf0 con 0x564c577addf0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.747+0000 7fda9b400700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c577addf0 0x564c577b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fda9b400700 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c577addf0 0x564c577b01e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e16d1713fc5ba126 server_cookie=b66c26dd93295cd6 in_seq=0 out_seq=0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fda85600700 10 client.27662 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fda85600700 1 -- 172.21.15.23:0/1876868573 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564c574dfcf0 con 0x564c577addf0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fda85600700 10 client.27662 handle_command_reply: tid=0 2024-03-30T04:09:38.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 client.27662 shutdown 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 msgr2=0x564c5774ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 0x564c5774ed70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fda88009fd0 tx=0x7fda88003780 comp rx=0 tx=0).stop 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 shutdown_connections 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x564c577addf0 0x564c577b01e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c57749df0 0x564c577ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c5773e5d0 0x564c5774e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 --2- 172.21.15.23:0/1876868573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c5773eef0 0x564c5774ed70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fda85600700 0 client.27662 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 >> 172.21.15.23:0/1876868573 conn(0x564c5769e6a0 msgr2=0x564c577551c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 shutdown_connections 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.748+0000 7fdab5e77b80 1 -- 172.21.15.23:0/1876868573 wait complete. 2024-03-30T04:09:38.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdab5e77b80 20 client.27662 trim_cache size 0 max 16384 2024-03-30T04:09:38.759 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 msgr2=0x7fda800a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 0x7fda800a3750 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fda8c006010 tx=0x7fda8c00b260 comp rx=0 tx=0).stop 2024-03-30T04:09:38.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 msgr2=0x7fda9c1999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fda8800ea30 tx=0x7fda8800ed40 comp rx=0 tx=0).stop 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 shutdown_connections 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda9c19dcf0 0x7fda9c19e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda9c100f70 0x7fda9c199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda800a1290 0x7fda800a3750 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 --2- 172.21.15.23:0/3421165325 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda9c106090 0x7fda9c1999a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 >> 172.21.15.23:0/3421165325 conn(0x7fda9c0fc7f0 msgr2=0x7fda9c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 shutdown_connections 2024-03-30T04:09:38.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.749+0000 7fdaa2400700 1 -- 172.21.15.23:0/3421165325 wait complete. 2024-03-30T04:09:38.766 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 -- 172.21.15.23:0/774119314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc106090 msgr2=0x7f33cc10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 --2- 172.21.15.23:0/774119314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc106090 0x7f33cc10dc10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f33c4009f90 tx=0x7f33c400b330 comp rx=0 tx=0).stop 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 -- 172.21.15.23:0/774119314 shutdown_connections 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 --2- 172.21.15.23:0/774119314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc106090 0x7f33cc10dc10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 --2- 172.21.15.23:0/774119314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc100f70 0x7f33cc101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 --2- 172.21.15.23:0/774119314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc101920 0x7f33cc105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 -- 172.21.15.23:0/774119314 >> 172.21.15.23:0/774119314 conn(0x7f33cc0fc7f0 msgr2=0x7f33cc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 -- 172.21.15.23:0/774119314 shutdown_connections 2024-03-30T04:09:38.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.981+0000 7f33d0c00700 1 -- 172.21.15.23:0/774119314 wait complete. 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.982+0000 7f33d0c00700 1 Processor -- start 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.982+0000 7f33d0c00700 1 -- start start 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc106090 0x7f33cc197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc19bbe0 0x7f33cc19c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc19c660 con 0x7f33cc19bbe0 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc19c7d0 con 0x7f33cc106090 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33d0c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33cc19c990 con 0x7f33cc100f70 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc106090 0x7f33cc197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33636/0 (socket says 172.21.15.23:33636) 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 -- 172.21.15.23:0/1920583728 learned_addr learned my addr 172.21.15.23:0/1920583728 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33ca800700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc19bbe0 0x7f33cc19c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 -- 172.21.15.23:0/1920583728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc106090 msgr2=0x7f33cc197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc106090 0x7f33cc197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 -- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc19bbe0 msgr2=0x7f33cc19c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.983+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc19bbe0 0x7f33cc19c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33c9e00700 1 -- 172.21.15.23:0/1920583728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c4009c40 con 0x7f33cc100f70 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00e9a0 tx=0x7f33bc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33c2a00700 1 -- 172.21.15.23:0/1920583728 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc009790 con 0x7f33cc100f70 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33c2a00700 1 -- 172.21.15.23:0/1920583728 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33bc00ce80 con 0x7f33cc100f70 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc1a3ef0 con 0x7f33cc100f70 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33c2a00700 1 -- 172.21.15.23:0/1920583728 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc018680 con 0x7f33cc100f70 2024-03-30T04:09:38.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.984+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1a4370 con 0x7f33cc100f70 2024-03-30T04:09:38.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.987+0000 7f33c2a00700 1 -- 172.21.15.23:0/1920583728 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f33bc010430 con 0x7f33cc100f70 2024-03-30T04:09:38.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33c2a00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 0x7f33b00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33e465db80 1 Processor -- start 2024-03-30T04:09:38.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33c2a00700 1 -- 172.21.15.23:0/1920583728 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f33bc014070 con 0x7f33cc100f70 2024-03-30T04:09:38.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33e465db80 1 -- start start 2024-03-30T04:09:38.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33c9400700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 0x7f33b00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33e465db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.989+0000 7f33e465db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33e465db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33e465db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b11608990 con 0x564b116afef0 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33e465db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b11606350 con 0x564b116af5d0 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33e465db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b11601c40 con 0x564b116bf200 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35172/0 (socket says 172.21.15.23:35172) 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 -- 172.21.15.23:0/1604209746 learned_addr learned my addr 172.21.15.23:0/1604209746 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9400700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33ca800700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 -- 172.21.15.23:0/1604209746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 msgr2=0x564b116c15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 -- 172.21.15.23:0/1604209746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 msgr2=0x564b116becc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116becc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9400700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 0x7f33b00a3750 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33cc198850 tx=0x7f33b8011440 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 -- 172.21.15.23:0/1604209746 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b114833b0 con 0x564b116af5d0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33ca800700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.990+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0a1f00 tx=0x7f33bc09a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d073ef899530500 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33b7000700 1 -- 172.21.15.23:0/1604209746 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc09dde0 con 0x564b116af5d0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33b7000700 1 -- 172.21.15.23:0/1604209746 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33bc0a2430 con 0x564b116af5d0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33b7000700 1 -- 172.21.15.23:0/1604209746 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc0996a0 con 0x564b116af5d0 2024-03-30T04:09:38.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 -- 172.21.15.23:0/1604209746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 msgr2=0x564b116af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0a1f00 tx=0x7f33bc09a730 comp rx=0 tx=0).stop 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 -- 172.21.15.23:0/1604209746 shutdown_connections 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116af5d0 0x564b116af9b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 --2- 172.21.15.23:0/1604209746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116becc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.991+0000 7f33e465db80 1 -- 172.21.15.23:0/1604209746 >> 172.21.15.23:0/1604209746 conn(0x564b1160f6a0 msgr2=0x564b116aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.992+0000 7f33e465db80 1 -- 172.21.15.23:0/1604209746 shutdown_connections 2024-03-30T04:09:38.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.992+0000 7f33e465db80 1 -- 172.21.15.23:0/1604209746 wait complete. 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.992+0000 7f33e465db80 1 -- start start 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.992+0000 7f33e465db80 1 Processor -- start 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116bbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116bc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b1159dd80 con 0x564b116afef0 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b1159da20 con 0x564b116bf200 2024-03-30T04:09:38.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b114856a0 con 0x564b116af5d0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33e465db80 10 client.0 _ll_register_callbacks cb 0x564b115a7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33658/0 (socket says 172.21.15.23:33658) 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116bbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 -- 172.21.15.23:0/893928529 learned_addr learned my addr 172.21.15.23:0/893928529 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33ca800700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116bc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 -- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 msgr2=0x564b116bc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116bc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 -- 172.21.15.23:0/893928529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 msgr2=0x564b116bbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116bbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.993+0000 7f33c9e00700 1 -- 172.21.15.23:0/893928529 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc1a4370 con 0x564b116af5d0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33c9e00700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0964b0 tx=0x7f33bc010a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b8b8334da7970d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33ab400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:38.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33bc09de60 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f33bc098350 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b115c2380 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33bc09a420 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33e465db80 10 client.27676 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.994+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564b1142fb60 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564b115c2380 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f33bc094c50 con 0x564b116af5d0 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 10 client.27676 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 10 client.27676 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 20 client.27676 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b116ce990 0x564b116d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 4 client.27676 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b115fa700 con 0x564b116ce990 2024-03-30T04:09:38.998 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.995+0000 7f33c9400700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b116ce990 0x564b116d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33c9400700 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b116ce990 0x564b116d0db0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e5f73b01fcc29ec server_cookie=3c408f97f4250b0a in_seq=0 out_seq=0 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33ab400700 10 client.27676 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33ab400700 1 -- 172.21.15.23:0/893928529 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564b115fa700 con 0x564b116ce990 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33ab400700 10 client.27676 handle_command_reply: tid=0 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 client.27676 shutdown 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 msgr2=0x564b116bec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f33bc0964b0 tx=0x7f33bc010a10 comp rx=0 tx=0).stop 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 shutdown_connections 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116bc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b116ce990 0x564b116d0db0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116af5d0 0x564b116bec30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 --2- 172.21.15.23:0/893928529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116afef0 0x564b116bbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:38.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33ab400700 0 client.27676 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:39.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.996+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 >> 172.21.15.23:0/893928529 conn(0x564b1160f6a0 msgr2=0x564b116b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.997+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 shutdown_connections 2024-03-30T04:09:39.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.997+0000 7f33e465db80 1 -- 172.21.15.23:0/893928529 wait complete. 2024-03-30T04:09:39.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:38.997+0000 7f33e465db80 20 client.27676 trim_cache size 0 max 16384 2024-03-30T04:09:39.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.011+0000 7f33e465db80 1 Processor -- start 2024-03-30T04:09:39.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 -- start start 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116ab530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116a7840 con 0x564b116af5d0 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116b02d0 con 0x564b116bf200 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33e465db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116bcea0 con 0x564b116afef0 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33c9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33c9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33668/0 (socket says 172.21.15.23:33668) 2024-03-30T04:09:39.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33c9400700 1 -- 172.21.15.23:0/1327840439 learned_addr learned my addr 172.21.15.23:0/1327840439 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33ca800700 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.012+0000 7f33c9e00700 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 -- 172.21.15.23:0/1327840439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 msgr2=0x564b116ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 -- 172.21.15.23:0/1327840439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 msgr2=0x564b116d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 -- 172.21.15.23:0/1327840439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b115c7ea0 con 0x564b116afef0 2024-03-30T04:09:39.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33c9400700 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f33cc06a3d0 tx=0x7f33b801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f15383c1821a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33b7a00700 1 -- 172.21.15.23:0/1327840439 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b802d070 con 0x564b116afef0 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33b7a00700 1 -- 172.21.15.23:0/1327840439 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33b8028e20 con 0x564b116afef0 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.013+0000 7f33b7a00700 1 -- 172.21.15.23:0/1327840439 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8023040 con 0x564b116afef0 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 -- 172.21.15.23:0/1327840439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 msgr2=0x564b116c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f33cc06a3d0 tx=0x7f33b801c000 comp rx=0 tx=0).stop 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 -- 172.21.15.23:0/1327840439 shutdown_connections 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116bf200 0x564b116ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 --2- 172.21.15.23:0/1327840439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116afef0 0x564b116c5e80 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 -- 172.21.15.23:0/1327840439 >> 172.21.15.23:0/1327840439 conn(0x564b1160f6a0 msgr2=0x564b116a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 -- 172.21.15.23:0/1327840439 shutdown_connections 2024-03-30T04:09:39.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.014+0000 7f33e465db80 1 -- 172.21.15.23:0/1327840439 wait complete. 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 -- start start 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 Processor -- start 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116c4740 con 0x564b116af5d0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116c48b0 con 0x564b116afef0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b116c4a70 con 0x564b116bf200 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33e465db80 10 client.0 _ll_register_callbacks cb 0x564b115a7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33c9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33c9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33ca800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33c9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35198/0 (socket says 172.21.15.23:35198) 2024-03-30T04:09:39.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33c9e00700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51676/0 (socket says 172.21.15.23:51676) 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.015+0000 7f33c9400700 1 -- 172.21.15.23:0/334395114 learned_addr learned my addr 172.21.15.23:0/334395114 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 -- 172.21.15.23:0/334395114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 msgr2=0x564b116c4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 -- 172.21.15.23:0/334395114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 msgr2=0x564b116c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 -- 172.21.15.23:0/334395114 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b8028900 con 0x564b116afef0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33ca800700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9e00700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33c9400700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f33b802c000 tx=0x7f33b8009290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54cf15577bc94f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33ab400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.016+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b802d040 con 0x564b116afef0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33bc0a2430 con 0x564b116afef0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f33b8005200 con 0x564b116afef0 2024-03-30T04:09:39.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33b8023040 con 0x564b116afef0 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 10 client.27680 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564b11487cd0 con 0x564b116afef0 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f33bc0a2430 con 0x564b116afef0 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f33b8026070 con 0x564b116afef0 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 10 client.27680 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 10 client.27680 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 20 client.27680 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b11723ad0 0x564b11725ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 4 client.27680 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:39.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.017+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564b11450cf0 con 0x564b11723ad0 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33c9e00700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b11723ad0 0x564b11725ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33ab400700 10 client.27680 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33c9e00700 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b11723ad0 0x564b11725ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8d4c320c55a53998 server_cookie=7c6b85527c4f8937 in_seq=0 out_seq=0 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33ab400700 1 -- 172.21.15.23:0/334395114 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564b11450cf0 con 0x564b11723ad0 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33ab400700 10 client.27680 handle_command_reply: tid=0 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.018+0000 7f33e465db80 1 client.27680 shutdown 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 msgr2=0x564b116c5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f33b802c000 tx=0x7f33b8009290 comp rx=0 tx=0).stop 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 shutdown_connections 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564b11723ad0 0x564b11725ef0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b116bf200 0x564b116c4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564b116af5d0 0x564b116c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33ab400700 0 client.27680 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:09:39.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 --2- 172.21.15.23:0/334395114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564b116afef0 0x564b116c5e70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 >> 172.21.15.23:0/334395114 conn(0x564b1160f6a0 msgr2=0x564b116ab4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 shutdown_connections 2024-03-30T04:09:39.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 1 -- 172.21.15.23:0/334395114 wait complete. 2024-03-30T04:09:39.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.019+0000 7f33e465db80 20 client.27680 trim_cache size 0 max 16384 2024-03-30T04:09:39.022 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 msgr2=0x7f33b00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 0x7f33b00a3750 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f33cc198850 tx=0x7f33b8011440 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 msgr2=0x7f33cc197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00e9a0 tx=0x7f33bc00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 shutdown_connections 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33cc19bbe0 0x7f33cc19c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f33cc100f70 0x7f33cc197300 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f33b00a1290 0x7f33b00a3750 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 --2- 172.21.15.23:0/1920583728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33cc106090 0x7f33cc197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 >> 172.21.15.23:0/1920583728 conn(0x7f33cc0fc7f0 msgr2=0x7f33cc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 shutdown_connections 2024-03-30T04:09:39.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.020+0000 7f33d0c00700 1 -- 172.21.15.23:0/1920583728 wait complete. 2024-03-30T04:09:39.037 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:09:39.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 764513 10.74 MB/sec execute 256 sec latency 19.321 ms 2024-03-30T04:09:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 -- 172.21.15.23:0/2104779779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c101920 msgr2=0x7f995c105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 --2- 172.21.15.23:0/2104779779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c101920 0x7f995c105960 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9948009990 tx=0x7f9948009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 -- 172.21.15.23:0/2104779779 shutdown_connections 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 --2- 172.21.15.23:0/2104779779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c106090 0x7f995c10dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 --2- 172.21.15.23:0/2104779779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c100f70 0x7f995c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 --2- 172.21.15.23:0/2104779779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c101920 0x7f995c105960 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 -- 172.21.15.23:0/2104779779 >> 172.21.15.23:0/2104779779 conn(0x7f995c0fc7f0 msgr2=0x7f995c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 -- 172.21.15.23:0/2104779779 shutdown_connections 2024-03-30T04:09:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.249+0000 7f9963400700 1 -- 172.21.15.23:0/2104779779 wait complete. 2024-03-30T04:09:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.250+0000 7f9963400700 1 Processor -- start 2024-03-30T04:09:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 -- start start 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c106090 0x7f995c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c073b30 0x7f995c1a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1a5990 con 0x7f995c106090 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1a5b00 con 0x7f995c100f70 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9963400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c1a5cc0 con 0x7f995c073b30 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9960a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9960a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35218/0 (socket says 172.21.15.23:35218) 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f9960a00700 1 -- 172.21.15.23:0/3279752774 learned_addr learned my addr 172.21.15.23:0/3279752774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9961400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c073b30 0x7f995c1a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.251+0000 7f995be00700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c106090 0x7f995c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 -- 172.21.15.23:0/3279752774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c073b30 msgr2=0x7f995c1a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c073b30 0x7f995c1a53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 -- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c106090 msgr2=0x7f995c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c106090 0x7f995c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 -- 172.21.15.23:0/3279752774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480095f0 con 0x7f995c100f70 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9960a00700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f995000e9a0 tx=0x7f995000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9959600700 1 -- 172.21.15.23:0/3279752774 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950009790 con 0x7f995c100f70 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9959600700 1 -- 172.21.15.23:0/3279752774 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9950010430 con 0x7f995c100f70 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9959600700 1 -- 172.21.15.23:0/3279752774 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950018680 con 0x7f995c100f70 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.252+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c1a5fa0 con 0x7f995c100f70 2024-03-30T04:09:39.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.253+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c1a63d0 con 0x7f995c100f70 2024-03-30T04:09:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.255+0000 7f9959600700 1 -- 172.21.15.23:0/3279752774 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f99500105a0 con 0x7f995c100f70 2024-03-30T04:09:39.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 Processor -- start 2024-03-30T04:09:39.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 -- start start 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec35d0 0x55d7daec39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daece8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dae1c990 con 0x55d7daec3ef0 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dae1a350 con 0x55d7daec35d0 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9976f52b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dae15c40 con 0x55d7daecedf0 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f9959600700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 0x7f99440a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.257+0000 7f995be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f9959600700 1 -- 172.21.15.23:0/3279752774 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9950014070 con 0x7f995c100f70 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33696/0 (socket says 172.21.15.23:33696) 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 -- 172.21.15.23:0/979043125 learned_addr learned my addr 172.21.15.23:0/979043125 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f9961400700 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec35d0 0x55d7daec39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f9960a00700 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daece8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f9961400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 0x7f99440a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 -- 172.21.15.23:0/979043125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec35d0 msgr2=0x55d7daec39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec35d0 0x55d7daec39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 -- 172.21.15.23:0/979043125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 msgr2=0x55d7daece8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daece8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 -- 172.21.15.23:0/979043125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7dac973b0 con 0x55d7daecedf0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f995be00700 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9948009990 tx=0x7f9948005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1147998ca247257 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.258+0000 7f9961400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 0x7f99440a37d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9954009ec0 tx=0x7f995400b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f994dc00700 1 -- 172.21.15.23:0/979043125 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994801d070 con 0x55d7daecedf0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f994dc00700 1 -- 172.21.15.23:0/979043125 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f994801e040 con 0x55d7daecedf0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f994dc00700 1 -- 172.21.15.23:0/979043125 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948004880 con 0x55d7daecedf0 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 -- 172.21.15.23:0/979043125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 msgr2=0x55d7daed11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f9948009990 tx=0x7f9948005ec0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 -- 172.21.15.23:0/979043125 shutdown_connections 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daecedf0 0x55d7daed11e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec35d0 0x55d7daec39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 --2- 172.21.15.23:0/979043125 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daece8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.259+0000 7f9976f52b80 1 -- 172.21.15.23:0/979043125 >> 172.21.15.23:0/979043125 conn(0x55d7dae236a0 msgr2=0x55d7daebfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.260+0000 7f9976f52b80 1 -- 172.21.15.23:0/979043125 shutdown_connections 2024-03-30T04:09:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.260+0000 7f9976f52b80 1 -- 172.21.15.23:0/979043125 wait complete. 2024-03-30T04:09:39.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.260+0000 7f9976f52b80 1 -- start start 2024-03-30T04:09:39.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.260+0000 7f9976f52b80 1 Processor -- start 2024-03-30T04:09:39.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec3ef0 0x55d7daecba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daecc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dadb1d80 con 0x55d7daec35d0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dadb1a20 con 0x55d7daecedf0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7dac996a0 con 0x55d7daec3ef0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9976f52b80 10 client.0 _ll_register_callbacks cb 0x55d7dadbb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9961400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daecc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f995be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec3ef0 0x55d7daecba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51722/0 (socket says 172.21.15.23:51722) 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9961400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daecc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35230/0 (socket says 172.21.15.23:35230) 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 -- 172.21.15.23:0/183798848 learned_addr learned my addr 172.21.15.23:0/183798848 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 -- 172.21.15.23:0/183798848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec3ef0 msgr2=0x55d7daecba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec3ef0 0x55d7daecba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 -- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 msgr2=0x55d7daecc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.261+0000 7f9960a00700 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daecc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9960a00700 1 -- 172.21.15.23:0/183798848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c10e480 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9960a00700 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f99500969a0 tx=0x7f99500caad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac502688d2e88d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9942000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995009a390 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9950099f30 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d7dadd6380 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995009dd60 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9976f52b80 10 client.17688 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.262+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d7dac43b60 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d7dadd6380 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9950099420 con 0x55d7daec35d0 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 10 client.17688 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 20 client.17688 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daede8b0 0x55d7daee0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 4 client.17688 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7dae0e700 con 0x55d7daede8b0 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.263+0000 7f995be00700 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daede8b0 0x55d7daee0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f995be00700 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daede8b0 0x55d7daee0ca0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d5dd2eb000fa5e4 server_cookie=234fa35bf59f7e1 in_seq=0 out_seq=0 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9942000700 10 client.17688 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9942000700 1 -- 172.21.15.23:0/183798848 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d7dae0e700 con 0x55d7daede8b0 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9942000700 10 client.17688 handle_command_reply: tid=0 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 client.17688 shutdown 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 msgr2=0x55d7daece7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f99500969a0 tx=0x7f99500caad0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 shutdown_connections 2024-03-30T04:09:39.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daecc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec35d0 0x55d7daece7d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.264+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daede8b0 0x55d7daee0ca0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9976f52b80 1 --2- 172.21.15.23:0/183798848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec3ef0 0x55d7daecba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9942000700 0 client.17688 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 >> 172.21.15.23:0/183798848 conn(0x55d7dae236a0 msgr2=0x55d7daec5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 shutdown_connections 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9976f52b80 1 -- 172.21.15.23:0/183798848 wait complete. 2024-03-30T04:09:39.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.265+0000 7f9976f52b80 20 client.17688 trim_cache size 0 max 16384 2024-03-30T04:09:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.278+0000 7f9976f52b80 1 Processor -- start 2024-03-30T04:09:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.278+0000 7f9976f52b80 1 -- start start 2024-03-30T04:09:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daec64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daebf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daed5f50 con 0x55d7daec3ef0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daec9fa0 con 0x55d7daecedf0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9976f52b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daecc640 con 0x55d7daec35d0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f995be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9960a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daec64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9961400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daebf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f995be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51736/0 (socket says 172.21.15.23:51736) 2024-03-30T04:09:39.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f995be00700 1 -- 172.21.15.23:0/394366969 learned_addr learned my addr 172.21.15.23:0/394366969 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9960a00700 1 -- 172.21.15.23:0/394366969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 msgr2=0x55d7daebf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9960a00700 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daebf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9960a00700 1 -- 172.21.15.23:0/394366969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 msgr2=0x55d7daed5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.279+0000 7f9960a00700 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f9960a00700 1 -- 172.21.15.23:0/394366969 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7daddbea0 con 0x55d7daec35d0 2024-03-30T04:09:39.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f995be00700 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f9960a00700 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daec64a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f995006d090 tx=0x7f9950010a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e49d388c8980a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f994e600700 1 -- 172.21.15.23:0/394366969 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995009a630 con 0x55d7daec35d0 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f994e600700 1 -- 172.21.15.23:0/394366969 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9950099f30 con 0x55d7daec35d0 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.280+0000 7f994e600700 1 -- 172.21.15.23:0/394366969 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500a23b0 con 0x55d7daec35d0 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- 172.21.15.23:0/394366969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 msgr2=0x55d7daec64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daec64a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f995006d090 tx=0x7f9950010a90 comp rx=0 tx=0).stop 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- 172.21.15.23:0/394366969 shutdown_connections 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daecedf0 0x55d7daebf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daec64a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 --2- 172.21.15.23:0/394366969 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daec3ef0 0x55d7daed5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- 172.21.15.23:0/394366969 >> 172.21.15.23:0/394366969 conn(0x55d7dae236a0 msgr2=0x55d7daeb85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- 172.21.15.23:0/394366969 shutdown_connections 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- 172.21.15.23:0/394366969 wait complete. 2024-03-30T04:09:39.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 -- start start 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.281+0000 7f9976f52b80 1 Processor -- start 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec3ef0 0x55d7daed3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 0x55d7daf316b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daf31bf0 con 0x55d7daecedf0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daf31d30 con 0x55d7daec3ef0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7daf31ec0 con 0x55d7daec35d0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9976f52b80 10 client.0 _ll_register_callbacks cb 0x55d7dadbb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9960a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f995be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec3ef0 0x55d7daed3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9960a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33720/0 (socket says 172.21.15.23:33720) 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9960a00700 1 -- 172.21.15.23:0/302133228 learned_addr learned my addr 172.21.15.23:0/302133228 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.282+0000 7f9961400700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 0x55d7daf316b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 -- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec3ef0 msgr2=0x55d7daed3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec3ef0 0x55d7daed3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 -- 172.21.15.23:0/302133228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 msgr2=0x55d7daf316b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 0x55d7daf316b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 -- 172.21.15.23:0/302133228 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7dac973b0 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9961400700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 0x55d7daf316b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9960a00700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9950096430 tx=0x7f9950099d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d434dff4e5e685fb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9942000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995009a630 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f99500a23b0 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f994801e040 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995009dd60 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9976f52b80 10 client.27700 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d7dac43b60 con 0x55d7daec35d0 2024-03-30T04:09:39.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.283+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f994801e040 con 0x55d7daec35d0 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9950095d20 con 0x55d7daec35d0 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 10 client.27700 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 10 client.27700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 20 client.27700 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daf331b0 0x55d7daf355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 4 client.27700 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d7dac64cf0 con 0x55d7daf331b0 2024-03-30T04:09:39.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f995be00700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daf331b0 0x55d7daf355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f9942000700 10 client.27700 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.284+0000 7f995be00700 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daf331b0 0x55d7daf355a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a84b8c114e85e9bb server_cookie=95494622a6145225 in_seq=0 out_seq=0 2024-03-30T04:09:39.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9942000700 1 -- 172.21.15.23:0/302133228 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d7dac64cf0 con 0x55d7daf331b0 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9942000700 10 client.27700 handle_command_reply: tid=0 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 client.27700 shutdown 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 msgr2=0x55d7daed3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9950096430 tx=0x7f9950099d00 comp rx=0 tx=0).stop 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 shutdown_connections 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d7daecedf0 0x55d7daf316b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d7daf331b0 0x55d7daf355a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d7daec35d0 0x55d7daed3620 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 --2- 172.21.15.23:0/302133228 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d7daec3ef0 0x55d7daed3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 >> 172.21.15.23:0/302133228 conn(0x55d7dae236a0 msgr2=0x55d7daebf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9942000700 0 client.27700 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 shutdown_connections 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 1 -- 172.21.15.23:0/302133228 wait complete. 2024-03-30T04:09:39.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.285+0000 7f9976f52b80 20 client.27700 trim_cache size 0 max 16384 2024-03-30T04:09:39.289 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:09:39.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 msgr2=0x7f99440a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 0x7f99440a37d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9954009ec0 tx=0x7f995400b040 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 msgr2=0x7f995c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f995000e9a0 tx=0x7f995000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 shutdown_connections 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f995c073b30 0x7f995c1a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99440a1310 0x7f99440a37d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f995c100f70 0x7f995c0730b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 --2- 172.21.15.23:0/3279752774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f995c106090 0x7f995c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 >> 172.21.15.23:0/3279752774 conn(0x7f995c0fc7f0 msgr2=0x7f995c0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 shutdown_connections 2024-03-30T04:09:39.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:39.286+0000 7f9963400700 1 -- 172.21.15.23:0/3279752774 wait complete. 2024-03-30T04:09:40.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 766758 10.75 MB/sec execute 257 sec latency 19.190 ms 2024-03-30T04:09:40.304 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:09:40.304 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:09:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:40 smithi154 ceph-mon[79161]: pgmap v238: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-03-30T04:09:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:40 smithi193 ceph-mon[79581]: pgmap v238: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-03-30T04:09:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:40 smithi023 ceph-mon[38254]: pgmap v238: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.5 MiB/s wr, 251 op/s 2024-03-30T04:09:40.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 -- 172.21.15.23:0/461659381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988100800 msgr2=0x7fb988104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 --2- 172.21.15.23:0/461659381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988100800 0x7fb988104830 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb974009990 tx=0x7fb974009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 -- 172.21.15.23:0/461659381 shutdown_connections 2024-03-30T04:09:40.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 --2- 172.21.15.23:0/461659381 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb988104f60 0x7fb98810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 --2- 172.21.15.23:0/461659381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9880ffe50 0x7fb988100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 --2- 172.21.15.23:0/461659381 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988100800 0x7fb988104830 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 -- 172.21.15.23:0/461659381 >> 172.21.15.23:0/461659381 conn(0x7fb988076d40 msgr2=0x7fb988077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 -- 172.21.15.23:0/461659381 shutdown_connections 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.527+0000 7fb98de00700 1 -- 172.21.15.23:0/461659381 wait complete. 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.528+0000 7fb98de00700 1 Processor -- start 2024-03-30T04:09:40.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.528+0000 7fb98de00700 1 -- start start 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988104f60 0x7fb98810fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb988110360 0x7fb9881a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881a59a0 con 0x7fb9880ffe50 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881a5b10 con 0x7fb988104f60 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb98de00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9881a5cd0 con 0x7fb988110360 2024-03-30T04:09:40.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51776/0 (socket says 172.21.15.23:51776) 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 -- 172.21.15.23:0/1078026941 learned_addr learned my addr 172.21.15.23:0/1078026941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb987400700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb988110360 0x7fb9881a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 -- 172.21.15.23:0/1078026941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb988110360 msgr2=0x7fb9881a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb988110360 0x7fb9881a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 -- 172.21.15.23:0/1078026941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988104f60 msgr2=0x7fb98810fe20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988104f60 0x7fb98810fe20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.529+0000 7fb986a00700 1 -- 172.21.15.23:0/1078026941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb980009c40 con 0x7fb9880ffe50 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb986a00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e9c0 tx=0x7fb97800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb98ca00700 1 -- 172.21.15.23:0/1078026941 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9780097b0 con 0x7fb9880ffe50 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9740095f0 con 0x7fb9880ffe50 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb98ca00700 1 -- 172.21.15.23:0/1078026941 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb978004cf0 con 0x7fb9880ffe50 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881a6250 con 0x7fb9880ffe50 2024-03-30T04:09:40.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.530+0000 7fb98ca00700 1 -- 172.21.15.23:0/1078026941 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9780055b0 con 0x7fb9880ffe50 2024-03-30T04:09:40.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.533+0000 7fb98ca00700 1 -- 172.21.15.23:0/1078026941 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb97800cc60 con 0x7fb9880ffe50 2024-03-30T04:09:40.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.534+0000 7fb9a183cb80 1 Processor -- start 2024-03-30T04:09:40.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.534+0000 7fb9a183cb80 1 -- start start 2024-03-30T04:09:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e82900aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb98ca00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 0x7fb96c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829018260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828f63a10 con 0x55e82900a650 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828f613d0 con 0x55e82900af70 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828f5ccc0 con 0x55e829015e70 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb98ca00700 1 -- 172.21.15.23:0/1078026941 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb978014070 con 0x7fb9880ffe50 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35266/0 (socket says 172.21.15.23:35266) 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 -- 172.21.15.23:0/3899768078 learned_addr learned my addr 172.21.15.23:0/3899768078 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986000700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 0x7fb96c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb987400700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829018260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb987400700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e82900aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 -- 172.21.15.23:0/3899768078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 msgr2=0x55e829018260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829018260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 -- 172.21.15.23:0/3899768078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 msgr2=0x55e82900aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e82900aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb986a00700 1 -- 172.21.15.23:0/3899768078 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e828dde430 con 0x55e82900af70 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.535+0000 7fb987400700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e82900aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb986a00700 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb978096c70 tx=0x7fb978099e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537ff1f99bcffade server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb986000700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 0x7fb96c0a3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9740098e0 tx=0x7fb974000bc0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb973e00700 1 -- 172.21.15.23:0/3899768078 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97809e030 con 0x55e82900af70 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb973e00700 1 -- 172.21.15.23:0/3899768078 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9780a2680 con 0x55e82900af70 2024-03-30T04:09:40.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb973e00700 1 -- 172.21.15.23:0/3899768078 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97809a460 con 0x55e82900af70 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3899768078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 msgr2=0x55e829015930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.536+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb978096c70 tx=0x7fb978099e40 comp rx=0 tx=0).stop 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3899768078 shutdown_connections 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829018260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e82900aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3899768078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900af70 0x55e829015930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3899768078 >> 172.21.15.23:0/3899768078 conn(0x55e828f6a720 msgr2=0x55e829006b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3899768078 shutdown_connections 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3899768078 wait complete. 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 -- start start 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.537+0000 7fb9a183cb80 1 Processor -- start 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900af70 0x55e829012b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828ef8e00 con 0x55e82900a650 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828ef8aa0 con 0x55e829015e70 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e828de0720 con 0x55e82900af70 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb9a183cb80 10 client.0 _ll_register_callbacks cb 0x55e828f024f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb986a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb987400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb986000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900af70 0x55e829012b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb987400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35268/0 (socket says 172.21.15.23:35268) 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb987400700 1 -- 172.21.15.23:0/4286015742 learned_addr learned my addr 172.21.15.23:0/4286015742 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.538+0000 7fb986a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51784/0 (socket says 172.21.15.23:51784) 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 -- 172.21.15.23:0/4286015742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900af70 msgr2=0x55e829012b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900af70 0x55e829012b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 -- 172.21.15.23:0/4286015742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 msgr2=0x55e829015840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 -- 172.21.15.23:0/4286015742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9881a6250 con 0x55e829015e70 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb986a00700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb987400700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb980007c10 tx=0x7fb980007c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf2731084fa15f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb970c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb98000e070 con 0x55e829015e70 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e828f1d400 con 0x55e829015e70 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.539+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb980015860 con 0x55e829015e70 2024-03-30T04:09:40.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb980015a20 con 0x55e829015e70 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 10 client.27707 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e828d8abe0 con 0x55e829015e70 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e828f1d400 con 0x55e829015e70 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb980004340 con 0x55e829015e70 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 10 client.27707 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 10 client.27707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 20 client.27707 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e8290259f0 0x55e829027e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 4 client.27707 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e828f55780 con 0x55e8290259f0 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.540+0000 7fb986a00700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e8290259f0 0x55e829027e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.541+0000 7fb986a00700 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e8290259f0 0x55e829027e10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f032ad1b5d3ee4f8 server_cookie=c529b29dd47794c1 in_seq=0 out_seq=0 2024-03-30T04:09:40.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.541+0000 7fb970c00700 10 client.27707 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.541+0000 7fb970c00700 1 -- 172.21.15.23:0/4286015742 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e828f55780 con 0x55e8290259f0 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.541+0000 7fb970c00700 10 client.27707 handle_command_reply: tid=0 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.541+0000 7fb9a183cb80 1 client.27707 shutdown 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 msgr2=0x55e8290131b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb980007c10 tx=0x7fb980007c40 comp rx=0 tx=0).stop 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 shutdown_connections 2024-03-30T04:09:40.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e8290131b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900a650 0x55e829015840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e8290259f0 0x55e829027e10 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/4286015742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900af70 0x55e829012b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 >> 172.21.15.23:0/4286015742 conn(0x55e828f6a720 msgr2=0x55e82900c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb970c00700 0 client.27707 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 shutdown_connections 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 1 -- 172.21.15.23:0/4286015742 wait complete. 2024-03-30T04:09:40.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.542+0000 7fb9a183cb80 20 client.27707 trim_cache size 0 max 16384 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 Processor -- start 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 -- start start 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e8290065b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829006af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e829013e80 con 0x55e82900af70 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82900d460 con 0x55e82900a650 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e829023120 con 0x55e829015e70 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb986a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.560+0000 7fb986000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e8290065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35284/0 (socket says 172.21.15.23:35284) 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 -- 172.21.15.23:0/1287696785 learned_addr learned my addr 172.21.15.23:0/1287696785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986000700 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e8290065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51792/0 (socket says 172.21.15.23:51792) 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb987400700 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829006af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 -- 172.21.15.23:0/1287696785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 msgr2=0x55e829006af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829006af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 -- 172.21.15.23:0/1287696785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 msgr2=0x55e8290065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e8290065b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 -- 172.21.15.23:0/1287696785 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e828f22f20 con 0x55e82900a650 2024-03-30T04:09:40.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb986a00700 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb97809a190 tx=0x7fb9780caf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae634e9039eb25b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb97d200700 1 -- 172.21.15.23:0/1287696785 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978064040 con 0x55e82900a650 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb97d200700 1 -- 172.21.15.23:0/1287696785 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9780985a0 con 0x55e82900a650 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.561+0000 7fb97d200700 1 -- 172.21.15.23:0/1287696785 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97809a460 con 0x55e82900a650 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- 172.21.15.23:0/1287696785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 msgr2=0x55e82901caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fb97809a190 tx=0x7fb9780caf90 comp rx=0 tx=0).stop 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- 172.21.15.23:0/1287696785 shutdown_connections 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e829015e70 0x55e829006af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e82900a650 0x55e82901caf0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/1287696785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e8290065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- 172.21.15.23:0/1287696785 >> 172.21.15.23:0/1287696785 conn(0x55e828f6a720 msgr2=0x55e829011400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- 172.21.15.23:0/1287696785 shutdown_connections 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- 172.21.15.23:0/1287696785 wait complete. 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.562+0000 7fb9a183cb80 1 -- start start 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 Processor -- start 2024-03-30T04:09:40.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e82901bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e829079830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82901c340 con 0x55e82900af70 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e82901c4b0 con 0x55e829015e70 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e829079da0 con 0x55e82900a650 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb9a183cb80 10 client.0 _ll_register_callbacks cb 0x55e828f024f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb986a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb986000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e82901bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb986a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33750/0 (socket says 172.21.15.23:33750) 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.563+0000 7fb986a00700 1 -- 172.21.15.23:0/3930316366 learned_addr learned my addr 172.21.15.23:0/3930316366 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb987400700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e829079830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 -- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 msgr2=0x55e829079830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e829079830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 -- 172.21.15.23:0/3930316366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 msgr2=0x55e82901bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e82901bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 -- 172.21.15.23:0/3930316366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9780b6040 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb986a00700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb978009230 tx=0x7fb97800ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c267b8814574ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb970c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb978099d40 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb9780a2680 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb978063070 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb9780caac0 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb9a183cb80 10 client.27715 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.564+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e828de2d50 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb9780a2680 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb978099320 con 0x55e82900a650 2024-03-30T04:09:40.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 10 client.27715 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 10 client.27715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 20 client.27715 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e82907b0d0 0x55e82907d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 4 client.27715 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e828dabd70 con 0x55e82907b0d0 2024-03-30T04:09:40.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.565+0000 7fb986000700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e82907b0d0 0x55e82907d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.566+0000 7fb986000700 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e82907b0d0 0x55e82907d4c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=788b5c7ca89e7707 server_cookie=9b814b901d0d3453 in_seq=0 out_seq=0 2024-03-30T04:09:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.566+0000 7fb970c00700 10 client.27715 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb970c00700 1 -- 172.21.15.23:0/3930316366 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e828dabd70 con 0x55e82907b0d0 2024-03-30T04:09:40.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb970c00700 10 client.27715 handle_command_reply: tid=0 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 client.27715 shutdown 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 msgr2=0x55e82901b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb978009230 tx=0x7fb97800ef50 comp rx=0 tx=0).stop 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 shutdown_connections 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e829015e70 0x55e829079830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e82907b0d0 0x55e82907d4c0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e82900a650 0x55e82901b630 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 --2- 172.21.15.23:0/3930316366 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e82900af70 0x55e82901bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 >> 172.21.15.23:0/3930316366 conn(0x55e828f6a720 msgr2=0x55e829006590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb970c00700 0 client.27715 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 shutdown_connections 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.567+0000 7fb9a183cb80 1 -- 172.21.15.23:0/3930316366 wait complete. 2024-03-30T04:09:40.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb9a183cb80 20 client.27715 trim_cache size 0 max 16384 2024-03-30T04:09:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:40.570 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:09:40.570 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "bfb96fb8-a3aa-41b8-b772-9cd4b3cb2e16", 2024-03-30T04:09:40.570 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:09:40.570 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 msgr2=0x7fb96c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 0x7fb96c0a3750 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb9740098e0 tx=0x7fb974000bc0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 msgr2=0x7fb98810f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fb97800e9c0 tx=0x7fb97800ed80 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 shutdown_connections 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb988110360 0x7fb9881a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9880ffe50 0x7fb98810f8e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb96c0a1290 0x7fb96c0a3750 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 --2- 172.21.15.23:0/1078026941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb988104f60 0x7fb98810fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 >> 172.21.15.23:0/1078026941 conn(0x7fb988076d40 msgr2=0x7fb9880fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 shutdown_connections 2024-03-30T04:09:40.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.568+0000 7fb98de00700 1 -- 172.21.15.23:0/1078026941 wait complete. 2024-03-30T04:09:40.585 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:09:40.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 -- 172.21.15.23:0/1827907011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58101910 msgr2=0x7f3a58105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1827907011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58101910 0x7f3a58105950 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009990 tx=0x7f3a44009ca0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 -- 172.21.15.23:0/1827907011 shutdown_connections 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1827907011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58106080 0x7f3a5810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1827907011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 0x7f3a58101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1827907011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58101910 0x7f3a58105950 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.795+0000 7f3a5d400700 1 -- 172.21.15.23:0/1827907011 >> 172.21.15.23:0/1827907011 conn(0x7f3a580fc7c0 msgr2=0x7f3a580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.796+0000 7f3a5d400700 1 -- 172.21.15.23:0/1827907011 shutdown_connections 2024-03-30T04:09:40.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.796+0000 7f3a5d400700 1 -- 172.21.15.23:0/1827907011 wait complete. 2024-03-30T04:09:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 Processor -- start 2024-03-30T04:09:40.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 -- start start 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 0x7f3a580730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 0x7f3a580735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581a5aa0 con 0x7f3a58106080 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581a5c10 con 0x7f3a58073b30 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a5d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a581a5dd0 con 0x7f3a58100f60 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 0x7f3a580735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a57200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.797+0000 7f3a56800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 0x7f3a580730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a57200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35306/0 (socket says 172.21.15.23:35306) 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a57200700 1 -- 172.21.15.23:0/1406892377 learned_addr learned my addr 172.21.15.23:0/1406892377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 -- 172.21.15.23:0/1406892377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 msgr2=0x7f3a580730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 0x7f3a580730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 -- 172.21.15.23:0/1406892377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 msgr2=0x7f3a581a54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 -- 172.21.15.23:0/1406892377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440095f0 con 0x7f3a58106080 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a57200700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a55e00700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 0x7f3a580735f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3a44006010 tx=0x7f3a44005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a4f400700 1 -- 172.21.15.23:0/1406892377 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4401d070 con 0x7f3a58106080 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.798+0000 7f3a4f400700 1 -- 172.21.15.23:0/1406892377 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a44021410 con 0x7f3a58106080 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.799+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a581a6050 con 0x7f3a58106080 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.799+0000 7f3a4f400700 1 -- 172.21.15.23:0/1406892377 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44017670 con 0x7f3a58106080 2024-03-30T04:09:40.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.799+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a581a6470 con 0x7f3a58106080 2024-03-30T04:09:40.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.803+0000 7f3a4f400700 1 -- 172.21.15.23:0/1406892377 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3a440217f0 con 0x7f3a58106080 2024-03-30T04:09:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.804+0000 7f3a70e65b80 1 Processor -- start 2024-03-30T04:09:40.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 -- start start 2024-03-30T04:09:40.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44baca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44ba15990 con 0x55d44bac7df0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44ba13350 con 0x55d44babcef0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44ba0ec40 con 0x55d44babc5d0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a56800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44baca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a56800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35312/0 (socket says 172.21.15.23:35312) 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a56800700 1 -- 172.21.15.23:0/3068278020 learned_addr learned my addr 172.21.15.23:0/3068278020 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a4f400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 0x7f3a3c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.805+0000 7f3a57200700 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 -- 172.21.15.23:0/3068278020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 msgr2=0x55d44babc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 -- 172.21.15.23:0/3068278020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 msgr2=0x55d44baca1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44baca1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a4f400700 1 -- 172.21.15.23:0/1406892377 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a440ce1a0 con 0x7f3a58106080 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 -- 172.21.15.23:0/3068278020 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d44b8903b0 con 0x55d44babcef0 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a57200700 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a57200700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 0x7f3a3c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a56800700 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009ad0 tx=0x7f3a48009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e032f5e8e294c6d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a43400700 1 -- 172.21.15.23:0/3068278020 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480095d0 con 0x55d44babcef0 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a43400700 1 -- 172.21.15.23:0/3068278020 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a4800ed90 con 0x55d44babcef0 2024-03-30T04:09:40.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.806+0000 7f3a43400700 1 -- 172.21.15.23:0/3068278020 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480123d0 con 0x55d44babcef0 2024-03-30T04:09:40.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 -- 172.21.15.23:0/3068278020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 msgr2=0x55d44bac78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a57200700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 0x7f3a3c0a3750 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009fd0 tx=0x7f3a50009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009ad0 tx=0x7f3a48009de0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 -- 172.21.15.23:0/3068278020 shutdown_connections 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44baca1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/3068278020 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac78b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 -- 172.21.15.23:0/3068278020 >> 172.21.15.23:0/3068278020 conn(0x55d44ba1c6a0 msgr2=0x55d44bab8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 -- 172.21.15.23:0/3068278020 shutdown_connections 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.807+0000 7f3a70e65b80 1 -- 172.21.15.23:0/3068278020 wait complete. 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 -- start start 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 Processor -- start 2024-03-30T04:09:40.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44bac77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44bac5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44b9aad80 con 0x55d44bac7df0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44b9aaa20 con 0x55d44babcef0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.808+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44b8926a0 con 0x55d44babc5d0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a70e65b80 10 client.0 _ll_register_callbacks cb 0x55d44b9b4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a57200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44bac5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35320/0 (socket says 172.21.15.23:35320) 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 -- 172.21.15.23:0/2904271213 learned_addr learned my addr 172.21.15.23:0/2904271213 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a56800700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44bac77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 -- 172.21.15.23:0/2904271213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 msgr2=0x55d44bac77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44bac77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 -- 172.21.15.23:0/2904271213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 msgr2=0x55d44bac5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44bac5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 -- 172.21.15.23:0/2904271213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a48009730 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a55e00700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3a44000c00 tx=0x7f3a4402e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68eb2789a9367e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.809+0000 7f3a37e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44021580 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a4409b600 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d44b9cf380 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a4409aa40 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 10 client.27725 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d44b83cb60 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d44b9cf380 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3a440a2e60 con 0x55d44babcef0 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 10 client.27725 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:40.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 10 client.27725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 20 client.27725 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bad7660 0x55d44bad9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 4 client.27725 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.810+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d44ba07700 con 0x55d44bad7660 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.811+0000 7f3a56800700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bad7660 0x55d44bad9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.811+0000 7f3a37e00700 10 client.27725 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.811+0000 7f3a56800700 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bad7660 0x55d44bad9a80 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e7f5548f9773bd server_cookie=ebac9df3f4bd4869 in_seq=0 out_seq=0 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a37e00700 1 -- 172.21.15.23:0/2904271213 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d44ba07700 con 0x55d44bad7660 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a37e00700 10 client.27725 handle_command_reply: tid=0 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 client.27725 shutdown 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 msgr2=0x55d44bac4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3a44000c00 tx=0x7f3a4402e000 comp rx=0 tx=0).stop 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 shutdown_connections 2024-03-30T04:09:40.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44bac7df0 0x55d44bac5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bad7660 0x55d44bad9a80 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44bac77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a37e00700 0 client.27725 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/2904271213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bac4b40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 >> 172.21.15.23:0/2904271213 conn(0x55d44ba1c6a0 msgr2=0x55d44babe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 shutdown_connections 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 1 -- 172.21.15.23:0/2904271213 wait complete. 2024-03-30T04:09:40.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.812+0000 7f3a70e65b80 20 client.27725 trim_cache size 0 max 16384 2024-03-30T04:09:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 Processor -- start 2024-03-30T04:09:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 -- start start 2024-03-30T04:09:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bacea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44bac7df0 0x55d44bab15a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44bacf040 con 0x55d44babc5d0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44bac30e0 con 0x55d44babcef0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44bac59d0 con 0x55d44bac7df0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a56800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a56800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51844/0 (socket says 172.21.15.23:51844) 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a56800700 1 -- 172.21.15.23:0/531283107 learned_addr learned my addr 172.21.15.23:0/531283107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.830+0000 7f3a57200700 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44bac7df0 0x55d44bab15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a55e00700 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bacea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 -- 172.21.15.23:0/531283107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44bac7df0 msgr2=0x55d44bab15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44bac7df0 0x55d44bab15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 -- 172.21.15.23:0/531283107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 msgr2=0x55d44bacea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bacea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 -- 172.21.15.23:0/531283107 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d44b9d4ea0 con 0x55d44babc5d0 2024-03-30T04:09:40.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a56800700 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006ae0 tx=0x7f3a48009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24b51d9d4ab9f37 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a43e00700 1 -- 172.21.15.23:0/531283107 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48021030 con 0x55d44babc5d0 2024-03-30T04:09:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a43e00700 1 -- 172.21.15.23:0/531283107 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a48022040 con 0x55d44babc5d0 2024-03-30T04:09:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.831+0000 7f3a43e00700 1 -- 172.21.15.23:0/531283107 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48013890 con 0x55d44babc5d0 2024-03-30T04:09:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- 172.21.15.23:0/531283107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 msgr2=0x55d44bad9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3a48006ae0 tx=0x7f3a48009fb0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- 172.21.15.23:0/531283107 shutdown_connections 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44bac7df0 0x55d44bab15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babc5d0 0x55d44bad9a50 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/531283107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44babcef0 0x55d44bacea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- 172.21.15.23:0/531283107 >> 172.21.15.23:0/531283107 conn(0x55d44ba1c6a0 msgr2=0x55d44bac75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- 172.21.15.23:0/531283107 shutdown_connections 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- 172.21.15.23:0/531283107 wait complete. 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.832+0000 7f3a70e65b80 1 -- start start 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 Processor -- start 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babcef0 0x55d44bacdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44bac7df0 0x55d44bacdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44babf290 con 0x55d44babcef0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44bace520 con 0x55d44bac7df0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d44bace6e0 con 0x55d44babc5d0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.833+0000 7f3a70e65b80 10 client.0 _ll_register_callbacks cb 0x55d44b9b4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a57200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44bac7df0 0x55d44bacdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:33804/0 (socket says 172.21.15.23:33804) 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 -- 172.21.15.23:0/4021101992 learned_addr learned my addr 172.21.15.23:0/4021101992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a55e00700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babcef0 0x55d44bacdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 -- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44bac7df0 msgr2=0x55d44bacdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44bac7df0 0x55d44bacdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 -- 172.21.15.23:0/4021101992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babcef0 msgr2=0x55d44bacdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babcef0 0x55d44bacdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 -- 172.21.15.23:0/4021101992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d44b8903b0 con 0x55d44babc5d0 2024-03-30T04:09:40.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a56800700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e4b0 tx=0x7f3a48028540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41d9f86d9323224c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a37e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.834+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48021050 con 0x55d44babc5d0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a48005a00 con 0x55d44babc5d0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a4800ed90 con 0x55d44babc5d0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a48005bc0 con 0x55d44babc5d0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 10 client.27730 fetch_fsmap learned FSMap version 30 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d44b894cd0 con 0x55d44babc5d0 2024-03-30T04:09:40.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a4800ed90 con 0x55d44babc5d0 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3a4802f030 con 0x55d44babc5d0 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 10 client.27730 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 10 client.27730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 20 client.27730 populate_metadata read hostname 'smithi023' 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bb2c390 0x55d44bb2e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:09:40.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 4 client.27730 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.835+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d44b85dcf0 con 0x55d44bb2c390 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.836+0000 7f3a55e00700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bb2c390 0x55d44bb2e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a37e00700 10 client.27730 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a55e00700 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bb2c390 0x55d44bb2e7b0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ba52c5c63748306 server_cookie=92e651105211427f in_seq=0 out_seq=0 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a37e00700 1 -- 172.21.15.23:0/4021101992 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d44b85dcf0 con 0x55d44bb2c390 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a37e00700 10 client.27730 handle_command_reply: tid=0 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 client.27730 shutdown 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 msgr2=0x55d44babe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e4b0 tx=0x7f3a48028540 comp rx=0 tx=0).stop 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 shutdown_connections 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d44bac7df0 0x55d44bacdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d44bb2c390 0x55d44bb2e7b0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d44babc5d0 0x55d44babe9b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 --2- 172.21.15.23:0/4021101992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d44babcef0 0x55d44bacdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a37e00700 0 client.27730 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 >> 172.21.15.23:0/4021101992 conn(0x55d44ba1c6a0 msgr2=0x55d44babfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 shutdown_connections 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 1 -- 172.21.15.23:0/4021101992 wait complete. 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.837+0000 7f3a70e65b80 20 client.27730 trim_cache size 0 max 16384 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:09:40.840 INFO:teuthology.orchestra.run.smithi023.stdout: "bfb96fb8-a3aa-41b8-b772-9cd4b3cb2e16": { 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "bfb96fb8-a3aa-41b8-b772-9cd4b3cb2e16", 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:09:40.841 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 msgr2=0x7f3a3c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 0x7f3a3c0a3750 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009fd0 tx=0x7f3a50009040 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 msgr2=0x7f3a580735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 0x7f3a580735f0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3a44006010 tx=0x7f3a44005740 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 shutdown_connections 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a3c0a1290 0x7f3a3c0a3750 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3a58073b30 0x7f3a581a54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a58100f60 0x7f3a580730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 --2- 172.21.15.23:0/1406892377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3a58106080 0x7f3a580735f0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 >> 172.21.15.23:0/1406892377 conn(0x7f3a580fc7c0 msgr2=0x7f3a580fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 shutdown_connections 2024-03-30T04:09:40.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:09:40.838+0000 7f3a5d400700 1 -- 172.21.15.23:0/1406892377 wait complete. 2024-03-30T04:09:40.854 INFO:tasks.cephfs.filesystem:scrub status for tag:bfb96fb8-a3aa-41b8-b772-9cd4b3cb2e16 - {'path': '/', 'tag': 'bfb96fb8-a3aa-41b8-b772-9cd4b3cb2e16', 'options': 'recursive,force'} 2024-03-30T04:09:41.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 767946 10.73 MB/sec execute 258 sec latency 33.876 ms 2024-03-30T04:09:42.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 770265 10.74 MB/sec execute 259 sec latency 18.593 ms 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: pgmap v239: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:42 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: pgmap v239: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:42 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: pgmap v239: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 166 op/s 2024-03-30T04:09:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:09:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:09:42.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:09:42.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:42.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:42 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:09:43.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 771383 10.72 MB/sec execute 260 sec latency 81.668 ms 2024-03-30T04:09:44.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 773007 10.71 MB/sec execute 261 sec latency 61.477 ms 2024-03-30T04:09:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:44 smithi154 ceph-mon[79161]: pgmap v240: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:44 smithi193 ceph-mon[79581]: pgmap v240: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:44 smithi023 ceph-mon[38254]: pgmap v240: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:45.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 774854 10.71 MB/sec execute 262 sec latency 20.849 ms 2024-03-30T04:09:45.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:45 smithi193 ceph-mon[79581]: pgmap v241: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 168 op/s 2024-03-30T04:09:45.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:45 smithi023 ceph-mon[38254]: pgmap v241: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 168 op/s 2024-03-30T04:09:45.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:45 smithi154 ceph-mon[79161]: pgmap v241: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 168 op/s 2024-03-30T04:09:46.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 776638 10.70 MB/sec execute 263 sec latency 91.071 ms 2024-03-30T04:09:47.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 778400 10.70 MB/sec execute 264 sec latency 24.198 ms 2024-03-30T04:09:48.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 780058 10.69 MB/sec execute 265 sec latency 105.259 ms 2024-03-30T04:09:48.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:48 smithi154 ceph-mon[79161]: pgmap v242: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-03-30T04:09:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:48 smithi193 ceph-mon[79581]: pgmap v242: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-03-30T04:09:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:48 smithi023 ceph-mon[38254]: pgmap v242: 129 pgs: 129 active+clean; 437 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-03-30T04:09:49.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 781815 10.69 MB/sec execute 266 sec latency 19.628 ms 2024-03-30T04:09:49.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:49 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:09:49.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:49 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:09:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:49 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:09:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:49 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:09:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:49 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:09:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:49 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:09:50.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 783629 10.68 MB/sec execute 267 sec latency 21.868 ms 2024-03-30T04:09:50.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:50 smithi154 ceph-mon[79161]: pgmap v243: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-30T04:09:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:50 smithi193 ceph-mon[79581]: pgmap v243: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-30T04:09:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:50 smithi023 ceph-mon[38254]: pgmap v243: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 247 op/s 2024-03-30T04:09:51.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 785404 10.68 MB/sec execute 268 sec latency 27.189 ms 2024-03-30T04:09:52.212 INFO:tasks.workunit.client.0.smithi023.stdout: 1 787284 10.67 MB/sec execute 269 sec latency 19.763 ms 2024-03-30T04:09:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:52 smithi154 ceph-mon[79161]: pgmap v244: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-03-30T04:09:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:52 smithi193 ceph-mon[79581]: pgmap v244: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-03-30T04:09:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:52 smithi023 ceph-mon[38254]: pgmap v244: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-03-30T04:09:53.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 788896 10.68 MB/sec execute 270 sec latency 34.542 ms 2024-03-30T04:09:54.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 790637 10.66 MB/sec execute 271 sec latency 35.396 ms 2024-03-30T04:09:54.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:54 smithi154 ceph-mon[79161]: pgmap v245: 129 pgs: 129 active+clean; 448 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:54 smithi193 ceph-mon[79581]: pgmap v245: 129 pgs: 129 active+clean; 448 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:54 smithi023 ceph-mon[38254]: pgmap v245: 129 pgs: 129 active+clean; 448 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 245 op/s 2024-03-30T04:09:55.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 792372 10.67 MB/sec execute 272 sec latency 24.689 ms 2024-03-30T04:09:56.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 794173 10.65 MB/sec execute 273 sec latency 20.219 ms 2024-03-30T04:09:56.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:56 smithi154 ceph-mon[79161]: pgmap v246: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-30T04:09:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:56 smithi193 ceph-mon[79581]: pgmap v246: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-30T04:09:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:56 smithi023 ceph-mon[38254]: pgmap v246: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 171 op/s 2024-03-30T04:09:57.213 INFO:tasks.workunit.client.0.smithi023.stdout: 1 795704 10.66 MB/sec execute 274 sec latency 219.769 ms 2024-03-30T04:09:58.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 797098 10.64 MB/sec execute 275 sec latency 25.790 ms 2024-03-30T04:09:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:09:58 smithi154 ceph-mon[79161]: pgmap v247: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-03-30T04:09:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:09:58 smithi193 ceph-mon[79581]: pgmap v247: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-03-30T04:09:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:09:58 smithi023 ceph-mon[38254]: pgmap v247: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 168 op/s 2024-03-30T04:09:59.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 798862 10.64 MB/sec execute 276 sec latency 235.084 ms 2024-03-30T04:10:00.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 800216 10.63 MB/sec execute 277 sec latency 24.559 ms 2024-03-30T04:10:00.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:00 smithi154 ceph-mon[79161]: pgmap v248: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 246 op/s 2024-03-30T04:10:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:00 smithi154 ceph-mon[79161]: overall HEALTH_OK 2024-03-30T04:10:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:00 smithi193 ceph-mon[79581]: pgmap v248: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 246 op/s 2024-03-30T04:10:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:00 smithi193 ceph-mon[79581]: overall HEALTH_OK 2024-03-30T04:10:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:00 smithi023 ceph-mon[38254]: pgmap v248: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 246 op/s 2024-03-30T04:10:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:00 smithi023 ceph-mon[38254]: overall HEALTH_OK 2024-03-30T04:10:01.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 801147 10.60 MB/sec execute 278 sec latency 75.489 ms 2024-03-30T04:10:02.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 802517 10.60 MB/sec execute 279 sec latency 76.053 ms 2024-03-30T04:10:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:02 smithi154 ceph-mon[79161]: pgmap v249: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 160 op/s 2024-03-30T04:10:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:02 smithi193 ceph-mon[79581]: pgmap v249: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 160 op/s 2024-03-30T04:10:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:02 smithi023 ceph-mon[38254]: pgmap v249: 129 pgs: 129 active+clean; 451 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 160 op/s 2024-03-30T04:10:03.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 803930 10.59 MB/sec execute 280 sec latency 23.076 ms 2024-03-30T04:10:04.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 806016 10.59 MB/sec execute 281 sec latency 112.277 ms 2024-03-30T04:10:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:04 smithi154 ceph-mon[79161]: pgmap v250: 129 pgs: 129 active+clean; 457 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 250 op/s 2024-03-30T04:10:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:04 smithi193 ceph-mon[79581]: pgmap v250: 129 pgs: 129 active+clean; 457 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 250 op/s 2024-03-30T04:10:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:04 smithi023 ceph-mon[38254]: pgmap v250: 129 pgs: 129 active+clean; 457 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 250 op/s 2024-03-30T04:10:05.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 807389 10.58 MB/sec execute 282 sec latency 23.014 ms 2024-03-30T04:10:06.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 809607 10.59 MB/sec execute 283 sec latency 31.697 ms 2024-03-30T04:10:06.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:06 smithi154 ceph-mon[79161]: pgmap v251: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-03-30T04:10:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:06 smithi193 ceph-mon[79581]: pgmap v251: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-03-30T04:10:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:06 smithi023 ceph-mon[38254]: pgmap v251: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 190 op/s 2024-03-30T04:10:07.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 811025 10.57 MB/sec execute 284 sec latency 21.241 ms 2024-03-30T04:10:08.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 813191 10.58 MB/sec execute 285 sec latency 37.243 ms 2024-03-30T04:10:08.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:08 smithi154 ceph-mon[79161]: pgmap v252: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 186 op/s 2024-03-30T04:10:08.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:08 smithi193 ceph-mon[79581]: pgmap v252: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 186 op/s 2024-03-30T04:10:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:08 smithi023 ceph-mon[38254]: pgmap v252: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 186 op/s 2024-03-30T04:10:09.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 814233 10.55 MB/sec execute 286 sec latency 58.385 ms 2024-03-30T04:10:10.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 816394 10.56 MB/sec execute 287 sec latency 22.777 ms 2024-03-30T04:10:10.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:10 smithi154 ceph-mon[79161]: pgmap v253: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 270 op/s 2024-03-30T04:10:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:10 smithi154 ceph-mon[79161]: osdmap e94: 12 total, 12 up, 12 in 2024-03-30T04:10:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:10 smithi193 ceph-mon[79581]: pgmap v253: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 270 op/s 2024-03-30T04:10:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:10 smithi193 ceph-mon[79581]: osdmap e94: 12 total, 12 up, 12 in 2024-03-30T04:10:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:10 smithi023 ceph-mon[38254]: pgmap v253: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 270 op/s 2024-03-30T04:10:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:10 smithi023 ceph-mon[38254]: osdmap e94: 12 total, 12 up, 12 in 2024-03-30T04:10:10.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 -- 172.21.15.23:0/865287371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630106080 msgr2=0x7f463010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 --2- 172.21.15.23:0/865287371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630106080 0x7f463010dc00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f462800b0a0 tx=0x7f462800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 -- 172.21.15.23:0/865287371 shutdown_connections 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 --2- 172.21.15.23:0/865287371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630106080 0x7f463010dc00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 --2- 172.21.15.23:0/865287371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630100f60 0x7f4630101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 --2- 172.21.15.23:0/865287371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630101910 0x7f4630105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 -- 172.21.15.23:0/865287371 >> 172.21.15.23:0/865287371 conn(0x7f46300fc7c0 msgr2=0x7f46300febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 -- 172.21.15.23:0/865287371 shutdown_connections 2024-03-30T04:10:11.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.119+0000 7f4636e00700 1 -- 172.21.15.23:0/865287371 wait complete. 2024-03-30T04:10:11.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.121+0000 7f4636e00700 1 Processor -- start 2024-03-30T04:10:11.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.121+0000 7f4636e00700 1 -- start start 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630100f60 0x7f46300730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630106080 0x7f46300735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46301a9f70 con 0x7f4630106080 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463019b4d0 con 0x7f4630074d70 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4636e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463019b690 con 0x7f4630100f60 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630106080 0x7f46300735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:39870/0 (socket says 172.21.15.23:39870) 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 -- 172.21.15.23:0/2147790597 learned_addr learned my addr 172.21.15.23:0/2147790597 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 -- 172.21.15.23:0/2147790597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630100f60 msgr2=0x7f46300730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630100f60 0x7f46300730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 -- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630106080 msgr2=0x7f46300735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.122+0000 7f4634e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630106080 0x7f46300735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f4634e00700 1 -- 172.21.15.23:0/2147790597 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4628009d00 con 0x7f4630074d70 2024-03-30T04:10:11.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f4634e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f462800b070 tx=0x7f462800be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f462cc00700 1 -- 172.21.15.23:0/2147790597 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462800e040 con 0x7f4630074d70 2024-03-30T04:10:11.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f462cc00700 1 -- 172.21.15.23:0/2147790597 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4628007e50 con 0x7f4630074d70 2024-03-30T04:10:11.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f463019b910 con 0x7f4630074d70 2024-03-30T04:10:11.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.123+0000 7f462cc00700 1 -- 172.21.15.23:0/2147790597 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462801dea0 con 0x7f4630074d70 2024-03-30T04:10:11.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.124+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f463019bdc0 con 0x7f4630074d70 2024-03-30T04:10:11.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.128+0000 7f462cc00700 1 -- 172.21.15.23:0/2147790597 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f462801d480 con 0x7f4630074d70 2024-03-30T04:10:11.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.128+0000 7f464a982b80 1 Processor -- start 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.128+0000 7f464a982b80 1 -- start start 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfbef0 0x560057d068b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057c54990 con 0x560057cfb5d0 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057c52350 con 0x560057d06df0 2024-03-30T04:10:11.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f464a982b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057c4dc40 con 0x560057cfbef0 2024-03-30T04:10:11.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60640/0 (socket says 172.21.15.23:60640) 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f462f400700 1 -- 172.21.15.23:0/125758437 learned_addr learned my addr 172.21.15.23:0/125758437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.129+0000 7f462f400700 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 -- 172.21.15.23:0/125758437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfbef0 msgr2=0x560057d068b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfbef0 0x560057d068b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 -- 172.21.15.23:0/125758437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 msgr2=0x560057d091e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d091e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 -- 172.21.15.23:0/125758437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560057acf3b0 con 0x560057cfb5d0 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462cc00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 0x7f46140a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462f400700 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f461c0126f0 tx=0x7f461c0096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd95f1bea80fbfef server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.130+0000 7f462cc00700 1 -- 172.21.15.23:0/2147790597 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f46280ce2c0 con 0x7f4630074d70 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.131+0000 7f4625600700 1 -- 172.21.15.23:0/125758437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c012d10 con 0x560057cfb5d0 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.131+0000 7f4625600700 1 -- 172.21.15.23:0/125758437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f461c012e70 con 0x560057cfb5d0 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.131+0000 7f4625600700 1 -- 172.21.15.23:0/125758437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c00f480 con 0x560057cfb5d0 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.131+0000 7f462fe00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 0x7f46140a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 -- 172.21.15.23:0/125758437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 msgr2=0x560057cfb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f461c0126f0 tx=0x7f461c0096f0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f462fe00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 0x7f46140a37d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4620005950 tx=0x7f46200058e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 -- 172.21.15.23:0/125758437 shutdown_connections 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfb5d0 0x560057cfb9b0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 --2- 172.21.15.23:0/125758437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfbef0 0x560057d068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 -- 172.21.15.23:0/125758437 >> 172.21.15.23:0/125758437 conn(0x560057c5b6a0 msgr2=0x560057cf7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 -- 172.21.15.23:0/125758437 shutdown_connections 2024-03-30T04:10:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.132+0000 7f464a982b80 1 -- 172.21.15.23:0/125758437 wait complete. 2024-03-30T04:10:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.133+0000 7f464a982b80 1 -- start start 2024-03-30T04:10:11.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 Processor -- start 2024-03-30T04:10:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d06820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 0x560057d03b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d042a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057be9d80 con 0x560057cfbef0 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057be9a20 con 0x560057d06df0 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.134+0000 7f464a982b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057ad16a0 con 0x560057cfb5d0 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f4634e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 0x560057d03b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f4634e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:39888/0 (socket says 172.21.15.23:39888) 2024-03-30T04:10:11.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f4634e00700 1 -- 172.21.15.23:0/3858852344 learned_addr learned my addr 172.21.15.23:0/3858852344 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462fe00700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d06820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 -- 172.21.15.23:0/3858852344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 msgr2=0x560057d06820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d06820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 -- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 msgr2=0x560057d042a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d042a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f462f400700 1 -- 172.21.15.23:0/3858852344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c01a040 con 0x560057cfbef0 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f464a982b80 10 client.0 _ll_register_callbacks cb 0x560057bf3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.135+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560057c0e380 con 0x560057cfbef0 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.136+0000 7f462fe00700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d06820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.136+0000 7f462f400700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 0x560057d03b60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f461c005920 tx=0x7f461c003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc79b977452349a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.136+0000 7f461a000700 10 client.17739 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:11.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.137+0000 7f461a000700 1 -- 172.21.15.23:0/3858852344 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c012ce0 con 0x560057cfbef0 2024-03-30T04:10:11.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.137+0000 7f461a000700 1 -- 172.21.15.23:0/3858852344 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f461c0042d0 con 0x560057cfbef0 2024-03-30T04:10:11.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.137+0000 7f461a000700 1 -- 172.21.15.23:0/3858852344 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f461c0044d0 con 0x560057cfbef0 2024-03-30T04:10:11.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.137+0000 7f464a982b80 10 client.17739 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:11.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.137+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=30}) v3 -- 0x560057a7bb60 con 0x560057cfbef0 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f461a000700 1 -- 172.21.15.23:0/3858852344 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f461c005030 con 0x560057cfbef0 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 10 client.17739 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 20 client.17739 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d16650 0x560057d18a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 4 client.17739 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.138+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560057c46700 con 0x560057d16650 2024-03-30T04:10:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.139+0000 7f462fe00700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d16650 0x560057d18a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.139+0000 7f461a000700 10 client.17739 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:11.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.139+0000 7f462fe00700 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d16650 0x560057d18a70 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8073590ca7eb726a server_cookie=67ffb266e835f9b2 in_seq=0 out_seq=0 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f461a000700 1 -- 172.21.15.23:0/3858852344 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x560057c46700 con 0x560057d16650 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f461a000700 10 client.17739 handle_command_reply: tid=0 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 client.17739 shutdown 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 msgr2=0x560057d03b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 0x560057d03b60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f461c005920 tx=0x7f461c003e20 comp rx=0 tx=0).stop 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 shutdown_connections 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d06df0 0x560057d042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d16650 0x560057d18a70 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d06820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 --2- 172.21.15.23:0/3858852344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057cfbef0 0x560057d03b60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f461a000700 0 client.17739 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.140+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 >> 172.21.15.23:0/3858852344 conn(0x560057c5b6a0 msgr2=0x560057cfd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.141+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 shutdown_connections 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.141+0000 7f464a982b80 1 -- 172.21.15.23:0/3858852344 wait complete. 2024-03-30T04:10:11.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.141+0000 7f464a982b80 20 client.17739 trim_cache size 0 max 16384 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 Processor -- start 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 -- start start 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d0da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057cfee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 0x560057cff3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057cfc850 con 0x560057d06df0 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057cf7dc0 con 0x560057d1d3d0 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f464a982b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057d0dfb0 con 0x560057cfb5d0 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057cfee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.166+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057cfee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60664/0 (socket says 172.21.15.23:60664) 2024-03-30T04:10:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f462f400700 1 -- 172.21.15.23:0/1291698618 learned_addr learned my addr 172.21.15.23:0/1291698618 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 0x560057cff3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f462fe00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d0da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 -- 172.21.15.23:0/1291698618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 msgr2=0x560057d0da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d0da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 -- 172.21.15.23:0/1291698618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 msgr2=0x560057cfee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057cfee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 -- 172.21.15.23:0/1291698618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560057c13ea0 con 0x560057d1d3d0 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f462fe00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d0da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.167+0000 7f4634e00700 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 0x560057cff3d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f462809af80 tx=0x7f462800b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2dfeb8a4a61313 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.168+0000 7f4626000700 1 -- 172.21.15.23:0/1291698618 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462809aff0 con 0x560057d1d3d0 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.168+0000 7f4626000700 1 -- 172.21.15.23:0/1291698618 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f462809ee20 con 0x560057d1d3d0 2024-03-30T04:10:11.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.168+0000 7f4626000700 1 -- 172.21.15.23:0/1291698618 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628097700 con 0x560057d1d3d0 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- 172.21.15.23:0/1291698618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 msgr2=0x560057cff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 0x560057cff3d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f462809af80 tx=0x7f462800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- 172.21.15.23:0/1291698618 shutdown_connections 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057d1d3d0 0x560057cff3d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057cfb5d0 0x560057d0da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 --2- 172.21.15.23:0/1291698618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057cfee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- 172.21.15.23:0/1291698618 >> 172.21.15.23:0/1291698618 conn(0x560057c5b6a0 msgr2=0x560057cf05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- 172.21.15.23:0/1291698618 shutdown_connections 2024-03-30T04:10:11.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- 172.21.15.23:0/1291698618 wait complete. 2024-03-30T04:10:11.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.169+0000 7f464a982b80 1 -- start start 2024-03-30T04:10:11.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.170+0000 7f464a982b80 1 Processor -- start 2024-03-30T04:10:11.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.170+0000 7f464a982b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 0x560057d64790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.170+0000 7f464a982b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f464a982b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057d1d3d0 0x560057d0c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f464a982b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057d0cd60 con 0x560057d06df0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f464a982b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057d0cea0 con 0x560057cfb5d0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f464a982b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560057d0d060 con 0x560057d1d3d0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f464a982b80 10 client.0 _ll_register_callbacks cb 0x560057bf3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 0x560057d64790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60680/0 (socket says 172.21.15.23:60680) 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462f400700 1 -- 172.21.15.23:0/4078366112 learned_addr learned my addr 172.21.15.23:0/4078366112 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f4634e00700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057d1d3d0 0x560057d0c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 -- 172.21.15.23:0/4078366112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057d1d3d0 msgr2=0x560057d0c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057d1d3d0 0x560057d0c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 -- 172.21.15.23:0/4078366112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 msgr2=0x560057d64cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:11.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.171+0000 7f462fe00700 1 -- 172.21.15.23:0/4078366112 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462000e960 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f462f400700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f462fe00700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 0x560057d64790 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x560057d16a30 tx=0x7f4620009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa013688434416e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f461a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620024a80 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f461c012e70 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f4620021040 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4620024be0 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f464a982b80 10 client.27749 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.172+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f463019bdc0 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f461c012e70 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f462001c420 con 0x560057cfb5d0 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 10 client.27749 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:11.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 10 client.27749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 20 client.27749 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d6c620 0x560057d6ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 4 client.27749 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:10:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560057a9ccf0 con 0x560057d6c620 2024-03-30T04:10:11.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.173+0000 7f462f400700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d6c620 0x560057d6ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f462f400700 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d6c620 0x560057d6ea10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd345b2c7bcdf639 server_cookie=9b9c9f3920c2de49 in_seq=0 out_seq=0 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f461a000700 10 client.27749 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f461a000700 1 -- 172.21.15.23:0/4078366112 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560057a9ccf0 con 0x560057d6c620 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f461a000700 10 client.27749 handle_command_reply: tid=0 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f464a982b80 1 client.27749 shutdown 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 msgr2=0x560057d64790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.174+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 0x560057d64790 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x560057d16a30 tx=0x7f4620009260 comp rx=0 tx=0).stop 2024-03-30T04:10:11.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 shutdown_connections 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x560057d6c620 0x560057d6ea10 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560057d1d3d0 0x560057d0c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x560057cfb5d0 0x560057d64790 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f461a000700 0 client.27749 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 --2- 172.21.15.23:0/4078366112 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x560057d06df0 0x560057d64cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 >> 172.21.15.23:0/4078366112 conn(0x560057c5b6a0 msgr2=0x560057cfee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 shutdown_connections 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 1 -- 172.21.15.23:0/4078366112 wait complete. 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.175+0000 7f464a982b80 20 client.27749 trim_cache size 0 max 16384 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:10:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:10:11.177 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:11.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 msgr2=0x7f46140a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 0x7f46140a37d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4620005950 tx=0x7f46200058e0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 msgr2=0x7f4630073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f462800b070 tx=0x7f462800be50 comp rx=0 tx=0).stop 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 shutdown_connections 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f46140a1310 0x7f46140a37d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4630074d70 0x7f4630073b70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4630100f60 0x7f46300730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 --2- 172.21.15.23:0/2147790597 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4630106080 0x7f46300735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 >> 172.21.15.23:0/2147790597 conn(0x7f46300fc7c0 msgr2=0x7f46300feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 shutdown_connections 2024-03-30T04:10:11.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.176+0000 7f4636e00700 1 -- 172.21.15.23:0/2147790597 wait complete. 2024-03-30T04:10:11.194 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:11 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:10:11.197 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:10:11.197 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:10:11.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 817662 10.55 MB/sec execute 288 sec latency 19.632 ms 2024-03-30T04:10:11.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:11 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 -- 172.21.15.23:0/2085711818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 msgr2=0x7fb13c10d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 --2- 172.21.15.23:0/2085711818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c10d650 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb13800b0a0 tx=0x7fb13800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 -- 172.21.15.23:0/2085711818 shutdown_connections 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 --2- 172.21.15.23:0/2085711818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c10d650 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 --2- 172.21.15.23:0/2085711818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 0x7fb13c102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 --2- 172.21.15.23:0/2085711818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 0x7fb13c10ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 -- 172.21.15.23:0/2085711818 >> 172.21.15.23:0/2085711818 conn(0x7fb13c0fb590 msgr2=0x7fb13c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 -- 172.21.15.23:0/2085711818 shutdown_connections 2024-03-30T04:10:11.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.456+0000 7fb143200700 1 -- 172.21.15.23:0/2085711818 wait complete. 2024-03-30T04:10:11.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.457+0000 7fb143200700 1 Processor -- start 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 -- start start 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 0x7fb13c0ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 0x7fb13c0fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1a55b0 con 0x7fb13c101d70 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1a5720 con 0x7fb13c102690 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb143200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c1a58e0 con 0x7fb13c10b190 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb141400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 0x7fb13c0fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35916/0 (socket says 172.21.15.23:35916) 2024-03-30T04:10:11.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 -- 172.21.15.23:0/3121009868 learned_addr learned my addr 172.21.15.23:0/3121009868 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb141e00700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 0x7fb13c0ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 -- 172.21.15.23:0/3121009868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 msgr2=0x7fb13c0fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 0x7fb13c0fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 -- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 msgr2=0x7fb13c0ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 0x7fb13c0ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.458+0000 7fb142800700 1 -- 172.21.15.23:0/3121009868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138009d00 con 0x7fb13c10b190 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb142800700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb13800b070 tx=0x7fb1380078c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13800e040 con 0x7fb13c10b190 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb13801d470 con 0x7fb13c10b190 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c1a5b60 con 0x7fb13c10b190 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138005280 con 0x7fb13c10b190 2024-03-30T04:10:11.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.459+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c1a6020 con 0x7fb13c10b190 2024-03-30T04:10:11.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.460+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c1062c0 con 0x7fb13c10b190 2024-03-30T04:10:11.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.462+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb13801d5e0 con 0x7fb13c10b190 2024-03-30T04:10:11.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.463+0000 7fb136a00700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 0x7fb1240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:11.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.464+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fb1380ce400 con 0x7fb13c10b190 2024-03-30T04:10:11.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.464+0000 7fb141e00700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 0x7fb1240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:11.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.464+0000 7fb141e00700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 0x7fb1240a3750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb130009c60 tx=0x7fb130009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:11.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.466+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb138097470 con 0x7fb13c10b190 2024-03-30T04:10:11.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:11 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:10:11.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.765+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb13c0686d0 con 0x7fb13c10b190 2024-03-30T04:10:11.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.766+0000 7fb136a00700 1 -- 172.21.15.23:0/3121009868 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fb13809aa90 con 0x7fb13c10b190 2024-03-30T04:10:11.765 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:10:11.765 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:10:11.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 msgr2=0x7fb1240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 0x7fb1240a3750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb130009c60 tx=0x7fb130009260 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 msgr2=0x7fb13c100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb13800b070 tx=0x7fb1380078c0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 shutdown_connections 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb13c10b190 0x7fb13c100440 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb13c101d70 0x7fb13c0ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb1240a1290 0x7fb1240a3750 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 --2- 172.21.15.23:0/3121009868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb13c102690 0x7fb13c0fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 >> 172.21.15.23:0/3121009868 conn(0x7fb13c0fb590 msgr2=0x7fb13c0fce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 shutdown_connections 2024-03-30T04:10:11.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:11.769+0000 7fb143200700 1 -- 172.21.15.23:0/3121009868 wait complete. 2024-03-30T04:10:11.769 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:10:11.787 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:10:12.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 -- 172.21.15.23:0/3081535266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e84067e30 msgr2=0x7f9e840682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 --2- 172.21.15.23:0/3081535266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e84067e30 0x7f9e840682b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f9e700099c0 tx=0x7f9e70009cd0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 -- 172.21.15.23:0/3081535266 shutdown_connections 2024-03-30T04:10:12.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 --2- 172.21.15.23:0/3081535266 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e840687f0 0x7f9e8410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 --2- 172.21.15.23:0/3081535266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84105ce0 0x7f9e841060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.056+0000 7f9e8a000700 1 --2- 172.21.15.23:0/3081535266 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e84067e30 0x7f9e840682b0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.057+0000 7f9e8a000700 1 -- 172.21.15.23:0/3081535266 >> 172.21.15.23:0/3081535266 conn(0x7f9e84076d40 msgr2=0x7f9e84077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.057+0000 7f9e8a000700 1 -- 172.21.15.23:0/3081535266 shutdown_connections 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.057+0000 7f9e8a000700 1 -- 172.21.15.23:0/3081535266 wait complete. 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.058+0000 7f9e8a000700 1 Processor -- start 2024-03-30T04:10:12.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.058+0000 7f9e8a000700 1 -- start start 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84067e30 0x7f9e840ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e84105ce0 0x7f9e841006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84101c40 con 0x7f9e84105ce0 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841a9c20 con 0x7f9e840687f0 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e8a000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e841a9de0 con 0x7f9e84067e30 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e84105ce0 0x7f9e841006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:39950/0 (socket says 172.21.15.23:39950) 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 -- 172.21.15.23:0/546473294 learned_addr learned my addr 172.21.15.23:0/546473294 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e83400700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84067e30 0x7f9e840ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 -- 172.21.15.23:0/546473294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84067e30 msgr2=0x7f9e840ffc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84067e30 0x7f9e840ffc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 -- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e84105ce0 msgr2=0x7f9e841006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e84105ce0 0x7f9e841006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.059+0000 7f9e82a00700 1 -- 172.21.15.23:0/546473294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c009c40 con 0x7f9e840687f0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e82a00700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000b580 tx=0x7f9e7000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e7be00700 1 -- 172.21.15.23:0/546473294 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7001c070 con 0x7f9e840687f0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e7be00700 1 -- 172.21.15.23:0/546473294 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e7000fe30 con 0x7f9e840687f0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e700095f0 con 0x7f9e840687f0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e7be00700 1 -- 172.21.15.23:0/546473294 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e70022c80 con 0x7f9e840687f0 2024-03-30T04:10:12.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.060+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e841aa350 con 0x7f9e840687f0 2024-03-30T04:10:12.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.063+0000 7f9e7be00700 1 -- 172.21.15.23:0/546473294 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9e70020070 con 0x7f9e840687f0 2024-03-30T04:10:12.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.065+0000 7f9e7be00700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 0x7f9e680a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.065+0000 7f9e7be00700 1 -- 172.21.15.23:0/546473294 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9e700cde10 con 0x7f9e840687f0 2024-03-30T04:10:12.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.065+0000 7f9e83400700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 0x7f9e680a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.066+0000 7f9e9db8bb80 1 Processor -- start 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.066+0000 7f9e9db8bb80 1 -- start start 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.066+0000 7f9e83400700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 0x7f9e680a37d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e74005950 tx=0x7f9e7400b410 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.066+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28b9a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.066+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28af3990 con 0x559d28ba5df0 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28af1350 con 0x559d28b9a5d0 2024-03-30T04:10:12.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28aecc40 con 0x559d28b9aef0 2024-03-30T04:10:12.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28b9a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35960/0 (socket says 172.21.15.23:35960) 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 -- 172.21.15.23:0/3833992894 learned_addr learned my addr 172.21.15.23:0/3833992894 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e83e00700 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 -- 172.21.15.23:0/3833992894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 msgr2=0x559d28b9a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28b9a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 -- 172.21.15.23:0/3833992894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 msgr2=0x559d28ba81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.067+0000 7f9e82a00700 1 -- 172.21.15.23:0/3833992894 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d2896e3b0 con 0x559d28b9aef0 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.068+0000 7f9e82a00700 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9e700975b0 tx=0x7f9e7009a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=201acd597ed48af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.068+0000 7f9e6fe00700 1 -- 172.21.15.23:0/3833992894 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7009ab40 con 0x559d28b9aef0 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.068+0000 7f9e6fe00700 1 -- 172.21.15.23:0/3833992894 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e7009e970 con 0x559d28b9aef0 2024-03-30T04:10:12.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.068+0000 7f9e6fe00700 1 -- 172.21.15.23:0/3833992894 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7009a030 con 0x559d28b9aef0 2024-03-30T04:10:12.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3833992894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 msgr2=0x559d28ba58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9e700975b0 tx=0x7f9e7009a680 comp rx=0 tx=0).stop 2024-03-30T04:10:12.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3833992894 shutdown_connections 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28b9a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3833992894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba58b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3833992894 >> 172.21.15.23:0/3833992894 conn(0x559d28afa6a0 msgr2=0x559d28b96a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3833992894 shutdown_connections 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.069+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3833992894 wait complete. 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.070+0000 7f9e9db8bb80 1 -- start start 2024-03-30T04:10:12.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.070+0000 7f9e9db8bb80 1 Processor -- start 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28a88d80 con 0x559d28ba5df0 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28a88a20 con 0x559d28b9a5d0 2024-03-30T04:10:12.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d289706a0 con 0x559d28b9aef0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e9db8bb80 10 client.0 _ll_register_callbacks cb 0x559d28a92470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:39958/0 (socket says 172.21.15.23:39958) 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 -- 172.21.15.23:0/4170234127 learned_addr learned my addr 172.21.15.23:0/4170234127 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e82a00700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83e00700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 -- 172.21.15.23:0/4170234127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 msgr2=0x559d28ba2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 -- 172.21.15.23:0/4170234127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 msgr2=0x559d28ba32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83400700 1 -- 172.21.15.23:0/4170234127 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e841aa350 con 0x559d28b9a5d0 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.071+0000 7f9e83e00700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e83400700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9e740116f0 tx=0x7f9e740047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43c01c7f641e85cd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74004a30 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d28aad380 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9e74017070 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e74004b90 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e9db8bb80 10 client.27767 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.072+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559d2891ab60 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559d28aad380 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e7401f430 con 0x559d28b9a5d0 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 10 client.27767 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 10 client.27767 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 20 client.27767 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28bb56d0 0x559d28bb7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 4 client.27767 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:12.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d28ae5700 con 0x559d28bb56d0 2024-03-30T04:10:12.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.073+0000 7f9e82a00700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28bb56d0 0x559d28bb7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.074+0000 7f9e82a00700 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28bb56d0 0x559d28bb7af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f1d5db6e57313a6 server_cookie=fa1e668adb4697e0 in_seq=0 out_seq=0 2024-03-30T04:10:12.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.074+0000 7f9e6cc00700 10 client.27767 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.074+0000 7f9e6cc00700 1 -- 172.21.15.23:0/4170234127 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559d28ae5700 con 0x559d28bb56d0 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.074+0000 7f9e6cc00700 10 client.27767 handle_command_reply: tid=0 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.074+0000 7f9e9db8bb80 1 client.27767 shutdown 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 msgr2=0x559d28ba5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9e740116f0 tx=0x7f9e740047c0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 shutdown_connections 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28ba5df0 0x559d28ba32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28ba5880 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28bb56d0 0x559d28bb7af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/4170234127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28b9aef0 0x559d28ba2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e6cc00700 0 client.27767 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 >> 172.21.15.23:0/4170234127 conn(0x559d28afa6a0 msgr2=0x559d28b9c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 shutdown_connections 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/4170234127 wait complete. 2024-03-30T04:10:12.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.075+0000 7f9e9db8bb80 20 client.27767 trim_cache size 0 max 16384 2024-03-30T04:10:12.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.090+0000 7f9e9db8bb80 1 Processor -- start 2024-03-30T04:10:12.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.090+0000 7f9e9db8bb80 1 -- start start 2024-03-30T04:10:12.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 0x559d28bb1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28ba2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28bb2f00 con 0x559d28b9a5d0 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28ba3640 con 0x559d28b9aef0 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28bb1bb0 con 0x559d28ba5df0 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 0x559d28bb1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e82a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:39970/0 (socket says 172.21.15.23:39970) 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e82a00700 1 -- 172.21.15.23:0/3397498681 learned_addr learned my addr 172.21.15.23:0/3397498681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83e00700 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28ba2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 -- 172.21.15.23:0/3397498681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 msgr2=0x559d28ba2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28ba2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 -- 172.21.15.23:0/3397498681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 msgr2=0x559d28bac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e83400700 1 -- 172.21.15.23:0/3397498681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d28ab2ea0 con 0x559d28b9a5d0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.091+0000 7f9e82a00700 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.092+0000 7f9e83400700 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 0x559d28bb1640 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9e7401f7c0 tx=0x7f9e7401f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d001f1e31f762c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.092+0000 7f9e79200700 1 -- 172.21.15.23:0/3397498681 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74011ab0 con 0x559d28b9a5d0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.092+0000 7f9e79200700 1 -- 172.21.15.23:0/3397498681 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e74015070 con 0x559d28b9a5d0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.092+0000 7f9e79200700 1 -- 172.21.15.23:0/3397498681 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74034d50 con 0x559d28b9a5d0 2024-03-30T04:10:12.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3397498681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 msgr2=0x559d28bb1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 0x559d28bb1640 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9e7401f7c0 tx=0x7f9e7401f9e0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3397498681 shutdown_connections 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28ba2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9a5d0 0x559d28bb1640 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/3397498681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9aef0 0x559d28bac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3397498681 >> 172.21.15.23:0/3397498681 conn(0x559d28afa6a0 msgr2=0x559d28bb0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3397498681 shutdown_connections 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.093+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/3397498681 wait complete. 2024-03-30T04:10:12.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 -- start start 2024-03-30T04:10:12.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 Processor -- start 2024-03-30T04:10:12.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28bac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9aef0 0x559d28baa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28bab3f0 con 0x559d28b9aef0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28bab530 con 0x559d28b9a5d0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d28bab6c0 con 0x559d28ba5df0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.094+0000 7f9e9db8bb80 10 client.0 _ll_register_callbacks cb 0x559d28a92470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28bac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35988/0 (socket says 172.21.15.23:35988) 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83e00700 1 -- 172.21.15.23:0/386816753 learned_addr learned my addr 172.21.15.23:0/386816753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e82a00700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9aef0 0x559d28baa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 -- 172.21.15.23:0/386816753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 msgr2=0x559d28baae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 -- 172.21.15.23:0/386816753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9aef0 msgr2=0x559d28baa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9aef0 0x559d28baa8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 -- 172.21.15.23:0/386816753 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e700b2040 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83e00700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e83400700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28bac4f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9e74012fd0 tx=0x7f9e74011b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34bf243f0ecd709d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.095+0000 7f9e6cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e74034990 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9e74017040 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e7009e970 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e74034af0 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e9db8bb80 10 client.27773 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.096+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559d2891ab60 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e7009e970 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e7403d030 con 0x559d28b9a5d0 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 10 client.27773 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 10 client.27773 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 20 client.27773 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28c0a600 0x559d28c0c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 4 client.27773 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d2893bcf0 con 0x559d28c0a600 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.097+0000 7f9e82a00700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28c0a600 0x559d28c0c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e82a00700 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28c0a600 0x559d28c0c9f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e52b312558c04ab server_cookie=c7ab2d914fcfbc36 in_seq=0 out_seq=0 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e6cc00700 10 client.27773 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e6cc00700 1 -- 172.21.15.23:0/386816753 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d2893bcf0 con 0x559d28c0a600 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e6cc00700 10 client.27773 handle_command_reply: tid=0 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 client.27773 shutdown 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 msgr2=0x559d28bac4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28bac4f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9e74012fd0 tx=0x7f9e74011b50 comp rx=0 tx=0).stop 2024-03-30T04:10:12.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 shutdown_connections 2024-03-30T04:10:12.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x559d28c0a600 0x559d28c0c9f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d28ba5df0 0x559d28baae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559d28b9a5d0 0x559d28bac4f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 --2- 172.21.15.23:0/386816753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d28b9aef0 0x559d28baa8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 >> 172.21.15.23:0/386816753 conn(0x559d28afa6a0 msgr2=0x559d28b9c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e6cc00700 0 client.27773 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 shutdown_connections 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.098+0000 7f9e9db8bb80 1 -- 172.21.15.23:0/386816753 wait complete. 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.099+0000 7f9e9db8bb80 20 client.27773 trim_cache size 0 max 16384 2024-03-30T04:10:12.101 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.099+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 msgr2=0x7f9e680a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.099+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 0x7f9e680a37d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9e74005950 tx=0x7f9e7400b410 comp rx=0 tx=0).stop 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.099+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 msgr2=0x7f9e841001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.099+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000b580 tx=0x7f9e7000f690 comp rx=0 tx=0).stop 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 shutdown_connections 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9e84105ce0 0x7f9e841006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e84067e30 0x7f9e840ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e680a1310 0x7f9e680a37d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 --2- 172.21.15.23:0/546473294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840687f0 0x7f9e841001b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 >> 172.21.15.23:0/546473294 conn(0x7f9e84076d40 msgr2=0x7f9e840feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 shutdown_connections 2024-03-30T04:10:12.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.100+0000 7f9e8a000700 1 -- 172.21.15.23:0/546473294 wait complete. 2024-03-30T04:10:12.119 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:10:12.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 819942 10.55 MB/sec execute 289 sec latency 15.940 ms 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 -- 172.21.15.23:0/1529421464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7654068fd0 msgr2=0x7f76540693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 --2- 172.21.15.23:0/1529421464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7654068fd0 0x7f76540693b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7644009990 tx=0x7f7644009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 -- 172.21.15.23:0/1529421464 shutdown_connections 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 --2- 172.21.15.23:0/1529421464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d750 0x7f765410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 --2- 172.21.15.23:0/1529421464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7654068fd0 0x7f76540693b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 --2- 172.21.15.23:0/1529421464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 0x7f765410d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 -- 172.21.15.23:0/1529421464 >> 172.21.15.23:0/1529421464 conn(0x7f7654076d80 msgr2=0x7f7654077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 -- 172.21.15.23:0/1529421464 shutdown_connections 2024-03-30T04:10:12.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.349+0000 7f7659c00700 1 -- 172.21.15.23:0/1529421464 wait complete. 2024-03-30T04:10:12.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.350+0000 7f7659c00700 1 Processor -- start 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.350+0000 7f7659c00700 1 -- start start 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 0x7f7654199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f765410d750 0x7f765419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765419e310 con 0x7f76540698f0 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765419e480 con 0x7f765410d750 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7659c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765419e640 con 0x7f7654068fd0 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36008/0 (socket says 172.21.15.23:36008) 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 -- 172.21.15.23:0/3219603912 learned_addr learned my addr 172.21.15.23:0/3219603912 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7653400700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f765410d750 0x7f765419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652000700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 0x7f7654199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 -- 172.21.15.23:0/3219603912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f765410d750 msgr2=0x7f765419dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f765410d750 0x7f765419dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 -- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 msgr2=0x7f7654199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 0x7f7654199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.351+0000 7f7652a00700 1 -- 172.21.15.23:0/3219603912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76440095f0 con 0x7f7654068fd0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f7652a00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7644005910 tx=0x7f7644004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f764b400700 1 -- 172.21.15.23:0/3219603912 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764401d070 con 0x7f7654068fd0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f764b400700 1 -- 172.21.15.23:0/3219603912 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7644022470 con 0x7f7654068fd0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f764b400700 1 -- 172.21.15.23:0/3219603912 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644017650 con 0x7f7654068fd0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765419e8c0 con 0x7f7654068fd0 2024-03-30T04:10:12.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.352+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765419eb40 con 0x7f7654068fd0 2024-03-30T04:10:12.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.356+0000 7f764b400700 1 -- 172.21.15.23:0/3219603912 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f76440225e0 con 0x7f7654068fd0 2024-03-30T04:10:12.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.358+0000 7f766d6ddb80 1 Processor -- start 2024-03-30T04:10:12.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.358+0000 7f766d6ddb80 1 -- start start 2024-03-30T04:10:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.358+0000 7f764b400700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 0x7f76380a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.358+0000 7f764b400700 1 -- 172.21.15.23:0/3219603912 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f76440ce1b0 con 0x7f7654068fd0 2024-03-30T04:10:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.358+0000 7f7653400700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 0x7f76380a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 0x561006a251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006970990 con 0x561006a175d0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56100696e350 con 0x561006a17ef0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006969c40 con 0x561006a22df0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60772/0 (socket says 172.21.15.23:60772) 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 -- 172.21.15.23:0/1829471734 learned_addr learned my addr 172.21.15.23:0/1829471734 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7653400700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 0x561006a251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652000700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 -- 172.21.15.23:0/1829471734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 msgr2=0x561006a251e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 0x561006a251e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 -- 172.21.15.23:0/1829471734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 msgr2=0x561006a228b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a228b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7652a00700 1 -- 172.21.15.23:0/1829471734 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610067eb3b0 con 0x561006a175d0 2024-03-30T04:10:12.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.359+0000 7f7653400700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 0x561006a251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.360+0000 7f7653400700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 0x7f76380a3750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7654067940 tx=0x7f764c00fc00 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.360+0000 7f7652a00700 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f76440a2f00 tx=0x7f7644009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94414f0cc656e460 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.360+0000 7f763fe00700 1 -- 172.21.15.23:0/1829471734 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76440ce030 con 0x561006a175d0 2024-03-30T04:10:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.360+0000 7f763fe00700 1 -- 172.21.15.23:0/1829471734 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f764409a9a0 con 0x561006a175d0 2024-03-30T04:10:12.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.360+0000 7f763fe00700 1 -- 172.21.15.23:0/1829471734 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644099470 con 0x561006a175d0 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- 172.21.15.23:0/1829471734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 msgr2=0x561006a179b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f76440a2f00 tx=0x7f7644009310 comp rx=0 tx=0).stop 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- 172.21.15.23:0/1829471734 shutdown_connections 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a22df0 0x561006a251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a175d0 0x561006a179b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/1829471734 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a228b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- 172.21.15.23:0/1829471734 >> 172.21.15.23:0/1829471734 conn(0x5610069776a0 msgr2=0x561006a13a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- 172.21.15.23:0/1829471734 shutdown_connections 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- 172.21.15.23:0/1829471734 wait complete. 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.361+0000 7f766d6ddb80 1 -- start start 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 Processor -- start 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a22710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 0x561006a1fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 0x561006a1ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006905d80 con 0x561006a17ef0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006905a20 con 0x561006a22df0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610067ed6a0 con 0x561006a175d0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f766d6ddb80 10 client.0 _ll_register_callbacks cb 0x56100690f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7652000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 0x561006a1fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7653400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 0x561006a1ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7652a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a22710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7652000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 0x561006a1fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60780/0 (socket says 172.21.15.23:60780) 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7652a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a22710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36028/0 (socket says 172.21.15.23:36028) 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.362+0000 7f7652000700 1 -- 172.21.15.23:0/3028158462 learned_addr learned my addr 172.21.15.23:0/3028158462 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 -- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 msgr2=0x561006a22710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a22710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 -- 172.21.15.23:0/3028158462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 msgr2=0x561006a1fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 0x561006a1fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 -- 172.21.15.23:0/3028158462 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765419e8c0 con 0x561006a22df0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f7653400700 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 0x561006a1ff90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f764c018b50 tx=0x7f764c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dda00156f0fc975 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f763cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c028070 con 0x561006a22df0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f764c0093f0 con 0x561006a22df0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.363+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56100692a380 con 0x561006a22df0 2024-03-30T04:10:12.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.364+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f764c00e040 con 0x561006a22df0 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.364+0000 7f766d6ddb80 10 client.27785 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.364+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561006797b60 con 0x561006a22df0 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.364+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56100692a380 con 0x561006a22df0 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.364+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f764c02b020 con 0x561006a22df0 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 10 client.27785 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 10 client.27785 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 20 client.27785 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a32680 0x561006a34aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 4 client.27785 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:12.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561006962700 con 0x561006a32680 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f7652a00700 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a32680 0x561006a34aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f763cc00700 10 client.27785 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.365+0000 7f7652a00700 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a32680 0x561006a34aa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39330ba4a7b78aab server_cookie=22070b9bf0fa93a6 in_seq=0 out_seq=0 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f763cc00700 1 -- 172.21.15.23:0/3028158462 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561006962700 con 0x561006a32680 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f763cc00700 10 client.27785 handle_command_reply: tid=0 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 client.27785 shutdown 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 msgr2=0x561006a1ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 0x561006a1ff90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f764c018b50 tx=0x7f764c004b40 comp rx=0 tx=0).stop 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 shutdown_connections 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a32680 0x561006a34aa0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a22df0 0x561006a1ff90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a22710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f763cc00700 0 client.27785 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3028158462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a17ef0 0x561006a1fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.366+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 >> 172.21.15.23:0/3028158462 conn(0x5610069776a0 msgr2=0x561006a19680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.367+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 shutdown_connections 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.367+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3028158462 wait complete. 2024-03-30T04:10:12.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.367+0000 7f766d6ddb80 20 client.27785 trim_cache size 0 max 16384 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 Processor -- start 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 -- start start 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a12f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a0cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a18850 con 0x561006a22df0 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a34e30 con 0x561006a17ef0 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a202f0 con 0x561006a175d0 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f7652000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f7652000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40024/0 (socket says 172.21.15.23:40024) 2024-03-30T04:10:12.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f7652000700 1 -- 172.21.15.23:0/3776675925 learned_addr learned my addr 172.21.15.23:0/3776675925 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f7653400700 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a0cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.382+0000 7f7652a00700 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a12f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 -- 172.21.15.23:0/3776675925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 msgr2=0x561006a12f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a12f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 -- 172.21.15.23:0/3776675925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 msgr2=0x561006a0cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a0cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 -- 172.21.15.23:0/3776675925 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56100692fea0 con 0x561006a17ef0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7652000700 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7640009e90 tx=0x7f7640009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb449f8adf7da02c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7648c00700 1 -- 172.21.15.23:0/3776675925 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640017070 con 0x561006a17ef0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7648c00700 1 -- 172.21.15.23:0/3776675925 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7640018040 con 0x561006a17ef0 2024-03-30T04:10:12.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.383+0000 7f7648c00700 1 -- 172.21.15.23:0/3776675925 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640003b10 con 0x561006a17ef0 2024-03-30T04:10:12.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3776675925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 msgr2=0x561006a297e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f7640009e90 tx=0x7f7640009260 comp rx=0 tx=0).stop 2024-03-30T04:10:12.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3776675925 shutdown_connections 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a0cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a175d0 0x561006a12f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/3776675925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a17ef0 0x561006a297e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3776675925 >> 172.21.15.23:0/3776675925 conn(0x5610069776a0 msgr2=0x561006a0c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3776675925 shutdown_connections 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.384+0000 7f766d6ddb80 1 -- 172.21.15.23:0/3776675925 wait complete. 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 -- start start 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 Processor -- start 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a17ef0 0x561006a27950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a27e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a28410 con 0x561006a22df0 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a28580 con 0x561006a175d0 2024-03-30T04:10:12.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561006a28740 con 0x561006a17ef0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f766d6ddb80 10 client.0 _ll_register_callbacks cb 0x56100690f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.385+0000 7f7652a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40026/0 (socket says 172.21.15.23:40026) 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 -- 172.21.15.23:0/762583942 learned_addr learned my addr 172.21.15.23:0/762583942 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7653400700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a27e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652000700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a17ef0 0x561006a27950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 -- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a17ef0 msgr2=0x561006a27950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a17ef0 0x561006a27950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 -- 172.21.15.23:0/762583942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 msgr2=0x561006a27e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a27e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 -- 172.21.15.23:0/762583942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7640009b40 con 0x561006a175d0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7653400700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a27e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f7652a00700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f764409afd0 tx=0x7f76440a1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5657d912e97ad7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f763cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764409b740 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f764409b040 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.386+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f764409a9a0 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76440a3430 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 10 client.27791 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5610067efcd0 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f764409a9a0 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f764409b2a0 con 0x561006a175d0 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 10 client.27791 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 10 client.27791 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 20 client.27791 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a87600 0x561006a899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 4 client.27791 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:12.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5610067b8cf0 con 0x561006a87600 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.387+0000 7f7652000700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a87600 0x561006a899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f7652000700 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a87600 0x561006a899f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91e7275175f27343 server_cookie=aa5b9a5e9e7bb6c0 in_seq=0 out_seq=0 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f763cc00700 10 client.27791 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f763cc00700 1 -- 172.21.15.23:0/762583942 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5610067b8cf0 con 0x561006a87600 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f763cc00700 10 client.27791 handle_command_reply: tid=0 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f766d6ddb80 1 client.27791 shutdown 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 msgr2=0x561006a294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.388+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f764409afd0 tx=0x7f76440a1040 comp rx=0 tx=0).stop 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 shutdown_connections 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x561006a87600 0x561006a899f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561006a22df0 0x561006a27e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561006a175d0 0x561006a294b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f763cc00700 0 client.27791 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:12.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 --2- 172.21.15.23:0/762583942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561006a17ef0 0x561006a27950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 >> 172.21.15.23:0/762583942 conn(0x5610069776a0 msgr2=0x561006a1e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 shutdown_connections 2024-03-30T04:10:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 1 -- 172.21.15.23:0/762583942 wait complete. 2024-03-30T04:10:12.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.389+0000 7f766d6ddb80 20 client.27791 trim_cache size 0 max 16384 2024-03-30T04:10:12.392 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 msgr2=0x7f76380a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 0x7f76380a3750 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7654067940 tx=0x7f764c00fc00 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 msgr2=0x7f76541993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7644005910 tx=0x7f7644004730 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 shutdown_connections 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76380a1290 0x7f76380a3750 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f765410d750 0x7f765419dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7654068fd0 0x7f76541993f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 --2- 172.21.15.23:0/3219603912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76540698f0 0x7f7654199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 >> 172.21.15.23:0/3219603912 conn(0x7f7654076d80 msgr2=0x7f765410ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 shutdown_connections 2024-03-30T04:10:12.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.390+0000 7f7659c00700 1 -- 172.21.15.23:0/3219603912 wait complete. 2024-03-30T04:10:12.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:12 smithi023 ceph-mon[38254]: pgmap v255: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-30T04:10:12.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:12 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:10:12.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:12 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:10:12.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:12 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:10:12.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:12 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3121009868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:12.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:10:12.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:12 smithi154 ceph-mon[79161]: pgmap v255: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-30T04:10:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:12 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:10:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:12 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:10:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:12 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:10:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:12 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3121009868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:12 smithi193 ceph-mon[79581]: pgmap v255: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.6 MiB/s wr, 230 op/s 2024-03-30T04:10:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:12 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:10:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:12 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:10:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:12 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:10:12.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:12 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3121009868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:12.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.629+0000 7fd61e600700 1 -- 172.21.15.23:0/2368696883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618100710 msgr2=0x7fd618100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.629+0000 7fd61e600700 1 --2- 172.21.15.23:0/2368696883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618100710 0x7fd618100af0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd608009990 tx=0x7fd608009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 -- 172.21.15.23:0/2368696883 shutdown_connections 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 --2- 172.21.15.23:0/2368696883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 0x7fd61810fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 --2- 172.21.15.23:0/2368696883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618100710 0x7fd618100af0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 --2- 172.21.15.23:0/2368696883 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618101030 0x7fd61810d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 -- 172.21.15.23:0/2368696883 >> 172.21.15.23:0/2368696883 conn(0x7fd618076d40 msgr2=0x7fd618077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 -- 172.21.15.23:0/2368696883 shutdown_connections 2024-03-30T04:10:12.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.630+0000 7fd61e600700 1 -- 172.21.15.23:0/2368696883 wait complete. 2024-03-30T04:10:12.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 Processor -- start 2024-03-30T04:10:12.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 -- start start 2024-03-30T04:10:12.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618101030 0x7fd618199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 0x7fd61819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61819e530 con 0x7fd618101030 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61819e6a0 con 0x7fd618100710 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd61e600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd61819e860 con 0x7fd61810d750 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd617400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd617400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40048/0 (socket says 172.21.15.23:40048) 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 -- 172.21.15.23:0/3042433174 learned_addr learned my addr 172.21.15.23:0/3042433174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.631+0000 7fd616a00700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618101030 0x7fd618199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617e00700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 0x7fd61819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 -- 172.21.15.23:0/3042433174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 msgr2=0x7fd61819def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 0x7fd61819def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 -- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618101030 msgr2=0x7fd618199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618101030 0x7fd618199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 -- 172.21.15.23:0/3042433174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080095f0 con 0x7fd618100710 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.632+0000 7fd617400700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd608006010 tx=0x7fd608003b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.633+0000 7fd60fe00700 1 -- 172.21.15.23:0/3042433174 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60801d070 con 0x7fd618100710 2024-03-30T04:10:12.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.633+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61819eae0 con 0x7fd618100710 2024-03-30T04:10:12.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.633+0000 7fd60fe00700 1 -- 172.21.15.23:0/3042433174 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd608005580 con 0x7fd618100710 2024-03-30T04:10:12.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.633+0000 7fd60fe00700 1 -- 172.21.15.23:0/3042433174 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60800fce0 con 0x7fd618100710 2024-03-30T04:10:12.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.633+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6181a6380 con 0x7fd618100710 2024-03-30T04:10:12.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.636+0000 7fd60fe00700 1 -- 172.21.15.23:0/3042433174 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd60800f460 con 0x7fd618100710 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd60fe00700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 0x7fd5fc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd60fe00700 1 -- 172.21.15.23:0/3042433174 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fd6080ce1c0 con 0x7fd618100710 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 Processor -- start 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd616a00700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 0x7fd5fc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 -- start start 2024-03-30T04:10:12.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af6a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4aeb5990 con 0x55cb4af67df0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4aeb3350 con 0x55cb4af5cef0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.638+0000 7fd63220ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4aeaec40 con 0x55cb4af5c5d0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 0x7fd5fc0a3750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd61819ab60 tx=0x7fd604009190 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40050/0 (socket says 172.21.15.23:40050) 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 -- 172.21.15.23:0/3844591827 learned_addr learned my addr 172.21.15.23:0/3844591827 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd617e00700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af6a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd617400700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 -- 172.21.15.23:0/3844591827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 msgr2=0x55cb4af5c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 -- 172.21.15.23:0/3844591827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 msgr2=0x55cb4af6a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af6a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd616a00700 1 -- 172.21.15.23:0/3844591827 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb4ad303b0 con 0x55cb4af5cef0 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.639+0000 7fd617400700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.640+0000 7fd616a00700 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd604006e80 tx=0x7fd604011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e481f8e29017898 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.640+0000 7fd60cc00700 1 -- 172.21.15.23:0/3844591827 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60400eeb0 con 0x55cb4af5cef0 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.640+0000 7fd60cc00700 1 -- 172.21.15.23:0/3844591827 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd604015940 con 0x55cb4af5cef0 2024-03-30T04:10:12.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.640+0000 7fd60cc00700 1 -- 172.21.15.23:0/3844591827 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040139f0 con 0x55cb4af5cef0 2024-03-30T04:10:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- 172.21.15.23:0/3844591827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 msgr2=0x55cb4af678b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd604006e80 tx=0x7fd604011000 comp rx=0 tx=0).stop 2024-03-30T04:10:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- 172.21.15.23:0/3844591827 shutdown_connections 2024-03-30T04:10:12.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af6a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3844591827 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af678b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- 172.21.15.23:0/3844591827 >> 172.21.15.23:0/3844591827 conn(0x55cb4aebc6a0 msgr2=0x55cb4af58a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- 172.21.15.23:0/3844591827 shutdown_connections 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- 172.21.15.23:0/3844591827 wait complete. 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 -- start start 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.641+0000 7fd63220ab80 1 Processor -- start 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5c5d0 0x55cb4af67840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af64b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4ae4ad80 con 0x55cb4af67df0 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4ae4aa20 con 0x55cb4af5c5d0 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4ad326a0 con 0x55cb4af5cef0 2024-03-30T04:10:12.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd63220ab80 10 client.0 _ll_register_callbacks cb 0x55cb4ae54470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd617e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd617e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60818/0 (socket says 172.21.15.23:60818) 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.642+0000 7fd617e00700 1 -- 172.21.15.23:0/3993222635 learned_addr learned my addr 172.21.15.23:0/3993222635 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd616a00700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af64b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617400700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5c5d0 0x55cb4af67840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 -- 172.21.15.23:0/3993222635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 msgr2=0x55cb4af64b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af64b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 -- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5c5d0 msgr2=0x55cb4af67840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5c5d0 0x55cb4af67840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 -- 172.21.15.23:0/3993222635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60400e700 con 0x55cb4af67df0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd617e00700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd61000ab30 tx=0x7fd61000aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d135ab7cf06cdda server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd601600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:12.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610010040 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fd610006e40 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.643+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb4ae6f380 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd610011ba0 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 10 client.17799 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cb4acdcb60 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cb4ae6f380 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd610008030 con 0x55cb4af67df0 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 10 client.17799 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 10 client.17799 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 20 client.17799 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4af77a20 0x55cb4af79e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 4 client.17799 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.644+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb4aea7700 con 0x55cb4af77a20 2024-03-30T04:10:12.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.645+0000 7fd617400700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4af77a20 0x55cb4af79e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.645+0000 7fd617400700 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4af77a20 0x55cb4af79e40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39bdf64165ed71f6 server_cookie=f0d5d93e8bff90bd in_seq=0 out_seq=0 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.645+0000 7fd601600700 10 client.17799 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd601600700 1 -- 172.21.15.23:0/3993222635 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cb4aea7700 con 0x55cb4af77a20 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd601600700 10 client.17799 handle_command_reply: tid=0 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 client.17799 shutdown 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 msgr2=0x55cb4af65270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd61000ab30 tx=0x7fd61000aef0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 shutdown_connections 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4af77a20 0x55cb4af79e40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af67df0 0x55cb4af65270 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5c5d0 0x55cb4af67840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd601600700 0 client.17799 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:12.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 --2- 172.21.15.23:0/3993222635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af64b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 >> 172.21.15.23:0/3993222635 conn(0x55cb4aebc6a0 msgr2=0x55cb4af5e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 shutdown_connections 2024-03-30T04:10:12.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.646+0000 7fd63220ab80 1 -- 172.21.15.23:0/3993222635 wait complete. 2024-03-30T04:10:12.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.647+0000 7fd63220ab80 20 client.17799 trim_cache size 0 max 16384 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 Processor -- start 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 -- start start 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af6ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af66160 con 0x55cb4af5c5d0 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af65880 con 0x55cb4af67df0 2024-03-30T04:10:12.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd63220ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af6f030 con 0x55cb4af5cef0 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd617e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd617400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd617e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40062/0 (socket says 172.21.15.23:40062) 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.661+0000 7fd617e00700 1 -- 172.21.15.23:0/1440600729 learned_addr learned my addr 172.21.15.23:0/1440600729 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd616a00700 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af6ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 -- 172.21.15.23:0/1440600729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 msgr2=0x55cb4af6ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af6ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 -- 172.21.15.23:0/1440600729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 msgr2=0x55cb4af5f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 -- 172.21.15.23:0/1440600729 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb4ae74ea0 con 0x55cb4af67df0 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd617e00700 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd610004c20 tx=0x7fd610011540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=772c32b490bac829 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd60d600700 1 -- 172.21.15.23:0/1440600729 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610010050 con 0x55cb4af67df0 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd60d600700 1 -- 172.21.15.23:0/1440600729 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd610011040 con 0x55cb4af67df0 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.662+0000 7fd60d600700 1 -- 172.21.15.23:0/1440600729 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd610016450 con 0x55cb4af67df0 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 -- 172.21.15.23:0/1440600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 msgr2=0x55cb4af731c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd610004c20 tx=0x7fd610011540 comp rx=0 tx=0).stop 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 -- 172.21.15.23:0/1440600729 shutdown_connections 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af67df0 0x55cb4af731c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af5f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 --2- 172.21.15.23:0/1440600729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af5cef0 0x55cb4af6ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 -- 172.21.15.23:0/1440600729 >> 172.21.15.23:0/1440600729 conn(0x55cb4aebc6a0 msgr2=0x55cb4af515a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 -- 172.21.15.23:0/1440600729 shutdown_connections 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.663+0000 7fd63220ab80 1 -- 172.21.15.23:0/1440600729 wait complete. 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 -- start start 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 Processor -- start 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af6e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af67df0 0x55cb4af6d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af6d5d0 con 0x55cb4af5c5d0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af6d740 con 0x55cb4af5cef0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb4af6d900 con 0x55cb4af67df0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.664+0000 7fd63220ab80 10 client.0 _ll_register_callbacks cb 0x55cb4ae54470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40070/0 (socket says 172.21.15.23:40070) 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 -- 172.21.15.23:0/504313832 learned_addr learned my addr 172.21.15.23:0/504313832 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd617400700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af6e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd617e00700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af67df0 0x55cb4af6d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 -- 172.21.15.23:0/504313832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af67df0 msgr2=0x55cb4af6d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af67df0 0x55cb4af6d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 -- 172.21.15.23:0/504313832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 msgr2=0x55cb4af6e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af6e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 -- 172.21.15.23:0/504313832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb4acdcb60 con 0x55cb4af5cef0 2024-03-30T04:10:12.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd616a00700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd60400e9f0 tx=0x7fd604015cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19c73dc9683654d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.665+0000 7fd601600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60402d070 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd604015940 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fd604013660 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd60400e4e0 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 10 client.27806 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cb4ad34cd0 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd604015940 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd604028820 con 0x55cb4af5cef0 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 10 client.27806 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 10 client.27806 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.666+0000 7fd63220ab80 20 client.27806 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4afcccc0 0x55cb4afcf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd63220ab80 4 client.27806 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:12.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cb4acfdcf0 con 0x55cb4afcccc0 2024-03-30T04:10:12.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd617400700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4afcccc0 0x55cb4afcf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd617400700 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4afcccc0 0x55cb4afcf0e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d3b2070acb1413d1 server_cookie=d8d4d48616496fa3 in_seq=0 out_seq=0 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd601600700 10 client.27806 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd601600700 1 -- 172.21.15.23:0/504313832 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cb4acfdcf0 con 0x55cb4afcccc0 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd601600700 10 client.27806 handle_command_reply: tid=0 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.667+0000 7fd63220ab80 1 client.27806 shutdown 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 msgr2=0x55cb4af6cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd60400e9f0 tx=0x7fd604015cc0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 shutdown_connections 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55cb4afcccc0 0x55cb4afcf0e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cb4af67df0 0x55cb4af6d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cb4af5c5d0 0x55cb4af6e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 --2- 172.21.15.23:0/504313832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cb4af5cef0 0x55cb4af6cb50 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 >> 172.21.15.23:0/504313832 conn(0x55cb4aebc6a0 msgr2=0x55cb4af731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.668+0000 7fd601600700 0 client.27806 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 shutdown_connections 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd63220ab80 1 -- 172.21.15.23:0/504313832 wait complete. 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd63220ab80 20 client.27806 trim_cache size 0 max 16384 2024-03-30T04:10:12.671 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 msgr2=0x7fd5fc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 0x7fd5fc0a3750 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd61819ab60 tx=0x7fd604009190 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.669+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 msgr2=0x7fd618199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd608006010 tx=0x7fd608003b60 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 shutdown_connections 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd61810d750 0x7fd61819def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5fc0a1290 0x7fd5fc0a3750 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd618100710 0x7fd618199610 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 --2- 172.21.15.23:0/3042433174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd618101030 0x7fd618199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 >> 172.21.15.23:0/3042433174 conn(0x7fd618076d40 msgr2=0x7fd61810eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 shutdown_connections 2024-03-30T04:10:12.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.670+0000 7fd61e600700 1 -- 172.21.15.23:0/3042433174 wait complete. 2024-03-30T04:10:12.688 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:10:12.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.910+0000 7f6122e00700 1 -- 172.21.15.23:0/2145297003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c100f60 msgr2=0x7f611c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.910+0000 7f6122e00700 1 --2- 172.21.15.23:0/2145297003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c100f60 0x7f611c101340 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f610c009990 tx=0x7f610c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 -- 172.21.15.23:0/2145297003 shutdown_connections 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 --2- 172.21.15.23:0/2145297003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c106080 0x7f611c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 --2- 172.21.15.23:0/2145297003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c100f60 0x7f611c101340 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 --2- 172.21.15.23:0/2145297003 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c101910 0x7f611c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 -- 172.21.15.23:0/2145297003 >> 172.21.15.23:0/2145297003 conn(0x7f611c0fc7c0 msgr2=0x7f611c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 -- 172.21.15.23:0/2145297003 shutdown_connections 2024-03-30T04:10:12.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.911+0000 7f6122e00700 1 -- 172.21.15.23:0/2145297003 wait complete. 2024-03-30T04:10:12.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.912+0000 7f6122e00700 1 Processor -- start 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.912+0000 7f6122e00700 1 -- start start 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c106080 0x7f611c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c073b30 0x7f611c1a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1a5980 con 0x7f611c106080 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1a5af0 con 0x7f611c073b30 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6122e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c1a5cb0 con 0x7f611c100f60 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36120/0 (socket says 172.21.15.23:36120) 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 -- 172.21.15.23:0/1384361419 learned_addr learned my addr 172.21.15.23:0/1384361419 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f6120e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c073b30 0x7f611c1a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611b400700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c106080 0x7f611c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 -- 172.21.15.23:0/1384361419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c073b30 msgr2=0x7f611c1a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c073b30 0x7f611c1a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 -- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c106080 msgr2=0x7f611c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.913+0000 7f611be00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c106080 0x7f611c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f611be00700 1 -- 172.21.15.23:0/1384361419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0095f0 con 0x7f611c100f60 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f611be00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f610c005770 tx=0x7f610c0057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f6118c00700 1 -- 172.21.15.23:0/1384361419 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c01d070 con 0x7f611c100f60 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f6118c00700 1 -- 172.21.15.23:0/1384361419 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f610c005da0 con 0x7f611c100f60 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f6118c00700 1 -- 172.21.15.23:0/1384361419 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610c021ea0 con 0x7f611c100f60 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c1a5f30 con 0x7f611c100f60 2024-03-30T04:10:12.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.914+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c1a6350 con 0x7f611c100f60 2024-03-30T04:10:12.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.918+0000 7f6118c00700 1 -- 172.21.15.23:0/1384361419 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f610c0215a0 con 0x7f611c100f60 2024-03-30T04:10:12.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 Processor -- start 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 -- start start 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6118c00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 0x7f61000a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24ae8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee23fc990 con 0x55aee24a3ef0 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee23fa350 con 0x55aee24aedf0 2024-03-30T04:10:12.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6136971b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee23f5c40 con 0x55aee24a35d0 2024-03-30T04:10:12.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.919+0000 7f6118c00700 1 -- 172.21.15.23:0/1384361419 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f610c0ce0e0 con 0x7f611c100f60 2024-03-30T04:10:12.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36124/0 (socket says 172.21.15.23:36124) 2024-03-30T04:10:12.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 -- 172.21.15.23:0/1692720439 learned_addr learned my addr 172.21.15.23:0/1692720439 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f611be00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 0x7f61000a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f611b400700 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24ae8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 -- 172.21.15.23:0/1692720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 msgr2=0x55aee24b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 -- 172.21.15.23:0/1692720439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 msgr2=0x55aee24ae8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24ae8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.920+0000 7f6120e00700 1 -- 172.21.15.23:0/1692720439 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aee22773b0 con 0x55aee24a35d0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.921+0000 7f6120e00700 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6114013ef0 tx=0x7f6114009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=927e57d47f7982bc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.921+0000 7f611be00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 0x7f61000a37d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f610c09ada0 tx=0x7f610c005f10 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.921+0000 7f6111600700 1 -- 172.21.15.23:0/1692720439 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611400e040 con 0x55aee24a35d0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.921+0000 7f6111600700 1 -- 172.21.15.23:0/1692720439 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f611400f040 con 0x55aee24a35d0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.921+0000 7f6111600700 1 -- 172.21.15.23:0/1692720439 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6114004020 con 0x55aee24a35d0 2024-03-30T04:10:12.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- 172.21.15.23:0/1692720439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 msgr2=0x55aee24a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6114013ef0 tx=0x7f6114009920 comp rx=0 tx=0).stop 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- 172.21.15.23:0/1692720439 shutdown_connections 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24a39b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 --2- 172.21.15.23:0/1692720439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24ae8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- 172.21.15.23:0/1692720439 >> 172.21.15.23:0/1692720439 conn(0x55aee24036a0 msgr2=0x55aee249fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- 172.21.15.23:0/1692720439 shutdown_connections 2024-03-30T04:10:12.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- 172.21.15.23:0/1692720439 wait complete. 2024-03-30T04:10:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.922+0000 7f6136971b80 1 -- start start 2024-03-30T04:10:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.923+0000 7f6136971b80 1 Processor -- start 2024-03-30T04:10:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a35d0 0x55aee24ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24ac320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee2391d80 con 0x55aee24a35d0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee2391a20 con 0x55aee24aedf0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee22796a0 con 0x55aee24a3ef0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6136971b80 10 client.0 _ll_register_callbacks cb 0x55aee239b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36132/0 (socket says 172.21.15.23:36132) 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 -- 172.21.15.23:0/1876565495 learned_addr learned my addr 172.21.15.23:0/1876565495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6120e00700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a35d0 0x55aee24ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f6120e00700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 -- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 msgr2=0x55aee24ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24ac320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.924+0000 7f611b400700 1 -- 172.21.15.23:0/1876565495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a35d0 msgr2=0x55aee24ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f611b400700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a35d0 0x55aee24ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f611b400700 1 -- 172.21.15.23:0/1876565495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6114013ba0 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f6120e00700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f611b400700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6108012700 tx=0x7f6108012a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3723b5bb368b663 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f6106000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108017070 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f6108004260 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.925+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aee23b6380 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6108004420 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 10 client.27808 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55aee2223b60 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aee23b6380 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6108011800 con 0x55aee24a3ef0 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 10 client.27808 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 10 client.27808 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 20 client.27808 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee24bea30 0x55aee24c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 4 client.27808 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.926+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aee23ee700 con 0x55aee24bea30 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.927+0000 7f6120e00700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee24bea30 0x55aee24c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.927+0000 7f6120e00700 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee24bea30 0x55aee24c0e50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=80acf86c352162c0 server_cookie=427f4ec75e1de554 in_seq=0 out_seq=0 2024-03-30T04:10:12.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.927+0000 7f6106000700 10 client.27808 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6106000700 1 -- 172.21.15.23:0/1876565495 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aee23ee700 con 0x55aee24bea30 2024-03-30T04:10:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6106000700 10 client.27808 handle_command_reply: tid=0 2024-03-30T04:10:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 client.27808 shutdown 2024-03-30T04:10:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 msgr2=0x55aee24abbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6108012700 tx=0x7f6108012a10 comp rx=0 tx=0).stop 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 shutdown_connections 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24ac320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a35d0 0x55aee24ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee24bea30 0x55aee24c0e50 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 --2- 172.21.15.23:0/1876565495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24abbe0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 >> 172.21.15.23:0/1876565495 conn(0x55aee24036a0 msgr2=0x55aee24a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6106000700 0 client.27808 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 shutdown_connections 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.928+0000 7f6136971b80 1 -- 172.21.15.23:0/1876565495 wait complete. 2024-03-30T04:10:12.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.929+0000 7f6136971b80 20 client.27808 trim_cache size 0 max 16384 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.944+0000 7f6136971b80 1 Processor -- start 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 -- start start 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24abba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24acee0 con 0x55aee24a3ef0 2024-03-30T04:10:12.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24c11e0 con 0x55aee24aedf0 2024-03-30T04:10:12.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6136971b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24a0660 con 0x55aee24a35d0 2024-03-30T04:10:12.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6120e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24abba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f611b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f611b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60866/0 (socket says 172.21.15.23:60866) 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f611b400700 1 -- 172.21.15.23:0/2486568033 learned_addr learned my addr 172.21.15.23:0/2486568033 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6120e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24abba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36144/0 (socket says 172.21.15.23:36144) 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.945+0000 7f6120e00700 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 -- 172.21.15.23:0/2486568033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 msgr2=0x55aee24abba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24abba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 -- 172.21.15.23:0/2486568033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 msgr2=0x55aee24b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 -- 172.21.15.23:0/2486568033 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aee23bbea0 con 0x55aee24a3ef0 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f6120e00700 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f611b400700 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6108001eb0 tx=0x7f6108012e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e228e1564ba24c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f6112000700 1 -- 172.21.15.23:0/2486568033 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108017040 con 0x55aee24a3ef0 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f6112000700 1 -- 172.21.15.23:0/2486568033 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f61080119c0 con 0x55aee24a3ef0 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.946+0000 7f6112000700 1 -- 172.21.15.23:0/2486568033 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108003cf0 con 0x55aee24a3ef0 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 -- 172.21.15.23:0/2486568033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 msgr2=0x55aee24a6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6108001eb0 tx=0x7f6108012e90 comp rx=0 tx=0).stop 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 -- 172.21.15.23:0/2486568033 shutdown_connections 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24aedf0 0x55aee24b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a35d0 0x55aee24abba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 --2- 172.21.15.23:0/2486568033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24a3ef0 0x55aee24a6670 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 -- 172.21.15.23:0/2486568033 >> 172.21.15.23:0/2486568033 conn(0x55aee24036a0 msgr2=0x55aee24ab470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 -- 172.21.15.23:0/2486568033 shutdown_connections 2024-03-30T04:10:12.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.947+0000 7f6136971b80 1 -- 172.21.15.23:0/2486568033 wait complete. 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 -- start start 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 Processor -- start 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24a35d0 0x55aee24b5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 0x55aee24b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24b4430 con 0x55aee24aedf0 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24b45a0 con 0x55aee24a35d0 2024-03-30T04:10:12.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aee24b4760 con 0x55aee24a3ef0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.948+0000 7f6136971b80 10 client.0 _ll_register_callbacks cb 0x55aee239b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f6120e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24a35d0 0x55aee24b5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f6120e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 0x55aee24b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36150/0 (socket says 172.21.15.23:36150) 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 -- 172.21.15.23:0/4253134137 learned_addr learned my addr 172.21.15.23:0/4253134137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 -- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24a35d0 msgr2=0x55aee24b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24a35d0 0x55aee24b5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 -- 172.21.15.23:0/4253134137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 msgr2=0x55aee24b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 0x55aee24b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 -- 172.21.15.23:0/4253134137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aee2223b60 con 0x55aee24a3ef0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f6120e00700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 0x55aee24b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f611b400700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6108009fb0 tx=0x7f6108012a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c6d679133a1d08f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:12.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.949+0000 7f6106000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6108017070 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aee2244cf0 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f6108011500 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61080116c0 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6136971b80 10 client.27814 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55aee227bcd0 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.950+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aee2244cf0 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6108030070 con 0x55aee24a3ef0 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 10 client.27814 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 10 client.27814 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 20 client.27814 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee25138a0 0x55aee2515c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 4 client.27814 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:12.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55aee2279ce0 con 0x55aee25138a0 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.951+0000 7f6120e00700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee25138a0 0x55aee2515c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6120e00700 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee25138a0 0x55aee2515c90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2b78ba9f589113d server_cookie=cac6bb404f4740fc in_seq=0 out_seq=0 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6106000700 10 client.27814 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6106000700 1 -- 172.21.15.23:0/4253134137 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55aee2279ce0 con 0x55aee25138a0 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6106000700 10 client.27814 handle_command_reply: tid=0 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 client.27814 shutdown 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 msgr2=0x55aee24b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6108009fb0 tx=0x7f6108012a10 comp rx=0 tx=0).stop 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 shutdown_connections 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aee24aedf0 0x55aee24b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aee24a35d0 0x55aee24b5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55aee25138a0 0x55aee2515c90 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 --2- 172.21.15.23:0/4253134137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aee24a3ef0 0x55aee24b39b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6106000700 0 client.27814 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:12.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 >> 172.21.15.23:0/4253134137 conn(0x55aee24036a0 msgr2=0x55aee24ba160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 shutdown_connections 2024-03-30T04:10:12.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.952+0000 7f6136971b80 1 -- 172.21.15.23:0/4253134137 wait complete. 2024-03-30T04:10:12.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6136971b80 20 client.27814 trim_cache size 0 max 16384 2024-03-30T04:10:12.955 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 msgr2=0x7f61000a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 0x7f61000a37d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f610c09ada0 tx=0x7f610c005f10 comp rx=0 tx=0).stop 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 msgr2=0x7f611c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f610c005770 tx=0x7f610c0057a0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 shutdown_connections 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f61000a1310 0x7f61000a37d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f611c073b30 0x7f611c1a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f611c100f60 0x7f611c0730b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 --2- 172.21.15.23:0/1384361419 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f611c106080 0x7f611c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:12.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.953+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 >> 172.21.15.23:0/1384361419 conn(0x7f611c0fc7c0 msgr2=0x7f611c0fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:12.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.954+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 shutdown_connections 2024-03-30T04:10:12.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:12.954+0000 7f6122e00700 1 -- 172.21.15.23:0/1384361419 wait complete. 2024-03-30T04:10:12.970 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:10:13.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 -- 172.21.15.23:0/652920331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee0106080 msgr2=0x7f0ee010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 --2- 172.21.15.23:0/652920331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee0106080 0x7f0ee010dc00 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8009f90 tx=0x7f0ed800b330 comp rx=0 tx=0).stop 2024-03-30T04:10:13.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 -- 172.21.15.23:0/652920331 shutdown_connections 2024-03-30T04:10:13.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 --2- 172.21.15.23:0/652920331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee0106080 0x7f0ee010dc00 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 --2- 172.21.15.23:0/652920331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0100f60 0x7f0ee0101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 --2- 172.21.15.23:0/652920331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0101910 0x7f0ee0105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.196+0000 7f0ee8000700 1 -- 172.21.15.23:0/652920331 >> 172.21.15.23:0/652920331 conn(0x7f0ee00fc7c0 msgr2=0x7f0ee00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.197+0000 7f0ee8000700 1 -- 172.21.15.23:0/652920331 shutdown_connections 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.197+0000 7f0ee8000700 1 -- 172.21.15.23:0/652920331 wait complete. 2024-03-30T04:10:13.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.198+0000 7f0ee8000700 1 Processor -- start 2024-03-30T04:10:13.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.198+0000 7f0ee8000700 1 -- start start 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee019de20 0x7f0ee019e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee019e8a0 con 0x7f0ee019de20 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee019ea10 con 0x7f0ee0106080 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee8000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee019ebd0 con 0x7f0ee0100f60 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee5600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40128/0 (socket says 172.21.15.23:40128) 2024-03-30T04:10:13.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee019de20 0x7f0ee019e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 -- 172.21.15.23:0/3353794146 learned_addr learned my addr 172.21.15.23:0/3353794146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee5600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36170/0 (socket says 172.21.15.23:36170) 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 -- 172.21.15.23:0/3353794146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 msgr2=0x7f0ee0199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 -- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee019de20 msgr2=0x7f0ee019e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee019de20 0x7f0ee019e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee4c00700 1 -- 172.21.15.23:0/3353794146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40095f0 con 0x7f0ee0106080 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.199+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:13.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00e970 tx=0x7f0ecc00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ede000700 1 -- 172.21.15.23:0/3353794146 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc009790 con 0x7f0ee0106080 2024-03-30T04:10:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ede000700 1 -- 172.21.15.23:0/3353794146 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ecc00c7e0 con 0x7f0ee0106080 2024-03-30T04:10:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed8009c40 con 0x7f0ee0106080 2024-03-30T04:10:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ede000700 1 -- 172.21.15.23:0/3353794146 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0054e0 con 0x7f0ee0106080 2024-03-30T04:10:13.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.200+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee01a6480 con 0x7f0ee0106080 2024-03-30T04:10:13.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.205+0000 7f0ede000700 1 -- 172.21.15.23:0/3353794146 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0ecc00c950 con 0x7f0ee0106080 2024-03-30T04:10:13.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.206+0000 7f0ede000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 0x7f0ec80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.206+0000 7f0efbb8eb80 1 Processor -- start 2024-03-30T04:10:13.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.206+0000 7f0ede000700 1 -- 172.21.15.23:0/3353794146 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f0ecc014070 con 0x7f0ee0106080 2024-03-30T04:10:13.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.206+0000 7f0efbb8eb80 1 -- start start 2024-03-30T04:10:13.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.206+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 0x7f0ec80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb8865d0 0x5580bb8869b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb8918b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8941e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb7df990 con 0x5580bb8865d0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb7dd350 con 0x5580bb891df0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb7d8c40 con 0x5580bb886ef0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb8918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb8918b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36176/0 (socket says 172.21.15.23:36176) 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0ee4c00700 1 -- 172.21.15.23:0/71704630 learned_addr learned my addr 172.21.15.23:0/71704630 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0ee5600700 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb8865d0 0x5580bb8869b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.207+0000 7f0ee6000700 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8941e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 0x7f0ec80a3750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4005fd0 tx=0x7f0ed400fa90 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 -- 172.21.15.23:0/71704630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 msgr2=0x5580bb8918b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb8918b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 -- 172.21.15.23:0/71704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb8865d0 msgr2=0x5580bb8869b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb8865d0 0x5580bb8869b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 -- 172.21.15.23:0/71704630 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580bb65a3b0 con 0x5580bb891df0 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ee6000700 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8941e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800b6c0 tx=0x7f0ed80094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=813174dc271aaba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ed2600700 1 -- 172.21.15.23:0/71704630 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800e040 con 0x5580bb891df0 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ed2600700 1 -- 172.21.15.23:0/71704630 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ed8007620 con 0x5580bb891df0 2024-03-30T04:10:13.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.208+0000 7f0ed2600700 1 -- 172.21.15.23:0/71704630 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800bbf0 con 0x5580bb891df0 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/71704630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 msgr2=0x5580bb8941e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8941e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800b6c0 tx=0x7f0ed80094a0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/71704630 shutdown_connections 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8941e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb8865d0 0x5580bb8869b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/71704630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb8918b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/71704630 >> 172.21.15.23:0/71704630 conn(0x5580bb7e66a0 msgr2=0x5580bb882a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/71704630 shutdown_connections 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.209+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/71704630 wait complete. 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 -- start start 2024-03-30T04:10:13.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 Processor -- start 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8917f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb774d80 con 0x5580bb891df0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb774a20 con 0x5580bb886ef0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb65c6a0 con 0x5580bb8865d0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.210+0000 7f0efbb8eb80 10 client.0 _ll_register_callbacks cb 0x5580bb77e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee6000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60930/0 (socket says 172.21.15.23:60930) 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee6000700 1 -- 172.21.15.23:0/4232530435 learned_addr learned my addr 172.21.15.23:0/4232530435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40138/0 (socket says 172.21.15.23:40138) 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee5600700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 -- 172.21.15.23:0/4232530435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 msgr2=0x5580bb8917f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8917f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 -- 172.21.15.23:0/4232530435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 msgr2=0x5580bb88f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 -- 172.21.15.23:0/4232530435 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee010e470 con 0x5580bb886ef0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee6000700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc096a10 tx=0x7f0ecc09a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c96bae7a5378083 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.211+0000 7f0ec6a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc0a2420 con 0x5580bb886ef0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f0ecc09a200 con 0x5580bb886ef0 2024-03-30T04:10:13.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5580bb799380 con 0x5580bb886ef0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ecc09a3c0 con 0x5580bb886ef0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 10 client.27836 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5580bb606b60 con 0x5580bb886ef0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5580bb799380 con 0x5580bb886ef0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0ecc0995b0 con 0x5580bb886ef0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 10 client.27836 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 10 client.27836 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 20 client.27836 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8a1920 0x5580bb8a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 4 client.27836 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.212+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580bb7d1700 con 0x5580bb8a1920 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.213+0000 7f0ee5600700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8a1920 0x5580bb8a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.213+0000 7f0ee5600700 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8a1920 0x5580bb8a3d40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9ec123a8b94004f server_cookie=7cf21d9c83256d25 in_seq=0 out_seq=0 2024-03-30T04:10:13.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.213+0000 7f0ec6a00700 10 client.27836 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.214+0000 7f0ec6a00700 1 -- 172.21.15.23:0/4232530435 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5580bb7d1700 con 0x5580bb8a1920 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.214+0000 7f0ec6a00700 10 client.27836 handle_command_reply: tid=0 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.214+0000 7f0efbb8eb80 1 client.27836 shutdown 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.214+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 msgr2=0x5580bb88eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.214+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc096a10 tx=0x7f0ecc09a6a0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 shutdown_connections 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb88f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8a1920 0x5580bb8a3d40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8917f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/4232530435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb886ef0 0x5580bb88eb30 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 >> 172.21.15.23:0/4232530435 conn(0x5580bb7e66a0 msgr2=0x5580bb8886c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0ec6a00700 0 client.27836 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:13.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 shutdown_connections 2024-03-30T04:10:13.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/4232530435 wait complete. 2024-03-30T04:10:13.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.215+0000 7f0efbb8eb80 20 client.27836 trim_cache size 0 max 16384 2024-03-30T04:10:13.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 821302 10.54 MB/sec execute 290 sec latency 21.955 ms 2024-03-30T04:10:13.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.237+0000 7f0efbb8eb80 1 Processor -- start 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.237+0000 7f0efbb8eb80 1 -- start start 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb8865d0 0x5580bb88f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb87bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb8841d0 con 0x5580bb891df0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb88f770 con 0x5580bb8865d0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb8a16a0 con 0x5580bb886ef0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.238+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36186/0 (socket says 172.21.15.23:36186) 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 -- 172.21.15.23:0/1274748585 learned_addr learned my addr 172.21.15.23:0/1274748585 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee5600700 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb8865d0 0x5580bb88f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee6000700 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb87bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 -- 172.21.15.23:0/1274748585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb8865d0 msgr2=0x5580bb88f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb8865d0 0x5580bb88f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 -- 172.21.15.23:0/1274748585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 msgr2=0x5580bb87bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb87bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 -- 172.21.15.23:0/1274748585 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580bb79eea0 con 0x5580bb886ef0 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.239+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc099f30 tx=0x7f0ecc099c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba6a78845e8ab91 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.240+0000 7f0ed3000700 1 -- 172.21.15.23:0/1274748585 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc09a710 con 0x5580bb886ef0 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.240+0000 7f0ed3000700 1 -- 172.21.15.23:0/1274748585 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ecc0a2420 con 0x5580bb886ef0 2024-03-30T04:10:13.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.240+0000 7f0ed3000700 1 -- 172.21.15.23:0/1274748585 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc09a460 con 0x5580bb886ef0 2024-03-30T04:10:13.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/1274748585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 msgr2=0x5580bb898a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc099f30 tx=0x7f0ecc099c10 comp rx=0 tx=0).stop 2024-03-30T04:10:13.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/1274748585 shutdown_connections 2024-03-30T04:10:13.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb891df0 0x5580bb87bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb8865d0 0x5580bb88f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/1274748585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb886ef0 0x5580bb898a10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/1274748585 >> 172.21.15.23:0/1274748585 conn(0x5580bb7e66a0 msgr2=0x5580bb87b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/1274748585 shutdown_connections 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.241+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/1274748585 wait complete. 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.242+0000 7f0efbb8eb80 1 -- start start 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.242+0000 7f0efbb8eb80 1 Processor -- start 2024-03-30T04:10:13.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb886ef0 0x5580bb8eef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8969f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb897040 con 0x5580bb886ef0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb8971b0 con 0x5580bb891df0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580bb897370 con 0x5580bb8865d0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.243+0000 7f0efbb8eb80 10 client.0 _ll_register_callbacks cb 0x5580bb77e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36200/0 (socket says 172.21.15.23:36200) 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 -- 172.21.15.23:0/3287759631 learned_addr learned my addr 172.21.15.23:0/3287759631 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb886ef0 0x5580bb8eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 -- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 msgr2=0x5580bb8969f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8969f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 -- 172.21.15.23:0/3287759631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb886ef0 msgr2=0x5580bb8eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:13.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb886ef0 0x5580bb8eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 -- 172.21.15.23:0/3287759631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580bb606b60 con 0x5580bb8865d0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.244+0000 7f0ee5600700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x5580bb8a4620 tx=0x7f0ed40047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec196ec5431fd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0ec6a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed4023040 con 0x5580bb8865d0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f0ed40043a0 con 0x5580bb8865d0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ed8007620 con 0x5580bb8865d0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ed4004560 con 0x5580bb8865d0 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0efbb8eb80 10 client.27832 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:13.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.245+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5580bb65ecd0 con 0x5580bb8865d0 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ed8007620 con 0x5580bb8865d0 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0ed4003b10 con 0x5580bb8865d0 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 10 client.27832 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 10 client.27832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 20 client.27832 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8f68d0 0x5580bb8f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 4 client.27832 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5580bb627cf0 con 0x5580bb8f68d0 2024-03-30T04:10:13.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.246+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8f68d0 0x5580bb8f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.247+0000 7f0ee4c00700 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8f68d0 0x5580bb8f8cf0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22171414a9f2f1b1 server_cookie=301fdb4a366cf60d in_seq=0 out_seq=0 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.247+0000 7f0ec6a00700 10 client.27832 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.248+0000 7f0ec6a00700 1 -- 172.21.15.23:0/3287759631 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5580bb627cf0 con 0x5580bb8f68d0 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.248+0000 7f0ec6a00700 10 client.27832 handle_command_reply: tid=0 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.248+0000 7f0efbb8eb80 1 client.27832 shutdown 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.248+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 msgr2=0x5580bb8eea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.248+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x5580bb8a4620 tx=0x7f0ed40047c0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 shutdown_connections 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5580bb891df0 0x5580bb8969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5580bb8f68d0 0x5580bb8f8cf0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0ec6a00700 0 client.27832 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5580bb8865d0 0x5580bb8eea50 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 --2- 172.21.15.23:0/3287759631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5580bb886ef0 0x5580bb8eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.249+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 >> 172.21.15.23:0/3287759631 conn(0x5580bb7e66a0 msgr2=0x5580bb88d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.250+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 shutdown_connections 2024-03-30T04:10:13.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.250+0000 7f0efbb8eb80 1 -- 172.21.15.23:0/3287759631 wait complete. 2024-03-30T04:10:13.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.250+0000 7f0efbb8eb80 20 client.27832 trim_cache size 0 max 16384 2024-03-30T04:10:13.249 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 msgr2=0x7f0ec80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 0x7f0ec80a3750 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4005fd0 tx=0x7f0ed400fa90 comp rx=0 tx=0).stop 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 msgr2=0x7f0ee0199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00e970 tx=0x7f0ecc00ec80 comp rx=0 tx=0).stop 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 shutdown_connections 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ee019de20 0x7f0ee019e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ee0100f60 0x7f0ee0199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0ec80a1290 0x7f0ec80a3750 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 --2- 172.21.15.23:0/3353794146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ee0106080 0x7f0ee0199ad0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.251+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 >> 172.21.15.23:0/3353794146 conn(0x7f0ee00fc7c0 msgr2=0x7f0ee00fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.253+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 shutdown_connections 2024-03-30T04:10:13.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:13.253+0000 7f0ee8000700 1 -- 172.21.15.23:0/3353794146 wait complete. 2024-03-30T04:10:14.214 INFO:tasks.workunit.client.0.smithi023.stdout: 1 823309 10.54 MB/sec execute 291 sec latency 108.147 ms 2024-03-30T04:10:14.285 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:10:14.286 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:10:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:14 smithi154 ceph-mon[79161]: pgmap v256: 129 pgs: 129 active+clean; 462 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-03-30T04:10:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:14 smithi193 ceph-mon[79581]: pgmap v256: 129 pgs: 129 active+clean; 462 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-03-30T04:10:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:14 smithi023 ceph-mon[38254]: pgmap v256: 129 pgs: 129 active+clean; 462 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.8 MiB/s wr, 215 op/s 2024-03-30T04:10:14.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.547+0000 7fc620e00700 1 -- 172.21.15.23:0/695959114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c101910 msgr2=0x7fc61c105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.547+0000 7fc620e00700 1 --2- 172.21.15.23:0/695959114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c101910 0x7fc61c105950 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc608009990 tx=0x7fc608009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 -- 172.21.15.23:0/695959114 shutdown_connections 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 --2- 172.21.15.23:0/695959114 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c106080 0x7fc61c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 --2- 172.21.15.23:0/695959114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c100f60 0x7fc61c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 --2- 172.21.15.23:0/695959114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c101910 0x7fc61c105950 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 -- 172.21.15.23:0/695959114 >> 172.21.15.23:0/695959114 conn(0x7fc61c0fc7c0 msgr2=0x7fc61c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 -- 172.21.15.23:0/695959114 shutdown_connections 2024-03-30T04:10:14.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.548+0000 7fc620e00700 1 -- 172.21.15.23:0/695959114 wait complete. 2024-03-30T04:10:14.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 Processor -- start 2024-03-30T04:10:14.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 -- start start 2024-03-30T04:10:14.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 0x7fc61c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c073b30 0x7fc61c1a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5a30 con 0x7fc61c100f60 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5ba0 con 0x7fc61c106080 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc620e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61c1a5d60 con 0x7fc61c073b30 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc619e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 0x7fc61c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc61a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c073b30 0x7fc61c1a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40148/0 (socket says 172.21.15.23:40148) 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.549+0000 7fc619400700 1 -- 172.21.15.23:0/3893247289 learned_addr learned my addr 172.21.15.23:0/3893247289 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 -- 172.21.15.23:0/3893247289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c073b30 msgr2=0x7fc61c1a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c073b30 0x7fc61c1a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 -- 172.21.15.23:0/3893247289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 msgr2=0x7fc61c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 -- 172.21.15.23:0/3893247289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6080095f0 con 0x7fc61c100f60 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619400700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc619e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 0x7fc61c0730b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00e9f0 tx=0x7fc60c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc612a00700 1 -- 172.21.15.23:0/3893247289 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c00c9c0 con 0x7fc61c100f60 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.550+0000 7fc612a00700 1 -- 172.21.15.23:0/3893247289 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc60c010430 con 0x7fc61c100f60 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.551+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61c1a5fe0 con 0x7fc61c100f60 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.551+0000 7fc612a00700 1 -- 172.21.15.23:0/3893247289 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c018680 con 0x7fc61c100f60 2024-03-30T04:10:14.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.551+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c1a64c0 con 0x7fc61c100f60 2024-03-30T04:10:14.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.554+0000 7fc612a00700 1 -- 172.21.15.23:0/3893247289 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc60c010860 con 0x7fc61c100f60 2024-03-30T04:10:14.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.555+0000 7fc612a00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 0x7fc6000a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.555+0000 7fc612a00700 1 -- 172.21.15.23:0/3893247289 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fc60c014070 con 0x7fc61c100f60 2024-03-30T04:10:14.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.555+0000 7fc6348eeb80 1 Processor -- start 2024-03-30T04:10:14.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.555+0000 7fc619400700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 0x7fc6000a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.555+0000 7fc6348eeb80 1 -- start start 2024-03-30T04:10:14.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cd650 0x55f9295cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295db260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f929526a10 con 0x55f9295d8e70 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9295243d0 con 0x55f9295cdf70 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92951fcc0 con 0x55f9295cd650 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc619400700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 0x7fc6000a3750 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0747e0 tx=0x7fc608005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc61a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295db260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40164/0 (socket says 172.21.15.23:40164) 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc619400700 1 -- 172.21.15.23:0/2989646428 learned_addr learned my addr 172.21.15.23:0/2989646428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.556+0000 7fc619e00700 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cd650 0x55f9295cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 -- 172.21.15.23:0/2989646428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cd650 msgr2=0x55f9295cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cd650 0x55f9295cda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 -- 172.21.15.23:0/2989646428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 msgr2=0x55f9295db260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295db260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 -- 172.21.15.23:0/2989646428 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9293a1430 con 0x55f9295cdf70 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc619400700 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc608005680 tx=0x7fc608020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16c01edde354df29 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc607400700 1 -- 172.21.15.23:0/2989646428 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60800fdd0 con 0x55f9295cdf70 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc607400700 1 -- 172.21.15.23:0/2989646428 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc60801e040 con 0x55f9295cdf70 2024-03-30T04:10:14.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.557+0000 7fc607400700 1 -- 172.21.15.23:0/2989646428 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608004c70 con 0x55f9295cdf70 2024-03-30T04:10:14.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 -- 172.21.15.23:0/2989646428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 msgr2=0x55f9295d8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fc608005680 tx=0x7fc608020000 comp rx=0 tx=0).stop 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 -- 172.21.15.23:0/2989646428 shutdown_connections 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295db260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cd650 0x55f9295cda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/2989646428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d8930 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 -- 172.21.15.23:0/2989646428 >> 172.21.15.23:0/2989646428 conn(0x55f92952d720 msgr2=0x55f9295c9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 -- 172.21.15.23:0/2989646428 shutdown_connections 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.558+0000 7fc6348eeb80 1 -- 172.21.15.23:0/2989646428 wait complete. 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 -- start start 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 Processor -- start 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295d5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295d61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9294bbe00 con 0x55f9295d8e70 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9294bbaa0 con 0x55f9295cd650 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.559+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9293a3720 con 0x55f9295cdf70 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc6348eeb80 10 client.0 _ll_register_callbacks cb 0x55f9294c54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40180/0 (socket says 172.21.15.23:40180) 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 -- 172.21.15.23:0/405730429 learned_addr learned my addr 172.21.15.23:0/405730429 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc61a800700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295d61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619400700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295d5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 -- 172.21.15.23:0/405730429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 msgr2=0x55f9295d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295d5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 -- 172.21.15.23:0/405730429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 msgr2=0x55f9295d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295d61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 -- 172.21.15.23:0/405730429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61c10e470 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc619e00700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0ca5a0 tx=0x7fc60c099b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e902499b865d3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.560+0000 7fc5fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0995f0 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fc60c099f00 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9294e0400 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc60c09dd30 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 10 client.27851 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f92934dbe0 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f9294e0400 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc60c094ba0 con 0x55f9295cd650 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 10 client.27851 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:14.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 10 client.27851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 20 client.27851 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f9295e8870 0x55f9295eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 4 client.27851 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.561+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f929518780 con 0x55f9295e8870 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.562+0000 7fc619400700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f9295e8870 0x55f9295eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.562+0000 7fc5fbe00700 10 client.27851 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.562+0000 7fc619400700 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f9295e8870 0x55f9295eac90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f493d57158f89452 server_cookie=39513991a1f1184c in_seq=0 out_seq=0 2024-03-30T04:10:14.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc5fbe00700 1 -- 172.21.15.23:0/405730429 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f929518780 con 0x55f9295e8870 2024-03-30T04:10:14.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc5fbe00700 10 client.27851 handle_command_reply: tid=0 2024-03-30T04:10:14.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 client.27851 shutdown 2024-03-30T04:10:14.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 msgr2=0x55f9295d8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0ca5a0 tx=0x7fc60c099b60 comp rx=0 tx=0).stop 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 shutdown_connections 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295d8e70 0x55f9295d61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cd650 0x55f9295d8830 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f9295e8870 0x55f9295eac90 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/405730429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295d5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc5fbe00700 0 client.27851 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 >> 172.21.15.23:0/405730429 conn(0x55f92952d720 msgr2=0x55f9295cf740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 shutdown_connections 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.563+0000 7fc6348eeb80 1 -- 172.21.15.23:0/405730429 wait complete. 2024-03-30T04:10:14.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.564+0000 7fc6348eeb80 20 client.27851 trim_cache size 0 max 16384 2024-03-30T04:10:14.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.583+0000 7fc6348eeb80 1 Processor -- start 2024-03-30T04:10:14.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.583+0000 7fc6348eeb80 1 -- start start 2024-03-30T04:10:14.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295d8e70 0x55f9295d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9295cf550 con 0x55f9295cd650 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9295e8690 con 0x55f9295cdf70 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9295ce6c0 con 0x55f9295d8e70 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40190/0 (socket says 172.21.15.23:40190) 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32768/0 (socket says 172.21.15.23:32768) 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619400700 1 -- 172.21.15.23:0/3759916254 learned_addr learned my addr 172.21.15.23:0/3759916254 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc61a800700 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295d8e70 0x55f9295d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 -- 172.21.15.23:0/3759916254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295d8e70 msgr2=0x55f9295d5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295d8e70 0x55f9295d5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 -- 172.21.15.23:0/3759916254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 msgr2=0x55f9295d5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619e00700 1 -- 172.21.15.23:0/3759916254 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9294e5f20 con 0x55f9295cd650 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.584+0000 7fc619400700 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.585+0000 7fc619e00700 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc60c098250 tx=0x7fc60c0107b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b99300a7ddef926 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.585+0000 7fc607e00700 1 -- 172.21.15.23:0/3759916254 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c099f00 con 0x55f9295cd650 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.585+0000 7fc607e00700 1 -- 172.21.15.23:0/3759916254 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc60c09dd30 con 0x55f9295cd650 2024-03-30T04:10:14.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.585+0000 7fc607e00700 1 -- 172.21.15.23:0/3759916254 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0993f0 con 0x55f9295cd650 2024-03-30T04:10:14.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3759916254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 msgr2=0x55f9295dfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc60c098250 tx=0x7fc60c0107b0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3759916254 shutdown_connections 2024-03-30T04:10:14.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295d8e70 0x55f9295d5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295dfa90 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3759916254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295cdf70 0x55f9295d5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3759916254 >> 172.21.15.23:0/3759916254 conn(0x55f92952d720 msgr2=0x55f9295d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3759916254 shutdown_connections 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3759916254 wait complete. 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.586+0000 7fc6348eeb80 1 -- start start 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 Processor -- start 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295dee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295d8e70 0x55f92963c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92963cbb0 con 0x55f9295cd650 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92963ccf0 con 0x55f9295d8e70 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f92963ce30 con 0x55f9295cdf70 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc6348eeb80 10 client.0 _ll_register_callbacks cb 0x55f9294c54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:14.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc619e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc619400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295dee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc619400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295dee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36254/0 (socket says 172.21.15.23:36254) 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc619e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:32774/0 (socket says 172.21.15.23:32774) 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.587+0000 7fc619400700 1 -- 172.21.15.23:0/3237527237 learned_addr learned my addr 172.21.15.23:0/3237527237 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc61a800700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295d8e70 0x55f92963c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 -- 172.21.15.23:0/3237527237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 msgr2=0x55f9295dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295dee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 -- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295d8e70 msgr2=0x55f92963c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295d8e70 0x55f92963c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 -- 172.21.15.23:0/3237527237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc608033040 con 0x55f9295cd650 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc619e00700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc60c096400 tx=0x7fc60c094d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2675e1c0f099af8a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc5fbe00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0a2380 con 0x55f9295cd650 2024-03-30T04:10:14.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fc60c099f00 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc60801e040 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.588+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc60c0993f0 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 10 client.17859 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f92934dbe0 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc60801e040 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc60c0b2bb0 con 0x55f9295cd650 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 10 client.17859 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 10 client.17859 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 20 client.17859 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f92963e110 0x55f929640500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 4 client.17859 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f92936ed70 con 0x55f92963e110 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.589+0000 7fc619400700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f92963e110 0x55f929640500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.590+0000 7fc619400700 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f92963e110 0x55f929640500 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b8b2b576d6a14c3 server_cookie=5b13bb7deca71071 in_seq=0 out_seq=0 2024-03-30T04:10:14.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.590+0000 7fc5fbe00700 10 client.17859 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc5fbe00700 1 -- 172.21.15.23:0/3237527237 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f92936ed70 con 0x55f92963e110 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc5fbe00700 10 client.17859 handle_command_reply: tid=0 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 client.17859 shutdown 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 msgr2=0x55f9295de910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc60c096400 tx=0x7fc60c094d20 comp rx=0 tx=0).stop 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 shutdown_connections 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f9295d8e70 0x55f92963c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f9295cd650 0x55f9295de910 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f92963e110 0x55f929640500 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 --2- 172.21.15.23:0/3237527237 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9295cdf70 0x55f9295dee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 >> 172.21.15.23:0/3237527237 conn(0x55f92952d720 msgr2=0x55f9295d5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc5fbe00700 0 client.17859 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 shutdown_connections 2024-03-30T04:10:14.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.591+0000 7fc6348eeb80 1 -- 172.21.15.23:0/3237527237 wait complete. 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc6348eeb80 20 client.17859 trim_cache size 0 max 16384 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "661fdbcd-38aa-4c59-a020-c872ec693ae0", 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:10:14.594 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 msgr2=0x7fc6000a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 0x7fc6000a3750 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0747e0 tx=0x7fc608005fd0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 msgr2=0x7fc61c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 0x7fc61c0730b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00e9f0 tx=0x7fc60c00ed00 comp rx=0 tx=0).stop 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 shutdown_connections 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc61c073b30 0x7fc61c1a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc61c100f60 0x7fc61c0730b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc6000a1290 0x7fc6000a3750 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 --2- 172.21.15.23:0/3893247289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc61c106080 0x7fc61c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.592+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 >> 172.21.15.23:0/3893247289 conn(0x7fc61c0fc7c0 msgr2=0x7fc61c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.593+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 shutdown_connections 2024-03-30T04:10:14.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.593+0000 7fc620e00700 1 -- 172.21.15.23:0/3893247289 wait complete. 2024-03-30T04:10:14.610 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:10:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 -- 172.21.15.23:0/1355802176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec106080 msgr2=0x7f1aec10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 --2- 172.21.15.23:0/1355802176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec106080 0x7f1aec10dc00 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400b0a0 tx=0x7f1ae400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 -- 172.21.15.23:0/1355802176 shutdown_connections 2024-03-30T04:10:14.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 --2- 172.21.15.23:0/1355802176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec106080 0x7f1aec10dc00 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 --2- 172.21.15.23:0/1355802176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec100f60 0x7f1aec101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 --2- 172.21.15.23:0/1355802176 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec101910 0x7f1aec105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.814+0000 7f1af3800700 1 -- 172.21.15.23:0/1355802176 >> 172.21.15.23:0/1355802176 conn(0x7f1aec0fc7c0 msgr2=0x7f1aec0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.815+0000 7f1af3800700 1 -- 172.21.15.23:0/1355802176 shutdown_connections 2024-03-30T04:10:14.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.815+0000 7f1af3800700 1 -- 172.21.15.23:0/1355802176 wait complete. 2024-03-30T04:10:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 Processor -- start 2024-03-30T04:10:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 -- start start 2024-03-30T04:10:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec100f60 0x7f1aec199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec19e680 con 0x7f1aec100f60 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec19e7f0 con 0x7f1aec19dc00 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af3800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec19e9b0 con 0x7f1aec106080 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec100f60 0x7f1aec199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1aebe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40230/0 (socket says 172.21.15.23:40230) 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1aebe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36270/0 (socket says 172.21.15.23:36270) 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.816+0000 7f1af1800700 1 -- 172.21.15.23:0/2950772566 learned_addr learned my addr 172.21.15.23:0/2950772566 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 -- 172.21.15.23:0/2950772566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 msgr2=0x7f1aec199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 -- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec100f60 msgr2=0x7f1aec199320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec100f60 0x7f1aec199320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 -- 172.21.15.23:0/2950772566 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae4009d00 con 0x7f1aec19dc00 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1aebe00700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af1800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400b070 tx=0x7f1ae40077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1ae9600700 1 -- 172.21.15.23:0/2950772566 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae400e040 con 0x7f1aec19dc00 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1ae9600700 1 -- 172.21.15.23:0/2950772566 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ae40094c0 con 0x7f1aec19dc00 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec1a5eb0 con 0x7f1aec19dc00 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.817+0000 7f1ae9600700 1 -- 172.21.15.23:0/2950772566 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4004310 con 0x7f1aec19dc00 2024-03-30T04:10:14.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.818+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec1a6330 con 0x7f1aec19dc00 2024-03-30T04:10:14.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.822+0000 7f1ae9600700 1 -- 172.21.15.23:0/2950772566 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1ae4003a00 con 0x7f1aec19dc00 2024-03-30T04:10:14.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 Processor -- start 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 -- start start 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1ae9600700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 0x7f1ad40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cbef0 0x5613800d68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800d6df0 0x5613800d91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561380024990 con 0x5613800d6df0 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561380022350 con 0x5613800cb5d0 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56138001dc40 con 0x5613800cbef0 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.823+0000 7f1ae9600700 1 -- 172.21.15.23:0/2950772566 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f1ae40ce200 con 0x7f1aec19dc00 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1aebe00700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 0x7f1ad40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40236/0 (socket says 172.21.15.23:40236) 2024-03-30T04:10:14.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 -- 172.21.15.23:0/1494763596 learned_addr learned my addr 172.21.15.23:0/1494763596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af1800700 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800d6df0 0x5613800d91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1aebe00700 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cbef0 0x5613800d68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 -- 172.21.15.23:0/1494763596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cbef0 msgr2=0x5613800d68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cbef0 0x5613800d68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 -- 172.21.15.23:0/1494763596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800d6df0 msgr2=0x5613800d91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800d6df0 0x5613800d91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.824+0000 7f1af0e00700 1 -- 172.21.15.23:0/1494763596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56137fe9f3b0 con 0x5613800cb5d0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.825+0000 7f1af0e00700 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009d80 tx=0x7f1ae0011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4283f800420efad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.825+0000 7f1aebe00700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 0x7f1ad40a37d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1aec19a870 tx=0x7f1ad8009ed0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.825+0000 7f1ade000700 1 -- 172.21.15.23:0/1494763596 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0004230 con 0x5613800cb5d0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.825+0000 7f1ade000700 1 -- 172.21.15.23:0/1494763596 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ae0004390 con 0x5613800cb5d0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.825+0000 7f1ade000700 1 -- 172.21.15.23:0/1494763596 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00197e0 con 0x5613800cb5d0 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- 172.21.15.23:0/1494763596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 msgr2=0x5613800cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009d80 tx=0x7f1ae0011720 comp rx=0 tx=0).stop 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- 172.21.15.23:0/1494763596 shutdown_connections 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800d6df0 0x5613800d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800cb9b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/1494763596 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cbef0 0x5613800d68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- 172.21.15.23:0/1494763596 >> 172.21.15.23:0/1494763596 conn(0x56138002b6a0 msgr2=0x5613800c7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- 172.21.15.23:0/1494763596 shutdown_connections 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- 172.21.15.23:0/1494763596 wait complete. 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 -- start start 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.826+0000 7f1b073c4b80 1 Processor -- start 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800d6df0 0x5613800d4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56137ffb9d80 con 0x5613800cbef0 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56137ffb9a20 con 0x5613800cb5d0 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56137fea16a0 con 0x5613800d6df0 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1b073c4b80 10 client.0 _ll_register_callbacks cb 0x56137ffc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1aebe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40246/0 (socket says 172.21.15.23:40246) 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1af0e00700 1 -- 172.21.15.23:0/2682911326 learned_addr learned my addr 172.21.15.23:0/2682911326 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.827+0000 7f1af1800700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800d6df0 0x5613800d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 -- 172.21.15.23:0/2682911326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800d6df0 msgr2=0x5613800d4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800d6df0 0x5613800d4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 -- 172.21.15.23:0/2682911326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 msgr2=0x5613800d3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 -- 172.21.15.23:0/2682911326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00099e0 con 0x5613800cb5d0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1af0e00700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0003f10 tx=0x7f1ae0009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f058c794e7f3648 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1ad2a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0003d30 con 0x5613800cb5d0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f1ae0005ce0 con 0x5613800cb5d0 2024-03-30T04:10:14.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.828+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56137ffde380 con 0x5613800cb5d0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ae0005040 con 0x5613800cb5d0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 10 client.27869 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56137fe4bb60 con 0x5613800cb5d0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56137ffde380 con 0x5613800cb5d0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1ae0021040 con 0x5613800cb5d0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 10 client.27869 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 10 client.27869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 20 client.27869 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5613800e68e0 0x5613800e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 4 client.27869 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561380016700 con 0x5613800e68e0 2024-03-30T04:10:14.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.829+0000 7f1aebe00700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5613800e68e0 0x5613800e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.830+0000 7f1aebe00700 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5613800e68e0 0x5613800e8d00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e15faf209c49fa35 server_cookie=6f1392b99fbf104d in_seq=0 out_seq=0 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.830+0000 7f1ad2a00700 10 client.27869 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1ad2a00700 1 -- 172.21.15.23:0/2682911326 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561380016700 con 0x5613800e68e0 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1ad2a00700 10 client.27869 handle_command_reply: tid=0 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 client.27869 shutdown 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 msgr2=0x5613800d6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0003f10 tx=0x7f1ae0009f40 comp rx=0 tx=0).stop 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 shutdown_connections 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5613800e68e0 0x5613800e8d00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800d6df0 0x5613800d4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800cb5d0 0x5613800d6810 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1ad2a00700 0 client.27869 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/2682911326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 >> 172.21.15.23:0/2682911326 conn(0x56138002b6a0 msgr2=0x5613800cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 shutdown_connections 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 1 -- 172.21.15.23:0/2682911326 wait complete. 2024-03-30T04:10:14.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.831+0000 7f1b073c4b80 20 client.27869 trim_cache size 0 max 16384 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 Processor -- start 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 -- start start 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800d33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800cc850 con 0x5613800cbef0 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800d4a60 con 0x5613800d6df0 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.849+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800ddfe0 con 0x5613800cb5d0 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36308/0 (socket says 172.21.15.23:36308) 2024-03-30T04:10:14.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 -- 172.21.15.23:0/696771077 learned_addr learned my addr 172.21.15.23:0/696771077 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1aebe00700 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 -- 172.21.15.23:0/696771077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 msgr2=0x5613800d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800d33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 -- 172.21.15.23:0/696771077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 msgr2=0x5613800d6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 -- 172.21.15.23:0/696771077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56137ffe3ea0 con 0x5613800cb5d0 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1af0e00700 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0004330 tx=0x7f1ae0011a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fb7ee6faa3059b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1adea00700 1 -- 172.21.15.23:0/696771077 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0003960 con 0x5613800cb5d0 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1adea00700 1 -- 172.21.15.23:0/696771077 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ae00236f0 con 0x5613800cb5d0 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.850+0000 7f1adea00700 1 -- 172.21.15.23:0/696771077 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000e540 con 0x5613800cb5d0 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- 172.21.15.23:0/696771077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 msgr2=0x5613800ddaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0004330 tx=0x7f1ae0011a70 comp rx=0 tx=0).stop 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- 172.21.15.23:0/696771077 shutdown_connections 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800d33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800ddaa0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/696771077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800d6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- 172.21.15.23:0/696771077 >> 172.21.15.23:0/696771077 conn(0x56138002b6a0 msgr2=0x5613800d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- 172.21.15.23:0/696771077 shutdown_connections 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- 172.21.15.23:0/696771077 wait complete. 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.851+0000 7f1b073c4b80 1 -- start start 2024-03-30T04:10:14.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 Processor -- start 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800dbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800dc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800dc8d0 con 0x5613800cbef0 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800dca40 con 0x5613800d6df0 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613800dcc00 con 0x5613800cb5d0 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1b073c4b80 10 client.0 _ll_register_callbacks cb 0x56137ffc3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1af0e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36310/0 (socket says 172.21.15.23:36310) 2024-03-30T04:10:14.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1af0e00700 1 -- 172.21.15.23:0/4007095704 learned_addr learned my addr 172.21.15.23:0/4007095704 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.852+0000 7f1af1800700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800dc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1aebe00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800dbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 -- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 msgr2=0x5613800dc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800dc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 -- 172.21.15.23:0/4007095704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 msgr2=0x5613800dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800dbdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 -- 172.21.15.23:0/4007095704 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56137fe4bb60 con 0x5613800cb5d0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1af0e00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0000c00 tx=0x7f1ae0009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0505f91f826ccc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1ad2a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.853+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae003e070 con 0x5613800cb5d0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ae0004390 con 0x5613800cb5d0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f1ae0000e40 con 0x5613800cb5d0 2024-03-30T04:10:14.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ae0023860 con 0x5613800cb5d0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 10 client.27868 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56137fea3cd0 con 0x5613800cb5d0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ae0004390 con 0x5613800cb5d0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1ae0021070 con 0x5613800cb5d0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 10 client.27868 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 10 client.27868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 20 client.27868 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56138013b8b0 0x56138013dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 4 client.27868 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.854+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56137fe6ccf0 con 0x56138013b8b0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1aebe00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56138013b8b0 0x56138013dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1ad2a00700 10 client.27868 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1aebe00700 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56138013b8b0 0x56138013dca0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b5d497d4592e4e9 server_cookie=e6fe7221f9f822e8 in_seq=0 out_seq=0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1ad2a00700 1 -- 172.21.15.23:0/4007095704 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56137fe6ccf0 con 0x56138013b8b0 2024-03-30T04:10:14.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1ad2a00700 10 client.27868 handle_command_reply: tid=0 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1b073c4b80 1 client.27868 shutdown 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 msgr2=0x5613800dd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.855+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0000c00 tx=0x7f1ae0009f40 comp rx=0 tx=0).stop 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 shutdown_connections 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5613800d6df0 0x5613800dc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56138013b8b0 0x56138013dca0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613800cb5d0 0x5613800dd920 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 --2- 172.21.15.23:0/4007095704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5613800cbef0 0x5613800dbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 >> 172.21.15.23:0/4007095704 conn(0x56138002b6a0 msgr2=0x5613800d33b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1ad2a00700 0 client.27868 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 shutdown_connections 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 1 -- 172.21.15.23:0/4007095704 wait complete. 2024-03-30T04:10:14.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1b073c4b80 20 client.27868 trim_cache size 0 max 16384 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "661fdbcd-38aa-4c59-a020-c872ec693ae0": { 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "661fdbcd-38aa-4c59-a020-c872ec693ae0", 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:10:14.861 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 msgr2=0x7f1ad40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 0x7f1ad40a37d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1aec19a870 tx=0x7f1ad8009ed0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 msgr2=0x7f1aec19e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.856+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400b070 tx=0x7f1ae40077f0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 shutdown_connections 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1ad40a1310 0x7f1ad40a37d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1aec19dc00 0x7f1aec19e080 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1aec100f60 0x7f1aec199320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 --2- 172.21.15.23:0/2950772566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1aec106080 0x7f1aec199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 >> 172.21.15.23:0/2950772566 conn(0x7f1aec0fc7c0 msgr2=0x7f1aec0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 shutdown_connections 2024-03-30T04:10:14.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:14.857+0000 7f1af3800700 1 -- 172.21.15.23:0/2950772566 wait complete. 2024-03-30T04:10:14.874 INFO:tasks.cephfs.filesystem:scrub status for tag:661fdbcd-38aa-4c59-a020-c872ec693ae0 - {'path': '/', 'tag': '661fdbcd-38aa-4c59-a020-c872ec693ae0', 'options': 'recursive,force'} 2024-03-30T04:10:15.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 824620 10.53 MB/sec execute 292 sec latency 20.202 ms 2024-03-30T04:10:16.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 826704 10.53 MB/sec execute 293 sec latency 19.309 ms 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: pgmap v257: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:16 smithi154 ceph-mon[79161]: osdmap e95: 12 total, 12 up, 12 in 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: pgmap v257: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:10:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:16 smithi193 ceph-mon[79581]: osdmap e95: 12 total, 12 up, 12 in 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: pgmap v257: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:10:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:16 smithi023 ceph-mon[38254]: osdmap e95: 12 total, 12 up, 12 in 2024-03-30T04:10:17.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 828286 10.53 MB/sec execute 294 sec latency 24.745 ms 2024-03-30T04:10:18.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 830437 10.53 MB/sec execute 295 sec latency 20.840 ms 2024-03-30T04:10:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:18 smithi154 ceph-mon[79161]: pgmap v259: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-30T04:10:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:18 smithi193 ceph-mon[79581]: pgmap v259: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-30T04:10:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:18 smithi023 ceph-mon[38254]: pgmap v259: 129 pgs: 129 active+clean; 462 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 123 op/s 2024-03-30T04:10:19.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 831650 10.51 MB/sec execute 296 sec latency 44.502 ms 2024-03-30T04:10:20.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 833560 10.51 MB/sec execute 297 sec latency 19.014 ms 2024-03-30T04:10:20.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:20 smithi154 ceph-mon[79161]: pgmap v260: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-30T04:10:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:20 smithi193 ceph-mon[79581]: pgmap v260: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-30T04:10:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:20 smithi023 ceph-mon[38254]: pgmap v260: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 222 op/s 2024-03-30T04:10:21.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 835190 10.51 MB/sec execute 298 sec latency 19.920 ms 2024-03-30T04:10:22.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 837142 10.51 MB/sec execute 299 sec latency 37.533 ms 2024-03-30T04:10:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:22 smithi154 ceph-mon[79161]: pgmap v261: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:22 smithi193 ceph-mon[79581]: pgmap v261: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:22 smithi023 ceph-mon[38254]: pgmap v261: 129 pgs: 129 active+clean; 468 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:23.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 838885 10.50 MB/sec execute 300 sec latency 20.362 ms 2024-03-30T04:10:24.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 840807 10.50 MB/sec execute 301 sec latency 34.415 ms 2024-03-30T04:10:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:24 smithi154 ceph-mon[79161]: pgmap v262: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-03-30T04:10:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:24 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:10:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:24 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:10:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:24 smithi193 ceph-mon[79581]: pgmap v262: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-03-30T04:10:24.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:24 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:10:24.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:24 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:10:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:24 smithi023 ceph-mon[38254]: pgmap v262: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.8 MiB/s wr, 203 op/s 2024-03-30T04:10:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:24 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:10:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:24 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:10:25.215 INFO:tasks.workunit.client.0.smithi023.stdout: 1 842181 10.49 MB/sec execute 302 sec latency 38.771 ms 2024-03-30T04:10:26.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 843921 10.48 MB/sec execute 303 sec latency 82.298 ms 2024-03-30T04:10:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:26 smithi154 ceph-mon[79161]: pgmap v263: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:26 smithi193 ceph-mon[79581]: pgmap v263: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:26.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:26 smithi023 ceph-mon[38254]: pgmap v263: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 200 op/s 2024-03-30T04:10:27.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 845742 10.49 MB/sec execute 304 sec latency 23.064 ms 2024-03-30T04:10:27.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:27 smithi154 ceph-mon[79161]: pgmap v264: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-30T04:10:27.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:27 smithi193 ceph-mon[79581]: pgmap v264: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-30T04:10:27.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:27 smithi023 ceph-mon[38254]: pgmap v264: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 198 op/s 2024-03-30T04:10:28.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 847640 10.48 MB/sec execute 305 sec latency 21.789 ms 2024-03-30T04:10:29.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 849281 10.48 MB/sec execute 306 sec latency 35.578 ms 2024-03-30T04:10:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:29 smithi023 ceph-mon[38254]: pgmap v265: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 248 op/s 2024-03-30T04:10:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:29 smithi154 ceph-mon[79161]: pgmap v265: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 248 op/s 2024-03-30T04:10:29.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:29 smithi193 ceph-mon[79581]: pgmap v265: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 248 op/s 2024-03-30T04:10:30.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 850543 10.47 MB/sec execute 307 sec latency 23.216 ms 2024-03-30T04:10:31.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 852585 10.48 MB/sec execute 308 sec latency 19.645 ms 2024-03-30T04:10:32.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 854090 10.46 MB/sec execute 309 sec latency 41.812 ms 2024-03-30T04:10:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:32 smithi154 ceph-mon[79161]: pgmap v266: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-03-30T04:10:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:32 smithi193 ceph-mon[79581]: pgmap v266: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-03-30T04:10:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:32 smithi023 ceph-mon[38254]: pgmap v266: 129 pgs: 129 active+clean; 472 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 163 op/s 2024-03-30T04:10:33.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 856209 10.47 MB/sec execute 310 sec latency 17.742 ms 2024-03-30T04:10:34.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 857632 10.45 MB/sec execute 311 sec latency 83.185 ms 2024-03-30T04:10:34.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:34 smithi154 ceph-mon[79161]: pgmap v267: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 242 op/s 2024-03-30T04:10:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:34 smithi193 ceph-mon[79581]: pgmap v267: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 242 op/s 2024-03-30T04:10:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:34 smithi023 ceph-mon[38254]: pgmap v267: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 242 op/s 2024-03-30T04:10:35.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 859574 10.46 MB/sec execute 312 sec latency 22.377 ms 2024-03-30T04:10:36.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 861097 10.45 MB/sec execute 313 sec latency 20.985 ms 2024-03-30T04:10:36.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:36 smithi154 ceph-mon[79161]: pgmap v268: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:10:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:36 smithi193 ceph-mon[79581]: pgmap v268: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:10:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:36 smithi023 ceph-mon[38254]: pgmap v268: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:10:37.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 863202 10.46 MB/sec execute 314 sec latency 25.366 ms 2024-03-30T04:10:38.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 864885 10.44 MB/sec execute 315 sec latency 21.992 ms 2024-03-30T04:10:38.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:38 smithi154 ceph-mon[79161]: pgmap v269: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 164 op/s 2024-03-30T04:10:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:38 smithi193 ceph-mon[79581]: pgmap v269: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 164 op/s 2024-03-30T04:10:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:38 smithi023 ceph-mon[38254]: pgmap v269: 129 pgs: 129 active+clean; 477 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 164 op/s 2024-03-30T04:10:39.216 INFO:tasks.workunit.client.0.smithi023.stdout: 1 866761 10.45 MB/sec execute 316 sec latency 54.186 ms 2024-03-30T04:10:40.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 868154 10.44 MB/sec execute 317 sec latency 71.305 ms 2024-03-30T04:10:40.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:40 smithi154 ceph-mon[79161]: pgmap v270: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 244 op/s 2024-03-30T04:10:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:40 smithi193 ceph-mon[79581]: pgmap v270: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 244 op/s 2024-03-30T04:10:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:40 smithi023 ceph-mon[38254]: pgmap v270: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 244 op/s 2024-03-30T04:10:41.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 870222 10.45 MB/sec execute 318 sec latency 19.675 ms 2024-03-30T04:10:42.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 871662 10.43 MB/sec execute 319 sec latency 22.569 ms 2024-03-30T04:10:42.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:42 smithi154 ceph-mon[79161]: pgmap v271: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-03-30T04:10:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:42 smithi193 ceph-mon[79581]: pgmap v271: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-03-30T04:10:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:42 smithi023 ceph-mon[38254]: pgmap v271: 129 pgs: 129 active+clean; 481 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-03-30T04:10:43.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 873838 10.44 MB/sec execute 320 sec latency 15.810 ms 2024-03-30T04:10:44.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 875363 10.43 MB/sec execute 321 sec latency 37.295 ms 2024-03-30T04:10:44.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:44 smithi154 ceph-mon[79161]: pgmap v272: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 239 op/s 2024-03-30T04:10:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:44 smithi193 ceph-mon[79581]: pgmap v272: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 239 op/s 2024-03-30T04:10:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:44 smithi023 ceph-mon[38254]: pgmap v272: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 239 op/s 2024-03-30T04:10:44.875 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 -- 172.21.15.23:0/3223214109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 msgr2=0x7f805c069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 --2- 172.21.15.23:0/3223214109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c069640 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8050009920 tx=0x7f8050009c30 comp rx=0 tx=0).stop 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 -- 172.21.15.23:0/3223214109 shutdown_connections 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 --2- 172.21.15.23:0/3223214109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c1060b0 0x7f805c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 --2- 172.21.15.23:0/3223214109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c069640 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 --2- 172.21.15.23:0/3223214109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c069c10 0x7f805c105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 -- 172.21.15.23:0/3223214109 >> 172.21.15.23:0/3223214109 conn(0x7f805c076d80 msgr2=0x7f805c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 -- 172.21.15.23:0/3223214109 shutdown_connections 2024-03-30T04:10:45.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.096+0000 7f8063800700 1 -- 172.21.15.23:0/3223214109 wait complete. 2024-03-30T04:10:45.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.097+0000 7f8063800700 1 Processor -- start 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.097+0000 7f8063800700 1 -- start start 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 0x7f805c199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c19dda0 0x7f805c19e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c19e820 con 0x7f805c069260 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c19e990 con 0x7f805c19dda0 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8063800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805c19eb50 con 0x7f805c1060b0 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f805be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 0x7f805c199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8061800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c19dda0 0x7f805c19e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38346/0 (socket says 172.21.15.23:38346) 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f805be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 0x7f805c199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37426/0 (socket says 172.21.15.23:37426) 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8060e00700 1 -- 172.21.15.23:0/573932368 learned_addr learned my addr 172.21.15.23:0/573932368 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8060e00700 1 -- 172.21.15.23:0/573932368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 msgr2=0x7f805c199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.098+0000 7f8060e00700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 0x7f805c199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8060e00700 1 -- 172.21.15.23:0/573932368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c19dda0 msgr2=0x7f805c19e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8060e00700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c19dda0 0x7f805c19e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8060e00700 1 -- 172.21.15.23:0/573932368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054009d00 con 0x7f805c069260 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8060e00700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8050009230 tx=0x7f805000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8059600700 1 -- 172.21.15.23:0/573932368 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805001c070 con 0x7f805c069260 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8059600700 1 -- 172.21.15.23:0/573932368 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f805000baa0 con 0x7f805c069260 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8050009520 con 0x7f805c069260 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.099+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c1a62e0 con 0x7f805c069260 2024-03-30T04:10:45.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.100+0000 7f8059600700 1 -- 172.21.15.23:0/573932368 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8050004930 con 0x7f805c069260 2024-03-30T04:10:45.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.103+0000 7f8059600700 1 -- 172.21.15.23:0/573932368 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8050020070 con 0x7f805c069260 2024-03-30T04:10:45.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.104+0000 7f8059600700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 0x7f80440a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.104+0000 7f8077230b80 1 Processor -- start 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.104+0000 7f8059600700 1 -- 172.21.15.23:0/573932368 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f80500cd9d0 con 0x7f805c069260 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.104+0000 7f8077230b80 1 -- start start 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f805be00700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 0x7f80440a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 0x55e7adb118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb11df0 0x55e7adb141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ada5f990 con 0x55e7adb065d0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ada5d350 con 0x55e7adb11df0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8077230b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ada58c40 con 0x55e7adb06ef0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f805be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 0x55e7adb118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f805be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 0x55e7adb118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37440/0 (socket says 172.21.15.23:37440) 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38362/0 (socket says 172.21.15.23:38362) 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f805be00700 1 -- 172.21.15.23:0/2599189668 learned_addr learned my addr 172.21.15.23:0/2599189668 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.105+0000 7f8061800700 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb11df0 0x55e7adb141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 -- 172.21.15.23:0/2599189668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 msgr2=0x55e7adb118b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 0x55e7adb118b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 -- 172.21.15.23:0/2599189668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb11df0 msgr2=0x55e7adb141e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb11df0 0x55e7adb141e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 -- 172.21.15.23:0/2599189668 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7ad8da3b0 con 0x55e7adb065d0 2024-03-30T04:10:45.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f805be00700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 0x7f80440a37d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8048011630 tx=0x7f80480119f0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f8060e00700 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f80500cace0 tx=0x7f8050009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca27242ff6d865b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f804e000700 1 -- 172.21.15.23:0/2599189668 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805009ae30 con 0x55e7adb065d0 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f804e000700 1 -- 172.21.15.23:0/2599189668 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f805009a730 con 0x55e7adb065d0 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.106+0000 7f804e000700 1 -- 172.21.15.23:0/2599189668 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805009e130 con 0x55e7adb065d0 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 -- 172.21.15.23:0/2599189668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 msgr2=0x55e7adb069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f80500cace0 tx=0x7f8050009c30 comp rx=0 tx=0).stop 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 -- 172.21.15.23:0/2599189668 shutdown_connections 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb11df0 0x55e7adb141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb069b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 --2- 172.21.15.23:0/2599189668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb06ef0 0x55e7adb118b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 -- 172.21.15.23:0/2599189668 >> 172.21.15.23:0/2599189668 conn(0x55e7ada666a0 msgr2=0x55e7adb02a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 -- 172.21.15.23:0/2599189668 shutdown_connections 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.107+0000 7f8077230b80 1 -- 172.21.15.23:0/2599189668 wait complete. 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.108+0000 7f8077230b80 1 -- start start 2024-03-30T04:10:45.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.108+0000 7f8077230b80 1 Processor -- start 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 0x55e7adb117d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 0x55e7adb0eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb0f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ad9f4d80 con 0x55e7adb06ef0 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ad9f4a20 con 0x55e7adb065d0 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7ad8dc6a0 con 0x55e7adb11df0 2024-03-30T04:10:45.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8060e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 0x55e7adb117d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8077230b80 10 client.0 _ll_register_callbacks cb 0x55e7ad9fe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8060e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 0x55e7adb117d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37524/0 (socket says 172.21.15.23:37524) 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8060e00700 1 -- 172.21.15.23:0/2579260178 learned_addr learned my addr 172.21.15.23:0/2579260178 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f8061800700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb0f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.109+0000 7f805be00700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 0x55e7adb0eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 -- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 msgr2=0x55e7adb117d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 0x55e7adb117d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 -- 172.21.15.23:0/2579260178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 msgr2=0x55e7adb0eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 0x55e7adb0eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 -- 172.21.15.23:0/2579260178 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f805c1a62e0 con 0x55e7adb11df0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f805be00700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 0x55e7adb0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8061800700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb0f090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8054009cd0 tx=0x7f8054007840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70e30528cd6e7e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8042a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:45.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805400e070 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7ada19380 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.110+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8054004db0 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8054004040 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 10 client.27880 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e7ad886b60 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e7ada19380 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8054005550 con 0x55e7adb11df0 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 10 client.27880 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 10 client.27880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 20 client.27880 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb20d40 0x55e7adb23160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 4 client.27880 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7ada51700 con 0x55e7adb20d40 2024-03-30T04:10:45.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.111+0000 7f8060e00700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb20d40 0x55e7adb23160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.112+0000 7f8042a00700 10 client.27880 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:45.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.112+0000 7f8060e00700 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb20d40 0x55e7adb23160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54eacead33a3e45b server_cookie=9b281aca5f4fc8bb in_seq=0 out_seq=0 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.112+0000 7f8042a00700 1 -- 172.21.15.23:0/2579260178 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e7ada51700 con 0x55e7adb20d40 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.112+0000 7f8042a00700 10 client.27880 handle_command_reply: tid=0 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.112+0000 7f8077230b80 1 client.27880 shutdown 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 msgr2=0x55e7adb0f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb0f090 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8054009cd0 tx=0x7f8054007840 comp rx=0 tx=0).stop 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 shutdown_connections 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb20d40 0x55e7adb23160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb0f090 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb065d0 0x55e7adb117d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8042a00700 0 client.27880 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:45.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 --2- 172.21.15.23:0/2579260178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb06ef0 0x55e7adb0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 >> 172.21.15.23:0/2579260178 conn(0x55e7ada666a0 msgr2=0x55e7adb086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 shutdown_connections 2024-03-30T04:10:45.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 1 -- 172.21.15.23:0/2579260178 wait complete. 2024-03-30T04:10:45.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.113+0000 7f8077230b80 20 client.27880 trim_cache size 0 max 16384 2024-03-30T04:10:45.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.135+0000 7f8077230b80 1 Processor -- start 2024-03-30T04:10:45.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 -- start start 2024-03-30T04:10:45.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb23130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb11300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb141e0 con 0x55e7adb065d0 2024-03-30T04:10:45.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb14350 con 0x55e7adb06ef0 2024-03-30T04:10:45.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8077230b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb11840 con 0x55e7adb11df0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f805be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f805be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37532/0 (socket says 172.21.15.23:37532) 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f805be00700 1 -- 172.21.15.23:0/3292675480 learned_addr learned my addr 172.21.15.23:0/3292675480 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8060e00700 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb23130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.136+0000 7f8061800700 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb11300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 -- 172.21.15.23:0/3292675480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 msgr2=0x55e7adb11300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb11300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 -- 172.21.15.23:0/3292675480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 msgr2=0x55e7adb23130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb23130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 -- 172.21.15.23:0/3292675480 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7ada1eea0 con 0x55e7adb06ef0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f805be00700 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8048004340 tx=0x7f804801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb139f833e4f85a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f804ea00700 1 -- 172.21.15.23:0/3292675480 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048003d10 con 0x55e7adb06ef0 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f804ea00700 1 -- 172.21.15.23:0/3292675480 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8048003e70 con 0x55e7adb06ef0 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.137+0000 7f804ea00700 1 -- 172.21.15.23:0/3292675480 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048021690 con 0x55e7adb06ef0 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 -- 172.21.15.23:0/3292675480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 msgr2=0x55e7adb0f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8048004340 tx=0x7f804801f000 comp rx=0 tx=0).stop 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 -- 172.21.15.23:0/3292675480 shutdown_connections 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb11300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb23130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 --2- 172.21.15.23:0/3292675480 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb0f050 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 -- 172.21.15.23:0/3292675480 >> 172.21.15.23:0/3292675480 conn(0x55e7ada666a0 msgr2=0x55e7adb2b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 -- 172.21.15.23:0/3292675480 shutdown_connections 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.138+0000 7f8077230b80 1 -- 172.21.15.23:0/3292675480 wait complete. 2024-03-30T04:10:45.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.139+0000 7f8077230b80 1 -- start start 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.139+0000 7f8077230b80 1 Processor -- start 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb167c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb17410 con 0x55e7adb065d0 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb17580 con 0x55e7adb06ef0 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7adb75680 con 0x55e7adb11df0 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8077230b80 10 client.0 _ll_register_callbacks cb 0x55e7ad9fe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38390/0 (socket says 172.21.15.23:38390) 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 -- 172.21.15.23:0/2943365095 learned_addr learned my addr 172.21.15.23:0/2943365095 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f805be00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb167c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8061800700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 -- 172.21.15.23:0/2943365095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 msgr2=0x55e7adb16d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb16d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 -- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 msgr2=0x55e7adb167c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb167c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8060e00700 1 -- 172.21.15.23:0/2943365095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8048004030 con 0x55e7adb065d0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.140+0000 7f8061800700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb16d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8060e00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f80500058b0 tx=0x7f8050005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74abe86831a067a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805009e5e0 con 0x55e7adb065d0 2024-03-30T04:10:45.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f80500a2b20 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f805009a730 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8050098ad0 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 10 client.17895 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e7ad8decd0 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8050080020 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f805009a990 con 0x55e7adb065d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 10 client.17895 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 10 client.17895 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.141+0000 7f8077230b80 20 client.17895 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb769d0 0x55e7adb78df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f8077230b80 4 client.17895 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e7ad8a7cf0 con 0x55e7adb769d0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f805be00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb769d0 0x55e7adb78df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f805be00700 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb769d0 0x55e7adb78df0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c98a1135e8cefb7 server_cookie=157fd994234bf87c in_seq=0 out_seq=0 2024-03-30T04:10:45.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.142+0000 7f8042a00700 10 client.17895 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8042a00700 1 -- 172.21.15.23:0/2943365095 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e7ad8a7cf0 con 0x55e7adb769d0 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8042a00700 10 client.17895 handle_command_reply: tid=0 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 client.17895 shutdown 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 msgr2=0x55e7adb189d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f80500058b0 tx=0x7f8050005e30 comp rx=0 tx=0).stop 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 shutdown_connections 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e7adb769d0 0x55e7adb78df0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e7adb11df0 0x55e7adb16d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e7adb065d0 0x55e7adb189d0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8042a00700 0 client.17895 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 --2- 172.21.15.23:0/2943365095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e7adb06ef0 0x55e7adb167c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 >> 172.21.15.23:0/2943365095 conn(0x55e7ada666a0 msgr2=0x55e7adb0d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 shutdown_connections 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 1 -- 172.21.15.23:0/2943365095 wait complete. 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.143+0000 7f8077230b80 20 client.17895 trim_cache size 0 max 16384 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:10:45.147 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 msgr2=0x7f80440a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 0x7f80440a37d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8048011630 tx=0x7f80480119f0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 msgr2=0x7f805c1994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f8050009230 tx=0x7f805000fa00 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 shutdown_connections 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f80440a1310 0x7f80440a37d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f805c19dda0 0x7f805c19e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f805c069260 0x7f805c1994c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 --2- 172.21.15.23:0/573932368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f805c1060b0 0x7f805c199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 >> 172.21.15.23:0/573932368 conn(0x7f805c076d80 msgr2=0x7f805c10c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 shutdown_connections 2024-03-30T04:10:45.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.144+0000 7f8063800700 1 -- 172.21.15.23:0/573932368 wait complete. 2024-03-30T04:10:45.163 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:10:45.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:10:45.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 877268 10.44 MB/sec execute 322 sec latency 43.970 ms 2024-03-30T04:10:45.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.415+0000 7f7b2d800700 1 -- 172.21.15.23:0/669384778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 msgr2=0x7f7b28104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.415+0000 7f7b2d800700 1 --2- 172.21.15.23:0/669384778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b28104980 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009990 tx=0x7f7b14009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 -- 172.21.15.23:0/669384778 shutdown_connections 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 --2- 172.21.15.23:0/669384778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b28105450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 --2- 172.21.15.23:0/669384778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b280fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 --2- 172.21.15.23:0/669384778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b28104980 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 -- 172.21.15.23:0/669384778 >> 172.21.15.23:0/669384778 conn(0x7f7b280fb5f0 msgr2=0x7f7b280fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 -- 172.21.15.23:0/669384778 shutdown_connections 2024-03-30T04:10:45.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.416+0000 7f7b2d800700 1 -- 172.21.15.23:0/669384778 wait complete. 2024-03-30T04:10:45.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.418+0000 7f7b2d800700 1 Processor -- start 2024-03-30T04:10:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.418+0000 7f7b2d800700 1 -- start start 2024-03-30T04:10:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b2819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b281a1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b281a25a0 con 0x7f7b28104fa0 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b281a2710 con 0x7f7b281004c0 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b2d800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b281a28d0 con 0x7f7b280ffba0 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b26a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b26a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37500/0 (socket says 172.21.15.23:37500) 2024-03-30T04:10:45.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.419+0000 7f7b26a00700 1 -- 172.21.15.23:0/734289400 learned_addr learned my addr 172.21.15.23:0/734289400 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:45.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26000700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b2819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b27400700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b281a1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26a00700 1 -- 172.21.15.23:0/734289400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 msgr2=0x7f7b2819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26a00700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b2819dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26a00700 1 -- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 msgr2=0x7f7b281a1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26a00700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b281a1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26a00700 1 -- 172.21.15.23:0/734289400 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b140095f0 con 0x7f7b280ffba0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b27400700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b281a1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.420+0000 7f7b26000700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b2819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b26a00700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c9b0 tx=0x7f7b1800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18007b00 con 0x7f7b280ffba0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7b18007c60 con 0x7f7b280ffba0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b18018660 con 0x7f7b280ffba0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b281a2bb0 con 0x7f7b280ffba0 2024-03-30T04:10:45.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.421+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b281aa3f0 con 0x7f7b280ffba0 2024-03-30T04:10:45.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.423+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b280686d0 con 0x7f7b280ffba0 2024-03-30T04:10:45.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.426+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f7b1800f450 con 0x7f7b280ffba0 2024-03-30T04:10:45.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.429+0000 7f7b1f400700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 0x7f7b0c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:45.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.429+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7b180cd010 con 0x7f7b280ffba0 2024-03-30T04:10:45.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.430+0000 7f7b26000700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 0x7f7b0c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:45.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.431+0000 7f7b26000700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 0x7f7b0c0a37d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b2819ebd0 tx=0x7f7b1401a040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:45.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.432+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7b18096080 con 0x7f7b280ffba0 2024-03-30T04:10:45.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.776+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b28067a10 con 0x7f7b280ffba0 2024-03-30T04:10:45.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.777+0000 7f7b1f400700 1 -- 172.21.15.23:0/734289400 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f7b180996a0 con 0x7f7b280ffba0 2024-03-30T04:10:45.777 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:10:45.777 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:10:45.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 msgr2=0x7f7b0c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 0x7f7b0c0a37d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7b2819ebd0 tx=0x7f7b1401a040 comp rx=0 tx=0).stop 2024-03-30T04:10:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 msgr2=0x7f7b2819d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:45.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800c9b0 tx=0x7f7b1800ccc0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 shutdown_connections 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7b28104fa0 0x7f7b281a1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b280ffba0 0x7f7b2819d680 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7b0c0a1310 0x7f7b0c0a37d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 --2- 172.21.15.23:0/734289400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b281004c0 0x7f7b2819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 >> 172.21.15.23:0/734289400 conn(0x7f7b280fb5f0 msgr2=0x7f7b280fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:45.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 shutdown_connections 2024-03-30T04:10:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:45.780+0000 7f7b2d800700 1 -- 172.21.15.23:0/734289400 wait complete. 2024-03-30T04:10:45.783 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:10:45.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:10:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 -- 172.21.15.23:0/1628233986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073a70 msgr2=0x7f7cf01117d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 --2- 172.21.15.23:0/1628233986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073a70 0x7f7cf01117d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009990 tx=0x7f7cdc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 -- 172.21.15.23:0/1628233986 shutdown_connections 2024-03-30T04:10:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 --2- 172.21.15.23:0/1628233986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 0x7f7cf0114100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 --2- 172.21.15.23:0/1628233986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073150 0x7f7cf0073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 --2- 172.21.15.23:0/1628233986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073a70 0x7f7cf01117d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.009+0000 7f7cf6200700 1 -- 172.21.15.23:0/1628233986 >> 172.21.15.23:0/1628233986 conn(0x7f7cf00fc7f0 msgr2=0x7f7cf00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.010+0000 7f7cf6200700 1 -- 172.21.15.23:0/1628233986 shutdown_connections 2024-03-30T04:10:46.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.010+0000 7f7cf6200700 1 -- 172.21.15.23:0/1628233986 wait complete. 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 Processor -- start 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 -- start start 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073a70 0x7f7cf01a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 0x7f7cf019d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf019dcc0 con 0x7f7cf0111d10 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf019de30 con 0x7f7cf0073a70 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cf6200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf019dff0 con 0x7f7cf0073150 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cef400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7ceea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073a70 0x7f7cf01a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cefe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 0x7f7cf019d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cef400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37516/0 (socket says 172.21.15.23:37516) 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.011+0000 7f7cef400700 1 -- 172.21.15.23:0/2815771897 learned_addr learned my addr 172.21.15.23:0/2815771897 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 -- 172.21.15.23:0/2815771897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073a70 msgr2=0x7f7cf01a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073a70 0x7f7cf01a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 -- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 msgr2=0x7f7cf019d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 0x7f7cf019d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 -- 172.21.15.23:0/2815771897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0095f0 con 0x7f7cf0073150 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cef400700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9a0 tx=0x7f7ce000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7ce7e00700 1 -- 172.21.15.23:0/2815771897 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0009790 con 0x7f7cf0073150 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7ce7e00700 1 -- 172.21.15.23:0/2815771897 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce0010430 con 0x7f7cf0073150 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf019e2d0 con 0x7f7cf0073150 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.012+0000 7f7ce7e00700 1 -- 172.21.15.23:0/2815771897 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0018680 con 0x7f7cf0073150 2024-03-30T04:10:46.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.013+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf01aa520 con 0x7f7cf0073150 2024-03-30T04:10:46.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.015+0000 7f7ce7e00700 1 -- 172.21.15.23:0/2815771897 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f7ce00105a0 con 0x7f7cf0073150 2024-03-30T04:10:46.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 Processor -- start 2024-03-30T04:10:46.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7ce7e00700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 0x7f7cd40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 -- start start 2024-03-30T04:10:46.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7ce7e00700 1 -- 172.21.15.23:0/2815771897 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7ce0014070 con 0x7f7cf0073150 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7cefe00700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 0x7f7cd40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3e2d990 con 0x55c2e3edfdf0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3e2b350 con 0x55c2e3ed45d0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.017+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3e26c40 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37584/0 (socket says 172.21.15.23:37584) 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 -- 172.21.15.23:0/2041139870 learned_addr learned my addr 172.21.15.23:0/2041139870 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7ceea00700 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 -- 172.21.15.23:0/2041139870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 msgr2=0x55c2e3edf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 -- 172.21.15.23:0/2041139870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 msgr2=0x55c2e3ee21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 -- 172.21.15.23:0/2041139870 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e3ca83b0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cefe00700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 0x7f7cd40a3750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7cf006a3d0 tx=0x7f7ce8009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cef400700 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0096a20 tx=0x7f7ce0099c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c62d1d0c614274 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.018+0000 7f7cdbe00700 1 -- 172.21.15.23:0/2041139870 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce009dde0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7cdbe00700 1 -- 172.21.15.23:0/2041139870 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce00a2430 con 0x55c2e3ed45d0 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7cdbe00700 1 -- 172.21.15.23:0/2041139870 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce0099700 con 0x55c2e3ed45d0 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/2041139870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 msgr2=0x55c2e3ed49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0096a20 tx=0x7f7ce0099c20 comp rx=0 tx=0).stop 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/2041139870 shutdown_connections 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ed49b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/2041139870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/2041139870 >> 172.21.15.23:0/2041139870 conn(0x55c2e3e346a0 msgr2=0x55c2e3ed0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/2041139870 shutdown_connections 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.019+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/2041139870 wait complete. 2024-03-30T04:10:46.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.020+0000 7f7d09ce8b80 1 -- start start 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.020+0000 7f7d09ce8b80 1 Processor -- start 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3dc2d80 con 0x55c2e3edfdf0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3dc2a20 con 0x55c2e3ed45d0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3caa6a0 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7d09ce8b80 10 client.0 _ll_register_callbacks cb 0x55c2e3dcc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cef400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7ceea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cef400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edf7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37594/0 (socket says 172.21.15.23:37594) 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38430/0 (socket says 172.21.15.23:38430) 2024-03-30T04:10:46.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cef400700 1 -- 172.21.15.23:0/1299802265 learned_addr learned my addr 172.21.15.23:0/1299802265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 -- 172.21.15.23:0/1299802265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 msgr2=0x55c2e3edca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 -- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 msgr2=0x55c2e3edf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.021+0000 7f7cefe00700 1 -- 172.21.15.23:0/1299802265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf01aa520 con 0x55c2e3edfdf0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cefe00700 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009fd0 tx=0x7f7ce8004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74e7ffa0831abaa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cd8c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800e040 con 0x55c2e3edfdf0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c2e3de7380 con 0x55c2e3edfdf0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7ce8003c00 con 0x55c2e3edfdf0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce8003dc0 con 0x55c2e3edfdf0 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7d09ce8b80 10 client.17907 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c2e3c54b60 con 0x55c2e3edfdf0 2024-03-30T04:10:46.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.022+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c2e3de7380 con 0x55c2e3edfdf0 2024-03-30T04:10:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ce8004510 con 0x55c2e3edfdf0 2024-03-30T04:10:46.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 10 client.17907 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 10 client.17907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 20 client.17907 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3eef9e0 0x55c2e3ef1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 4 client.17907 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2e3e1f700 con 0x55c2e3eef9e0 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7cef400700 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3eef9e0 0x55c2e3ef1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7cef400700 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3eef9e0 0x55c2e3ef1e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fde31d839bc4eab8 server_cookie=6fd352aff667b1d3 in_seq=0 out_seq=0 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.023+0000 7f7cd8c00700 10 client.17907 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7cd8c00700 1 -- 172.21.15.23:0/1299802265 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c2e3e1f700 con 0x55c2e3eef9e0 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7cd8c00700 10 client.17907 handle_command_reply: tid=0 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 client.17907 shutdown 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 msgr2=0x55c2e3edcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009fd0 tx=0x7f7ce8004b40 comp rx=0 tx=0).stop 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 shutdown_connections 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edcfe0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3eef9e0 0x55c2e3ef1e00 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/1299802265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3edca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 >> 172.21.15.23:0/1299802265 conn(0x55c2e3e346a0 msgr2=0x55c2e3ed6510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.024+0000 7f7cd8c00700 0 client.17907 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.025+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 shutdown_connections 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.025+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/1299802265 wait complete. 2024-03-30T04:10:46.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.025+0000 7f7d09ce8b80 20 client.17907 trim_cache size 0 max 16384 2024-03-30T04:10:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.039+0000 7f7d09ce8b80 1 Processor -- start 2024-03-30T04:10:46.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.039+0000 7f7d09ce8b80 1 -- start start 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3ed2160 con 0x55c2e3ed45d0 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3ee6f80 con 0x55c2e3edfdf0 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3ed02d0 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37552/0 (socket says 172.21.15.23:37552) 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 -- 172.21.15.23:0/3605852126 learned_addr learned my addr 172.21.15.23:0/3605852126 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7cefe00700 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 -- 172.21.15.23:0/3605852126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3edfdf0 msgr2=0x55c2e3edb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 -- 172.21.15.23:0/3605852126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 msgr2=0x55c2e3edcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.040+0000 7f7ceea00700 1 -- 172.21.15.23:0/3605852126 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2e3decea0 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7ceea00700 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009990 tx=0x7f7cdc00b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afd04ed96a92d8e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7ce5200700 1 -- 172.21.15.23:0/3605852126 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc01d030 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7ce5200700 1 -- 172.21.15.23:0/3605852126 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cdc01e040 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7ce5200700 1 -- 172.21.15.23:0/3605852126 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc004760 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/3605852126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 msgr2=0x55c2e3ee6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009990 tx=0x7f7cdc00b260 comp rx=0 tx=0).stop 2024-03-30T04:10:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/3605852126 shutdown_connections 2024-03-30T04:10:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3edfdf0 0x55c2e3edb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3edcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/3605852126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee6a40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.041+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/3605852126 >> 172.21.15.23:0/3605852126 conn(0x55c2e3e346a0 msgr2=0x55c2e3ed0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.042+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/3605852126 shutdown_connections 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.042+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/3605852126 wait complete. 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.042+0000 7f7d09ce8b80 1 -- start start 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.042+0000 7f7d09ce8b80 1 Processor -- start 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3ee5e90 con 0x55c2e3ed45d0 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3f43820 con 0x55c2e3ed4ef0 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2e3f43960 con 0x55c2e3edfdf0 2024-03-30T04:10:46.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7d09ce8b80 10 client.0 _ll_register_callbacks cb 0x55c2e3dcc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7ceea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38450/0 (socket says 172.21.15.23:38450) 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 -- 172.21.15.23:0/305408697 learned_addr learned my addr 172.21.15.23:0/305408697 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cefe00700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 -- 172.21.15.23:0/305408697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3edfdf0 msgr2=0x55c2e3ee5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 -- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 msgr2=0x55c2e3ee5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 -- 172.21.15.23:0/305408697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf0114ab0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7ceea00700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cef400700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0099cb0 tx=0x7f7ce000e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d511f0a0c5555c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.043+0000 7f7cd8c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:10:46.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00996a0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7ce009a210 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ce00a2430 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce009a3d0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 10 client.17913 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c2e3caccd0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce00a2430 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ce0094bc0 con 0x55c2e3ed45d0 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 10 client.17913 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 10 client.17913 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 20 client.17913 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3f44eb0 0x55c2e3f472d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 4 client.17913 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:46.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.044+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c2e3c75cf0 con 0x55c2e3f44eb0 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7ceea00700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3f44eb0 0x55c2e3f472d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7ceea00700 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3f44eb0 0x55c2e3f472d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=eff216a23d933916 server_cookie=932bcb46f3b07a07 in_seq=0 out_seq=0 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7cd8c00700 10 client.17913 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7cd8c00700 1 -- 172.21.15.23:0/305408697 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c2e3c75cf0 con 0x55c2e3f44eb0 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7cd8c00700 10 client.17913 handle_command_reply: tid=0 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7d09ce8b80 1 client.17913 shutdown 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.045+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 msgr2=0x55c2e3ee6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0099cb0 tx=0x7f7ce000e8f0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 shutdown_connections 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55c2e3f44eb0 0x55c2e3f472d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c2e3edfdf0 0x55c2e3ee5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c2e3ed45d0 0x55c2e3ee6a40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 --2- 172.21.15.23:0/305408697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c2e3ed4ef0 0x55c2e3ee5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 >> 172.21.15.23:0/305408697 conn(0x55c2e3e346a0 msgr2=0x55c2e3edb260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7cd8c00700 0 client.17913 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:10:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 shutdown_connections 2024-03-30T04:10:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 1 -- 172.21.15.23:0/305408697 wait complete. 2024-03-30T04:10:46.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7d09ce8b80 20 client.17913 trim_cache size 0 max 16384 2024-03-30T04:10:46.050 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 msgr2=0x7f7cd40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 0x7f7cd40a3750 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7cf006a3d0 tx=0x7f7ce8009920 comp rx=0 tx=0).stop 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 msgr2=0x7f7cf01a3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.046+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7ce000e9a0 tx=0x7f7ce000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 shutdown_connections 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7cf0111d10 0x7f7cf019d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7cf0073150 0x7f7cf01a3530 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7cd40a1290 0x7f7cd40a3750 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 --2- 172.21.15.23:0/2815771897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf0073a70 0x7f7cf01a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 >> 172.21.15.23:0/2815771897 conn(0x7f7cf00fc7f0 msgr2=0x7f7cf00fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 shutdown_connections 2024-03-30T04:10:46.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.047+0000 7f7cf6200700 1 -- 172.21.15.23:0/2815771897 wait complete. 2024-03-30T04:10:46.065 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:10:46.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 878619 10.42 MB/sec execute 323 sec latency 20.553 ms 2024-03-30T04:10:46.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.275+0000 7f4336400700 1 -- 172.21.15.23:0/1143845518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 msgr2=0x7f433010d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.275+0000 7f4336400700 1 --2- 172.21.15.23:0/1143845518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f433010d060 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f431c009990 tx=0x7f431c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 -- 172.21.15.23:0/1143845518 shutdown_connections 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 --2- 172.21.15.23:0/1143845518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 0x7f433010f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 --2- 172.21.15.23:0/1143845518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330073540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 --2- 172.21.15.23:0/1143845518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f433010d060 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 -- 172.21.15.23:0/1143845518 >> 172.21.15.23:0/1143845518 conn(0x7f43300fc7f0 msgr2=0x7f43300fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 -- 172.21.15.23:0/1143845518 shutdown_connections 2024-03-30T04:10:46.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.276+0000 7f4336400700 1 -- 172.21.15.23:0/1143845518 wait complete. 2024-03-30T04:10:46.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 Processor -- start 2024-03-30T04:10:46.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 -- start start 2024-03-30T04:10:46.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f4330199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 0x7f433019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019e430 con 0x7f433010d5a0 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019e5a0 con 0x7f4330073160 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.277+0000 7f4336400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f433019e760 con 0x7f4330073a80 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37648/0 (socket says 172.21.15.23:37648) 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 -- 172.21.15.23:0/3405087036 learned_addr learned my addr 172.21.15.23:0/3405087036 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432fe00700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 0x7f433019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432ea00700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f4330199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 -- 172.21.15.23:0/3405087036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 msgr2=0x7f4330199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f4330199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 -- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 msgr2=0x7f433019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 0x7f433019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 -- 172.21.15.23:0/3405087036 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4328009c40 con 0x7f4330073160 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.278+0000 7f432f400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f432000e9a0 tx=0x7f432000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.279+0000 7f4327e00700 1 -- 172.21.15.23:0/3405087036 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320009790 con 0x7f4330073160 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.279+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c0095f0 con 0x7f4330073160 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.279+0000 7f4327e00700 1 -- 172.21.15.23:0/3405087036 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f432000ce80 con 0x7f4330073160 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.279+0000 7f4327e00700 1 -- 172.21.15.23:0/3405087036 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320018680 con 0x7f4330073160 2024-03-30T04:10:46.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.279+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433019ec30 con 0x7f4330073160 2024-03-30T04:10:46.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.284+0000 7f4327e00700 1 -- 172.21.15.23:0/3405087036 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f4320010430 con 0x7f4330073160 2024-03-30T04:10:46.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.285+0000 7f4349f91b80 1 Processor -- start 2024-03-30T04:10:46.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.285+0000 7f4327e00700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 0x7f43140a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.285+0000 7f4327e00700 1 -- 172.21.15.23:0/3405087036 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4320014070 con 0x7f4330073160 2024-03-30T04:10:46.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.285+0000 7f4349f91b80 1 -- start start 2024-03-30T04:10:46.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.285+0000 7f432fe00700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 0x7f43140a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a0845d0 0x564a0a0849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a0921e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09fdd990 con 0x564a0a0845d0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09fdb350 con 0x564a0a084ef0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f4349f91b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09fd6c40 con 0x564a0a08fdf0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a0845d0 0x564a0a0849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37654/0 (socket says 172.21.15.23:37654) 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 -- 172.21.15.23:0/283428235 learned_addr learned my addr 172.21.15.23:0/283428235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432fe00700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 0x7f43140a37d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4328009fd0 tx=0x7f432800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432fe00700 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a0921e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 -- 172.21.15.23:0/283428235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 msgr2=0x564a0a0921e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a0921e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 -- 172.21.15.23:0/283428235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a0845d0 msgr2=0x564a0a0849b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a0845d0 0x564a0a0849b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.286+0000 7f432ea00700 1 -- 172.21.15.23:0/283428235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a09e583b0 con 0x564a0a084ef0 2024-03-30T04:10:46.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.287+0000 7f432ea00700 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f431c015000 tx=0x7f431c005ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3d7d8515018afaa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.287+0000 7f4324c00700 1 -- 172.21.15.23:0/283428235 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c01d070 con 0x564a0a084ef0 2024-03-30T04:10:46.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.287+0000 7f4324c00700 1 -- 172.21.15.23:0/283428235 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f431c01e040 con 0x564a0a084ef0 2024-03-30T04:10:46.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.287+0000 7f4324c00700 1 -- 172.21.15.23:0/283428235 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c004920 con 0x564a0a084ef0 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- 172.21.15.23:0/283428235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 msgr2=0x564a0a08f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f431c015000 tx=0x7f431c005ec0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- 172.21.15.23:0/283428235 shutdown_connections 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a0921e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a0845d0 0x564a0a0849b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 --2- 172.21.15.23:0/283428235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a084ef0 0x564a0a08f8b0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- 172.21.15.23:0/283428235 >> 172.21.15.23:0/283428235 conn(0x564a09fe46a0 msgr2=0x564a0a080a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- 172.21.15.23:0/283428235 shutdown_connections 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- 172.21.15.23:0/283428235 wait complete. 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.288+0000 7f4349f91b80 1 -- start start 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 Processor -- start 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a084ef0 0x564a0a08ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09f72d80 con 0x564a0a084ef0 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09f72a20 con 0x564a0a0845d0 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a09e5a6a0 con 0x564a0a08fdf0 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f4349f91b80 10 client.0 _ll_register_callbacks cb 0x564a09f7c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.289+0000 7f432ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a084ef0 0x564a0a08ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37666/0 (socket says 172.21.15.23:37666) 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 -- 172.21.15.23:0/2727929954 learned_addr learned my addr 172.21.15.23:0/2727929954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432fe00700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 -- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 msgr2=0x564a0a08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 -- 172.21.15.23:0/2727929954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a084ef0 msgr2=0x564a0a08ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a084ef0 0x564a0a08ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 -- 172.21.15.23:0/2727929954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4330110650 con 0x564a0a0845d0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f432f400700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43200cab50 tx=0x7f4320099b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6039974d494b73f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f4319600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320099600 con 0x564a0a0845d0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f432009dd40 con 0x564a0a0845d0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.290+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a09f97380 con 0x564a0a0845d0 2024-03-30T04:10:46.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43200a2390 con 0x564a0a0845d0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 10 client.27926 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564a09e04b60 con 0x564a0a0845d0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564a09f97380 con 0x564a0a0845d0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4320095e70 con 0x564a0a0845d0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 10 client.27926 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 10 client.27926 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 20 client.27926 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a09f8e0 0x564a0a0a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 4 client.27926 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a09fcf700 con 0x564a0a09f8e0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.291+0000 7f432ea00700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a09f8e0 0x564a0a0a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.292+0000 7f432ea00700 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a09f8e0 0x564a0a0a1d00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=50b761e22c1a4ab8 server_cookie=c082b445880b028c in_seq=0 out_seq=0 2024-03-30T04:10:46.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.292+0000 7f4319600700 10 client.27926 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4319600700 1 -- 172.21.15.23:0/2727929954 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564a09fcf700 con 0x564a0a09f8e0 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4319600700 10 client.27926 handle_command_reply: tid=0 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 client.27926 shutdown 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 msgr2=0x564a0a08f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f43200cab50 tx=0x7f4320099b60 comp rx=0 tx=0).stop 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 shutdown_connections 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a09f8e0 0x564a0a0a1d00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a08fdf0 0x564a0a08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a08f770 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4319600700 0 client.27926 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2727929954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a084ef0 0x564a0a08ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 >> 172.21.15.23:0/2727929954 conn(0x564a09fe46a0 msgr2=0x564a0a086530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 shutdown_connections 2024-03-30T04:10:46.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 1 -- 172.21.15.23:0/2727929954 wait complete. 2024-03-30T04:10:46.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.293+0000 7f4349f91b80 20 client.27926 trim_cache size 0 max 16384 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 Processor -- start 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 -- start start 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a0a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a096a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a08b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a08d600 con 0x564a0a08fdf0 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a08e060 con 0x564a0a0845d0 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f4349f91b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a0817f0 con 0x564a0a084ef0 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a08b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a0a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a096a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a08b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38490/0 (socket says 172.21.15.23:38490) 2024-03-30T04:10:46.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a0a1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37676/0 (socket says 172.21.15.23:37676) 2024-03-30T04:10:46.309 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.308+0000 7f432fe00700 1 -- 172.21.15.23:0/3282246047 learned_addr learned my addr 172.21.15.23:0/3282246047 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 -- 172.21.15.23:0/3282246047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 msgr2=0x564a0a0a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a0a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 -- 172.21.15.23:0/3282246047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 msgr2=0x564a0a08b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a08b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 -- 172.21.15.23:0/3282246047 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a09f9cea0 con 0x564a0a084ef0 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f432ea00700 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a096a70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f431c005f50 tx=0x7f431c00b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a5121899fa0c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f4325600700 1 -- 172.21.15.23:0/3282246047 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c01d040 con 0x564a0a084ef0 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f4325600700 1 -- 172.21.15.23:0/3282246047 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f431c005320 con 0x564a0a084ef0 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.309+0000 7f4325600700 1 -- 172.21.15.23:0/3282246047 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431c00bb70 con 0x564a0a084ef0 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- 172.21.15.23:0/3282246047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 msgr2=0x564a0a096a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a096a70 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f431c005f50 tx=0x7f431c00b260 comp rx=0 tx=0).stop 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- 172.21.15.23:0/3282246047 shutdown_connections 2024-03-30T04:10:46.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a08b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a0a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 --2- 172.21.15.23:0/3282246047 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a096a70 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- 172.21.15.23:0/3282246047 >> 172.21.15.23:0/3282246047 conn(0x564a09fe46a0 msgr2=0x564a0a0795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- 172.21.15.23:0/3282246047 shutdown_connections 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- 172.21.15.23:0/3282246047 wait complete. 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.310+0000 7f4349f91b80 1 -- start start 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 Processor -- start 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a095180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a0f2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a0f2c50 con 0x564a0a08fdf0 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a0f2d90 con 0x564a0a0845d0 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a0a0f2f50 con 0x564a0a084ef0 2024-03-30T04:10:46.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f4349f91b80 10 client.0 _ll_register_callbacks cb 0x564a09f7c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f432ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a095180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f432f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37682/0 (socket says 172.21.15.23:37682) 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f432f400700 1 -- 172.21.15.23:0/2376756146 learned_addr learned my addr 172.21.15.23:0/2376756146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.311+0000 7f432fe00700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a0f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 -- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 msgr2=0x564a0a095180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a095180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 -- 172.21.15.23:0/2376756146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 msgr2=0x564a0a0f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a0f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 -- 172.21.15.23:0/2376756146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a09e583b0 con 0x564a0a0845d0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432ea00700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a095180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f432f400700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4320099c10 tx=0x7f432000e930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97f58c0093a1210f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4319600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:46.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4320099600 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f432009dd40 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f431c01e040 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43200b8030 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4349f91b80 10 client.27932 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.312+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564a09e5ccd0 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f431c01e040 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4320095d90 con 0x564a0a0845d0 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 10 client.27932 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 10 client.27932 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 20 client.27932 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a0f4290 0x564a0a0f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 4 client.27932 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564a09e25cf0 con 0x564a0a0f4290 2024-03-30T04:10:46.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.313+0000 7f432ea00700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a0f4290 0x564a0a0f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f432ea00700 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a0f4290 0x564a0a0f66b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19cd7b9b9ba8b6d server_cookie=62a4a7304527914d in_seq=0 out_seq=0 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4319600700 10 client.27932 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4319600700 1 -- 172.21.15.23:0/2376756146 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564a09e25cf0 con 0x564a0a0f4290 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4319600700 10 client.27932 handle_command_reply: tid=0 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 client.27932 shutdown 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 msgr2=0x564a0a094c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4320099c10 tx=0x7f432000e930 comp rx=0 tx=0).stop 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 shutdown_connections 2024-03-30T04:10:46.318 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x564a0a0f4290 0x564a0a0f66b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564a0a08fdf0 0x564a0a0f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564a0a0845d0 0x564a0a094c40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 --2- 172.21.15.23:0/2376756146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564a0a084ef0 0x564a0a095180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4319600700 0 client.27932 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 >> 172.21.15.23:0/2376756146 conn(0x564a09fe46a0 msgr2=0x564a0a08c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 shutdown_connections 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.314+0000 7f4349f91b80 1 -- 172.21.15.23:0/2376756146 wait complete. 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4349f91b80 20 client.27932 trim_cache size 0 max 16384 2024-03-30T04:10:46.319 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:46.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 msgr2=0x7f43140a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 0x7f43140a37d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4328009fd0 tx=0x7f432800b040 comp rx=0 tx=0).stop 2024-03-30T04:10:46.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 msgr2=0x7f4330199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f432000e9a0 tx=0x7f432000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 shutdown_connections 2024-03-30T04:10:46.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f433010d5a0 0x7f433019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f43140a1310 0x7f43140a37d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4330073160 0x7f4330199510 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 --2- 172.21.15.23:0/3405087036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4330073a80 0x7f4330199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 >> 172.21.15.23:0/3405087036 conn(0x7f43300fc7f0 msgr2=0x7f43300fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 shutdown_connections 2024-03-30T04:10:46.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.315+0000 7f4336400700 1 -- 172.21.15.23:0/3405087036 wait complete. 2024-03-30T04:10:46.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:46 smithi023 ceph-mon[38254]: pgmap v273: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 164 op/s 2024-03-30T04:10:46.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:46 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/734289400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:46.333 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:10:46.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:46 smithi154 ceph-mon[79161]: pgmap v273: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 164 op/s 2024-03-30T04:10:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:46 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/734289400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:46 smithi193 ceph-mon[79581]: pgmap v273: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 164 op/s 2024-03-30T04:10:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:46 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/734289400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 -- 172.21.15.23:0/3630797088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4101910 msgr2=0x7f84f4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 --2- 172.21.15.23:0/3630797088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4101910 0x7f84f4105950 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009990 tx=0x7f84e0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 -- 172.21.15.23:0/3630797088 shutdown_connections 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 --2- 172.21.15.23:0/3630797088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4106080 0x7f84f410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 --2- 172.21.15.23:0/3630797088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4100f60 0x7f84f4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 --2- 172.21.15.23:0/3630797088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4101910 0x7f84f4105950 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 -- 172.21.15.23:0/3630797088 >> 172.21.15.23:0/3630797088 conn(0x7f84f40fc7c0 msgr2=0x7f84f40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 -- 172.21.15.23:0/3630797088 shutdown_connections 2024-03-30T04:10:46.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.540+0000 7f84f9e00700 1 -- 172.21.15.23:0/3630797088 wait complete. 2024-03-30T04:10:46.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.542+0000 7f84f9e00700 1 Processor -- start 2024-03-30T04:10:46.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.542+0000 7f84f9e00700 1 -- start start 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.542+0000 7f84f9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.542+0000 7f84f9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4106080 0x7f84f40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.542+0000 7f84f9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4073b30 0x7f84f41a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f9e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41a59f0 con 0x7f84f4073b30 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f9e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41a5b60 con 0x7f84f4100f60 2024-03-30T04:10:46.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f9e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f41a5d20 con 0x7f84f4106080 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4106080 0x7f84f40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37700/0 (socket says 172.21.15.23:37700) 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 -- 172.21.15.23:0/2606596656 learned_addr learned my addr 172.21.15.23:0/2606596656 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f3400700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4073b30 0x7f84f41a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 -- 172.21.15.23:0/2606596656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4106080 msgr2=0x7f84f40735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4106080 0x7f84f40735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 -- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4073b30 msgr2=0x7f84f41a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4073b30 0x7f84f41a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 -- 172.21.15.23:0/2606596656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00095f0 con 0x7f84f4100f60 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.543+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f84e400e9a0 tx=0x7f84e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.544+0000 7f84f8a00700 1 -- 172.21.15.23:0/2606596656 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4009790 con 0x7f84f4100f60 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.544+0000 7f84f8a00700 1 -- 172.21.15.23:0/2606596656 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84e4010430 con 0x7f84f4100f60 2024-03-30T04:10:46.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.544+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f41a5fa0 con 0x7f84f4100f60 2024-03-30T04:10:46.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.544+0000 7f84f8a00700 1 -- 172.21.15.23:0/2606596656 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e4018680 con 0x7f84f4100f60 2024-03-30T04:10:46.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.544+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f41a6470 con 0x7f84f4100f60 2024-03-30T04:10:46.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.547+0000 7f84f8a00700 1 -- 172.21.15.23:0/2606596656 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f84e40105a0 con 0x7f84f4100f60 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.548+0000 7f850d9dcb80 1 Processor -- start 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.548+0000 7f84f8a00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 0x7f84d80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.548+0000 7f850d9dcb80 1 -- start start 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f8a00700 1 -- 172.21.15.23:0/2606596656 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f84e4014070 con 0x7f84f4100f60 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f3400700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 0x7f84d80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec808675d0 0x55ec808679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec808751e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec807c0990 con 0x55ec808675d0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec807be350 con 0x55ec80872df0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec807b9c40 con 0x55ec80867ef0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37638/0 (socket says 172.21.15.23:37638) 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f2000700 1 -- 172.21.15.23:0/2761603189 learned_addr learned my addr 172.21.15.23:0/2761603189 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec808675d0 0x55ec808679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.549+0000 7f84f3400700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec808751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 -- 172.21.15.23:0/2761603189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 msgr2=0x55ec808751e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec808751e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 -- 172.21.15.23:0/2761603189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec808675d0 msgr2=0x55ec808679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec808675d0 0x55ec808679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 -- 172.21.15.23:0/2761603189 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec8063b3b0 con 0x55ec80867ef0 2024-03-30T04:10:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f3400700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec808751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:10:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f3400700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 0x7f84d80a37d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00f580 tx=0x7f84ec00b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84f2000700 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009990 tx=0x7f84e0005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d481b3344c351525 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84dfe00700 1 -- 172.21.15.23:0/2761603189 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e001d070 con 0x55ec80867ef0 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84dfe00700 1 -- 172.21.15.23:0/2761603189 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84e001e040 con 0x55ec80867ef0 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.550+0000 7f84dfe00700 1 -- 172.21.15.23:0/2761603189 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0004880 con 0x55ec80867ef0 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2761603189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 msgr2=0x55ec808728b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009990 tx=0x7f84e0005ec0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2761603189 shutdown_connections 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec808751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec808675d0 0x55ec808679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2761603189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec80867ef0 0x55ec808728b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2761603189 >> 172.21.15.23:0/2761603189 conn(0x55ec807c76a0 msgr2=0x55ec80863a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2761603189 shutdown_connections 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.551+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2761603189 wait complete. 2024-03-30T04:10:46.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 -- start start 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 Processor -- start 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec80872750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80872df0 0x55ec8086ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec80755d80 con 0x55ec80872df0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec80755a20 con 0x55ec80867ef0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec8063d6a0 con 0x55ec808675d0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.552+0000 7f850d9dcb80 10 client.0 _ll_register_callbacks cb 0x55ec8075f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37722/0 (socket says 172.21.15.23:37722) 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 -- 172.21.15.23:0/2287277124 learned_addr learned my addr 172.21.15.23:0/2287277124 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f3400700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80872df0 0x55ec8086ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec80872750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 -- 172.21.15.23:0/2287277124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 msgr2=0x55ec80872750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec80872750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 -- 172.21.15.23:0/2287277124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80872df0 msgr2=0x55ec8086ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80872df0 0x55ec8086ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 -- 172.21.15.23:0/2287277124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f410e470 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84f2000700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f84e000bf80 tx=0x7f84e000b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bafbd956ed2b8f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.553+0000 7f84dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e001d030 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec8077a380 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f84e0005980 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84e0005bc0 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 10 client.27941 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ec805e7b60 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec8077a380 con 0x55ec80867ef0 2024-03-30T04:10:46.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f84e000f9c0 con 0x55ec80867ef0 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 10 client.27941 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 10 client.27941 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.554+0000 7f850d9dcb80 20 client.27941 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808823c0 0x55ec808847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f850d9dcb80 4 client.27941 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec807b2700 con 0x55ec808823c0 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808823c0 0x55ec808847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f84f2a00700 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808823c0 0x55ec808847b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e5b09b69c4b3dc1 server_cookie=7e69524e6f91c01a in_seq=0 out_seq=0 2024-03-30T04:10:46.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.555+0000 7f84dcc00700 10 client.27941 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f84dcc00700 1 -- 172.21.15.23:0/2287277124 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ec807b2700 con 0x55ec808823c0 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f84dcc00700 10 client.27941 handle_command_reply: tid=0 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 client.27941 shutdown 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 msgr2=0x55ec8086fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f84e000bf80 tx=0x7f84e000b9e0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 shutdown_connections 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808823c0 0x55ec808847b0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80872df0 0x55ec8086ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec80872750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/2287277124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80867ef0 0x55ec8086fa40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f84dcc00700 0 client.27941 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 >> 172.21.15.23:0/2287277124 conn(0x55ec807c76a0 msgr2=0x55ec808696c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 shutdown_connections 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 1 -- 172.21.15.23:0/2287277124 wait complete. 2024-03-30T04:10:46.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.556+0000 7f850d9dcb80 20 client.27941 trim_cache size 0 max 16384 2024-03-30T04:10:46.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.571+0000 7f850d9dcb80 1 Processor -- start 2024-03-30T04:10:46.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 -- start start 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec8085f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec8085c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec80871040 con 0x55ec80867ef0 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec808647f0 con 0x55ec80872df0 2024-03-30T04:10:46.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec80868850 con 0x55ec808675d0 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f84f2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f84f3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec8085c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f84f2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38550/0 (socket says 172.21.15.23:38550) 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.572+0000 7f84f2000700 1 -- 172.21.15.23:0/1761282250 learned_addr learned my addr 172.21.15.23:0/1761282250 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2a00700 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec8085f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 -- 172.21.15.23:0/1761282250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 msgr2=0x55ec8085f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec8085f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 -- 172.21.15.23:0/1761282250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 msgr2=0x55ec8085c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec8085c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 -- 172.21.15.23:0/1761282250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec8077fea0 con 0x55ec80867ef0 2024-03-30T04:10:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f3400700 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec8085c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84f2000700 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009ce0 tx=0x7f84e000bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86d31526fd0e7d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84e9200700 1 -- 172.21.15.23:0/1761282250 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e001d040 con 0x55ec80867ef0 2024-03-30T04:10:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84e9200700 1 -- 172.21.15.23:0/1761282250 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84e0005bc0 con 0x55ec80867ef0 2024-03-30T04:10:46.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.573+0000 7f84e9200700 1 -- 172.21.15.23:0/1761282250 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0004970 con 0x55ec80867ef0 2024-03-30T04:10:46.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 -- 172.21.15.23:0/1761282250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 msgr2=0x55ec80879a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009ce0 tx=0x7f84e000bcd0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 -- 172.21.15.23:0/1761282250 shutdown_connections 2024-03-30T04:10:46.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec8085c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec8085f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/1761282250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80879a10 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 -- 172.21.15.23:0/1761282250 >> 172.21.15.23:0/1761282250 conn(0x55ec807c76a0 msgr2=0x55ec80889110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 -- 172.21.15.23:0/1761282250 shutdown_connections 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.574+0000 7f850d9dcb80 1 -- 172.21.15.23:0/1761282250 wait complete. 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 -- start start 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 Processor -- start 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec808779c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80877f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec808d52d0 con 0x55ec80867ef0 2024-03-30T04:10:46.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec808d5440 con 0x55ec80872df0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec808d5600 con 0x55ec808675d0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.575+0000 7f850d9dcb80 10 client.0 _ll_register_callbacks cb 0x55ec8075f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80877f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37744/0 (socket says 172.21.15.23:37744) 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 -- 172.21.15.23:0/3074315784 learned_addr learned my addr 172.21.15.23:0/3074315784 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f2a00700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec808779c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 -- 172.21.15.23:0/3074315784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 msgr2=0x55ec808779c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec808779c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 -- 172.21.15.23:0/3074315784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 msgr2=0x55ec80877f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80877f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 -- 172.21.15.23:0/3074315784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ec009040 con 0x55ec80872df0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f2a00700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec808779c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84f3400700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f84ec008ed0 tx=0x7f84ec004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90bf71474e50b530 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.576+0000 7f84dcc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec01c040 con 0x55ec80872df0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f84ec02d470 con 0x55ec80872df0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec80608cf0 con 0x55ec80872df0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84ec00e040 con 0x55ec80872df0 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 10 client.27947 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ec805e7b60 con 0x55ec80872df0 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec80608cf0 con 0x55ec80872df0 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f84ec0154b0 con 0x55ec80872df0 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 10 client.27947 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 10 client.27947 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 20 client.27947 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808d6e70 0x55ec808d9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 4 client.27947 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.577+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec8063dce0 con 0x55ec808d6e70 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.578+0000 7f84f2a00700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808d6e70 0x55ec808d9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.578+0000 7f84f2a00700 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808d6e70 0x55ec808d9290 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ea86540a87d96d3c server_cookie=420095abe0c3505f in_seq=0 out_seq=0 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.578+0000 7f84dcc00700 10 client.27947 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:46.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.578+0000 7f84dcc00700 1 -- 172.21.15.23:0/3074315784 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec8063dce0 con 0x55ec808d6e70 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f84dcc00700 10 client.27947 handle_command_reply: tid=0 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 client.27947 shutdown 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 msgr2=0x55ec80878440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f84ec008ed0 tx=0x7f84ec004ad0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 shutdown_connections 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ec808d6e70 0x55ec808d9290 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ec80872df0 0x55ec80878440 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec808675d0 0x55ec808779c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 --2- 172.21.15.23:0/3074315784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec80867ef0 0x55ec80877f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f84dcc00700 0 client.27947 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 >> 172.21.15.23:0/3074315784 conn(0x55ec807c76a0 msgr2=0x55ec8085c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 shutdown_connections 2024-03-30T04:10:46.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 1 -- 172.21.15.23:0/3074315784 wait complete. 2024-03-30T04:10:46.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.579+0000 7f850d9dcb80 20 client.27947 trim_cache size 0 max 16384 2024-03-30T04:10:46.586 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 msgr2=0x7f84d80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 0x7f84d80a37d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00f580 tx=0x7f84ec00b040 comp rx=0 tx=0).stop 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 msgr2=0x7f84f40730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f84e400e9a0 tx=0x7f84e400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 shutdown_connections 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f84f4073b30 0x7f84f41a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f84d80a1310 0x7f84d80a37d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f4100f60 0x7f84f40730b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 --2- 172.21.15.23:0/2606596656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84f4106080 0x7f84f40735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 >> 172.21.15.23:0/2606596656 conn(0x7f84f40fc7c0 msgr2=0x7f84f40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 shutdown_connections 2024-03-30T04:10:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.580+0000 7f84f9e00700 1 -- 172.21.15.23:0/2606596656 wait complete. 2024-03-30T04:10:46.598 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:10:46.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 -- 172.21.15.23:0/1254783276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c073150 msgr2=0x7f5b1c073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 --2- 172.21.15.23:0/1254783276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c073150 0x7f5b1c073530 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009990 tx=0x7f5b0c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 -- 172.21.15.23:0/1254783276 shutdown_connections 2024-03-30T04:10:46.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 --2- 172.21.15.23:0/1254783276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c111d00 0x7f5b1c1140f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 --2- 172.21.15.23:0/1254783276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c073150 0x7f5b1c073530 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 --2- 172.21.15.23:0/1254783276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1117c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.814+0000 7f5b23000700 1 -- 172.21.15.23:0/1254783276 >> 172.21.15.23:0/1254783276 conn(0x7f5b1c0fc7c0 msgr2=0x7f5b1c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.815+0000 7f5b23000700 1 -- 172.21.15.23:0/1254783276 shutdown_connections 2024-03-30T04:10:46.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.815+0000 7f5b23000700 1 -- 172.21.15.23:0/1254783276 wait complete. 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 Processor -- start 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 -- start start 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 0x7f5b1c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19dce0 con 0x7f5b1c073150 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19de50 con 0x7f5b1c073a70 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b23000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c19e010 con 0x7f5b1c111d00 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.816+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 0x7f5b1c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38578/0 (socket says 172.21.15.23:38578) 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b1be00700 1 -- 172.21.15.23:0/2079719370 learned_addr learned my addr 172.21.15.23:0/2079719370 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b1b400700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 -- 172.21.15.23:0/2079719370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 msgr2=0x7f5b1c1a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 -- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 msgr2=0x7f5b1c1a3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 -- 172.21.15.23:0/2079719370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c0095f0 con 0x7f5b1c111d00 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b1b400700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b1be00700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b21000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 0x7f5b1c19d680 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400ef90 tx=0x7f5b1400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b18c00700 1 -- 172.21.15.23:0/2079719370 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14018070 con 0x7f5b1c111d00 2024-03-30T04:10:46.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b18c00700 1 -- 172.21.15.23:0/2079719370 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b1400f040 con 0x7f5b1c111d00 2024-03-30T04:10:46.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b18c00700 1 -- 172.21.15.23:0/2079719370 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1400cc30 con 0x7f5b1c111d00 2024-03-30T04:10:46.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.817+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c19e290 con 0x7f5b1c111d00 2024-03-30T04:10:46.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.818+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c1aa540 con 0x7f5b1c111d00 2024-03-30T04:10:46.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.822+0000 7f5b18c00700 1 -- 172.21.15.23:0/2079719370 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f5b14003a00 con 0x7f5b1c111d00 2024-03-30T04:10:46.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 Processor -- start 2024-03-30T04:10:46.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b18c00700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 0x7f5b000a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 -- start start 2024-03-30T04:10:46.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b18c00700 1 -- 172.21.15.23:0/2079719370 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5b140ccaf0 con 0x7f5b1c111d00 2024-03-30T04:10:46.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b1b400700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 0x7f5b000a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c768b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c791e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3bc4990 con 0x5602c3c6bef0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3bc2350 con 0x5602c3c6b5d0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3bbdc40 con 0x5602c3c76df0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37786/0 (socket says 172.21.15.23:37786) 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b1be00700 1 -- 172.21.15.23:0/15776987 learned_addr learned my addr 172.21.15.23:0/15776987 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.824+0000 7f5b1b400700 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b21000700 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c791e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 -- 172.21.15.23:0/15776987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 msgr2=0x5602c3c791e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c791e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 -- 172.21.15.23:0/15776987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 msgr2=0x5602c3c768b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c768b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 -- 172.21.15.23:0/15776987 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602c3a3f3b0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1b400700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 0x7f5b000a37d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c19ed90 tx=0x7f5b08013c70 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b1be00700 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00b660 tx=0x7f5b0c00b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48184077403cd9b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b11600700 1 -- 172.21.15.23:0/15776987 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c01d070 con 0x5602c3c6b5d0 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.825+0000 7f5b11600700 1 -- 172.21.15.23:0/15776987 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b0c01e040 con 0x5602c3c6b5d0 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b11600700 1 -- 172.21.15.23:0/15776987 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c004cd0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 -- 172.21.15.23:0/15776987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 msgr2=0x5602c3c6b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00b660 tx=0x7f5b0c00b690 comp rx=0 tx=0).stop 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 -- 172.21.15.23:0/15776987 shutdown_connections 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c791e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c6b9b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/15776987 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c768b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 -- 172.21.15.23:0/15776987 >> 172.21.15.23:0/15776987 conn(0x5602c3bcb6a0 msgr2=0x5602c3c67a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 -- 172.21.15.23:0/15776987 shutdown_connections 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.826+0000 7f5b36bedb80 1 -- 172.21.15.23:0/15776987 wait complete. 2024-03-30T04:10:46.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.827+0000 7f5b36bedb80 1 -- start start 2024-03-30T04:10:46.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.827+0000 7f5b36bedb80 1 Processor -- start 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c766f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3c73fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3b59d80 con 0x5602c3c76df0 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3b59a20 con 0x5602c3c6bef0 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3a416a0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b36bedb80 10 client.0 _ll_register_callbacks cb 0x5602c3b63470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c766f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b21000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3c73fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37794/0 (socket says 172.21.15.23:37794) 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 -- 172.21.15.23:0/3414062049 learned_addr learned my addr 172.21.15.23:0/3414062049 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 -- 172.21.15.23:0/3414062049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 msgr2=0x5602c3c766f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c766f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 -- 172.21.15.23:0/3414062049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 msgr2=0x5602c3c73fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3c73fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.828+0000 7f5b1b400700 1 -- 172.21.15.23:0/3414062049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c114aa0 con 0x5602c3c6bef0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b1b400700 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5b08004470 tx=0x7f5b080047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b403ca19f987860 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b06000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08004560 con 0x5602c3c6bef0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602c3b7e380 con 0x5602c3c6bef0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5b08003890 con 0x5602c3c6bef0 2024-03-30T04:10:46.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b08003a50 con 0x5602c3c6bef0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b36bedb80 10 client.27959 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5602c39ebb60 con 0x5602c3c6bef0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.829+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5602c3b7e380 con 0x5602c3c6bef0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5b08020c00 con 0x5602c3c6bef0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 10 client.27959 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 10 client.27959 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 20 client.27959 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3c869c0 0x5602c3c88de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 4 client.27959 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602c3bb6700 con 0x5602c3c869c0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.830+0000 7f5b1be00700 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3c869c0 0x5602c3c88de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b1be00700 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3c869c0 0x5602c3c88de0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49c3016a1c18f8e6 server_cookie=bfd21fdef90f2569 in_seq=0 out_seq=0 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b06000700 10 client.27959 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:46.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b06000700 1 -- 172.21.15.23:0/3414062049 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5602c3bb6700 con 0x5602c3c869c0 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b06000700 10 client.27959 handle_command_reply: tid=0 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b36bedb80 1 client.27959 shutdown 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 msgr2=0x5602c3c73a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.831+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f5b08004470 tx=0x7f5b080047c0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 shutdown_connections 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3c73fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3c869c0 0x5602c3c88de0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c766f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/3414062049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c73a30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b06000700 0 client.27959 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 >> 172.21.15.23:0/3414062049 conn(0x5602c3bcb6a0 msgr2=0x5602c3c6d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 shutdown_connections 2024-03-30T04:10:46.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 1 -- 172.21.15.23:0/3414062049 wait complete. 2024-03-30T04:10:46.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.832+0000 7f5b36bedb80 20 client.27959 trim_cache size 0 max 16384 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 Processor -- start 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 -- start start 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c7da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c6ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3c89170 con 0x5602c3c6bef0 2024-03-30T04:10:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3c7df50 con 0x5602c3c6b5d0 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.848+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3c71f80 con 0x5602c3c76df0 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37798/0 (socket says 172.21.15.23:37798) 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 -- 172.21.15.23:0/2184569990 learned_addr learned my addr 172.21.15.23:0/2184569990 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b21000700 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c6ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1b400700 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c7da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 -- 172.21.15.23:0/2184569990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 msgr2=0x5602c3c6ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c6ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 -- 172.21.15.23:0/2184569990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 msgr2=0x5602c3c7da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c7da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 -- 172.21.15.23:0/2184569990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602c3b83ea0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.849+0000 7f5b1be00700 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00b660 tx=0x7f5b0c01a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddceed1c812993bf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b12000700 1 -- 172.21.15.23:0/2184569990 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c01d040 con 0x5602c3c6b5d0 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b12000700 1 -- 172.21.15.23:0/2184569990 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b0c00be90 con 0x5602c3c6b5d0 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b12000700 1 -- 172.21.15.23:0/2184569990 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c004680 con 0x5602c3c6b5d0 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 -- 172.21.15.23:0/2184569990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 msgr2=0x5602c3c66f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00b660 tx=0x7f5b0c01a040 comp rx=0 tx=0).stop 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 -- 172.21.15.23:0/2184569990 shutdown_connections 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c76df0 0x5602c3c6ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6b5d0 0x5602c3c66f10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/2184569990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c6bef0 0x5602c3c7da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 -- 172.21.15.23:0/2184569990 >> 172.21.15.23:0/2184569990 conn(0x5602c3bcb6a0 msgr2=0x5602c3c605a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 -- 172.21.15.23:0/2184569990 shutdown_connections 2024-03-30T04:10:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.850+0000 7f5b36bedb80 1 -- 172.21.15.23:0/2184569990 wait complete. 2024-03-30T04:10:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.851+0000 7f5b36bedb80 1 -- start start 2024-03-30T04:10:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.851+0000 7f5b36bedb80 1 Processor -- start 2024-03-30T04:10:46.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c7c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3cd3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3cd3dd0 con 0x5602c3c76df0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3cd3f40 con 0x5602c3c6bef0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602c3cd4100 con 0x5602c3c6b5d0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b36bedb80 10 client.0 _ll_register_callbacks cb 0x5602c3b63470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b1b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c7c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b1b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c7c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37802/0 (socket says 172.21.15.23:37802) 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b1be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37706/0 (socket says 172.21.15.23:37706) 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b1b400700 1 -- 172.21.15.23:0/1899952989 learned_addr learned my addr 172.21.15.23:0/1899952989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.852+0000 7f5b21000700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3cd3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 -- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 msgr2=0x5602c3c7c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c7c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 -- 172.21.15.23:0/1899952989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 msgr2=0x5602c3cd3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3cd3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 -- 172.21.15.23:0/1899952989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602c3a3f3b0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b1be00700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009ce0 tx=0x7f5b0c0343e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68f1eddaa2a7448d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b06000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c01d070 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b0c01e040 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.853+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5b0c031040 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b0c0056b0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 10 client.27958 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5602c3a43cd0 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b0c01e040 con 0x5602c3c6b5d0 2024-03-30T04:10:46.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5b0c00f460 con 0x5602c3c6b5d0 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 10 client.27958 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 10 client.27958 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 20 client.27958 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3cdb560 0x5602c3cdd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 4 client.27958 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5602c3a0ccf0 con 0x5602c3cdb560 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.854+0000 7f5b1b400700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3cdb560 0x5602c3cdd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b06000700 10 client.27958 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b1b400700 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3cdb560 0x5602c3cdd950 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c6b25418b0bb56c server_cookie=82bf64c33be53b06 in_seq=0 out_seq=0 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b06000700 1 -- 172.21.15.23:0/1899952989 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5602c3a0ccf0 con 0x5602c3cdb560 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b06000700 10 client.27958 handle_command_reply: tid=0 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 client.27958 shutdown 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 msgr2=0x5602c3c7bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009ce0 tx=0x7f5b0c0343e0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 shutdown_connections 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5602c3c76df0 0x5602c3cd3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5602c3cdb560 0x5602c3cdd950 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602c3c6b5d0 0x5602c3c7bc40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b06000700 0 client.27958 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 --2- 172.21.15.23:0/1899952989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5602c3c6bef0 0x5602c3c7c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 >> 172.21.15.23:0/1899952989 conn(0x5602c3bcb6a0 msgr2=0x5602c3c6ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.855+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 shutdown_connections 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b36bedb80 1 -- 172.21.15.23:0/1899952989 wait complete. 2024-03-30T04:10:46.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b36bedb80 20 client.27958 trim_cache size 0 max 16384 2024-03-30T04:10:46.860 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 msgr2=0x7f5b000a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 0x7f5b000a37d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c19ed90 tx=0x7f5b08013c70 comp rx=0 tx=0).stop 2024-03-30T04:10:46.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 msgr2=0x7f5b1c19d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 0x7f5b1c19d680 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400ef90 tx=0x7f5b1400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 shutdown_connections 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b1c111d00 0x7f5b1c19d680 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b1c073150 0x7f5b1c1a3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5b000a1310 0x7f5b000a37d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 --2- 172.21.15.23:0/2079719370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b1c073a70 0x7f5b1c1a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.856+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 >> 172.21.15.23:0/2079719370 conn(0x7f5b1c0fc7c0 msgr2=0x7f5b1c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.857+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 shutdown_connections 2024-03-30T04:10:46.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:46.857+0000 7f5b23000700 1 -- 172.21.15.23:0/2079719370 wait complete. 2024-03-30T04:10:46.876 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:10:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 -- 172.21.15.23:0/2876613164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 msgr2=0x7f8580101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 --2- 172.21.15.23:0/2876613164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 0x7f8580101340 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8570009990 tx=0x7f8570009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 -- 172.21.15.23:0/2876613164 shutdown_connections 2024-03-30T04:10:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 --2- 172.21.15.23:0/2876613164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8580106080 0x7f858010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 --2- 172.21.15.23:0/2876613164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 0x7f8580101340 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 --2- 172.21.15.23:0/2876613164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580101910 0x7f8580105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 -- 172.21.15.23:0/2876613164 >> 172.21.15.23:0/2876613164 conn(0x7f85800fc7c0 msgr2=0x7f85800febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 -- 172.21.15.23:0/2876613164 shutdown_connections 2024-03-30T04:10:47.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.084+0000 7f8585e00700 1 -- 172.21.15.23:0/2876613164 wait complete. 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.085+0000 7f8585e00700 1 Processor -- start 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 -- start start 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 0x7f85801994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858019e810 con 0x7f8580106080 2024-03-30T04:10:47.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858019e980 con 0x7f858019dd90 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f8585e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858019eb40 con 0x7f8580100f60 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f857e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f857f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f857f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37820/0 (socket says 172.21.15.23:37820) 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f857f400700 1 -- 172.21.15.23:0/2321563032 learned_addr learned my addr 172.21.15.23:0/2321563032 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.086+0000 7f857e000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38652/0 (socket says 172.21.15.23:38652) 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 -- 172.21.15.23:0/2321563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 msgr2=0x7f85801994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 0x7f85801994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 -- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 msgr2=0x7f85801999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 -- 172.21.15.23:0/2321563032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85700095f0 con 0x7f858019dd90 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857e000700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:47.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f857f400700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f857800ef30 tx=0x7f857800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f8584a00700 1 -- 172.21.15.23:0/2321563032 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578018070 con 0x7f858019dd90 2024-03-30T04:10:47.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f8584a00700 1 -- 172.21.15.23:0/2321563032 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857800f040 con 0x7f858019dd90 2024-03-30T04:10:47.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85801a6040 con 0x7f858019dd90 2024-03-30T04:10:47.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.087+0000 7f8584a00700 1 -- 172.21.15.23:0/2321563032 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857800cdb0 con 0x7f858019dd90 2024-03-30T04:10:47.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.088+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85801a6520 con 0x7f858019dd90 2024-03-30T04:10:47.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.091+0000 7f8584a00700 1 -- 172.21.15.23:0/2321563032 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8578003a00 con 0x7f858019dd90 2024-03-30T04:10:47.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 Processor -- start 2024-03-30T04:10:47.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 -- start start 2024-03-30T04:10:47.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f8584a00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 0x7f85640a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f8584a00700 1 -- 172.21.15.23:0/2321563032 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f85780ccc70 con 0x7f858019dd90 2024-03-30T04:10:47.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f857e000700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 0x7f85640a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455d21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614551d990 con 0x5561455c45d0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614551b350 con 0x5561455cfdf0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.093+0000 7f85999a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556145516c40 con 0x5561455c4ef0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455d21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38658/0 (socket says 172.21.15.23:38658) 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857ea00700 1 -- 172.21.15.23:0/1350880447 learned_addr learned my addr 172.21.15.23:0/1350880447 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857e000700 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 -- 172.21.15.23:0/1350880447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 msgr2=0x5561455cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 -- 172.21.15.23:0/1350880447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 msgr2=0x5561455c49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857e000700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 0x7f85640a3750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f858019aa00 tx=0x7f856c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 -- 172.21.15.23:0/1350880447 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5561453983b0 con 0x5561455cfdf0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857ea00700 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.094+0000 7f857f400700 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455d21e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f85780c9f80 tx=0x7f8578099820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6366e0f7e0dd5c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f856be00700 1 -- 172.21.15.23:0/1350880447 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578004ad0 con 0x5561455cfdf0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f856be00700 1 -- 172.21.15.23:0/1350880447 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857801b320 con 0x5561455cfdf0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f856be00700 1 -- 172.21.15.23:0/1350880447 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85780960e0 con 0x5561455cfdf0 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 -- 172.21.15.23:0/1350880447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 msgr2=0x5561455d21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455d21e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f85780c9f80 tx=0x7f8578099820 comp rx=0 tx=0).stop 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 -- 172.21.15.23:0/1350880447 shutdown_connections 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455d21e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455c49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1350880447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 -- 172.21.15.23:0/1350880447 >> 172.21.15.23:0/1350880447 conn(0x5561455246a0 msgr2=0x5561455c0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.095+0000 7f85999a3b80 1 -- 172.21.15.23:0/1350880447 shutdown_connections 2024-03-30T04:10:47.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.096+0000 7f85999a3b80 1 -- 172.21.15.23:0/1350880447 wait complete. 2024-03-30T04:10:47.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.096+0000 7f85999a3b80 1 -- start start 2024-03-30T04:10:47.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.096+0000 7f85999a3b80 1 Processor -- start 2024-03-30T04:10:47.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561455cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561454b2d80 con 0x5561455cfdf0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561454b2a20 con 0x5561455c45d0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55614539a6a0 con 0x5561455c4ef0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f85999a3b80 10 client.0 _ll_register_callbacks cb 0x5561454bc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37840/0 (socket says 172.21.15.23:37840) 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 -- 172.21.15.23:0/1913826395 learned_addr learned my addr 172.21.15.23:0/1913826395 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857f400700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561455cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857e000700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 -- 172.21.15.23:0/1913826395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 msgr2=0x5561455cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 -- 172.21.15.23:0/1913826395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 msgr2=0x5561455cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561455cd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.097+0000 7f857ea00700 1 -- 172.21.15.23:0/1913826395 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858010e470 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.098+0000 7f857ea00700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85700056e0 tx=0x7f8570005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=682314240d73f178 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.098+0000 7f8568c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.098+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857001f040 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.098+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8570004500 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.098+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5561454d7380 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85700046c0 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 10 client.27977 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f85801a6520 con 0x5561455c45d0 2024-03-30T04:10:47.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5561454d7380 con 0x5561455c45d0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f857000f580 con 0x5561455c45d0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 10 client.27977 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 10 client.27977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 20 client.27977 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5561455dfbd0 0x5561455e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 4 client.27977 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55614550f700 con 0x5561455dfbd0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.099+0000 7f857e000700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5561455dfbd0 0x5561455e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.100+0000 7f857e000700 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5561455dfbd0 0x5561455e1ff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b6108aeb17e7dbc server_cookie=a8f571dbf0393818 in_seq=0 out_seq=0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.100+0000 7f8568c00700 10 client.27977 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f8568c00700 1 -- 172.21.15.23:0/1913826395 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55614550f700 con 0x5561455dfbd0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f8568c00700 10 client.27977 handle_command_reply: tid=0 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 client.27977 shutdown 2024-03-30T04:10:47.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 msgr2=0x5561455ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85700056e0 tx=0x7f8570005710 comp rx=0 tx=0).stop 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 shutdown_connections 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561455cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c45d0 0x5561455ccca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5561455dfbd0 0x5561455e1ff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 --2- 172.21.15.23:0/1913826395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f8568c00700 0 client.27977 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 >> 172.21.15.23:0/1913826395 conn(0x5561455246a0 msgr2=0x5561455c66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 shutdown_connections 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 1 -- 172.21.15.23:0/1913826395 wait complete. 2024-03-30T04:10:47.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.101+0000 7f85999a3b80 20 client.27977 trim_cache size 0 max 16384 2024-03-30T04:10:47.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.115+0000 7f85999a3b80 1 Processor -- start 2024-03-30T04:10:47.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 -- start start 2024-03-30T04:10:47.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455d6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561455c5640 con 0x5561455c45d0 2024-03-30T04:10:47.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561455ce250 con 0x5561455cfdf0 2024-03-30T04:10:47.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f85999a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561455bc700 con 0x5561455c4ef0 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f857e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f857f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f857e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37734/0 (socket says 172.21.15.23:37734) 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f857ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455d6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.116+0000 7f857e000700 1 -- 172.21.15.23:0/2284339790 learned_addr learned my addr 172.21.15.23:0/2284339790 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 -- 172.21.15.23:0/2284339790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 msgr2=0x5561455c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 -- 172.21.15.23:0/2284339790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 msgr2=0x5561455d6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455d6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 -- 172.21.15.23:0/2284339790 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556145344b60 con 0x5561455cfdf0 2024-03-30T04:10:47.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857e000700 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857ea00700 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455d6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f857f400700 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455c84d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f85780c9fb0 tx=0x7f8578099610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4684c552d35b9047 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f8575200700 1 -- 172.21.15.23:0/2284339790 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857809a0d0 con 0x5561455cfdf0 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f8575200700 1 -- 172.21.15.23:0/2284339790 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f85780999d0 con 0x5561455cfdf0 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.117+0000 7f8575200700 1 -- 172.21.15.23:0/2284339790 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8578097e00 con 0x5561455cfdf0 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- 172.21.15.23:0/2284339790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 msgr2=0x5561455c84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455c84d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f85780c9fb0 tx=0x7f8578099610 comp rx=0 tx=0).stop 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- 172.21.15.23:0/2284339790 shutdown_connections 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455cfdf0 0x5561455c84d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455c45d0 0x5561455d6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 --2- 172.21.15.23:0/2284339790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c4ef0 0x5561455c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- 172.21.15.23:0/2284339790 >> 172.21.15.23:0/2284339790 conn(0x5561455246a0 msgr2=0x5561455b9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- 172.21.15.23:0/2284339790 shutdown_connections 2024-03-30T04:10:47.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- 172.21.15.23:0/2284339790 wait complete. 2024-03-30T04:10:47.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.118+0000 7f85999a3b80 1 -- start start 2024-03-30T04:10:47.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 Processor -- start 2024-03-30T04:10:47.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 0x556145628080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561456329d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561456287d0 con 0x5561455cfdf0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556145632f10 con 0x5561455c4ef0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5561456330d0 con 0x5561455c45d0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f85999a3b80 10 client.0 _ll_register_callbacks cb 0x5561454bc470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.119+0000 7f857ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 0x556145628080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:37736/0 (socket says 172.21.15.23:37736) 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857ea00700 1 -- 172.21.15.23:0/7437279 learned_addr learned my addr 172.21.15.23:0/7437279 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857f400700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561456329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 -- 172.21.15.23:0/7437279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 msgr2=0x556145627b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 -- 172.21.15.23:0/7437279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 msgr2=0x5561456329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561456329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 -- 172.21.15.23:0/7437279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85801a6520 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857ea00700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f857e000700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 0x556145628080 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f856c004720 tx=0x7f856c013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c0b9a7dcb69e350 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f8568c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856c021030 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.120+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f857801b320 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f856c02a520 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f856c02ae60 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 10 client.27983 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f856c016030 con 0x5561455c4ef0 2024-03-30T04:10:47.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f857801b320 con 0x5561455c4ef0 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f856c01c7f0 con 0x5561455c4ef0 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 10 client.27983 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 10 client.27983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 20 client.27983 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556145634850 0x556145636c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 4 client.27983 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.121+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556145365cf0 con 0x556145634850 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f857ea00700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556145634850 0x556145636c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f857ea00700 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556145634850 0x556145636c40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82a4228fc1264e1 server_cookie=c75a8f537788859d in_seq=0 out_seq=0 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f8568c00700 10 client.27983 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f8568c00700 1 -- 172.21.15.23:0/7437279 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556145365cf0 con 0x556145634850 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f8568c00700 10 client.27983 handle_command_reply: tid=0 2024-03-30T04:10:47.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f85999a3b80 1 client.27983 shutdown 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 msgr2=0x556145628080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.122+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 0x556145628080 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f856c004720 tx=0x7f856c013040 comp rx=0 tx=0).stop 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 shutdown_connections 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5561455cfdf0 0x5561456329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556145634850 0x556145636c40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5561455c45d0 0x556145627b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 --2- 172.21.15.23:0/7437279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5561455c4ef0 0x556145628080 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8568c00700 0 client.27983 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 >> 172.21.15.23:0/7437279 conn(0x5561455246a0 msgr2=0x5561455c7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 shutdown_connections 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 1 -- 172.21.15.23:0/7437279 wait complete. 2024-03-30T04:10:47.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f85999a3b80 20 client.27983 trim_cache size 0 max 16384 2024-03-30T04:10:47.128 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 msgr2=0x7f85640a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 0x7f85640a3750 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f858019aa00 tx=0x7f856c009260 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 msgr2=0x7f858019e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f857800ef30 tx=0x7f857800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 shutdown_connections 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85640a1290 0x7f85640a3750 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f858019dd90 0x7f858019e210 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8580100f60 0x7f85801994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 --2- 172.21.15.23:0/2321563032 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8580106080 0x7f85801999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.123+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 >> 172.21.15.23:0/2321563032 conn(0x7f85800fc7c0 msgr2=0x7f85800fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.124+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 shutdown_connections 2024-03-30T04:10:47.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:47.124+0000 7f8585e00700 1 -- 172.21.15.23:0/2321563032 wait complete. 2024-03-30T04:10:47.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 880850 10.43 MB/sec execute 324 sec latency 18.592 ms 2024-03-30T04:10:48.143 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:10:48.144 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:10:48.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 882392 10.42 MB/sec execute 325 sec latency 19.575 ms 2024-03-30T04:10:48.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.361+0000 7fc8e4a00700 1 -- 172.21.15.23:0/2428392123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 msgr2=0x7fc8e010dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.361+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/2428392123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e010dbf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8009f90 tx=0x7fc8d800b330 comp rx=0 tx=0).stop 2024-03-30T04:10:48.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 -- 172.21.15.23:0/2428392123 shutdown_connections 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/2428392123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e010dbf0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/2428392123 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e0101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/2428392123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0101900 0x7fc8e0105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 -- 172.21.15.23:0/2428392123 >> 172.21.15.23:0/2428392123 conn(0x7fc8e00fc7f0 msgr2=0x7fc8e00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 -- 172.21.15.23:0/2428392123 shutdown_connections 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.362+0000 7fc8e4a00700 1 -- 172.21.15.23:0/2428392123 wait complete. 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.363+0000 7fc8e4a00700 1 Processor -- start 2024-03-30T04:10:48.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.363+0000 7fc8e4a00700 1 -- start start 2024-03-30T04:10:48.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 0x7fc8e01a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e01a5af0 con 0x7fc8e0106070 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e01a5c60 con 0x7fc8e0073b30 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8e4a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e01a5e20 con 0x7fc8e0100f50 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8dde00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8dde00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35490/0 (socket says 172.21.15.23:35490) 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8dde00700 1 -- 172.21.15.23:0/437307589 learned_addr learned my addr 172.21.15.23:0/437307589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.364+0000 7fc8de800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 0x7fc8e01a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 -- 172.21.15.23:0/437307589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 msgr2=0x7fc8e01a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 0x7fc8e01a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 -- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 msgr2=0x7fc8e00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 -- 172.21.15.23:0/437307589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8009c40 con 0x7fc8e0100f50 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8de800700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 0x7fc8e01a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:48.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8dde00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e9c0 tx=0x7fc8d000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8d6a00700 1 -- 172.21.15.23:0/437307589 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0004a30 con 0x7fc8e0100f50 2024-03-30T04:10:48.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8d6a00700 1 -- 172.21.15.23:0/437307589 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8d000cc80 con 0x7fc8e0100f50 2024-03-30T04:10:48.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8d6a00700 1 -- 172.21.15.23:0/437307589 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d00054e0 con 0x7fc8e0100f50 2024-03-30T04:10:48.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.365+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e01a6100 con 0x7fc8e0100f50 2024-03-30T04:10:48.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.366+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e01a65c0 con 0x7fc8e0100f50 2024-03-30T04:10:48.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.369+0000 7fc8d6a00700 1 -- 172.21.15.23:0/437307589 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc8d0004b90 con 0x7fc8e0100f50 2024-03-30T04:10:48.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.370+0000 7fc8f84fab80 1 Processor -- start 2024-03-30T04:10:48.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.370+0000 7fc8f84fab80 1 -- start start 2024-03-30T04:10:48.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8d6a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 0x7fc8c40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8d6a00700 1 -- 172.21.15.23:0/437307589 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc8d0014070 con 0x7fc8e0100f50 2024-03-30T04:10:48.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a8397b0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8396fba10 con 0x55a8397a2650 2024-03-30T04:10:48.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8396f93d0 con 0x55a8397a2f70 2024-03-30T04:10:48.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8396f4cc0 con 0x55a8397ade70 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:33998/0 (socket says 172.21.15.23:33998) 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8dd400700 1 -- 172.21.15.23:0/656596071 learned_addr learned my addr 172.21.15.23:0/656596071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.371+0000 7fc8de800700 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a8397b0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8de800700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 0x7fc8c40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dde00700 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 -- 172.21.15.23:0/656596071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 msgr2=0x55a8397b0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a8397b0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 -- 172.21.15.23:0/656596071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 msgr2=0x55a8397a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397a2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 -- 172.21.15.23:0/656596071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a839576430 con 0x55a8397a2f70 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8dd400700 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009cb0 tx=0x7fc8cc011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb895ef80153782 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8de800700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 0x7fc8c40a37d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80079a0 tx=0x7fc8d8009700 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.372+0000 7fc8cb000700 1 -- 172.21.15.23:0/656596071 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc011d20 con 0x55a8397a2f70 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8cb000700 1 -- 172.21.15.23:0/656596071 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8cc011e80 con 0x55a8397a2f70 2024-03-30T04:10:48.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8cb000700 1 -- 172.21.15.23:0/656596071 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc005280 con 0x55a8397a2f70 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 -- 172.21.15.23:0/656596071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 msgr2=0x55a8397ad930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009cb0 tx=0x7fc8cc011440 comp rx=0 tx=0).stop 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 -- 172.21.15.23:0/656596071 shutdown_connections 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a8397b0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/656596071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397ad930 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.373+0000 7fc8f84fab80 1 -- 172.21.15.23:0/656596071 >> 172.21.15.23:0/656596071 conn(0x55a839702720 msgr2=0x55a83979eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.374+0000 7fc8f84fab80 1 -- 172.21.15.23:0/656596071 shutdown_connections 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.374+0000 7fc8f84fab80 1 -- 172.21.15.23:0/656596071 wait complete. 2024-03-30T04:10:48.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.374+0000 7fc8f84fab80 1 -- start start 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.374+0000 7fc8f84fab80 1 Processor -- start 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 0x55a8397ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 0x55a8397aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a839690e00 con 0x55a8397a2f70 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a839690aa0 con 0x55a8397ade70 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a839578720 con 0x55a8397a2650 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8f84fab80 10 client.0 _ll_register_callbacks cb 0x55a83969a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8de800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 0x55a8397aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8de800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34012/0 (socket says 172.21.15.23:34012) 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.375+0000 7fc8de800700 1 -- 172.21.15.23:0/1101942465 learned_addr learned my addr 172.21.15.23:0/1101942465 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 -- 172.21.15.23:0/1101942465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 msgr2=0x55a8397ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8dde00700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 0x55a8397ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 0x55a8397ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 -- 172.21.15.23:0/1101942465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 msgr2=0x55a8397aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 0x55a8397aaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 -- 172.21.15.23:0/1101942465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc0098e0 con 0x55a8397ade70 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8dd400700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 0x55a8397aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8dde00700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 0x55a8397ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8de800700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8015000 tx=0x7fc8d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70f1e35eed47128f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8bf400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8026070 con 0x55a8397ade70 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc8d8007df0 con 0x55a8397ade70 2024-03-30T04:10:48.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8396ed780 con 0x55a8397ade70 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.376+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8d800e040 con 0x55a8397ade70 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 10 client.27995 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a839522be0 con 0x55a8397ade70 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8d8036020 con 0x55a8397ade70 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc8d801e950 con 0x55a8397ade70 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 10 client.27995 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 10 client.27995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 20 client.27995 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8397bd9c0 0x55a8397bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 4 client.27995 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:48.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.377+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a839543d70 con 0x55a8397bd9c0 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.378+0000 7fc8dde00700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8397bd9c0 0x55a8397bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.378+0000 7fc8bf400700 10 client.27995 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.378+0000 7fc8dde00700 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8397bd9c0 0x55a8397bfde0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d0e90df5a5aabc0 server_cookie=d2f8742e1f488830 in_seq=0 out_seq=0 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8bf400700 1 -- 172.21.15.23:0/1101942465 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a839543d70 con 0x55a8397bd9c0 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8bf400700 10 client.27995 handle_command_reply: tid=0 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 client.27995 shutdown 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 msgr2=0x55a8397ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc8d8015000 tx=0x7fc8d800b330 comp rx=0 tx=0).stop 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 shutdown_connections 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397ade70 0x55a8397ab090 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8397bd9c0 0x55a8397bfde0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2650 0x55a8397ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1101942465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2f70 0x55a8397aaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 >> 172.21.15.23:0/1101942465 conn(0x55a839702720 msgr2=0x55a8397a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8bf400700 0 client.27995 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.380+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 shutdown_connections 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.381+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1101942465 wait complete. 2024-03-30T04:10:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.381+0000 7fc8f84fab80 20 client.27995 trim_cache size 0 max 16384 2024-03-30T04:10:48.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.400+0000 7fc8f84fab80 1 Processor -- start 2024-03-30T04:10:48.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 -- start start 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2650 0x55a8397ab000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397ade70 0x55a8397b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8397b9e90 con 0x55a8397ade70 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8397a38d0 con 0x55a8397a2650 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8397ba530 con 0x55a8397a2f70 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35502/0 (socket says 172.21.15.23:35502) 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8dd400700 1 -- 172.21.15.23:0/1708330723 learned_addr learned my addr 172.21.15.23:0/1708330723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8de800700 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397ade70 0x55a8397b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.401+0000 7fc8dde00700 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2650 0x55a8397ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 -- 172.21.15.23:0/1708330723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2650 msgr2=0x55a8397ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2650 0x55a8397ab000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 -- 172.21.15.23:0/1708330723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397ade70 msgr2=0x55a8397b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397ade70 0x55a8397b4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 -- 172.21.15.23:0/1708330723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8396baf20 con 0x55a8397a2f70 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8dd400700 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011790 tx=0x7fc8cc004b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27a2c001bd4cf0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8cba00700 1 -- 172.21.15.23:0/1708330723 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc011b20 con 0x55a8397a2f70 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8cba00700 1 -- 172.21.15.23:0/1708330723 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8cc011c80 con 0x55a8397a2f70 2024-03-30T04:10:48.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.402+0000 7fc8cba00700 1 -- 172.21.15.23:0/1708330723 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc003e70 con 0x55a8397a2f70 2024-03-30T04:10:48.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1708330723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 msgr2=0x55a8397b9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc011790 tx=0x7fc8cc004b00 comp rx=0 tx=0).stop 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1708330723 shutdown_connections 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397ade70 0x55a8397b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2650 0x55a8397ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/1708330723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397a2f70 0x55a8397b9950 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1708330723 >> 172.21.15.23:0/1708330723 conn(0x55a839702720 msgr2=0x55a8397b9240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1708330723 shutdown_connections 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.403+0000 7fc8f84fab80 1 -- 172.21.15.23:0/1708330723 wait complete. 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.404+0000 7fc8f84fab80 1 -- start start 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.404+0000 7fc8f84fab80 1 Processor -- start 2024-03-30T04:10:48.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.404+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.404+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.404+0000 7fc8f84fab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a83980b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83980bd80 con 0x55a8397a2650 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83980bef0 con 0x55a8397a2f70 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8f84fab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a83980c0b0 con 0x55a8397ade70 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8f84fab80 10 client.0 _ll_register_callbacks cb 0x55a83969a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dde00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34030/0 (socket says 172.21.15.23:34030) 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 -- 172.21.15.23:0/169332971 learned_addr learned my addr 172.21.15.23:0/169332971 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8de800700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a83980b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 -- 172.21.15.23:0/169332971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 msgr2=0x55a83980b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a83980b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 -- 172.21.15.23:0/169332971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 msgr2=0x55a8397b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dd400700 1 -- 172.21.15.23:0/169332971 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8cc01d040 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.405+0000 7fc8dde00700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8dd400700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009bd0 tx=0x7fc8cc003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f5e4068cc4088 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8bf400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc032070 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a8396b5400 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc8cc004050 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8cc011b20 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8f84fab80 10 client.28001 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a83957ad50 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.406+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a8396b5400 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc8cc035020 con 0x55a8397a2f70 2024-03-30T04:10:48.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 10 client.28001 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 10 client.28001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 20 client.28001 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8398136b0 0x55a839815ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 4 client.28001 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a839578d60 con 0x55a8398136b0 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8dde00700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8398136b0 0x55a839815ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8dde00700 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8398136b0 0x55a839815ad0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb92c5615370c047 server_cookie=528ad67967e386e3 in_seq=0 out_seq=0 2024-03-30T04:10:48.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.407+0000 7fc8bf400700 10 client.28001 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.408+0000 7fc8bf400700 1 -- 172.21.15.23:0/169332971 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a839578d60 con 0x55a8398136b0 2024-03-30T04:10:48.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.408+0000 7fc8bf400700 10 client.28001 handle_command_reply: tid=0 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 client.28001 shutdown 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 msgr2=0x55a8397b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009bd0 tx=0x7fc8cc003e00 comp rx=0 tx=0).stop 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 shutdown_connections 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55a8398136b0 0x55a839815ad0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a8397ade70 0x55a83980b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a8397a2650 0x55a8397b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 --2- 172.21.15.23:0/169332971 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a8397a2f70 0x55a8397b4130 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8bf400700 0 client.28001 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 >> 172.21.15.23:0/169332971 conn(0x55a839702720 msgr2=0x55a8397a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 shutdown_connections 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 1 -- 172.21.15.23:0/169332971 wait complete. 2024-03-30T04:10:48.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.409+0000 7fc8f84fab80 20 client.28001 trim_cache size 0 max 16384 2024-03-30T04:10:48.413 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:48.413 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:10:48.413 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "c93a953a-45a0-4c74-8a4c-31b16351ed40", 2024-03-30T04:10:48.413 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:10:48.413 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 msgr2=0x7fc8c40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 0x7fc8c40a37d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80079a0 tx=0x7fc8d8009700 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 msgr2=0x7fc8e00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e9c0 tx=0x7fc8d000ecd0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 shutdown_connections 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc8c40a1310 0x7fc8c40a37d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8e0073b30 0x7fc8e01a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8e0100f50 0x7fc8e00730b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 --2- 172.21.15.23:0/437307589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc8e0106070 0x7fc8e00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 >> 172.21.15.23:0/437307589 conn(0x7fc8e00fc7f0 msgr2=0x7fc8e00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 shutdown_connections 2024-03-30T04:10:48.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.410+0000 7fc8e4a00700 1 -- 172.21.15.23:0/437307589 wait complete. 2024-03-30T04:10:48.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:48 smithi023 ceph-mon[38254]: pgmap v274: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2024-03-30T04:10:48.429 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:10:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:48 smithi154 ceph-mon[79161]: pgmap v274: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2024-03-30T04:10:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:48 smithi193 ceph-mon[79581]: pgmap v274: 129 pgs: 129 active+clean; 486 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 161 op/s 2024-03-30T04:10:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 -- 172.21.15.23:0/458733556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680100f70 msgr2=0x7f9680101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 --2- 172.21.15.23:0/458733556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680100f70 0x7f9680101350 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9670009990 tx=0x7f9670009ca0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 -- 172.21.15.23:0/458733556 shutdown_connections 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 --2- 172.21.15.23:0/458733556 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f968010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 --2- 172.21.15.23:0/458733556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9680100f70 0x7f9680101350 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 --2- 172.21.15.23:0/458733556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680101920 0x7f9680105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 -- 172.21.15.23:0/458733556 >> 172.21.15.23:0/458733556 conn(0x7f96800fc7f0 msgr2=0x7f96800fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 -- 172.21.15.23:0/458733556 shutdown_connections 2024-03-30T04:10:48.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.650+0000 7f9687200700 1 -- 172.21.15.23:0/458733556 wait complete. 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.651+0000 7f9687200700 1 Processor -- start 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 -- start start 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 0x7f9680199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 0x7f968019e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019e750 con 0x7f9680106090 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019e8c0 con 0x7f968019dcd0 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9687200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968019ea80 con 0x7f9680100f70 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f967f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f967f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53980/0 (socket says 172.21.15.23:53980) 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f967f400700 1 -- 172.21.15.23:0/1872267885 learned_addr learned my addr 172.21.15.23:0/1872267885 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f967fe00700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 0x7f9680199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.652+0000 7f9685200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 0x7f968019e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 -- 172.21.15.23:0/1872267885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 msgr2=0x7f968019e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 0x7f968019e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 -- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 msgr2=0x7f9680199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 -- 172.21.15.23:0/1872267885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96700095f0 con 0x7f9680100f70 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967f400700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:48.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f9685200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 0x7f968019e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967fe00700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 0x7f9680199440 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9670000c00 tx=0x7f9670004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967cc00700 1 -- 172.21.15.23:0/1872267885 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967001d070 con 0x7f9680100f70 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967cc00700 1 -- 172.21.15.23:0/1872267885 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9670021410 con 0x7f9680100f70 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96801a5f80 con 0x7f9680100f70 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f967cc00700 1 -- 172.21.15.23:0/1872267885 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670021ea0 con 0x7f9680100f70 2024-03-30T04:10:48.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.653+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96801a63b0 con 0x7f9680100f70 2024-03-30T04:10:48.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.656+0000 7f967cc00700 1 -- 172.21.15.23:0/1872267885 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9670021580 con 0x7f9680100f70 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.658+0000 7f967cc00700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 0x7f96640a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.658+0000 7f969ac9db80 1 Processor -- start 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.658+0000 7f967cc00700 1 -- 172.21.15.23:0/1872267885 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f96700cdfa0 con 0x7f9680100f70 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.658+0000 7f969ac9db80 1 -- start start 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.658+0000 7f967f400700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 0x7f96640a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e58ef0 0x55f235e638b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e661e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235db1990 con 0x55f235e58ef0 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235daf350 con 0x55f235e585d0 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f969ac9db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235daac40 con 0x55f235e63df0 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34050/0 (socket says 172.21.15.23:34050) 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f967fe00700 1 -- 172.21.15.23:0/3577380448 learned_addr learned my addr 172.21.15.23:0/3577380448 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f9685200700 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f967f400700 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e58ef0 0x55f235e638b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.659+0000 7f967fe00700 1 -- 172.21.15.23:0/3577380448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 msgr2=0x55f235e661e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967fe00700 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e661e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967fe00700 1 -- 172.21.15.23:0/3577380448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e58ef0 msgr2=0x55f235e638b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967fe00700 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e58ef0 0x55f235e638b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967f400700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 0x7f96640a37d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f968019a940 tx=0x7f966c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967fe00700 1 -- 172.21.15.23:0/3577380448 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f235c2c3b0 con 0x55f235e585d0 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f967fe00700 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9670097200 tx=0x7f967009aad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585360cbb2a3c02b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f9675600700 1 -- 172.21.15.23:0/3577380448 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967009b400 con 0x55f235e585d0 2024-03-30T04:10:48.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f9675600700 1 -- 172.21.15.23:0/3577380448 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f967009ad00 con 0x55f235e585d0 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.660+0000 7f9675600700 1 -- 172.21.15.23:0/3577380448 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967009b180 con 0x55f235e585d0 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 -- 172.21.15.23:0/3577380448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 msgr2=0x55f235e589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9670097200 tx=0x7f967009aad0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 -- 172.21.15.23:0/3577380448 shutdown_connections 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e661e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e589b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3577380448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e58ef0 0x55f235e638b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 -- 172.21.15.23:0/3577380448 >> 172.21.15.23:0/3577380448 conn(0x55f235db86a0 msgr2=0x55f235e54a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 -- 172.21.15.23:0/3577380448 shutdown_connections 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.661+0000 7f969ac9db80 1 -- 172.21.15.23:0/3577380448 wait complete. 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 -- start start 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 Processor -- start 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e60b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235d46d80 con 0x55f235e63df0 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235d46a20 con 0x55f235e585d0 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235c2e6a0 con 0x55f235e58ef0 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.662+0000 7f969ac9db80 10 client.0 _ll_register_callbacks cb 0x55f235d50470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34066/0 (socket says 172.21.15.23:34066) 2024-03-30T04:10:48.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f9685200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 -- 172.21.15.23:0/3728063943 learned_addr learned my addr 172.21.15.23:0/3728063943 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967f400700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e60b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 -- 172.21.15.23:0/3728063943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 msgr2=0x55f235e60b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e60b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 -- 172.21.15.23:0/3728063943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 msgr2=0x55f235e610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 -- 172.21.15.23:0/3728063943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96801a63b0 con 0x55f235e585d0 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f967fe00700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9670009990 tx=0x7f967009a720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1be04f87d7b4c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.663+0000 7f966a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:48.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96700b7070 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f967009a400 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f235d6b380 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f967009b400 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 10 client.28013 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f235bd8b60 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f235d6b380 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f967000fb10 con 0x55f235e585d0 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 10 client.28013 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 10 client.28013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:48.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 20 client.28013 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235e73100 0x55f235e75520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 4 client.28013 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.664+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f235da3700 con 0x55f235e73100 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.665+0000 7f967f400700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235e73100 0x55f235e75520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.665+0000 7f967f400700 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235e73100 0x55f235e75520 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=645c47babb62da99 server_cookie=717084a1de66f96f in_seq=0 out_seq=0 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.665+0000 7f966a000700 10 client.28013 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f966a000700 1 -- 172.21.15.23:0/3728063943 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f235da3700 con 0x55f235e73100 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f966a000700 10 client.28013 handle_command_reply: tid=0 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 client.28013 shutdown 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 msgr2=0x55f235e63770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9670009990 tx=0x7f967009a720 comp rx=0 tx=0).stop 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 shutdown_connections 2024-03-30T04:10:48.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e63770 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235e73100 0x55f235e75520 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3728063943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e60b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f966a000700 0 client.28013 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.666+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 >> 172.21.15.23:0/3728063943 conn(0x55f235db86a0 msgr2=0x55f235e5a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.667+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 shutdown_connections 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.667+0000 7f969ac9db80 1 -- 172.21.15.23:0/3728063943 wait complete. 2024-03-30T04:10:48.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.667+0000 7f969ac9db80 20 client.28013 trim_cache size 0 max 16384 2024-03-30T04:10:48.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 Processor -- start 2024-03-30T04:10:48.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 -- start start 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e6a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e5f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e5f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235e70130 con 0x55f235e63df0 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235e5b960 con 0x55f235e585d0 2024-03-30T04:10:48.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.686+0000 7f969ac9db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235e5b4a0 con 0x55f235e58ef0 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e6a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e5f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e6a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34078/0 (socket says 172.21.15.23:34078) 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967fe00700 1 -- 172.21.15.23:0/3322984033 learned_addr learned my addr 172.21.15.23:0/3322984033 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f9685200700 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e5f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 -- 172.21.15.23:0/3322984033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 msgr2=0x55f235e6a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e6a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 -- 172.21.15.23:0/3322984033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 msgr2=0x55f235e5f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e5f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 -- 172.21.15.23:0/3322984033 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f235d70ea0 con 0x55f235e58ef0 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f967f400700 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e5f3f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55f235e75d90 tx=0x7f966c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec2ec24033005cb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f9676000700 1 -- 172.21.15.23:0/3322984033 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c023040 con 0x55f235e58ef0 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f9676000700 1 -- 172.21.15.23:0/3322984033 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f966c029410 con 0x55f235e58ef0 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.687+0000 7f9676000700 1 -- 172.21.15.23:0/3322984033 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c01e720 con 0x55f235e58ef0 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 -- 172.21.15.23:0/3322984033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 msgr2=0x55f235e5f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e5f3f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x55f235e75d90 tx=0x7f966c014000 comp rx=0 tx=0).stop 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 -- 172.21.15.23:0/3322984033 shutdown_connections 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e63df0 0x55f235e5f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e585d0 0x55f235e6a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 --2- 172.21.15.23:0/3322984033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e58ef0 0x55f235e5f3f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 -- 172.21.15.23:0/3322984033 >> 172.21.15.23:0/3322984033 conn(0x55f235db86a0 msgr2=0x55f235e4d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 -- 172.21.15.23:0/3322984033 shutdown_connections 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.688+0000 7f969ac9db80 1 -- 172.21.15.23:0/3322984033 wait complete. 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 -- start start 2024-03-30T04:10:48.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 Processor -- start 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e585d0 0x55f235e6a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e69260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235e699c0 con 0x55f235e585d0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235e69b30 con 0x55f235e58ef0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f235ec6f00 con 0x55f235e63df0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.689+0000 7f969ac9db80 10 client.0 _ll_register_callbacks cb 0x55f235d50470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e585d0 0x55f235e6a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:34092/0 (socket says 172.21.15.23:34092) 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 -- 172.21.15.23:0/4040478976 learned_addr learned my addr 172.21.15.23:0/4040478976 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f9685200700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e69260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 -- 172.21.15.23:0/4040478976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 msgr2=0x55f235e69260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e69260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 -- 172.21.15.23:0/4040478976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e585d0 msgr2=0x55f235e6a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e585d0 0x55f235e6a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 -- 172.21.15.23:0/4040478976 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966c017070 con 0x55f235e58ef0 2024-03-30T04:10:48.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f967f400700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x55f235e734e0 tx=0x7f966c009ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=783b51a11bef0e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.690+0000 7f966a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966c023040 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f966c003880 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f967009ad00 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f966c029e10 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f969ac9db80 10 client.28019 fetch_fsmap learned FSMap version 30 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f235c30cd0 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f967009ad00 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.691+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f966c033c70 con 0x55f235e58ef0 2024-03-30T04:10:48.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 10 client.28019 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 10 client.28019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 20 client.28019 populate_metadata read hostname 'smithi023' 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235ec85d0 0x55f235eca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 4 client.28019 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f235bf9cf0 con 0x55f235ec85d0 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f967fe00700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235ec85d0 0x55f235eca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f967fe00700 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235ec85d0 0x55f235eca9f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4321f603c8e396c server_cookie=18cda0f8e4d04e12 in_seq=0 out_seq=0 2024-03-30T04:10:48.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.692+0000 7f966a000700 10 client.28019 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f966a000700 1 -- 172.21.15.23:0/4040478976 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f235bf9cf0 con 0x55f235ec85d0 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f966a000700 10 client.28019 handle_command_reply: tid=0 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 client.28019 shutdown 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 msgr2=0x55f235e68d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x55f235e734e0 tx=0x7f966c009ec0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 shutdown_connections 2024-03-30T04:10:48.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f235ec85d0 0x55f235eca9f0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f235e63df0 0x55f235e69260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f966a000700 0 client.28019 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f235e585d0 0x55f235e6a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 --2- 172.21.15.23:0/4040478976 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f235e58ef0 0x55f235e68d00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 >> 172.21.15.23:0/4040478976 conn(0x55f235db86a0 msgr2=0x55f235e6ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 shutdown_connections 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 1 -- 172.21.15.23:0/4040478976 wait complete. 2024-03-30T04:10:48.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.693+0000 7f969ac9db80 20 client.28019 trim_cache size 0 max 16384 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout: "c93a953a-45a0-4c74-8a4c-31b16351ed40": { 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:10:48.700 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "c93a953a-45a0-4c74-8a4c-31b16351ed40", 2024-03-30T04:10:48.701 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:10:48.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:10:48.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:10:48.701 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:10:48.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 msgr2=0x7f96640a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 0x7f96640a37d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f968019a940 tx=0x7f966c009260 comp rx=0 tx=0).stop 2024-03-30T04:10:48.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 msgr2=0x7f9680199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:10:48.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 0x7f9680199440 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9670000c00 tx=0x7f9670004730 comp rx=0 tx=0).stop 2024-03-30T04:10:48.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 shutdown_connections 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f96640a1310 0x7f96640a37d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f968019dcd0 0x7f968019e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9680100f70 0x7f9680199440 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 --2- 172.21.15.23:0/1872267885 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9680106090 0x7f9680199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 >> 172.21.15.23:0/1872267885 conn(0x7f96800fc7f0 msgr2=0x7f96800fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 shutdown_connections 2024-03-30T04:10:48.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:10:48.694+0000 7f9687200700 1 -- 172.21.15.23:0/1872267885 wait complete. 2024-03-30T04:10:48.715 INFO:tasks.cephfs.filesystem:scrub status for tag:c93a953a-45a0-4c74-8a4c-31b16351ed40 - {'path': '/', 'tag': 'c93a953a-45a0-4c74-8a4c-31b16351ed40', 'options': 'recursive,force'} 2024-03-30T04:10:49.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 884495 10.42 MB/sec execute 326 sec latency 13.282 ms 2024-03-30T04:10:49.434 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:49 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:10:49.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:49 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:49 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:10:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:49 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:49 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:10:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:49 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 885932 10.41 MB/sec execute 327 sec latency 45.646 ms 2024-03-30T04:10:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:50 smithi154 ceph-mon[79161]: pgmap v275: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 244 op/s 2024-03-30T04:10:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:50 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:10:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:50 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:50 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:10:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:50 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:50 smithi193 ceph-mon[79581]: pgmap v275: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 244 op/s 2024-03-30T04:10:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:50 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:10:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:50 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:50 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:10:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:50 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:50 smithi023 ceph-mon[38254]: pgmap v275: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 244 op/s 2024-03-30T04:10:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:50 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:10:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:50 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:50 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:10:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:50 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:10:51.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 888006 10.42 MB/sec execute 328 sec latency 20.824 ms 2024-03-30T04:10:52.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 889657 10.41 MB/sec execute 329 sec latency 23.359 ms 2024-03-30T04:10:52.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:52 smithi154 ceph-mon[79161]: pgmap v276: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-03-30T04:10:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:52 smithi193 ceph-mon[79581]: pgmap v276: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-03-30T04:10:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:52 smithi023 ceph-mon[38254]: pgmap v276: 129 pgs: 129 active+clean; 491 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 164 op/s 2024-03-30T04:10:53.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 891667 10.41 MB/sec execute 330 sec latency 16.089 ms 2024-03-30T04:10:54.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 893512 10.40 MB/sec execute 331 sec latency 22.289 ms 2024-03-30T04:10:54.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:54 smithi154 ceph-mon[79161]: pgmap v277: 129 pgs: 129 active+clean; 491 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 226 op/s 2024-03-30T04:10:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:54 smithi193 ceph-mon[79581]: pgmap v277: 129 pgs: 129 active+clean; 491 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 226 op/s 2024-03-30T04:10:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:54 smithi023 ceph-mon[38254]: pgmap v277: 129 pgs: 129 active+clean; 491 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.1 MiB/s wr, 226 op/s 2024-03-30T04:10:55.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 895163 10.41 MB/sec execute 332 sec latency 78.440 ms 2024-03-30T04:10:56.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 896636 10.39 MB/sec execute 333 sec latency 29.481 ms 2024-03-30T04:10:56.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:56 smithi154 ceph-mon[79161]: pgmap v278: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-30T04:10:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:56 smithi193 ceph-mon[79581]: pgmap v278: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-30T04:10:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:56 smithi023 ceph-mon[38254]: pgmap v278: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 171 op/s 2024-03-30T04:10:57.217 INFO:tasks.workunit.client.0.smithi023.stdout: 1 898685 10.40 MB/sec execute 334 sec latency 17.188 ms 2024-03-30T04:10:58.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 900499 10.39 MB/sec execute 335 sec latency 21.961 ms 2024-03-30T04:10:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:58 smithi154 ceph-mon[79161]: pgmap v279: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 167 op/s 2024-03-30T04:10:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:58 smithi193 ceph-mon[79581]: pgmap v279: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 167 op/s 2024-03-30T04:10:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:58 smithi023 ceph-mon[38254]: pgmap v279: 129 pgs: 129 active+clean; 495 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.0 MiB/s wr, 167 op/s 2024-03-30T04:10:59.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 902341 10.40 MB/sec execute 336 sec latency 11.544 ms 2024-03-30T04:10:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:10:59.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:10:59 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:10:59.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:10:59.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:10:59 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:10:59.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:10:59.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:10:59 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:11:00.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 903840 10.39 MB/sec execute 337 sec latency 56.930 ms 2024-03-30T04:11:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:00 smithi154 ceph-mon[79161]: pgmap v280: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-03-30T04:11:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:00 smithi193 ceph-mon[79581]: pgmap v280: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-03-30T04:11:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:00 smithi023 ceph-mon[38254]: pgmap v280: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.6 MiB/s wr, 253 op/s 2024-03-30T04:11:01.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 905002 10.38 MB/sec execute 338 sec latency 77.051 ms 2024-03-30T04:11:02.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 905764 10.36 MB/sec execute 339 sec latency 73.220 ms 2024-03-30T04:11:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:02 smithi154 ceph-mon[79161]: pgmap v281: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-30T04:11:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:02 smithi193 ceph-mon[79581]: pgmap v281: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-30T04:11:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:02 smithi023 ceph-mon[38254]: pgmap v281: 129 pgs: 129 active+clean; 500 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.0 MiB/s wr, 169 op/s 2024-03-30T04:11:03.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 907269 10.35 MB/sec execute 340 sec latency 33.092 ms 2024-03-30T04:11:04.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 909262 10.36 MB/sec execute 341 sec latency 19.005 ms 2024-03-30T04:11:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:04 smithi154 ceph-mon[79161]: pgmap v282: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 257 op/s 2024-03-30T04:11:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:04 smithi193 ceph-mon[79581]: pgmap v282: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 257 op/s 2024-03-30T04:11:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:04 smithi023 ceph-mon[38254]: pgmap v282: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 257 op/s 2024-03-30T04:11:05.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 910534 10.34 MB/sec execute 342 sec latency 60.930 ms 2024-03-30T04:11:06.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 912612 10.35 MB/sec execute 343 sec latency 47.064 ms 2024-03-30T04:11:06.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:06 smithi154 ceph-mon[79161]: pgmap v283: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 213 op/s 2024-03-30T04:11:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:06 smithi193 ceph-mon[79581]: pgmap v283: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 213 op/s 2024-03-30T04:11:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:06 smithi023 ceph-mon[38254]: pgmap v283: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.3 MiB/s wr, 213 op/s 2024-03-30T04:11:07.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 913887 10.34 MB/sec execute 344 sec latency 22.220 ms 2024-03-30T04:11:08.218 INFO:tasks.workunit.client.0.smithi023.stdout: 1 915927 10.34 MB/sec execute 345 sec latency 34.956 ms 2024-03-30T04:11:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:08 smithi154 ceph-mon[79161]: pgmap v284: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:11:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:08 smithi193 ceph-mon[79581]: pgmap v284: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:11:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:08 smithi023 ceph-mon[38254]: pgmap v284: 129 pgs: 129 active+clean; 503 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.0 MiB/s wr, 192 op/s 2024-03-30T04:11:09.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 916817 10.34 MB/sec execute 346 sec latency 68.219 ms 2024-03-30T04:11:10.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 917387 10.35 MB/sec execute 347 sec latency 211.326 ms 2024-03-30T04:11:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:10 smithi193 ceph-mon[79581]: pgmap v285: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-30T04:11:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:10 smithi193 ceph-mon[79581]: osdmap e96: 12 total, 12 up, 12 in 2024-03-30T04:11:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:10 smithi023 ceph-mon[38254]: pgmap v285: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-30T04:11:10.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:10 smithi023 ceph-mon[38254]: osdmap e96: 12 total, 12 up, 12 in 2024-03-30T04:11:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:10 smithi154 ceph-mon[79161]: pgmap v285: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.7 MiB/s wr, 275 op/s 2024-03-30T04:11:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:10 smithi154 ceph-mon[79161]: osdmap e96: 12 total, 12 up, 12 in 2024-03-30T04:11:11.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 918291 10.34 MB/sec execute 348 sec latency 433.212 ms 2024-03-30T04:11:12.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 919657 10.33 MB/sec execute 349 sec latency 49.219 ms 2024-03-30T04:11:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:12 smithi154 ceph-mon[79161]: pgmap v287: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 227 op/s 2024-03-30T04:11:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:12 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:11:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:12 smithi193 ceph-mon[79581]: pgmap v287: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 227 op/s 2024-03-30T04:11:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:12 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:11:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:12 smithi023 ceph-mon[38254]: pgmap v287: 129 pgs: 129 active+clean; 510 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.8 MiB/s wr, 227 op/s 2024-03-30T04:11:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:12 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:11:13.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 921240 10.32 MB/sec execute 350 sec latency 23.813 ms 2024-03-30T04:11:13.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:13 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:11:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:13 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:11:13.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:13 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:11:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:13 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:11:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:13 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:11:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:13 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:11:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:13 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:11:13.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:13 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:11:13.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:13 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:11:14.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 922654 10.31 MB/sec execute 351 sec latency 228.771 ms 2024-03-30T04:11:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:14 smithi154 ceph-mon[79161]: pgmap v288: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 314 op/s 2024-03-30T04:11:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:14 smithi193 ceph-mon[79581]: pgmap v288: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 314 op/s 2024-03-30T04:11:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:14 smithi023 ceph-mon[38254]: pgmap v288: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 314 op/s 2024-03-30T04:11:15.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 924250 10.31 MB/sec execute 352 sec latency 58.767 ms 2024-03-30T04:11:16.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 925449 10.30 MB/sec execute 353 sec latency 30.339 ms 2024-03-30T04:11:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:16 smithi154 ceph-mon[79161]: pgmap v289: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.6 MiB/s wr, 309 op/s 2024-03-30T04:11:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:16 smithi154 ceph-mon[79161]: osdmap e97: 12 total, 12 up, 12 in 2024-03-30T04:11:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:16 smithi193 ceph-mon[79581]: pgmap v289: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.6 MiB/s wr, 309 op/s 2024-03-30T04:11:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:16 smithi193 ceph-mon[79581]: osdmap e97: 12 total, 12 up, 12 in 2024-03-30T04:11:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:16 smithi023 ceph-mon[38254]: pgmap v289: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 7.6 MiB/s wr, 309 op/s 2024-03-30T04:11:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:16 smithi023 ceph-mon[38254]: osdmap e97: 12 total, 12 up, 12 in 2024-03-30T04:11:17.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 927575 10.31 MB/sec execute 354 sec latency 28.199 ms 2024-03-30T04:11:18.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 928986 10.29 MB/sec execute 355 sec latency 25.048 ms 2024-03-30T04:11:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:18 smithi154 ceph-mon[79161]: pgmap v291: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 261 op/s 2024-03-30T04:11:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:18 smithi193 ceph-mon[79581]: pgmap v291: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 261 op/s 2024-03-30T04:11:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:18 smithi023 ceph-mon[38254]: pgmap v291: 129 pgs: 129 active+clean; 504 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.9 MiB/s wr, 261 op/s 2024-03-30T04:11:18.716 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:11:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 -- 172.21.15.23:0/2163631388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78106080 msgr2=0x7f7c7810dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 --2- 172.21.15.23:0/2163631388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78106080 0x7f7c7810dc00 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000b0a0 tx=0x7f7c7000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 -- 172.21.15.23:0/2163631388 shutdown_connections 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 --2- 172.21.15.23:0/2163631388 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78106080 0x7f7c7810dc00 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 --2- 172.21.15.23:0/2163631388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78100f60 0x7f7c78101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 --2- 172.21.15.23:0/2163631388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c78101910 0x7f7c78105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 -- 172.21.15.23:0/2163631388 >> 172.21.15.23:0/2163631388 conn(0x7f7c780fc7c0 msgr2=0x7f7c780febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 -- 172.21.15.23:0/2163631388 shutdown_connections 2024-03-30T04:11:18.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.939+0000 7f7c7f800700 1 -- 172.21.15.23:0/2163631388 wait complete. 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 Processor -- start 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 -- start start 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 0x7f7c781998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c7819dc90 0x7f7c7819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819e710 con 0x7f7c78106080 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819e880 con 0x7f7c78100f60 2024-03-30T04:11:18.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.941+0000 7f7c7f800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819ea40 con 0x7f7c7819dc90 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 0x7f7c781998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7d800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c7819dc90 0x7f7c7819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38040/0 (socket says 172.21.15.23:38040) 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1662605425 learned_addr learned my addr 172.21.15.23:0/1662605425 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1662605425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c7819dc90 msgr2=0x7f7c7819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c7819dc90 0x7f7c7819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 msgr2=0x7f7c781998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 0x7f7c781998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1662605425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70009d00 con 0x7f7c78100f60 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c77e00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 0x7f7c781998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00e9a0 tx=0x7f7c6c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c75600700 1 -- 172.21.15.23:0/1662605425 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c009790 con 0x7f7c78100f60 2024-03-30T04:11:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c75600700 1 -- 172.21.15.23:0/1662605425 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6c010430 con 0x7f7c78100f60 2024-03-30T04:11:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.942+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c781a5fa0 con 0x7f7c78100f60 2024-03-30T04:11:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.943+0000 7f7c75600700 1 -- 172.21.15.23:0/1662605425 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c018680 con 0x7f7c78100f60 2024-03-30T04:11:18.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.943+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c781a6470 con 0x7f7c78100f60 2024-03-30T04:11:18.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.945+0000 7f7c75600700 1 -- 172.21.15.23:0/1662605425 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f7c6c0105a0 con 0x7f7c78100f60 2024-03-30T04:11:18.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.948+0000 7f7c75600700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 0x7f7c600a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.948+0000 7f7c75600700 1 -- 172.21.15.23:0/1662605425 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7c6c014070 con 0x7f7c78100f60 2024-03-30T04:11:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.948+0000 7f7c77e00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 0x7f7c600a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.948+0000 7f7c93420b80 1 Processor -- start 2024-03-30T04:11:18.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.948+0000 7f7c93420b80 1 -- start start 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3bef0 0x55e99af468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c77e00700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 0x7f7c600a37d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c7819a900 tx=0x7f7c6400b410 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af491e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae94990 con 0x55e99af3b5d0 2024-03-30T04:11:18.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae92350 con 0x55e99af3bef0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c93420b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae8dc40 con 0x55e99af46df0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c77e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3bef0 0x55e99af468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42244/0 (socket says 172.21.15.23:42244) 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1961348476 learned_addr learned my addr 172.21.15.23:0/1961348476 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1961348476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 msgr2=0x55e99af491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1961348476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3bef0 msgr2=0x55e99af468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3bef0 0x55e99af468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7ce00700 1 -- 172.21.15.23:0/1961348476 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e99ad0f3b0 con 0x55e99af3b5d0 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.949+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:18.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.950+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0969a0 tx=0x7f7c6c099ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511b21c4d420ebbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.950+0000 7f7c6a000700 1 -- 172.21.15.23:0/1961348476 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c09dd60 con 0x55e99af3b5d0 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.950+0000 7f7c6a000700 1 -- 172.21.15.23:0/1961348476 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6c0a23b0 con 0x55e99af3b5d0 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.950+0000 7f7c6a000700 1 -- 172.21.15.23:0/1961348476 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c099420 con 0x55e99af3b5d0 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- 172.21.15.23:0/1961348476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 msgr2=0x55e99af3b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0969a0 tx=0x7f7c6c099ba0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- 172.21.15.23:0/1961348476 shutdown_connections 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3b5d0 0x55e99af3b9b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1961348476 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3bef0 0x55e99af468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- 172.21.15.23:0/1961348476 >> 172.21.15.23:0/1961348476 conn(0x55e99ae9b6a0 msgr2=0x55e99af37a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- 172.21.15.23:0/1961348476 shutdown_connections 2024-03-30T04:11:18.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- 172.21.15.23:0/1961348476 wait complete. 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.951+0000 7f7c93420b80 1 -- start start 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 Processor -- start 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af43b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae29d80 con 0x55e99af3bef0 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae29a20 con 0x55e99af46df0 2024-03-30T04:11:18.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ad116a0 con 0x55e99af3b5d0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c93420b80 10 client.0 _ll_register_callbacks cb 0x55e99ae33470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c7d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af43b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c7d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38058/0 (socket says 172.21.15.23:38058) 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c7d800700 1 -- 172.21.15.23:0/1215997339 learned_addr learned my addr 172.21.15.23:0/1215997339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.952+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41746/0 (socket says 172.21.15.23:41746) 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 -- 172.21.15.23:0/1215997339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 msgr2=0x55e99af46860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 -- 172.21.15.23:0/1215997339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 msgr2=0x55e99af43b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af43b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 -- 172.21.15.23:0/1215997339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7810e470 con 0x55e99af46df0 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:18.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c77e00700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af43b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000b0a0 tx=0x7f7c7000bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53b710d6e08dd6c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c5ea00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.953+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000e040 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e99ae4e380 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7c70003c00 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c70003e00 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 10 client.28031 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f7c781a6470 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e99ae4e380 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7c70019070 con 0x55e99af46df0 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 10 client.28031 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 10 client.28031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 20 client.28031 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:18.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99af56aa0 0x55e99af58ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 4 client.28031 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.954+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e99ae86700 con 0x55e99af56aa0 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.955+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99af56aa0 0x55e99af58ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.955+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99af56aa0 0x55e99af58ec0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=752dddffd7ddec71 server_cookie=a38d65ae21ea2340 in_seq=0 out_seq=0 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.955+0000 7f7c5ea00700 10 client.28031 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1215997339 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e99ae86700 con 0x55e99af56aa0 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c5ea00700 10 client.28031 handle_command_reply: tid=0 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 client.28031 shutdown 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 msgr2=0x55e99af44240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000b0a0 tx=0x7f7c7000bd50 comp rx=0 tx=0).stop 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 shutdown_connections 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af46df0 0x55e99af44240 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99af56aa0 0x55e99af58ec0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af3b5d0 0x55e99af46860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1215997339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af43b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c5ea00700 0 client.28031 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 >> 172.21.15.23:0/1215997339 conn(0x55e99ae9b6a0 msgr2=0x55e99af3d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 shutdown_connections 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.956+0000 7f7c93420b80 1 -- 172.21.15.23:0/1215997339 wait complete. 2024-03-30T04:11:18.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.957+0000 7f7c93420b80 20 client.28031 trim_cache size 0 max 16384 2024-03-30T04:11:18.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 Processor -- start 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 -- start start 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af42440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af42980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99af3cf90 con 0x55e99af3bef0 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99ae86730 con 0x55e99af3b5d0 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c93420b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99af3c850 con 0x55e99af46df0 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af42440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c7ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38060/0 (socket says 172.21.15.23:38060) 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c7ce00700 1 -- 172.21.15.23:0/741679506 learned_addr learned my addr 172.21.15.23:0/741679506 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:18.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.975+0000 7f7c7d800700 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af42980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 -- 172.21.15.23:0/741679506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 msgr2=0x55e99af42980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af42980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 -- 172.21.15.23:0/741679506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 msgr2=0x55e99af42440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af42440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 -- 172.21.15.23:0/741679506 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e99acbbb60 con 0x55e99af3b5d0 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c099ec0 tx=0x7f7c6c099a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee9ae1ed3a655c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c6aa00700 1 -- 172.21.15.23:0/741679506 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0982d0 con 0x55e99af3b5d0 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c6aa00700 1 -- 172.21.15.23:0/741679506 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6c010880 con 0x55e99af3b5d0 2024-03-30T04:11:18.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.976+0000 7f7c6aa00700 1 -- 172.21.15.23:0/741679506 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0a1f20 con 0x55e99af3b5d0 2024-03-30T04:11:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- 172.21.15.23:0/741679506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 msgr2=0x55e99af4da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c099ec0 tx=0x7f7c6c099a70 comp rx=0 tx=0).stop 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- 172.21.15.23:0/741679506 shutdown_connections 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99af42980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4da70 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 --2- 172.21.15.23:0/741679506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af42440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- 172.21.15.23:0/741679506 >> 172.21.15.23:0/741679506 conn(0x55e99ae9b6a0 msgr2=0x55e99af3ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- 172.21.15.23:0/741679506 shutdown_connections 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- 172.21.15.23:0/741679506 wait complete. 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.977+0000 7f7c93420b80 1 -- start start 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 Processor -- start 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af4c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99afa3de0 con 0x55e99af3bef0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99afa3f50 con 0x55e99af3b5d0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e99afa4110 con 0x55e99af46df0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c93420b80 10 client.0 _ll_register_callbacks cb 0x55e99ae33470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c7d800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af4c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c7d800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41762/0 (socket says 172.21.15.23:41762) 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.978+0000 7f7c7d800700 1 -- 172.21.15.23:0/1781911321 learned_addr learned my addr 172.21.15.23:0/1781911321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 -- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 msgr2=0x55e99af4bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 -- 172.21.15.23:0/1781911321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 msgr2=0x55e99af4c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af4c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 -- 172.21.15.23:0/1781911321 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e99ad0f3b0 con 0x55e99af46df0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c7d800700 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c70026ae0 tx=0x7f7c700266b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9028096228623bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:18.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c5ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000e040 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c6c0a23b0 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.979+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7c70003e10 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c700074a0 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 10 client.28024 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e99ae53ea0 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c6c0a23b0 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7c70019070 con 0x55e99af46df0 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 10 client.28024 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 10 client.28024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 20 client.28024 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99afab5d0 0x55e99afad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 4 client.28024 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:11:18.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e99acdccf0 con 0x55e99afab5d0 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.980+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99afab5d0 0x55e99afad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c7ce00700 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99afab5d0 0x55e99afad9c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9bc81a3a6d7311e server_cookie=dde244f86cb68d6b in_seq=0 out_seq=0 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c5ea00700 10 client.28024 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c5ea00700 1 -- 172.21.15.23:0/1781911321 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e99acdccf0 con 0x55e99afab5d0 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c5ea00700 10 client.28024 handle_command_reply: tid=0 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 client.28024 shutdown 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 msgr2=0x55e99afa38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c70026ae0 tx=0x7f7c700266b0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 shutdown_connections 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e99afab5d0 0x55e99afad9c0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e99af46df0 0x55e99afa38a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e99af3b5d0 0x55e99af4bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c5ea00700 0 client.28024 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:18.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 --2- 172.21.15.23:0/1781911321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e99af3bef0 0x55e99af4c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 >> 172.21.15.23:0/1781911321 conn(0x55e99ae9b6a0 msgr2=0x55e99af423c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.981+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 shutdown_connections 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c93420b80 1 -- 172.21.15.23:0/1781911321 wait complete. 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c93420b80 20 client.28024 trim_cache size 0 max 16384 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:11:18.986 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 msgr2=0x7f7c600a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 0x7f7c600a37d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7c7819a900 tx=0x7f7c6400b410 comp rx=0 tx=0).stop 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 msgr2=0x7f7c781993b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00e9a0 tx=0x7f7c6c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 shutdown_connections 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c7819dc90 0x7f7c7819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7c600a1310 0x7f7c600a37d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c78100f60 0x7f7c781993b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 --2- 172.21.15.23:0/1662605425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c78106080 0x7f7c781998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:18.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.982+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 >> 172.21.15.23:0/1662605425 conn(0x7f7c780fc7c0 msgr2=0x7f7c780fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:18.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.983+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 shutdown_connections 2024-03-30T04:11:18.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:18.983+0000 7f7c7f800700 1 -- 172.21.15.23:0/1662605425 wait complete. 2024-03-30T04:11:18.999 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:11:18.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 -- 172.21.15.23:0/89571257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358100e60 msgr2=0x7f3358101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 --2- 172.21.15.23:0/89571257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358100e60 0x7f3358101240 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3348009990 tx=0x7f3348009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 -- 172.21.15.23:0/89571257 shutdown_connections 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 --2- 172.21.15.23:0/89571257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358105f80 0x7f335810db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 --2- 172.21.15.23:0/89571257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358100e60 0x7f3358101240 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 --2- 172.21.15.23:0/89571257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358101810 0x7f3358105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 -- 172.21.15.23:0/89571257 >> 172.21.15.23:0/89571257 conn(0x7f33580fc760 msgr2=0x7f33580feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 -- 172.21.15.23:0/89571257 shutdown_connections 2024-03-30T04:11:19.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.208+0000 7f335ce00700 1 -- 172.21.15.23:0/89571257 wait complete. 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.209+0000 7f335ce00700 1 Processor -- start 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.209+0000 7f335ce00700 1 -- start start 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 0x7f33580730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 0x7f33580735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 0x7f33581a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581a59c0 con 0x7f3358105f80 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581a5b00 con 0x7f3358073b30 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f335ce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33581a5cc0 con 0x7f3358100e60 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3355400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 0x7f33580735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 0x7f33581a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3355400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 0x7f33580735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42298/0 (socket says 172.21.15.23:42298) 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3355400700 1 -- 172.21.15.23:0/2895925269 learned_addr learned my addr 172.21.15.23:0/2895925269 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3355e00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 0x7f33580730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 -- 172.21.15.23:0/2895925269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 msgr2=0x7f33580730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 0x7f33580730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 -- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 msgr2=0x7f33580735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 0x7f33580735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3356800700 1 -- 172.21.15.23:0/2895925269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480095f0 con 0x7f3358073b30 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.210+0000 7f3355e00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 0x7f33580730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:19.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f3356800700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 0x7f33581a53f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f335000efc0 tx=0x7f335000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3350018070 con 0x7f3358073b30 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f335000f040 con 0x7f3358073b30 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335000cdf0 con 0x7f3358073b30 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33581a5f40 con 0x7f3358073b30 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.211+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33581a63d0 con 0x7f3358073b30 2024-03-30T04:11:19.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.212+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33581091f0 con 0x7f3358073b30 2024-03-30T04:11:19.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.214+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3350003a00 con 0x7f3358073b30 2024-03-30T04:11:19.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.216+0000 7f334ea00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 0x7f333c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.216+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f33500cccb0 con 0x7f3358073b30 2024-03-30T04:11:19.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.216+0000 7f3355e00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 0x7f333c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.217+0000 7f3355e00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 0x7f333c0a37d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f334800b2e0 tx=0x7f3348005fb0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.220+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3350095d50 con 0x7f3358073b30 2024-03-30T04:11:19.219 INFO:tasks.workunit.client.0.smithi023.stdout: 1 931187 10.30 MB/sec execute 356 sec latency 19.947 ms 2024-03-30T04:11:19.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.535+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3358102330 con 0x7f3358073b30 2024-03-30T04:11:19.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.536+0000 7f334ea00700 1 -- 172.21.15.23:0/2895925269 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f3350003cb0 con 0x7f3358073b30 2024-03-30T04:11:19.535 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:11:19.535 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 msgr2=0x7f333c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 0x7f333c0a37d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f334800b2e0 tx=0x7f3348005fb0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 msgr2=0x7f33581a53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 0x7f33581a53f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f335000efc0 tx=0x7f335000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 shutdown_connections 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f333c0a1310 0x7f333c0a37d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3358073b30 0x7f33581a53f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3358100e60 0x7f33580730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 --2- 172.21.15.23:0/2895925269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3358105f80 0x7f33580735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.539+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 >> 172.21.15.23:0/2895925269 conn(0x7f33580fc760 msgr2=0x7f33580fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.540+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 shutdown_connections 2024-03-30T04:11:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.540+0000 7f335ce00700 1 -- 172.21.15.23:0/2895925269 wait complete. 2024-03-30T04:11:19.539 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:11:19.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:11:19.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 -- 172.21.15.23:0/4238434492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 msgr2=0x7f44a0103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4238434492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0103540 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4490009990 tx=0x7f4490009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 -- 172.21.15.23:0/4238434492 shutdown_connections 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4238434492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 0x7f44a010fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4238434492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0103540 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4238434492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a0103b10 0x7f44a0107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.792+0000 7f44a4c00700 1 -- 172.21.15.23:0/4238434492 >> 172.21.15.23:0/4238434492 conn(0x7f44a00fe9c0 msgr2=0x7f44a0100de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.793+0000 7f44a4c00700 1 -- 172.21.15.23:0/4238434492 shutdown_connections 2024-03-30T04:11:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.793+0000 7f44a4c00700 1 -- 172.21.15.23:0/4238434492 wait complete. 2024-03-30T04:11:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 Processor -- start 2024-03-30T04:11:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 -- start start 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 0x7f44a0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a019dd30 0x7f44a019e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a019e7b0 con 0x7f44a019dd30 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a019e920 con 0x7f44a0108280 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f44a4c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a019eae0 con 0x7f44a0103160 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f449de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f449d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 0x7f44a0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f449de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41790/0 (socket says 172.21.15.23:41790) 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f449de00700 1 -- 172.21.15.23:0/4280263373 learned_addr learned my addr 172.21.15.23:0/4280263373 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.794+0000 7f449e800700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a019dd30 0x7f44a019e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 -- 172.21.15.23:0/4280263373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 msgr2=0x7f44a0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 0x7f44a0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 -- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a019dd30 msgr2=0x7f44a019e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a019dd30 0x7f44a019e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 -- 172.21.15.23:0/4280263373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900095f0 con 0x7f44a0103160 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f449de00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f4490005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f4496a00700 1 -- 172.21.15.23:0/4280263373 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449001d070 con 0x7f44a0103160 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f4496a00700 1 -- 172.21.15.23:0/4280263373 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4490022470 con 0x7f44a0103160 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f4496a00700 1 -- 172.21.15.23:0/4280263373 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490017650 con 0x7f44a0103160 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.795+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a01a5fe0 con 0x7f44a0103160 2024-03-30T04:11:19.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.796+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a01a6350 con 0x7f44a0103160 2024-03-30T04:11:19.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.799+0000 7f4496a00700 1 -- 172.21.15.23:0/4280263373 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f44900225e0 con 0x7f44a0103160 2024-03-30T04:11:19.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 Processor -- start 2024-03-30T04:11:19.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 -- start start 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd55d0 0x561c02bd59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02b2e990 con 0x561c02bd55d0 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02b2c350 con 0x561c02bd5ef0 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02b27c40 con 0x561c02be0df0 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f4496a00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 0x7f44840a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.800+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38102/0 (socket says 172.21.15.23:38102) 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 -- 172.21.15.23:0/3345932012 learned_addr learned my addr 172.21.15.23:0/3345932012 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449d400700 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449e800700 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd55d0 0x561c02bd59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f4496a00700 1 -- 172.21.15.23:0/4280263373 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f44900ce330 con 0x7f44a0103160 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449e800700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 0x7f44840a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 -- 172.21.15.23:0/3345932012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 msgr2=0x561c02be31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 -- 172.21.15.23:0/3345932012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd55d0 msgr2=0x561c02bd59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd55d0 0x561c02bd59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.801+0000 7f449de00700 1 -- 172.21.15.23:0/3345932012 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c029a93b0 con 0x561c02bd5ef0 2024-03-30T04:11:19.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.802+0000 7f449de00700 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f44900cb6a0 tx=0x7f4490009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8042d3934597b099 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.802+0000 7f448b000700 1 -- 172.21.15.23:0/3345932012 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449009af80 con 0x561c02bd5ef0 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.802+0000 7f449e800700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 0x7f44840a3750 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4498009ec0 tx=0x7f449800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.802+0000 7f448b000700 1 -- 172.21.15.23:0/3345932012 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f449009acf0 con 0x561c02bd5ef0 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.802+0000 7f448b000700 1 -- 172.21.15.23:0/3345932012 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449009b240 con 0x561c02bd5ef0 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3345932012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 msgr2=0x561c02be08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f44900cb6a0 tx=0x7f4490009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3345932012 shutdown_connections 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd55d0 0x561c02bd59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3345932012 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd5ef0 0x561c02be08b0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3345932012 >> 172.21.15.23:0/3345932012 conn(0x561c02b356a0 msgr2=0x561c02bd1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3345932012 shutdown_connections 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3345932012 wait complete. 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.803+0000 7f44b87c0b80 1 -- start start 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 Processor -- start 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02bddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02bde020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02ac3d80 con 0x561c02bd5ef0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02ac3a20 con 0x561c02bd55d0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c029ab6a0 con 0x561c02be0df0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f44b87c0b80 10 client.0 _ll_register_callbacks cb 0x561c02acd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f449d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02bddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38110/0 (socket says 172.21.15.23:38110) 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.804+0000 7f449de00700 1 -- 172.21.15.23:0/3657421967 learned_addr learned my addr 172.21.15.23:0/3657421967 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449e800700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02bde020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 -- 172.21.15.23:0/3657421967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 msgr2=0x561c02bde020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02bde020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 -- 172.21.15.23:0/3657421967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 msgr2=0x561c02bddae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02bddae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 -- 172.21.15.23:0/3657421967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a01a6350 con 0x561c02bd55d0 2024-03-30T04:11:19.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449d400700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02bddae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f449de00700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f449009b140 tx=0x7f44900057b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593478e6a30a89e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f447f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900b7030 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f44900b8070 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.805+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c02ae8380 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f449009efe0 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 10 client.28052 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561c02955b60 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561c02ae8380 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f44900b9280 con 0x561c02bd55d0 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 10 client.28052 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 10 client.28052 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:11:19.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 20 client.28052 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02bf0200 0x561c02bf2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 4 client.28052 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c02b20700 con 0x561c02bf0200 2024-03-30T04:11:19.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.806+0000 7f449d400700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02bf0200 0x561c02bf2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.807+0000 7f449d400700 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02bf0200 0x561c02bf2620 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2520a340eefa1d2c server_cookie=9e2ae7f32d1b14ac in_seq=0 out_seq=0 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.807+0000 7f447f400700 10 client.28052 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f447f400700 1 -- 172.21.15.23:0/3657421967 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561c02b20700 con 0x561c02bf0200 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f447f400700 10 client.28052 handle_command_reply: tid=0 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 client.28052 shutdown 2024-03-30T04:11:19.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 msgr2=0x561c02be06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f449009b140 tx=0x7f44900057b0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 shutdown_connections 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02bf0200 0x561c02bf2620 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02bde020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be06f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/3657421967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02bddae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f447f400700 0 client.28052 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 >> 172.21.15.23:0/3657421967 conn(0x561c02b356a0 msgr2=0x561c02bd74d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 shutdown_connections 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.808+0000 7f44b87c0b80 1 -- 172.21.15.23:0/3657421967 wait complete. 2024-03-30T04:11:19.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.809+0000 7f44b87c0b80 20 client.28052 trim_cache size 0 max 16384 2024-03-30T04:11:19.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 Processor -- start 2024-03-30T04:11:19.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 -- start start 2024-03-30T04:11:19.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02be0df0 0x561c02bd8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02bd5d70 con 0x561c02bd5ef0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02bd6640 con 0x561c02be0df0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.823+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02be7d20 con 0x561c02bd55d0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02be0df0 0x561c02bd8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41820/0 (socket says 172.21.15.23:41820) 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 -- 172.21.15.23:0/428965599 learned_addr learned my addr 172.21.15.23:0/428965599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449d400700 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 -- 172.21.15.23:0/428965599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02be0df0 msgr2=0x561c02bd8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02be0df0 0x561c02bd8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 -- 172.21.15.23:0/428965599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 msgr2=0x561c02be77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.824+0000 7f449de00700 1 -- 172.21.15.23:0/428965599 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c02aedea0 con 0x561c02bd55d0 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.825+0000 7f449de00700 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f44900976b0 tx=0x7f44900cb660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f321695860aa14 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.825+0000 7f448ba00700 1 -- 172.21.15.23:0/428965599 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900b7030 con 0x561c02bd55d0 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.825+0000 7f448ba00700 1 -- 172.21.15.23:0/428965599 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f449009a7c0 con 0x561c02bd55d0 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.825+0000 7f448ba00700 1 -- 172.21.15.23:0/428965599 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449009b1b0 con 0x561c02bd55d0 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- 172.21.15.23:0/428965599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 msgr2=0x561c02bd8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f44900976b0 tx=0x7f44900cb660 comp rx=0 tx=0).stop 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- 172.21.15.23:0/428965599 shutdown_connections 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02be0df0 0x561c02bd8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02bd55d0 0x561c02bd8550 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/428965599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- 172.21.15.23:0/428965599 >> 172.21.15.23:0/428965599 conn(0x561c02b356a0 msgr2=0x561c02bf6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- 172.21.15.23:0/428965599 shutdown_connections 2024-03-30T04:11:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- 172.21.15.23:0/428965599 wait complete. 2024-03-30T04:11:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.826+0000 7f44b87c0b80 1 -- start start 2024-03-30T04:11:19.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 Processor -- start 2024-03-30T04:11:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02c43030 con 0x561c02bd5ef0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02c43170 con 0x561c02bd55d0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c02c432e0 con 0x561c02be0df0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f44b87c0b80 10 client.0 _ll_register_callbacks cb 0x561c02acd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f449de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38134/0 (socket says 172.21.15.23:38134) 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f449de00700 1 -- 172.21.15.23:0/4053107177 learned_addr learned my addr 172.21.15.23:0/4053107177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.827+0000 7f449e800700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449d400700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 -- 172.21.15.23:0/4053107177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 msgr2=0x561c02be6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 -- 172.21.15.23:0/4053107177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 msgr2=0x561c02be5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:19.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 -- 172.21.15.23:0/4053107177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561c02955b60 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f449de00700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f449000c010 tx=0x7f4490009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd2391f5fffc7605 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f447f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44900b7050 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f44900b8070 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f449009acf0 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f449009b8b0 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f44b87c0b80 10 client.28058 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.828+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561c029adcd0 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f449009acf0 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4490096fa0 con 0x561c02bd55d0 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 10 client.28058 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 10 client.28058 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 20 client.28058 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:19.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02c44590 0x561c02c46980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 4 client.28058 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c02976cf0 con 0x561c02c44590 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.829+0000 7f449d400700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02c44590 0x561c02c46980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f449d400700 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02c44590 0x561c02c46980 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb5daa70374df577 server_cookie=f4cb9462efcb194 in_seq=0 out_seq=0 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f447f400700 10 client.28058 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f447f400700 1 -- 172.21.15.23:0/4053107177 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c02976cf0 con 0x561c02c44590 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f447f400700 10 client.28058 handle_command_reply: tid=0 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 client.28058 shutdown 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 msgr2=0x561c02be57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f449000c010 tx=0x7f4490009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 shutdown_connections 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561c02c44590 0x561c02c46980 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561c02be0df0 0x561c02be6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561c02bd55d0 0x561c02be57c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 --2- 172.21.15.23:0/4053107177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561c02bd5ef0 0x561c02be5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 >> 172.21.15.23:0/4053107177 conn(0x561c02b356a0 msgr2=0x561c02bd8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.830+0000 7f447f400700 0 client.28058 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 shutdown_connections 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44b87c0b80 1 -- 172.21.15.23:0/4053107177 wait complete. 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44b87c0b80 20 client.28058 trim_cache size 0 max 16384 2024-03-30T04:11:19.833 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 msgr2=0x7f44840a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 0x7f44840a3750 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4498009ec0 tx=0x7f449800b040 comp rx=0 tx=0).stop 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 msgr2=0x7f44a0199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.831+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4490000c00 tx=0x7f4490005740 comp rx=0 tx=0).stop 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 shutdown_connections 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44a019dd30 0x7f44a019e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44a0103160 0x7f44a0199450 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f44840a1290 0x7f44840a3750 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 --2- 172.21.15.23:0/4280263373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44a0108280 0x7f44a0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 >> 172.21.15.23:0/4280263373 conn(0x7f44a00fe9c0 msgr2=0x7f44a00ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 shutdown_connections 2024-03-30T04:11:19.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:19.832+0000 7f44a4c00700 1 -- 172.21.15.23:0/4280263373 wait complete. 2024-03-30T04:11:19.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 -- 172.21.15.23:0/4038231912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e8101910 msgr2=0x7fa6e8105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 --2- 172.21.15.23:0/4038231912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e8101910 0x7fa6e8105950 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa6d40098f0 tx=0x7fa6d4009c00 comp rx=0 tx=0).stop 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 -- 172.21.15.23:0/4038231912 shutdown_connections 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 --2- 172.21.15.23:0/4038231912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8106080 0x7fa6e810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 --2- 172.21.15.23:0/4038231912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8100f60 0x7fa6e8101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 --2- 172.21.15.23:0/4038231912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e8101910 0x7fa6e8105950 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 -- 172.21.15.23:0/4038231912 >> 172.21.15.23:0/4038231912 conn(0x7fa6e80fc7c0 msgr2=0x7fa6e80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 -- 172.21.15.23:0/4038231912 shutdown_connections 2024-03-30T04:11:20.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.066+0000 7fa6ef800700 1 -- 172.21.15.23:0/4038231912 wait complete. 2024-03-30T04:11:20.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 Processor -- start 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 -- start start 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8106080 0x7fa6e8199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e819dda0 0x7fa6e819e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.067+0000 7fa6ef800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e819e820 con 0x7fa6e819dda0 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ef800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e819e990 con 0x7fa6e8106080 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ef800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e819eb50 con 0x7fa6e8100f60 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41858/0 (socket says 172.21.15.23:41858) 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 -- 172.21.15.23:0/1451243285 learned_addr learned my addr 172.21.15.23:0/1451243285 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ed800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e819dda0 0x7fa6e819e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8106080 0x7fa6e8199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 -- 172.21.15.23:0/1451243285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8106080 msgr2=0x7fa6e8199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8106080 0x7fa6e8199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 -- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e819dda0 msgr2=0x7fa6e819e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e819dda0 0x7fa6e819e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.068+0000 7fa6ece00700 1 -- 172.21.15.23:0/1451243285 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc0095f0 con 0x7fa6e8100f60 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6ece00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00ec10 tx=0x7fa6dc00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6e5600700 1 -- 172.21.15.23:0/1451243285 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00cc90 con 0x7fa6e8100f60 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6e5600700 1 -- 172.21.15.23:0/1451243285 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6dc0044e0 con 0x7fa6e8100f60 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6e5600700 1 -- 172.21.15.23:0/1451243285 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc010430 con 0x7fa6e8100f60 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d4009520 con 0x7fa6e8100f60 2024-03-30T04:11:20.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.069+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e81a63b0 con 0x7fa6e8100f60 2024-03-30T04:11:20.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.072+0000 7fa6e5600700 1 -- 172.21.15.23:0/1451243285 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa6dc003fb0 con 0x7fa6e8100f60 2024-03-30T04:11:20.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.073+0000 7fa6e5600700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 0x7fa6d00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.073+0000 7fa6e5600700 1 -- 172.21.15.23:0/1451243285 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6dc014070 con 0x7fa6e8100f60 2024-03-30T04:11:20.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 Processor -- start 2024-03-30T04:11:20.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 0x7fa6d00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 -- start start 2024-03-30T04:11:20.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345108b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x5567345131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55673445e990 con 0x5567345055d0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55673445c350 con 0x556734510df0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa703304b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556734457c40 con 0x556734505ef0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.074+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 0x7fa6d00a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4005e90 tx=0x7fa6d4019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42400/0 (socket says 172.21.15.23:42400) 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 -- 172.21.15.23:0/310603148 learned_addr learned my addr 172.21.15.23:0/310603148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 -- 172.21.15.23:0/310603148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 msgr2=0x5567345108b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345108b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 -- 172.21.15.23:0/310603148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 msgr2=0x5567345131e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x5567345131e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 -- 172.21.15.23:0/310603148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567342d93b0 con 0x5567345055d0 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6ece00700 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005d70 tx=0x7fa6dc09a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157373734773d18b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6da000700 1 -- 172.21.15.23:0/310603148 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc09aa20 con 0x5567345055d0 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6da000700 1 -- 172.21.15.23:0/310603148 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6dc09a320 con 0x5567345055d0 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.075+0000 7fa6da000700 1 -- 172.21.15.23:0/310603148 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc096a30 con 0x5567345055d0 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 -- 172.21.15.23:0/310603148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 msgr2=0x5567345059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc005d70 tx=0x7fa6dc09a1c0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 -- 172.21.15.23:0/310603148 shutdown_connections 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x5567345131e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x5567345059b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 --2- 172.21.15.23:0/310603148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 -- 172.21.15.23:0/310603148 >> 172.21.15.23:0/310603148 conn(0x5567344656a0 msgr2=0x556734501a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 -- 172.21.15.23:0/310603148 shutdown_connections 2024-03-30T04:11:20.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.076+0000 7fa703304b80 1 -- 172.21.15.23:0/310603148 wait complete. 2024-03-30T04:11:20.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.077+0000 7fa703304b80 1 -- start start 2024-03-30T04:11:20.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.077+0000 7fa703304b80 1 Processor -- start 2024-03-30T04:11:20.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.077+0000 7fa703304b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.077+0000 7fa703304b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x55673450da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.077+0000 7fa703304b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x55673450e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa703304b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567343f3d80 con 0x5567345055d0 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa703304b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567343f3a20 con 0x556734510df0 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa703304b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567342db6a0 con 0x556734505ef0 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa703304b80 10 client.0 _ll_register_callbacks cb 0x5567343fd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42414/0 (socket says 172.21.15.23:42414) 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 -- 172.21.15.23:0/398786986 learned_addr learned my addr 172.21.15.23:0/398786986 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 -- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 msgr2=0x55673450da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x55673450da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 -- 172.21.15.23:0/398786986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 msgr2=0x55673450e0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x55673450e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 -- 172.21.15.23:0/398786986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e81a63b0 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6ece00700 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0a2270 tx=0x7fa6dc009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbed61ef4bbeabb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.078+0000 7fa6cea00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc0a27a0 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6dc004260 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556734418380 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6dc0cd1d0 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 10 client.18072 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556734285b60 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556734418380 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa6dc099810 con 0x5567345055d0 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 10 client.18072 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 10 client.18072 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 20 client.18072 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5567345206f0 0x556734522ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 4 client.18072 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:20.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.079+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556734450700 con 0x5567345206f0 2024-03-30T04:11:20.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.081+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5567345206f0 0x556734522ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.085+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5567345206f0 0x556734522ae0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3eb89a0e1832b754 server_cookie=bd8a31cb0170154a in_seq=0 out_seq=0 2024-03-30T04:11:20.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.085+0000 7fa6cea00700 10 client.18072 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa6cea00700 1 -- 172.21.15.23:0/398786986 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556734450700 con 0x5567345206f0 2024-03-30T04:11:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa6cea00700 10 client.18072 handle_command_reply: tid=0 2024-03-30T04:11:20.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 client.18072 shutdown 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 msgr2=0x556734510800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc0a2270 tx=0x7fa6dc009ac0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 shutdown_connections 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x5567345206f0 0x556734522ae0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734510df0 0x55673450e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x556734510800 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa703304b80 1 --2- 172.21.15.23:0/398786986 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x55673450da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.087+0000 7fa6cea00700 0 client.18072 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.088+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 >> 172.21.15.23:0/398786986 conn(0x5567344656a0 msgr2=0x5567345076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.088+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 shutdown_connections 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.088+0000 7fa703304b80 1 -- 172.21.15.23:0/398786986 wait complete. 2024-03-30T04:11:20.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.088+0000 7fa703304b80 20 client.18072 trim_cache size 0 max 16384 2024-03-30T04:11:20.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.101+0000 7fa703304b80 1 Processor -- start 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.101+0000 7fa703304b80 1 -- start start 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734505ef0 0x556734517a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 0x5567344faf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556734501f20 con 0x5567345055d0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556734509660 con 0x556734505ef0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa703304b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556734507070 con 0x556734510df0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 0x5567344faf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6e7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734505ef0 0x556734517a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ece00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42422/0 (socket says 172.21.15.23:42422) 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ece00700 1 -- 172.21.15.23:0/522414716 learned_addr learned my addr 172.21.15.23:0/522414716 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 -- 172.21.15.23:0/522414716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734505ef0 msgr2=0x556734517a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734505ef0 0x556734517a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 -- 172.21.15.23:0/522414716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 msgr2=0x55673450e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ed800700 1 -- 172.21.15.23:0/522414716 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55673441dea0 con 0x556734510df0 2024-03-30T04:11:20.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.102+0000 7fa6ece00700 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.103+0000 7fa6ed800700 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 0x5567344faf70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0003d80 tx=0x7fa6e0009ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76766a6b56b18755 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.103+0000 7fa6daa00700 1 -- 172.21.15.23:0/522414716 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0019070 con 0x556734510df0 2024-03-30T04:11:20.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.103+0000 7fa6daa00700 1 -- 172.21.15.23:0/522414716 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6e000f040 con 0x556734510df0 2024-03-30T04:11:20.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.103+0000 7fa6daa00700 1 -- 172.21.15.23:0/522414716 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0007760 con 0x556734510df0 2024-03-30T04:11:20.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- 172.21.15.23:0/522414716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 msgr2=0x5567344faf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 0x5567344faf70 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0003d80 tx=0x7fa6e0009ac0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- 172.21.15.23:0/522414716 shutdown_connections 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734510df0 0x5567344faf70 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5567345055d0 0x55673450e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 --2- 172.21.15.23:0/522414716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556734505ef0 0x556734517a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- 172.21.15.23:0/522414716 >> 172.21.15.23:0/522414716 conn(0x5567344656a0 msgr2=0x5567344fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- 172.21.15.23:0/522414716 shutdown_connections 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- 172.21.15.23:0/522414716 wait complete. 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 -- start start 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.104+0000 7fa703304b80 1 Processor -- start 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567345055d0 0x556734515c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 0x55673456d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55673456db00 con 0x556734510df0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55673456dc70 con 0x5567345055d0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55673456de30 con 0x556734505ef0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa703304b80 10 client.0 _ll_register_callbacks cb 0x5567343fd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41904/0 (socket says 172.21.15.23:41904) 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 -- 172.21.15.23:0/4636170 learned_addr learned my addr 172.21.15.23:0/4636170 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6ece00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567345055d0 0x556734515c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6ed800700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 0x55673456d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 -- 172.21.15.23:0/4636170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567345055d0 msgr2=0x556734515c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567345055d0 0x556734515c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 -- 172.21.15.23:0/4636170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 msgr2=0x55673456d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 0x55673456d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6e7e00700 1 -- 172.21.15.23:0/4636170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556734285b60 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.105+0000 7fa6ed800700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 0x55673456d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6e7e00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa6e806a380 tx=0x7fa6d4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaeadba7164db660 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6cea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d402a040 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa6d40050d0 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6dc09a320 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6d400f910 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa703304b80 10 client.28063 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fa6e0003ac0 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.106+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6dc09a320 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa6d4032210 con 0x556734505ef0 2024-03-30T04:11:20.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 10 client.28063 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 10 client.28063 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 20 client.28063 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556734575710 0x556734577b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 4 client.28063 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5567342a6cf0 con 0x556734575710 2024-03-30T04:11:20.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa6ece00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556734575710 0x556734577b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa6ece00700 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556734575710 0x556734577b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9599c8e4f043a631 server_cookie=cdcbed52d7677694 in_seq=0 out_seq=0 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.107+0000 7fa6cea00700 10 client.28063 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa6cea00700 1 -- 172.21.15.23:0/4636170 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5567342a6cf0 con 0x556734575710 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa6cea00700 10 client.28063 handle_command_reply: tid=0 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 client.28063 shutdown 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 msgr2=0x5567345161e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa6e806a380 tx=0x7fa6d4009c00 comp rx=0 tx=0).stop 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 shutdown_connections 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556734575710 0x556734577b30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556734510df0 0x55673456d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5567345055d0 0x556734515c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa6cea00700 0 client.28063 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 --2- 172.21.15.23:0/4636170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556734505ef0 0x5567345161e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 >> 172.21.15.23:0/4636170 conn(0x5567344656a0 msgr2=0x5567344fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 shutdown_connections 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 1 -- 172.21.15.23:0/4636170 wait complete. 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.108+0000 7fa703304b80 20 client.28063 trim_cache size 0 max 16384 2024-03-30T04:11:20.111 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 msgr2=0x7fa6d00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 0x7fa6d00a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4005e90 tx=0x7fa6d4019040 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 msgr2=0x7fa6e81994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00ec10 tx=0x7fa6dc00ef20 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 shutdown_connections 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa6e819dda0 0x7fa6e819e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6e8100f60 0x7fa6e81994c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa6d00a1290 0x7fa6d00a3750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 --2- 172.21.15.23:0/1451243285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6e8106080 0x7fa6e8199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 >> 172.21.15.23:0/1451243285 conn(0x7fa6e80fc7c0 msgr2=0x7fa6e80febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 shutdown_connections 2024-03-30T04:11:20.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.109+0000 7fa6ef800700 1 -- 172.21.15.23:0/1451243285 wait complete. 2024-03-30T04:11:20.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:11:20.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 932551 10.29 MB/sec execute 357 sec latency 39.398 ms 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 -- 172.21.15.23:0/2549395243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 msgr2=0x7fad2410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 --2- 172.21.15.23:0/2549395243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad2410cff0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fad10009990 tx=0x7fad10009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 -- 172.21.15.23:0/2549395243 shutdown_connections 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 --2- 172.21.15.23:0/2549395243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad2410d530 0x7fad2410f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 --2- 172.21.15.23:0/2549395243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad240731e0 0x7fad240735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 --2- 172.21.15.23:0/2549395243 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad2410cff0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 -- 172.21.15.23:0/2549395243 >> 172.21.15.23:0/2549395243 conn(0x7fad240fc7c0 msgr2=0x7fad240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 -- 172.21.15.23:0/2549395243 shutdown_connections 2024-03-30T04:11:20.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.336+0000 7fad2c000700 1 -- 172.21.15.23:0/2549395243 wait complete. 2024-03-30T04:11:20.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.338+0000 7fad2c000700 1 Processor -- start 2024-03-30T04:11:20.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.338+0000 7fad2c000700 1 -- start start 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.338+0000 7fad2c000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad2c000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad24199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad2c000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad2410d530 0x7fad2419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad2c000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419e360 con 0x7fad24073b00 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad2c000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419e4d0 con 0x7fad240731e0 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad2c000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2419e690 con 0x7fad2410d530 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad24199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38212/0 (socket says 172.21.15.23:38212) 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad29600700 1 -- 172.21.15.23:0/882670591 learned_addr learned my addr 172.21.15.23:0/882670591 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 -- 172.21.15.23:0/882670591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad2410d530 msgr2=0x7fad2419dd20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad2410d530 0x7fad2419dd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 -- 172.21.15.23:0/882670591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 msgr2=0x7fad24199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad28c00700 1 -- 172.21.15.23:0/882670591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad18009520 con 0x7fad24073b00 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.339+0000 7fad29600700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad28c00700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad24199980 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fad10009990 tx=0x7fad10005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad22000700 1 -- 172.21.15.23:0/882670591 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1001d070 con 0x7fad24073b00 2024-03-30T04:11:20.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad22000700 1 -- 172.21.15.23:0/882670591 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad1000fe90 con 0x7fad24073b00 2024-03-30T04:11:20.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad22000700 1 -- 172.21.15.23:0/882670591 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10022dc0 con 0x7fad24073b00 2024-03-30T04:11:20.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad100095f0 con 0x7fad24073b00 2024-03-30T04:11:20.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.340+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad241a5fd0 con 0x7fad24073b00 2024-03-30T04:11:20.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.344+0000 7fad22000700 1 -- 172.21.15.23:0/882670591 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fad10022470 con 0x7fad24073b00 2024-03-30T04:11:20.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.346+0000 7fad3fa1fb80 1 Processor -- start 2024-03-30T04:11:20.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.346+0000 7fad3fa1fb80 1 -- start start 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.346+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb2d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfa7b990 con 0x5631dfb22ef0 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfa79350 con 0x5631dfb225d0 2024-03-30T04:11:20.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfa74c40 con 0x5631dfb2ddf0 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad2a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad29600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb2d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad2a000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38228/0 (socket says 172.21.15.23:38228) 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad2a000700 1 -- 172.21.15.23:0/124674539 learned_addr learned my addr 172.21.15.23:0/124674539 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad28c00700 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.347+0000 7fad22000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 0x7fad0c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 -- 172.21.15.23:0/124674539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 msgr2=0x5631dfb301e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb301e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 -- 172.21.15.23:0/124674539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 msgr2=0x5631dfb2d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb2d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 -- 172.21.15.23:0/124674539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631df8f63b0 con 0x5631dfb225d0 2024-03-30T04:11:20.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad22000700 1 -- 172.21.15.23:0/882670591 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fad100ce530 con 0x7fad24073b00 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad28c00700 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e380 tx=0x7fad1c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=868c0009087ef328 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad2a000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 0x7fad0c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad16600700 1 -- 172.21.15.23:0/124674539 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c010040 con 0x5631dfb225d0 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad16600700 1 -- 172.21.15.23:0/124674539 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad1c004dd0 con 0x5631dfb225d0 2024-03-30T04:11:20.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.348+0000 7fad16600700 1 -- 172.21.15.23:0/124674539 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c014660 con 0x5631dfb225d0 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.349+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/124674539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 msgr2=0x5631dfb229b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.349+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e380 tx=0x7fad1c00b040 comp rx=0 tx=0).stop 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.349+0000 7fad2a000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 0x7fad0c0a3750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad1c004fd0 tx=0x7fad1c0148a0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/124674539 shutdown_connections 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb301e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb229b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/124674539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb2d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/124674539 >> 172.21.15.23:0/124674539 conn(0x5631dfa826a0 msgr2=0x5631dfb1ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/124674539 shutdown_connections 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/124674539 wait complete. 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.350+0000 7fad3fa1fb80 1 -- start start 2024-03-30T04:11:20.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 Processor -- start 2024-03-30T04:11:20.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 0x5631dfb2aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfa10d80 con 0x5631dfb2ddf0 2024-03-30T04:11:20.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfa10a20 con 0x5631dfb225d0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.351+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631df8f86a0 con 0x5631dfb22ef0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad28c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 0x5631dfb2aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad3fa1fb80 10 client.0 _ll_register_callbacks cb 0x5631dfa1a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad28c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 0x5631dfb2aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41956/0 (socket says 172.21.15.23:41956) 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38236/0 (socket says 172.21.15.23:38236) 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad28c00700 1 -- 172.21.15.23:0/3076454213 learned_addr learned my addr 172.21.15.23:0/3076454213 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad2a000700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 -- 172.21.15.23:0/3076454213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 msgr2=0x5631dfb2aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 0x5631dfb2aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 -- 172.21.15.23:0/3076454213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 msgr2=0x5631dfb2b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 -- 172.21.15.23:0/3076454213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1c00e0c0 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad29600700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fad18012e90 tx=0x7fad1800f6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af5cba22240ff4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.352+0000 7fad0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18017070 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fad1800fb80 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631dfa35380 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad180122e0 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 10 client.28082 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5631df8a2b60 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631dfa35380 con 0x5631dfb225d0 2024-03-30T04:11:20.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fad18010430 con 0x5631dfb225d0 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 10 client.28082 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 10 client.28082 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.353+0000 7fad3fa1fb80 20 client.28082 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb3d9b0 0x5631dfb3fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad3fa1fb80 4 client.28082 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631dfa6d700 con 0x5631dfb3d9b0 2024-03-30T04:11:20.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad28c00700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb3d9b0 0x5631dfb3fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad28c00700 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb3d9b0 0x5631dfb3fdd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac8a4ad2a4461685 server_cookie=17f51849d8473d9e in_seq=0 out_seq=0 2024-03-30T04:11:20.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.354+0000 7fad0aa00700 10 client.28082 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:20.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad0aa00700 1 -- 172.21.15.23:0/3076454213 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5631dfa6d700 con 0x5631dfb3d9b0 2024-03-30T04:11:20.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad0aa00700 10 client.28082 handle_command_reply: tid=0 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 client.28082 shutdown 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 msgr2=0x5631dfb2d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fad18012e90 tx=0x7fad1800f6a0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 shutdown_connections 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb3d9b0 0x5631dfb3fdd0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb2d7e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad0aa00700 0 client.28082 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3076454213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb22ef0 0x5631dfb2aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.355+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 >> 172.21.15.23:0/3076454213 conn(0x5631dfa826a0 msgr2=0x5631dfb246c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.356+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 shutdown_connections 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.356+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3076454213 wait complete. 2024-03-30T04:11:20.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.356+0000 7fad3fa1fb80 20 client.28082 trim_cache size 0 max 16384 2024-03-30T04:11:20.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.370+0000 7fad3fa1fb80 1 Processor -- start 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 -- start start 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 0x5631dfb2aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 0x5631dfb34a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb2c0d0 con 0x5631dfb2ddf0 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb23850 con 0x5631dfb22ef0 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb2a8c0 con 0x5631dfb225d0 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad2a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad2a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42472/0 (socket says 172.21.15.23:42472) 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad2a000700 1 -- 172.21.15.23:0/3265192707 learned_addr learned my addr 172.21.15.23:0/3265192707 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad28c00700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 0x5631dfb34a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.371+0000 7fad29600700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 0x5631dfb2aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 -- 172.21.15.23:0/3265192707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 msgr2=0x5631dfb2aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 0x5631dfb2aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 -- 172.21.15.23:0/3265192707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 msgr2=0x5631dfb34a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 0x5631dfb34a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 -- 172.21.15.23:0/3265192707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631dfa3aea0 con 0x5631dfb2ddf0 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad28c00700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 0x5631dfb34a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad29600700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 0x5631dfb2aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad2a000700 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00f010 tx=0x7fad1c022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e97e44671b9258 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad17000700 1 -- 172.21.15.23:0/3265192707 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c010030 con 0x5631dfb2ddf0 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad17000700 1 -- 172.21.15.23:0/3265192707 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad1c024070 con 0x5631dfb2ddf0 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.372+0000 7fad17000700 1 -- 172.21.15.23:0/3265192707 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0090d0 con 0x5631dfb2ddf0 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3265192707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 msgr2=0x5631dfb2d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00f010 tx=0x7fad1c022000 comp rx=0 tx=0).stop 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3265192707 shutdown_connections 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb2ddf0 0x5631dfb2d310 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb225d0 0x5631dfb2aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/3265192707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb22ef0 0x5631dfb34a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3265192707 >> 172.21.15.23:0/3265192707 conn(0x5631dfa826a0 msgr2=0x5631dfb175a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3265192707 shutdown_connections 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.373+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/3265192707 wait complete. 2024-03-30T04:11:20.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 -- start start 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 Processor -- start 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb348a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb32a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb33040 con 0x5631dfb22ef0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb331b0 con 0x5631dfb225d0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631dfb33370 con 0x5631dfb2ddf0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad3fa1fb80 10 client.0 _ll_register_callbacks cb 0x5631dfa1a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.374+0000 7fad28c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38256/0 (socket says 172.21.15.23:38256) 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 -- 172.21.15.23:0/1815814477 learned_addr learned my addr 172.21.15.23:0/1815814477 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad2a000700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb32a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 -- 172.21.15.23:0/1815814477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 msgr2=0x5631dfb32a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb32a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 -- 172.21.15.23:0/1815814477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 msgr2=0x5631dfb348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb348a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 -- 172.21.15.23:0/1815814477 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5631df8a2b60 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad28c00700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb348a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad29600700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad18012e60 tx=0x7fad18010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36198acfb56b4aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad0aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad18017070 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fad18010ab0 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.375+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad1c004dd0 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad180050e0 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 10 client.28088 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fad241a5fd0 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad1c004dd0 con 0x5631dfb225d0 2024-03-30T04:11:20.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fad18003b50 con 0x5631dfb225d0 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 10 client.28088 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 10 client.28088 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 20 client.28088 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb929f0 0x5631dfb94e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 4 client.28088 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5631df8c3cf0 con 0x5631dfb929f0 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.376+0000 7fad28c00700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb929f0 0x5631dfb94e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.377+0000 7fad28c00700 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb929f0 0x5631dfb94e10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19bf302ac6ea497e server_cookie=5a6f581e1e4877ae in_seq=0 out_seq=0 2024-03-30T04:11:20.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.377+0000 7fad0aa00700 10 client.28088 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:20.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad0aa00700 1 -- 172.21.15.23:0/1815814477 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5631df8c3cf0 con 0x5631dfb929f0 2024-03-30T04:11:20.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad0aa00700 10 client.28088 handle_command_reply: tid=0 2024-03-30T04:11:20.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 client.28088 shutdown 2024-03-30T04:11:20.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 msgr2=0x5631dfb34360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fad18012e60 tx=0x7fad18010660 comp rx=0 tx=0).stop 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 shutdown_connections 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5631dfb929f0 0x5631dfb94e10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631dfb2ddf0 0x5631dfb32a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5631dfb225d0 0x5631dfb34360 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad0aa00700 0 client.28088 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 --2- 172.21.15.23:0/1815814477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5631dfb22ef0 0x5631dfb348a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 >> 172.21.15.23:0/1815814477 conn(0x5631dfa826a0 msgr2=0x5631dfb25e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 shutdown_connections 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 1 -- 172.21.15.23:0/1815814477 wait complete. 2024-03-30T04:11:20.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.378+0000 7fad3fa1fb80 20 client.28088 trim_cache size 0 max 16384 2024-03-30T04:11:20.382 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:20.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 msgr2=0x7fad0c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 0x7fad0c0a3750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fad1c004fd0 tx=0x7fad1c0148a0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 msgr2=0x7fad24199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad24199980 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fad10009990 tx=0x7fad10005740 comp rx=0 tx=0).stop 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 shutdown_connections 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad2410d530 0x7fad2419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fad0c0a1290 0x7fad0c0a3750 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad240731e0 0x7fad24199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 --2- 172.21.15.23:0/882670591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fad24073b00 0x7fad24199980 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 >> 172.21.15.23:0/882670591 conn(0x7fad240fc7c0 msgr2=0x7fad240fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 shutdown_connections 2024-03-30T04:11:20.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.379+0000 7fad2c000700 1 -- 172.21.15.23:0/882670591 wait complete. 2024-03-30T04:11:20.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:20 smithi023 ceph-mon[38254]: pgmap v292: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.8 MiB/s wr, 323 op/s 2024-03-30T04:11:20.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:20 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2895925269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:20.396 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:11:20.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:20 smithi154 ceph-mon[79161]: pgmap v292: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.8 MiB/s wr, 323 op/s 2024-03-30T04:11:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:20 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2895925269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:20 smithi193 ceph-mon[79581]: pgmap v292: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 9.8 MiB/s wr, 323 op/s 2024-03-30T04:11:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:20 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2895925269' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 -- 172.21.15.23:0/3123347521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b4101920 msgr2=0x7fe8b4105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 --2- 172.21.15.23:0/3123347521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b4101920 0x7fe8b4105960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009990 tx=0x7fe8a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 -- 172.21.15.23:0/3123347521 shutdown_connections 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 --2- 172.21.15.23:0/3123347521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4106090 0x7fe8b410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 --2- 172.21.15.23:0/3123347521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4100f70 0x7fe8b4101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 --2- 172.21.15.23:0/3123347521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b4101920 0x7fe8b4105960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 -- 172.21.15.23:0/3123347521 >> 172.21.15.23:0/3123347521 conn(0x7fe8b40fc7f0 msgr2=0x7fe8b40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 -- 172.21.15.23:0/3123347521 shutdown_connections 2024-03-30T04:11:20.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.615+0000 7fe8bba00700 1 -- 172.21.15.23:0/3123347521 wait complete. 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.616+0000 7fe8bba00700 1 Processor -- start 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.616+0000 7fe8bba00700 1 -- start start 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.616+0000 7fe8bba00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8bba00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 0x7fe8b41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8bba00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b419dd10 0x7fe8b419e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8bba00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b419e790 con 0x7fe8b4100f70 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8bba00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b419e900 con 0x7fe8b4106090 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8bba00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b419eac0 con 0x7fe8b419dd10 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 0x7fe8b41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42504/0 (socket says 172.21.15.23:42504) 2024-03-30T04:11:20.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b9000700 1 -- 172.21.15.23:0/1380865231 learned_addr learned my addr 172.21.15.23:0/1380865231 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b9a00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b419dd10 0x7fe8b419e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 -- 172.21.15.23:0/1380865231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b419dd10 msgr2=0x7fe8b419e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b419dd10 0x7fe8b419e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 -- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 msgr2=0x7fe8b4199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b3e00700 1 -- 172.21.15.23:0/1380865231 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a00095f0 con 0x7fe8b4106090 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.617+0000 7fe8b9000700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 0x7fe8b41999c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00098e0 tx=0x7fe8a0005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8b1600700 1 -- 172.21.15.23:0/1380865231 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a001d070 con 0x7fe8b4106090 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8b1600700 1 -- 172.21.15.23:0/1380865231 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8a0022470 con 0x7fe8b4106090 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b41a5fc0 con 0x7fe8b4106090 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8b1600700 1 -- 172.21.15.23:0/1380865231 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0017650 con 0x7fe8b4106090 2024-03-30T04:11:20.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.618+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b41a63e0 con 0x7fe8b4106090 2024-03-30T04:11:20.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.621+0000 7fe8b1600700 1 -- 172.21.15.23:0/1380865231 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe8a0022850 con 0x7fe8b4106090 2024-03-30T04:11:20.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.623+0000 7fe8cf59eb80 1 Processor -- start 2024-03-30T04:11:20.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.623+0000 7fe8b1600700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 0x7fe89c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.623+0000 7fe8b1600700 1 -- 172.21.15.23:0/1380865231 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe8a00ce2b0 con 0x7fe8b4106090 2024-03-30T04:11:20.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.623+0000 7fe8cf59eb80 1 -- start start 2024-03-30T04:11:20.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.623+0000 7fe8b9a00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 0x7fe89c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca798b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca7c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c9c7990 con 0x56510ca6eef0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c9c5350 con 0x56510ca79df0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c9c0c40 con 0x56510ca6e5d0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:42006/0 (socket says 172.21.15.23:42006) 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 -- 172.21.15.23:0/480383141 learned_addr learned my addr 172.21.15.23:0/480383141 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca798b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9a00700 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca7c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 -- 172.21.15.23:0/480383141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 msgr2=0x56510ca7c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca7c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 -- 172.21.15.23:0/480383141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 msgr2=0x56510ca798b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca798b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9a00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 0x7fe89c0a37d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009550 tx=0x7fe8ac009130 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.624+0000 7fe8b9000700 1 -- 172.21.15.23:0/480383141 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56510c8423b0 con 0x56510ca6e5d0 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.625+0000 7fe8b9000700 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80117e0 tx=0x7fe8a8011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5600a3f43016e0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.625+0000 7fe8a6000700 1 -- 172.21.15.23:0/480383141 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a800f480 con 0x56510ca6e5d0 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.625+0000 7fe8a6000700 1 -- 172.21.15.23:0/480383141 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8a800faa0 con 0x56510ca6e5d0 2024-03-30T04:11:20.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.625+0000 7fe8a6000700 1 -- 172.21.15.23:0/480383141 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8019ae0 con 0x56510ca6e5d0 2024-03-30T04:11:20.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/480383141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 msgr2=0x56510ca6e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80117e0 tx=0x7fe8a8011af0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/480383141 shutdown_connections 2024-03-30T04:11:20.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca7c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca6e9b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/480383141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca798b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/480383141 >> 172.21.15.23:0/480383141 conn(0x56510c9ce6a0 msgr2=0x56510ca6aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/480383141 shutdown_connections 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/480383141 wait complete. 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.626+0000 7fe8cf59eb80 1 -- start start 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 Processor -- start 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca796f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca77000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c95cd80 con 0x56510ca6eef0 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c95ca20 con 0x56510ca79df0 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510c8446a0 con 0x56510ca6e5d0 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8cf59eb80 10 client.0 _ll_register_callbacks cb 0x56510c966470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.627+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42532/0 (socket says 172.21.15.23:42532) 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 -- 172.21.15.23:0/3266658573 learned_addr learned my addr 172.21.15.23:0/3266658573 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 -- 172.21.15.23:0/3266658573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 msgr2=0x56510ca796f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca796f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 -- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 msgr2=0x56510ca77000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca77000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 -- 172.21.15.23:0/3266658573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a8011440 con 0x56510ca6eef0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0000c00 tx=0x7fe8a009aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe73e592dc04529 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe89aa00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a009d070 con 0x56510ca6eef0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe8a009b120 con 0x56510ca6eef0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.628+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56510c981380 con 0x56510ca6eef0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8a009ef50 con 0x56510ca6eef0 2024-03-30T04:11:20.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 10 client.18108 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56510c7eeb60 con 0x56510ca6eef0 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56510c981380 con 0x56510ca6eef0 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe8a009a610 con 0x56510ca6eef0 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 10 client.18108 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 10 client.18108 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 20 client.18108 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510ca89570 0x56510ca8b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 4 client.18108 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.629+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56510c9b9700 con 0x56510ca89570 2024-03-30T04:11:20.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.630+0000 7fe8b9000700 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510ca89570 0x56510ca8b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.631+0000 7fe8b9000700 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510ca89570 0x56510ca8b990 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bae5fdf33771cfca server_cookie=c8a27402a0562e83 in_seq=0 out_seq=0 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.631+0000 7fe89aa00700 10 client.18108 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.631+0000 7fe89aa00700 1 -- 172.21.15.23:0/3266658573 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56510c9b9700 con 0x56510ca89570 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.631+0000 7fe89aa00700 10 client.18108 handle_command_reply: tid=0 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.631+0000 7fe8cf59eb80 1 client.18108 shutdown 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 msgr2=0x56510ca76a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0000c00 tx=0x7fe8a009aa80 comp rx=0 tx=0).stop 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 shutdown_connections 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca79df0 0x56510ca77000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510ca89570 0x56510ca8b990 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca6e5d0 0x56510ca796f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe89aa00700 0 client.18108 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3266658573 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6eef0 0x56510ca76a80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 >> 172.21.15.23:0/3266658573 conn(0x56510c9ce6a0 msgr2=0x56510ca706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 shutdown_connections 2024-03-30T04:11:20.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3266658573 wait complete. 2024-03-30T04:11:20.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.632+0000 7fe8cf59eb80 20 client.18108 trim_cache size 0 max 16384 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.646+0000 7fe8cf59eb80 1 Processor -- start 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.646+0000 7fe8cf59eb80 1 -- start start 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca80ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca79df0 0x56510ca71db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510ca81050 con 0x56510ca6e5d0 2024-03-30T04:11:20.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510ca77fb0 con 0x56510ca6eef0 2024-03-30T04:11:20.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510ca6af20 con 0x56510ca79df0 2024-03-30T04:11:20.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca80ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca80ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38298/0 (socket says 172.21.15.23:38298) 2024-03-30T04:11:20.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42540/0 (socket says 172.21.15.23:42540) 2024-03-30T04:11:20.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b3e00700 1 -- 172.21.15.23:0/3533790239 learned_addr learned my addr 172.21.15.23:0/3533790239 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9a00700 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca79df0 0x56510ca71db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 -- 172.21.15.23:0/3533790239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca79df0 msgr2=0x56510ca71db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca79df0 0x56510ca71db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 -- 172.21.15.23:0/3533790239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 msgr2=0x56510ca80ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca80ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.647+0000 7fe8b9000700 1 -- 172.21.15.23:0/3533790239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56510c986ea0 con 0x56510ca6e5d0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.648+0000 7fe8b9000700 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x56510ca8c270 tx=0x7fe8a80196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cac7e236e69f602 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.648+0000 7fe8a6a00700 1 -- 172.21.15.23:0/3533790239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8019410 con 0x56510ca6e5d0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.648+0000 7fe8a6a00700 1 -- 172.21.15.23:0/3533790239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8a800fdb0 con 0x56510ca6e5d0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.648+0000 7fe8a6a00700 1 -- 172.21.15.23:0/3533790239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a800f770 con 0x56510ca6e5d0 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3533790239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 msgr2=0x56510ca8b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x56510ca8c270 tx=0x7fe8a80196c0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3533790239 shutdown_connections 2024-03-30T04:11:20.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca79df0 0x56510ca71db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca6e5d0 0x56510ca8b960 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/3533790239 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca80ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3533790239 >> 172.21.15.23:0/3533790239 conn(0x56510c9ce6a0 msgr2=0x56510ca635a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3533790239 shutdown_connections 2024-03-30T04:11:20.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/3533790239 wait complete. 2024-03-30T04:11:20.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.649+0000 7fe8cf59eb80 1 -- start start 2024-03-30T04:11:20.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 Processor -- start 2024-03-30T04:11:20.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 0x56510ca7f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca7fb50 0x56510cadd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510ca80d00 con 0x56510ca79df0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510ca80e70 con 0x56510ca6eef0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56510cadd8f0 con 0x56510ca7fb50 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8cf59eb80 10 client.0 _ll_register_callbacks cb 0x56510c966470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8b3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 0x56510ca7f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8b9a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca7fb50 0x56510cadd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8b9000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38314/0 (socket says 172.21.15.23:38314) 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.650+0000 7fe8b9000700 1 -- 172.21.15.23:0/1712832805 learned_addr learned my addr 172.21.15.23:0/1712832805 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 -- 172.21.15.23:0/1712832805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca7fb50 msgr2=0x56510cadd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca7fb50 0x56510cadd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 -- 172.21.15.23:0/1712832805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 msgr2=0x56510ca7f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 0x56510ca7f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 -- 172.21.15.23:0/1712832805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56510c8423b0 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 0x56510ca7f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8b9000700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800a010 tx=0x7fe8a8011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d95306ce4f019a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe89aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8027160 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe8a80272c0 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.651+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe8a800faa0 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8a8027b30 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 10 client.28103 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56510c846cd0 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8a800faa0 con 0x56510ca6eef0 2024-03-30T04:11:20.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe8a802e070 con 0x56510ca6eef0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 10 client.28103 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 10 client.28103 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 20 client.28103 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510cadeba0 0x56510cae0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 4 client.28103 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56510c80fcf0 con 0x56510cadeba0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.652+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510cadeba0 0x56510cae0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8b3e00700 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510cadeba0 0x56510cae0f90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9f03822cc2a2d51 server_cookie=a1c403c31098927d in_seq=0 out_seq=0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe89aa00700 10 client.28103 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe89aa00700 1 -- 172.21.15.23:0/1712832805 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56510c80fcf0 con 0x56510cadeba0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe89aa00700 10 client.28103 handle_command_reply: tid=0 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 client.28103 shutdown 2024-03-30T04:11:20.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 msgr2=0x56510ca7f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800a010 tx=0x7fe8a8011af0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 shutdown_connections 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x56510cadeba0 0x56510cae0f90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56510ca7fb50 0x56510cadd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe89aa00700 0 client.28103 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56510ca6eef0 0x56510ca7f0d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 --2- 172.21.15.23:0/1712832805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56510ca79df0 0x56510ca7f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 >> 172.21.15.23:0/1712832805 conn(0x56510c9ce6a0 msgr2=0x56510ca71db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.653+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 shutdown_connections 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8cf59eb80 1 -- 172.21.15.23:0/1712832805 wait complete. 2024-03-30T04:11:20.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8cf59eb80 20 client.28103 trim_cache size 0 max 16384 2024-03-30T04:11:20.657 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:20.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 msgr2=0x7fe89c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 0x7fe89c0a37d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009550 tx=0x7fe8ac009130 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 msgr2=0x7fe8b41999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 0x7fe8b41999c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe8a00098e0 tx=0x7fe8a0005740 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 shutdown_connections 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8b419dd10 0x7fe8b419e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8b4100f70 0x7fe8b4199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe89c0a1310 0x7fe89c0a37d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 --2- 172.21.15.23:0/1380865231 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b4106090 0x7fe8b41999c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 >> 172.21.15.23:0/1380865231 conn(0x7fe8b40fc7f0 msgr2=0x7fe8b40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 shutdown_connections 2024-03-30T04:11:20.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.654+0000 7fe8bba00700 1 -- 172.21.15.23:0/1380865231 wait complete. 2024-03-30T04:11:20.673 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:11:20.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.903+0000 7f70d6800700 1 -- 172.21.15.23:0/2666905454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0101490 msgr2=0x7f70d010d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.903+0000 7f70d6800700 1 --2- 172.21.15.23:0/2666905454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0101490 0x7f70d010d9a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b0a0 tx=0x7f70c800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 -- 172.21.15.23:0/2666905454 shutdown_connections 2024-03-30T04:11:20.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 --2- 172.21.15.23:0/2666905454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0101490 0x7f70d010d9a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 --2- 172.21.15.23:0/2666905454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 0x7f70d0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 --2- 172.21.15.23:0/2666905454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70d0100ad0 0x7f70d0100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 -- 172.21.15.23:0/2666905454 >> 172.21.15.23:0/2666905454 conn(0x7f70d00fc5e0 msgr2=0x7f70d00fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 -- 172.21.15.23:0/2666905454 shutdown_connections 2024-03-30T04:11:20.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.904+0000 7f70d6800700 1 -- 172.21.15.23:0/2666905454 wait complete. 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 Processor -- start 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 -- start start 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70d0101490 0x7f70d00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 0x7f70d0073b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d01a52e0 con 0x7f70d0100ad0 2024-03-30T04:11:20.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d01a5450 con 0x7f70d0101490 2024-03-30T04:11:20.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d6800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d01a5610 con 0x7f70d0106b70 2024-03-30T04:11:20.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42562/0 (socket says 172.21.15.23:42562) 2024-03-30T04:11:20.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.905+0000 7f70d5400700 1 -- 172.21.15.23:0/1521031788 learned_addr learned my addr 172.21.15.23:0/1521031788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5e00700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 0x7f70d0073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 -- 172.21.15.23:0/1521031788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 msgr2=0x7f70d0073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 0x7f70d0073b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 -- 172.21.15.23:0/1521031788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70d0101490 msgr2=0x7f70d00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70d0101490 0x7f70d00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 -- 172.21.15.23:0/1521031788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8009d00 con 0x7f70d0100ad0 2024-03-30T04:11:20.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d5400700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c9b0 tx=0x7f70c400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70ce000700 1 -- 172.21.15.23:0/1521031788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4007b00 con 0x7f70d0100ad0 2024-03-30T04:11:20.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70ce000700 1 -- 172.21.15.23:0/1521031788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70c4007c60 con 0x7f70d0100ad0 2024-03-30T04:11:20.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d01a58f0 con 0x7f70d0100ad0 2024-03-30T04:11:20.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.906+0000 7f70ce000700 1 -- 172.21.15.23:0/1521031788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4018660 con 0x7f70d0100ad0 2024-03-30T04:11:20.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.907+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d01a5dd0 con 0x7f70d0100ad0 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.910+0000 7f70ce000700 1 -- 172.21.15.23:0/1521031788 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f70c400f450 con 0x7f70d0100ad0 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.912+0000 7f70eb92eb80 1 Processor -- start 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.912+0000 7f70eb92eb80 1 -- start start 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.912+0000 7f70ce000700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 0x7f70b80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70ce000700 1 -- 172.21.15.23:0/1521031788 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f70c40cd070 con 0x7f70d0100ad0 2024-03-30T04:11:20.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d5e00700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 0x7f70b80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ec03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1e0b990 con 0x5568f1eb30c0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1e09350 con 0x5568f1eb27a0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1e04c40 con 0x5568f1ebdfc0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ec03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38332/0 (socket says 172.21.15.23:38332) 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d5400700 1 -- 172.21.15.23:0/3957380780 learned_addr learned my addr 172.21.15.23:0/3957380780 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.913+0000 7f70d4a00700 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 -- 172.21.15.23:0/3957380780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 msgr2=0x5568f1eb2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 -- 172.21.15.23:0/3957380780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 msgr2=0x5568f1ebda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 -- 172.21.15.23:0/3957380780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568f1c863b0 con 0x5568f1ebdfc0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5400700 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 0x7f70b80a3750 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b830 tx=0x7f70c8004af0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70d5e00700 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ec03b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f70d0074090 tx=0x7f70c8028000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a7cf736faa1ede7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70c2600700 1 -- 172.21.15.23:0/3957380780 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c800e040 con 0x5568f1ebdfc0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70c2600700 1 -- 172.21.15.23:0/3957380780 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70c801f930 con 0x5568f1ebdfc0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.914+0000 7f70c2600700 1 -- 172.21.15.23:0/3957380780 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c80053a0 con 0x5568f1ebdfc0 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 -- 172.21.15.23:0/3957380780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 msgr2=0x5568f1ec03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ec03b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f70d0074090 tx=0x7f70c8028000 comp rx=0 tx=0).stop 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 -- 172.21.15.23:0/3957380780 shutdown_connections 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ec03b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eb2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/3957380780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 -- 172.21.15.23:0/3957380780 >> 172.21.15.23:0/3957380780 conn(0x5568f1e126a0 msgr2=0x5568f1eaec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 -- 172.21.15.23:0/3957380780 shutdown_connections 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.915+0000 7f70eb92eb80 1 -- 172.21.15.23:0/3957380780 wait complete. 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 -- start start 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 Processor -- start 2024-03-30T04:11:20.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1da0d80 con 0x5568f1ebdfc0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1da0a20 con 0x5568f1eb27a0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1c886a0 con 0x5568f1eb30c0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.916+0000 7f70eb92eb80 10 client.0 _ll_register_callbacks cb 0x5568f1daa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38348/0 (socket says 172.21.15.23:38348) 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 -- 172.21.15.23:0/2214258909 learned_addr learned my addr 172.21.15.23:0/2214258909 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5e00700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d4a00700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 -- 172.21.15.23:0/2214258909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1eb30c0 msgr2=0x5568f1ebad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 -- 172.21.15.23:0/2214258909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1ebdfc0 msgr2=0x5568f1ebb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 -- 172.21.15.23:0/2214258909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d01a5dd0 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70d5400700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f70c40074a0 tx=0x7f70c409a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1aa85ab9db42981 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.917+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c400f900 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f70c409db70 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5568f1dc5380 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70c4063030 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 10 client.28112 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f70c8026050 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5568f1dc5380 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f70c40a1d30 con 0x5568f1eb27a0 2024-03-30T04:11:20.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 10 client.28112 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 10 client.28112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 20 client.28112 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1ecda10 0x5568f1ecfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 4 client.28112 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.918+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5568f1dfd700 con 0x5568f1ecda10 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.919+0000 7f70d4a00700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1ecda10 0x5568f1ecfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.919+0000 7f70d4a00700 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1ecda10 0x5568f1ecfe30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0790ab8d37d3b38 server_cookie=240565beaca08cd0 in_seq=0 out_seq=0 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.919+0000 7f70b6a00700 10 client.28112 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:20.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70b6a00700 1 -- 172.21.15.23:0/2214258909 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5568f1dfd700 con 0x5568f1ecda10 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70b6a00700 10 client.28112 handle_command_reply: tid=0 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 client.28112 shutdown 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 msgr2=0x5568f1ebda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f70c40074a0 tx=0x7f70c409a580 comp rx=0 tx=0).stop 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 shutdown_connections 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1ebda10 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1ecda10 0x5568f1ecfe30 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/2214258909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70b6a00700 0 client.28112 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 >> 172.21.15.23:0/2214258909 conn(0x5568f1e126a0 msgr2=0x5568f1eb4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 shutdown_connections 2024-03-30T04:11:20.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.920+0000 7f70eb92eb80 1 -- 172.21.15.23:0/2214258909 wait complete. 2024-03-30T04:11:20.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.921+0000 7f70eb92eb80 20 client.28112 trim_cache size 0 max 16384 2024-03-30T04:11:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 Processor -- start 2024-03-30T04:11:20.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 -- start start 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1eb3ff0 con 0x5568f1eb30c0 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1ed01c0 con 0x5568f1eb27a0 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.938+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1ebba90 con 0x5568f1ebdfc0 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:42080/0 (socket says 172.21.15.23:42080) 2024-03-30T04:11:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5e00700 1 -- 172.21.15.23:0/5957538 learned_addr learned my addr 172.21.15.23:0/5957538 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d4a00700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 -- 172.21.15.23:0/5957538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 msgr2=0x5568f1ebb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 -- 172.21.15.23:0/5957538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 msgr2=0x5568f1ebad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 -- 172.21.15.23:0/5957538 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5568f1c32b60 con 0x5568f1eb27a0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5e00700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.939+0000 7f70d5400700 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eae160 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f70c4099f00 tx=0x7f70c40982f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=513de321a71f261d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70c3000700 1 -- 172.21.15.23:0/5957538 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4099490 con 0x5568f1eb27a0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70c3000700 1 -- 172.21.15.23:0/5957538 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70c40cae90 con 0x5568f1eb27a0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70c3000700 1 -- 172.21.15.23:0/5957538 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c40cab20 con 0x5568f1eb27a0 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 -- 172.21.15.23:0/5957538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 msgr2=0x5568f1eae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eae160 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f70c4099f00 tx=0x7f70c40982f0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 -- 172.21.15.23:0/5957538 shutdown_connections 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1ebb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb27a0 0x5568f1eae160 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/5957538 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb30c0 0x5568f1ebad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 -- 172.21.15.23:0/5957538 >> 172.21.15.23:0/5957538 conn(0x5568f1e126a0 msgr2=0x5568f1eba5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.940+0000 7f70eb92eb80 1 -- 172.21.15.23:0/5957538 shutdown_connections 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.941+0000 7f70eb92eb80 1 -- 172.21.15.23:0/5957538 wait complete. 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.941+0000 7f70eb92eb80 1 -- start start 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.941+0000 7f70eb92eb80 1 Processor -- start 2024-03-30T04:11:20.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.941+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb27a0 0x5568f1ec2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1f20840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1f20d80 con 0x5568f1eb27a0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1f20ec0 con 0x5568f1eb30c0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5568f1f21050 con 0x5568f1ebdfc0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70eb92eb80 10 client.0 _ll_register_callbacks cb 0x5568f1daa470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38362/0 (socket says 172.21.15.23:38362) 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 -- 172.21.15.23:0/4002750469 learned_addr learned my addr 172.21.15.23:0/4002750469 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d5e00700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1f20840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d5400700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb27a0 0x5568f1ec2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 -- 172.21.15.23:0/4002750469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 msgr2=0x5568f1f20840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1f20840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 -- 172.21.15.23:0/4002750469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb27a0 msgr2=0x5568f1ec2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:20.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb27a0 0x5568f1ec2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.942+0000 7f70d4a00700 1 -- 172.21.15.23:0/4002750469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5568f1c863b0 con 0x5568f1eb30c0 2024-03-30T04:11:20.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70d4a00700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f70d00747e0 tx=0x7f70bc007fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1280b146b3eaa682 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:20.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc016040 con 0x5568f1eb30c0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f70bc01ce10 con 0x5568f1eb30c0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70c801f930 con 0x5568f1eb30c0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70bc004a30 con 0x5568f1eb30c0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70eb92eb80 10 client.28118 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f70c8026050 con 0x5568f1eb30c0 2024-03-30T04:11:20.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70c801f930 con 0x5568f1eb30c0 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.943+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f70bc0116e0 con 0x5568f1eb30c0 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 10 client.28118 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 10 client.28118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 20 client.28118 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1f22930 0x5568f1f24d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 4 client.28118 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5568f1c53cf0 con 0x5568f1f22930 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70d5400700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1f22930 0x5568f1f24d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70d5400700 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1f22930 0x5568f1f24d50 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7264922b87e040a8 server_cookie=d00c25f85af4893a in_seq=0 out_seq=0 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70b6a00700 10 client.28118 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70b6a00700 1 -- 172.21.15.23:0/4002750469 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5568f1c53cf0 con 0x5568f1f22930 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.944+0000 7f70b6a00700 10 client.28118 handle_command_reply: tid=0 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 client.28118 shutdown 2024-03-30T04:11:20.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 msgr2=0x5568f1ec32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f70d00747e0 tx=0x7f70bc007fd0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 shutdown_connections 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5568f1f22930 0x5568f1f24d50 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5568f1ebdfc0 0x5568f1f20840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5568f1eb27a0 0x5568f1ec2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70b6a00700 0 client.28118 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 --2- 172.21.15.23:0/4002750469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5568f1eb30c0 0x5568f1ec32f0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 >> 172.21.15.23:0/4002750469 conn(0x5568f1e126a0 msgr2=0x5568f1ea8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 shutdown_connections 2024-03-30T04:11:20.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 1 -- 172.21.15.23:0/4002750469 wait complete. 2024-03-30T04:11:20.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.945+0000 7f70eb92eb80 20 client.28118 trim_cache size 0 max 16384 2024-03-30T04:11:20.949 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 msgr2=0x7f70b80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 0x7f70b80a3750 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f70c800b830 tx=0x7f70c8004af0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 msgr2=0x7f70d00730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f70c400c9b0 tx=0x7f70c400ccc0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 shutdown_connections 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70d0106b70 0x7f70d0073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70d0100ad0 0x7f70d00730b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f70b80a1290 0x7f70b80a3750 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 --2- 172.21.15.23:0/1521031788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70d0101490 0x7f70d00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:20.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 >> 172.21.15.23:0/1521031788 conn(0x7f70d00fc5e0 msgr2=0x7f70d00fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:20.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 shutdown_connections 2024-03-30T04:11:20.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:20.946+0000 7f70d6800700 1 -- 172.21.15.23:0/1521031788 wait complete. 2024-03-30T04:11:21.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 934707 10.30 MB/sec execute 358 sec latency 18.069 ms 2024-03-30T04:11:21.965 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:11:21.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 -- 172.21.15.23:0/388986916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 msgr2=0x7f3f2810db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/388986916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f2810db50 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3f2000b0d0 tx=0x7f3f2000b3e0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 -- 172.21.15.23:0/388986916 shutdown_connections 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/388986916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f2810db50 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/388986916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f281060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/388986916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f280682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 -- 172.21.15.23:0/388986916 >> 172.21.15.23:0/388986916 conn(0x7f3f28076d40 msgr2=0x7f3f28077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 -- 172.21.15.23:0/388986916 shutdown_connections 2024-03-30T04:11:22.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.182+0000 7f3f2fc00700 1 -- 172.21.15.23:0/388986916 wait complete. 2024-03-30T04:11:22.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 Processor -- start 2024-03-30T04:11:22.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 -- start start 2024-03-30T04:11:22.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f28199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f2819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2819e460 con 0x7f3f28105ce0 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2819e5d0 con 0x7f3f28067e30 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.184+0000 7f3f2fc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2819e790 con 0x7f3f280687f0 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38376/0 (socket says 172.21.15.23:38376) 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 -- 172.21.15.23:0/2610922894 learned_addr learned my addr 172.21.15.23:0/2610922894 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f27e00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f28199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f2819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 -- 172.21.15.23:0/2610922894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 msgr2=0x7f3f28199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f28199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 -- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 msgr2=0x7f3f2819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f2819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 -- 172.21.15.23:0/2610922894 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f20009d00 con 0x7f3f28067e30 2024-03-30T04:11:22.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f2819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:22.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.185+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00e9a0 tx=0x7f3f1c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.186+0000 7f3f25600700 1 -- 172.21.15.23:0/2610922894 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c009790 con 0x7f3f28067e30 2024-03-30T04:11:22.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.186+0000 7f3f25600700 1 -- 172.21.15.23:0/2610922894 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f1c00cbd0 con 0x7f3f28067e30 2024-03-30T04:11:22.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.186+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2819ea70 con 0x7f3f28067e30 2024-03-30T04:11:22.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.186+0000 7f3f25600700 1 -- 172.21.15.23:0/2610922894 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c018680 con 0x7f3f28067e30 2024-03-30T04:11:22.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.186+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f281a62b0 con 0x7f3f28067e30 2024-03-30T04:11:22.187 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.189+0000 7f3f25600700 1 -- 172.21.15.23:0/2610922894 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3f1c00cd40 con 0x7f3f28067e30 2024-03-30T04:11:22.188 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.190+0000 7f3f436a6b80 1 Processor -- start 2024-03-30T04:11:22.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.190+0000 7f3f436a6b80 1 -- start start 2024-03-30T04:11:22.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 0x55d1d222ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d2236930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 0x55d1d2239260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d2184a10 con 0x55d1d222bf70 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d21823d0 con 0x55d1d222b650 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d217dcc0 con 0x55d1d2236e70 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f25600700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 0x7f3f100a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f2dc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 0x55d1d222ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d2236930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f2dc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 0x55d1d222ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38382/0 (socket says 172.21.15.23:38382) 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f2dc00700 1 -- 172.21.15.23:0/1652636323 learned_addr learned my addr 172.21.15.23:0/1652636323 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f25600700 1 -- 172.21.15.23:0/2610922894 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f1c014070 con 0x7f3f28067e30 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f27e00700 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 0x55d1d2239260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.191+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 0x7f3f100a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 -- 172.21.15.23:0/1652636323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 msgr2=0x55d1d222ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 0x55d1d222ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 -- 172.21.15.23:0/1652636323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 msgr2=0x55d1d2236930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d2236930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 -- 172.21.15.23:0/1652636323 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1d1fff430 con 0x55d1d2236e70 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f27e00700 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 0x55d1d2239260 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009c80 tx=0x7f3f14009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84d1bdbf38bbe9fe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 0x7f3f100a3750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f20008ff0 tx=0x7f3f2000b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f1a000700 1 -- 172.21.15.23:0/1652636323 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14011cb0 con 0x55d1d2236e70 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f1a000700 1 -- 172.21.15.23:0/1652636323 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f14011e10 con 0x55d1d2236e70 2024-03-30T04:11:22.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.192+0000 7f3f1a000700 1 -- 172.21.15.23:0/1652636323 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14005280 con 0x55d1d2236e70 2024-03-30T04:11:22.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 -- 172.21.15.23:0/1652636323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 msgr2=0x55d1d2239260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 0x55d1d2239260 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3f14009c80 tx=0x7f3f14009f90 comp rx=0 tx=0).stop 2024-03-30T04:11:22.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 -- 172.21.15.23:0/1652636323 shutdown_connections 2024-03-30T04:11:22.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d2236e70 0x55d1d2239260 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222b650 0x55d1d222ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/1652636323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d2236930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 -- 172.21.15.23:0/1652636323 >> 172.21.15.23:0/1652636323 conn(0x55d1d218b720 msgr2=0x55d1d2227b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 -- 172.21.15.23:0/1652636323 shutdown_connections 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.193+0000 7f3f436a6b80 1 -- 172.21.15.23:0/1652636323 wait complete. 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 -- start start 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 Processor -- start 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d2236810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 0x55d1d2233ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 0x55d1d2234160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d2119e00 con 0x55d1d2236e70 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d2119aa0 con 0x55d1d222bf70 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d2001720 con 0x55d1d222b650 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.194+0000 7f3f436a6b80 10 client.0 _ll_register_callbacks cb 0x55d1d21234f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:22.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 0x55d1d2233ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 0x55d1d2234160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d2236810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 0x55d1d2233ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38390/0 (socket says 172.21.15.23:38390) 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f27e00700 1 -- 172.21.15.23:0/4246756655 learned_addr learned my addr 172.21.15.23:0/4246756655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 -- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 msgr2=0x55d1d2233ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 0x55d1d2233ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 -- 172.21.15.23:0/4246756655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 msgr2=0x55d1d2234160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 0x55d1d2234160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 -- 172.21.15.23:0/4246756655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f140098e0 con 0x55d1d222b650 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 0x55d1d2234160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f2d200700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d2236810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0ca6a0 tx=0x7f3f1c014420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10c3274cb5bec27 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.195+0000 7f3f0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:22.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c09a760 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f1c09a060 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d1d213e400 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f1c09de90 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 10 client.28120 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d1d1fabbe0 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d1d213e400 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f1c099550 con 0x55d1d222b650 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 10 client.28120 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 10 client.28120 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 20 client.28120 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d2246800 0x55d1d2248c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 4 client.28120 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.196+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1d2176780 con 0x55d1d2246800 2024-03-30T04:11:22.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.197+0000 7f3f27e00700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d2246800 0x55d1d2248c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.197+0000 7f3f0ea00700 10 client.28120 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.197+0000 7f3f27e00700 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d2246800 0x55d1d2248c20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b517df4b102cb3a0 server_cookie=593cacad57e814c9 in_seq=0 out_seq=0 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.198+0000 7f3f0ea00700 1 -- 172.21.15.23:0/4246756655 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d1d2176780 con 0x55d1d2246800 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.198+0000 7f3f0ea00700 10 client.28120 handle_command_reply: tid=0 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.198+0000 7f3f436a6b80 1 client.28120 shutdown 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.198+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 msgr2=0x55d1d2236810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.198+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d2236810 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0ca6a0 tx=0x7f3f1c014420 comp rx=0 tx=0).stop 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 shutdown_connections 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d2236e70 0x55d1d2234160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d2246800 0x55d1d2248c20 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d2236810 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/4246756655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d222bf70 0x55d1d2233ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f0ea00700 0 client.28120 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 >> 172.21.15.23:0/4246756655 conn(0x55d1d218b720 msgr2=0x55d1d222d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 shutdown_connections 2024-03-30T04:11:22.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 1 -- 172.21.15.23:0/4246756655 wait complete. 2024-03-30T04:11:22.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.199+0000 7f3f436a6b80 20 client.28120 trim_cache size 0 max 16384 2024-03-30T04:11:22.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 Processor -- start 2024-03-30T04:11:22.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 -- start start 2024-03-30T04:11:22.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222bf70 0x55d1d2233b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d22340a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d222c8d0 con 0x55d1d222b650 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d223e030 con 0x55d1d2236e70 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.220+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d22320f0 con 0x55d1d222bf70 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42634/0 (socket says 172.21.15.23:42634) 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 -- 172.21.15.23:0/2783008487 learned_addr learned my addr 172.21.15.23:0/2783008487 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d22340a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f27e00700 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222bf70 0x55d1d2233b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 -- 172.21.15.23:0/2783008487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222bf70 msgr2=0x55d1d2233b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222bf70 0x55d1d2233b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 -- 172.21.15.23:0/2783008487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 msgr2=0x55d1d22340a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d22340a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 -- 172.21.15.23:0/2783008487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d1d2143f20 con 0x55d1d222b650 2024-03-30T04:11:22.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 936081 10.29 MB/sec execute 359 sec latency 55.590 ms 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.221+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c099d60 tx=0x7f3f1c066000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ca60b945fed0786 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.222+0000 7f3f1aa00700 1 -- 172.21.15.23:0/2783008487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c09a510 con 0x55d1d222b650 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.222+0000 7f3f1aa00700 1 -- 172.21.15.23:0/2783008487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f1c09a760 con 0x55d1d222b650 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.222+0000 7f3f1aa00700 1 -- 172.21.15.23:0/2783008487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c099550 con 0x55d1d222b650 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2783008487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 msgr2=0x55d1d223daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c099d60 tx=0x7f3f1c066000 comp rx=0 tx=0).stop 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2783008487 shutdown_connections 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d22340a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222b650 0x55d1d223daf0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2783008487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222bf70 0x55d1d2233b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2783008487 >> 172.21.15.23:0/2783008487 conn(0x55d1d218b720 msgr2=0x55d1d2220620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2783008487 shutdown_connections 2024-03-30T04:11:22.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.223+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2783008487 wait complete. 2024-03-30T04:11:22.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.224+0000 7f3f436a6b80 1 -- start start 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.224+0000 7f3f436a6b80 1 Processor -- start 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.224+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.224+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d223c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f436a6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d223cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d223dff0 con 0x55d1d222bf70 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d229a600 con 0x55d1d2236e70 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f436a6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1d229a7c0 con 0x55d1d222b650 2024-03-30T04:11:22.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f436a6b80 10 client.0 _ll_register_callbacks cb 0x55d1d21234f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:42156/0 (socket says 172.21.15.23:42156) 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 -- 172.21.15.23:0/2532932046 learned_addr learned my addr 172.21.15.23:0/2532932046 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f27e00700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d223c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2dc00700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d223cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 -- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 msgr2=0x55d1d223cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d223cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 -- 172.21.15.23:0/2532932046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 msgr2=0x55d1d223c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d223c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 -- 172.21.15.23:0/2532932046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d1d1fabbe0 con 0x55d1d222b650 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f2d200700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c096ad0 tx=0x7f3f1c098610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a2166faa55a9203 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.225+0000 7f3f0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c09a4c0 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f14011e10 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f1c0a24e0 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f1c09a060 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 10 client.28126 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d1d2003d50 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f14011e10 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f1c0990e0 con 0x55d1d222b650 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 10 client.28126 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 10 client.28126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 20 client.28126 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d229baf0 0x55d1d229dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 4 client.28126 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.226+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d1d1fccd70 con 0x55d1d229baf0 2024-03-30T04:11:22.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.227+0000 7f3f27e00700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d229baf0 0x55d1d229dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.227+0000 7f3f27e00700 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d229baf0 0x55d1d229dee0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4400ae600171e7f server_cookie=ad565981cac62492 in_seq=0 out_seq=0 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.227+0000 7f3f0ea00700 10 client.28126 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.228+0000 7f3f0ea00700 1 -- 172.21.15.23:0/2532932046 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d1d1fccd70 con 0x55d1d229baf0 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.228+0000 7f3f0ea00700 10 client.28126 handle_command_reply: tid=0 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.228+0000 7f3f436a6b80 1 client.28126 shutdown 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.228+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 msgr2=0x55d1d223c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.228+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c096ad0 tx=0x7f3f1c098610 comp rx=0 tx=0).stop 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 shutdown_connections 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d1d2236e70 0x55d1d223cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55d1d229baf0 0x55d1d229dee0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d1d222b650 0x55d1d223c150 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f0ea00700 0 client.28126 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 --2- 172.21.15.23:0/2532932046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d1d222bf70 0x55d1d223c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 >> 172.21.15.23:0/2532932046 conn(0x55d1d218b720 msgr2=0x55d1d2233ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 shutdown_connections 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 1 -- 172.21.15.23:0/2532932046 wait complete. 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.229+0000 7f3f436a6b80 20 client.28126 trim_cache size 0 max 16384 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "322a4ee0-e230-4dac-8442-f99d815db5e5", 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:11:22.230 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 msgr2=0x7f3f100a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 0x7f3f100a3750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f20008ff0 tx=0x7f3f2000b040 comp rx=0 tx=0).stop 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 msgr2=0x7f3f28199540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00e9a0 tx=0x7f3f1c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 shutdown_connections 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f28105ce0 0x7f3f2819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f100a1290 0x7f3f100a3750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f28067e30 0x7f3f28199540 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 --2- 172.21.15.23:0/2610922894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f280687f0 0x7f3f28199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 >> 172.21.15.23:0/2610922894 conn(0x7f3f28076d40 msgr2=0x7f3f2810c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 shutdown_connections 2024-03-30T04:11:22.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.230+0000 7f3f2fc00700 1 -- 172.21.15.23:0/2610922894 wait complete. 2024-03-30T04:11:22.250 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:11:22.314 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:22 smithi023 ceph-mon[38254]: pgmap v293: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 8.8 MiB/s wr, 292 op/s 2024-03-30T04:11:22.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:22 smithi154 ceph-mon[79161]: pgmap v293: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 8.8 MiB/s wr, 292 op/s 2024-03-30T04:11:22.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 -- 172.21.15.23:0/1896709508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c0731e0 msgr2=0x7fb66c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 --2- 172.21.15.23:0/1896709508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c0731e0 0x7fb66c0735c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb6600098f0 tx=0x7fb660009c00 comp rx=0 tx=0).stop 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 -- 172.21.15.23:0/1896709508 shutdown_connections 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 --2- 172.21.15.23:0/1896709508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c10d530 0x7fb66c10f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 --2- 172.21.15.23:0/1896709508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c0731e0 0x7fb66c0735c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 --2- 172.21.15.23:0/1896709508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c073b00 0x7fb66c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 -- 172.21.15.23:0/1896709508 >> 172.21.15.23:0/1896709508 conn(0x7fb66c0fc7c0 msgr2=0x7fb66c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 -- 172.21.15.23:0/1896709508 shutdown_connections 2024-03-30T04:11:22.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.470+0000 7fb673c00700 1 -- 172.21.15.23:0/1896709508 wait complete. 2024-03-30T04:11:22.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.471+0000 7fb673c00700 1 Processor -- start 2024-03-30T04:11:22.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.471+0000 7fb673c00700 1 -- start start 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 0x7fb66c199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c10d530 0x7fb66c19dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c19e300 con 0x7fb66c10d530 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c19e470 con 0x7fb66c0731e0 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb673c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c19e630 con 0x7fb66c073b00 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb671200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb671200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38410/0 (socket says 172.21.15.23:38410) 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb66be00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 0x7fb66c199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb671200700 1 -- 172.21.15.23:0/696318236 learned_addr learned my addr 172.21.15.23:0/696318236 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.472+0000 7fb671c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c10d530 0x7fb66c19dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 -- 172.21.15.23:0/696318236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 msgr2=0x7fb66c1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 -- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c10d530 msgr2=0x7fb66c19dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c10d530 0x7fb66c19dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 -- 172.21.15.23:0/696318236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6580095f0 con 0x7fb66c073b00 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb671200700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb66be00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 0x7fb66c199920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb65800eb90 tx=0x7fb65800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb669600700 1 -- 172.21.15.23:0/696318236 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658009970 con 0x7fb66c073b00 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb660009520 con 0x7fb66c073b00 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb669600700 1 -- 172.21.15.23:0/696318236 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6580044e0 con 0x7fb66c073b00 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.473+0000 7fb669600700 1 -- 172.21.15.23:0/696318236 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658003e40 con 0x7fb66c073b00 2024-03-30T04:11:22.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.474+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c1a5f70 con 0x7fb66c073b00 2024-03-30T04:11:22.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.477+0000 7fb669600700 1 -- 172.21.15.23:0/696318236 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb658003fe0 con 0x7fb66c073b00 2024-03-30T04:11:22.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.478+0000 7fb687758b80 1 Processor -- start 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.478+0000 7fb687758b80 1 -- start start 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.478+0000 7fb687758b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e13479b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.478+0000 7fb687758b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e13528b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb687758b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb687758b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e12a0990 con 0x55f4e13475d0 2024-03-30T04:11:22.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb687758b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e129e350 con 0x55f4e1352df0 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb687758b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1299c40 con 0x55f4e1347ef0 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb671200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e13528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38426/0 (socket says 172.21.15.23:38426) 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 -- 172.21.15.23:0/3333432844 learned_addr learned my addr 172.21.15.23:0/3333432844 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb671c00700 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e13479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 -- 172.21.15.23:0/3333432844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 msgr2=0x55f4e13528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb669600700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 0x7fb6540a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e13528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 -- 172.21.15.23:0/3333432844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 msgr2=0x55f4e13479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e13479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.479+0000 7fb66be00700 1 -- 172.21.15.23:0/3333432844 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4e111b3b0 con 0x55f4e1352df0 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.480+0000 7fb671c00700 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e13479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.480+0000 7fb669600700 1 -- 172.21.15.23:0/696318236 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb658014070 con 0x7fb66c073b00 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.480+0000 7fb671c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 0x7fb6540a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.480+0000 7fb66be00700 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6580ca890 tx=0x7fb6580695d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f724658a907f092d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.480+0000 7fb65e000700 1 -- 172.21.15.23:0/3333432844 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65809a0e0 con 0x55f4e1352df0 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb65e000700 1 -- 172.21.15.23:0/3333432844 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb658099e50 con 0x55f4e1352df0 2024-03-30T04:11:22.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb65e000700 1 -- 172.21.15.23:0/3333432844 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb658066a20 con 0x55f4e1352df0 2024-03-30T04:11:22.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb671c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 0x7fb6540a3750 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb664009ec0 tx=0x7fb664009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 -- 172.21.15.23:0/3333432844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 msgr2=0x55f4e13551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6580ca890 tx=0x7fb6580695d0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 -- 172.21.15.23:0/3333432844 shutdown_connections 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e13551e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e13479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 --2- 172.21.15.23:0/3333432844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e13528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.481+0000 7fb687758b80 1 -- 172.21.15.23:0/3333432844 >> 172.21.15.23:0/3333432844 conn(0x55f4e12a76a0 msgr2=0x55f4e1343a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.482+0000 7fb687758b80 1 -- 172.21.15.23:0/3333432844 shutdown_connections 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.482+0000 7fb687758b80 1 -- 172.21.15.23:0/3333432844 wait complete. 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.482+0000 7fb687758b80 1 -- start start 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.482+0000 7fb687758b80 1 Processor -- start 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e134faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e1350120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1235d80 con 0x55f4e13475d0 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1235a20 con 0x55f4e1352df0 2024-03-30T04:11:22.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e111d6a0 con 0x55f4e1347ef0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb687758b80 10 client.0 _ll_register_callbacks cb 0x55f4e123f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb671200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb671200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42676/0 (socket says 172.21.15.23:42676) 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb671200700 1 -- 172.21.15.23:0/3017991678 learned_addr learned my addr 172.21.15.23:0/3017991678 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb671c00700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e1350120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.483+0000 7fb66be00700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e134faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 -- 172.21.15.23:0/3017991678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 msgr2=0x55f4e134faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e134faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 -- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 msgr2=0x55f4e1350120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e1350120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 -- 172.21.15.23:0/3017991678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c1a5f70 con 0x55f4e13475d0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb66be00700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e134faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb671200700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b7c0 tx=0x7fb66000b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a983492b2f9d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb652a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66001c040 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.484+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb66000bba0 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4e125a380 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb66000f980 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 10 client.18162 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f4e10c7b60 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f4e125a380 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb660015020 con 0x55f4e13475d0 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 10 client.18162 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:22.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 20 client.18162 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13629d0 0x55f4e1364df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 4 client.18162 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.485+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4e1292700 con 0x55f4e13629d0 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.486+0000 7fb66be00700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13629d0 0x55f4e1364df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.486+0000 7fb66be00700 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13629d0 0x55f4e1364df0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f4583819c65f87f server_cookie=341b694016c8e53f in_seq=0 out_seq=0 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.486+0000 7fb652a00700 10 client.18162 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb652a00700 1 -- 172.21.15.23:0/3017991678 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f4e1292700 con 0x55f4e13629d0 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb652a00700 10 client.18162 handle_command_reply: tid=0 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 client.18162 shutdown 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 msgr2=0x55f4e1352800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb66000b7c0 tx=0x7fb66000b7f0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 shutdown_connections 2024-03-30T04:11:22.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1352df0 0x55f4e1350120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e13475d0 0x55f4e1352800 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13629d0 0x55f4e1364df0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 --2- 172.21.15.23:0/3017991678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1347ef0 0x55f4e134faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb652a00700 0 client.18162 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 >> 172.21.15.23:0/3017991678 conn(0x55f4e12a76a0 msgr2=0x55f4e13496c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.487+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 shutdown_connections 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.488+0000 7fb687758b80 1 -- 172.21.15.23:0/3017991678 wait complete. 2024-03-30T04:11:22.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.488+0000 7fb687758b80 20 client.18162 trim_cache size 0 max 16384 2024-03-30T04:11:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:22 smithi193 ceph-mon[79581]: pgmap v293: 129 pgs: 129 active+clean; 522 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 8.8 MiB/s wr, 292 op/s 2024-03-30T04:11:22.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.506+0000 7fb687758b80 1 Processor -- start 2024-03-30T04:11:22.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.506+0000 7fb687758b80 1 -- start start 2024-03-30T04:11:22.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 0x55f4e134e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1352df0 0x55f4e134e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1345160 con 0x55f4e1352df0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1344660 con 0x55f4e1347ef0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb687758b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e1359fa0 con 0x55f4e13475d0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb671200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb66be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 0x55f4e134e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb671200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:42216/0 (socket says 172.21.15.23:42216) 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.507+0000 7fb671200700 1 -- 172.21.15.23:0/3561911869 learned_addr learned my addr 172.21.15.23:0/3561911869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671c00700 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1352df0 0x55f4e134e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 -- 172.21.15.23:0/3561911869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 msgr2=0x55f4e134e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 0x55f4e134e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 -- 172.21.15.23:0/3561911869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1352df0 msgr2=0x55f4e134e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1352df0 0x55f4e134e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 -- 172.21.15.23:0/3561911869 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4e125fea0 con 0x55f4e13475d0 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb66be00700 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 0x55f4e134e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb671200700 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb660023040 tx=0x7fb660009290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25774e08e16b91af server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb65ea00700 1 -- 172.21.15.23:0/3561911869 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66001c040 con 0x55f4e13475d0 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb65ea00700 1 -- 172.21.15.23:0/3561911869 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb66001d040 con 0x55f4e13475d0 2024-03-30T04:11:22.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.508+0000 7fb65ea00700 1 -- 172.21.15.23:0/3561911869 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb660003880 con 0x55f4e13475d0 2024-03-30T04:11:22.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 -- 172.21.15.23:0/3561911869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 msgr2=0x55f4e1359a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb660023040 tx=0x7fb660009290 comp rx=0 tx=0).stop 2024-03-30T04:11:22.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 -- 172.21.15.23:0/3561911869 shutdown_connections 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1352df0 0x55f4e134e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e13475d0 0x55f4e1359a60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 --2- 172.21.15.23:0/3561911869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e1347ef0 0x55f4e134e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 -- 172.21.15.23:0/3561911869 >> 172.21.15.23:0/3561911869 conn(0x55f4e12a76a0 msgr2=0x55f4e133c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 -- 172.21.15.23:0/3561911869 shutdown_connections 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.509+0000 7fb687758b80 1 -- 172.21.15.23:0/3561911869 wait complete. 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 -- start start 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 Processor -- start 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 0x55f4e1357980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1352df0 0x55f4e13be030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e13be570 con 0x55f4e1347ef0 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e13be6b0 con 0x55f4e13475d0 2024-03-30T04:11:22.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4e13be870 con 0x55f4e1352df0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb687758b80 10 client.0 _ll_register_callbacks cb 0x55f4e123f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.510+0000 7fb66be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb66be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42702/0 (socket says 172.21.15.23:42702) 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 0x55f4e1357980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb66be00700 1 -- 172.21.15.23:0/2350204909 learned_addr learned my addr 172.21.15.23:0/2350204909 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671c00700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1352df0 0x55f4e13be030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 -- 172.21.15.23:0/2350204909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1352df0 msgr2=0x55f4e13be030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1352df0 0x55f4e13be030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 -- 172.21.15.23:0/2350204909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 msgr2=0x55f4e1357ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 -- 172.21.15.23:0/2350204909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4e111b3b0 con 0x55f4e13475d0 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb66be00700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:22.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb671200700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 0x55f4e1357980 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb660003bf0 tx=0x7fb6600038b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4416291d057305d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb652a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.511+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66001c070 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb658099e50 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb660000e40 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb660003a20 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 10 client.28154 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f4e10c7b60 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb658099e50 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb66003e070 con 0x55f4e13475d0 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 10 client.28154 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 10 client.28154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 20 client.28154 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13bfb50 0x55f4e13c1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:22.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 4 client.28154 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f4e10e8cf0 con 0x55f4e13bfb50 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.512+0000 7fb66be00700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13bfb50 0x55f4e13c1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb652a00700 10 client.28154 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb66be00700 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13bfb50 0x55f4e13c1f40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f40bb9108b3c966 server_cookie=5aca3894b969fdcc in_seq=0 out_seq=0 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb652a00700 1 -- 172.21.15.23:0/2350204909 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f4e10e8cf0 con 0x55f4e13bfb50 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb652a00700 10 client.28154 handle_command_reply: tid=0 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb687758b80 1 client.28154 shutdown 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 msgr2=0x55f4e1357980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.513+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 0x55f4e1357980 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb660003bf0 tx=0x7fb6600038b0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 shutdown_connections 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4e13bfb50 0x55f4e13c1f40 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4e1352df0 0x55f4e13be030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4e13475d0 0x55f4e1357980 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 --2- 172.21.15.23:0/2350204909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4e1347ef0 0x55f4e1357ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb652a00700 0 client.28154 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 >> 172.21.15.23:0/2350204909 conn(0x55f4e12a76a0 msgr2=0x55f4e134e320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 shutdown_connections 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 1 -- 172.21.15.23:0/2350204909 wait complete. 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb687758b80 20 client.28154 trim_cache size 0 max 16384 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout: "322a4ee0-e230-4dac-8442-f99d815db5e5": { 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:11:22.517 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "322a4ee0-e230-4dac-8442-f99d815db5e5", 2024-03-30T04:11:22.518 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:11:22.518 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:11:22.518 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:11:22.518 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:22.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 msgr2=0x7fb6540a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 0x7fb6540a3750 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb664009ec0 tx=0x7fb664009040 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 msgr2=0x7fb66c199920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.514+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 0x7fb66c199920 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb65800eb90 tx=0x7fb65800eea0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 shutdown_connections 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb66c10d530 0x7fb66c19dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb6540a1290 0x7fb6540a3750 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb66c0731e0 0x7fb66c1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 --2- 172.21.15.23:0/696318236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb66c073b00 0x7fb66c199920 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 >> 172.21.15.23:0/696318236 conn(0x7fb66c0fc7c0 msgr2=0x7fb66c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 shutdown_connections 2024-03-30T04:11:22.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:22.515+0000 7fb673c00700 1 -- 172.21.15.23:0/696318236 wait complete. 2024-03-30T04:11:22.533 INFO:tasks.cephfs.filesystem:scrub status for tag:322a4ee0-e230-4dac-8442-f99d815db5e5 - {'path': '/', 'tag': '322a4ee0-e230-4dac-8442-f99d815db5e5', 'options': 'recursive,force'} 2024-03-30T04:11:23.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 938301 10.30 MB/sec execute 360 sec latency 23.715 ms 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:23 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:23 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:11:23.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:11:23.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:11:23.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:23.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:23 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:24.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 939704 10.28 MB/sec execute 361 sec latency 21.630 ms 2024-03-30T04:11:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:24 smithi154 ceph-mon[79161]: pgmap v294: 129 pgs: 129 active+clean; 532 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 166 op/s 2024-03-30T04:11:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:24 smithi193 ceph-mon[79581]: pgmap v294: 129 pgs: 129 active+clean; 532 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 166 op/s 2024-03-30T04:11:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:24 smithi023 ceph-mon[38254]: pgmap v294: 129 pgs: 129 active+clean; 532 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 5.0 MiB/s wr, 166 op/s 2024-03-30T04:11:25.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 941870 10.29 MB/sec execute 362 sec latency 32.125 ms 2024-03-30T04:11:26.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 943255 10.28 MB/sec execute 363 sec latency 22.974 ms 2024-03-30T04:11:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:26 smithi154 ceph-mon[79161]: pgmap v295: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 158 op/s 2024-03-30T04:11:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:26 smithi193 ceph-mon[79581]: pgmap v295: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 158 op/s 2024-03-30T04:11:26.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:26 smithi023 ceph-mon[38254]: pgmap v295: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 4.5 MiB/s wr, 158 op/s 2024-03-30T04:11:27.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 945425 10.29 MB/sec execute 364 sec latency 21.676 ms 2024-03-30T04:11:28.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 946809 10.27 MB/sec execute 365 sec latency 27.045 ms 2024-03-30T04:11:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:28 smithi154 ceph-mon[79161]: pgmap v296: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 157 op/s 2024-03-30T04:11:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:28 smithi193 ceph-mon[79581]: pgmap v296: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 157 op/s 2024-03-30T04:11:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:28 smithi023 ceph-mon[38254]: pgmap v296: 129 pgs: 129 active+clean; 529 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 4.4 MiB/s wr, 157 op/s 2024-03-30T04:11:29.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 948939 10.28 MB/sec execute 366 sec latency 29.830 ms 2024-03-30T04:11:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:29 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:11:29.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:29 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:11:29.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:29 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:11:29.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:29 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:11:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:29 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:11:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:29 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:11:30.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 950344 10.27 MB/sec execute 367 sec latency 23.360 ms 2024-03-30T04:11:30.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:30 smithi154 ceph-mon[79161]: pgmap v297: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 195 op/s 2024-03-30T04:11:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:30 smithi193 ceph-mon[79581]: pgmap v297: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 195 op/s 2024-03-30T04:11:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:30 smithi023 ceph-mon[38254]: pgmap v297: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 195 op/s 2024-03-30T04:11:31.220 INFO:tasks.workunit.client.0.smithi023.stdout: 1 952513 10.28 MB/sec execute 368 sec latency 19.965 ms 2024-03-30T04:11:32.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 953891 10.27 MB/sec execute 369 sec latency 25.377 ms 2024-03-30T04:11:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:32 smithi154 ceph-mon[79161]: pgmap v298: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:32 smithi193 ceph-mon[79581]: pgmap v298: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:32 smithi023 ceph-mon[38254]: pgmap v298: 129 pgs: 129 active+clean; 535 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:33.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 956067 10.28 MB/sec execute 370 sec latency 18.694 ms 2024-03-30T04:11:34.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 957448 10.26 MB/sec execute 371 sec latency 31.853 ms 2024-03-30T04:11:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:34 smithi154 ceph-mon[79161]: pgmap v299: 129 pgs: 129 active+clean; 537 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-03-30T04:11:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:34 smithi193 ceph-mon[79581]: pgmap v299: 129 pgs: 129 active+clean; 537 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-03-30T04:11:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:34 smithi023 ceph-mon[38254]: pgmap v299: 129 pgs: 129 active+clean; 537 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-03-30T04:11:35.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 959627 10.27 MB/sec execute 372 sec latency 21.407 ms 2024-03-30T04:11:36.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 961061 10.26 MB/sec execute 373 sec latency 26.296 ms 2024-03-30T04:11:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:36 smithi193 ceph-mon[79581]: pgmap v300: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-03-30T04:11:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:36 smithi023 ceph-mon[38254]: pgmap v300: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-03-30T04:11:36.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:36 smithi154 ceph-mon[79161]: pgmap v300: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 135 op/s 2024-03-30T04:11:37.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 963259 10.27 MB/sec execute 374 sec latency 20.055 ms 2024-03-30T04:11:37.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:37 smithi193 ceph-mon[79581]: pgmap v301: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 127 op/s 2024-03-30T04:11:37.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:37 smithi023 ceph-mon[38254]: pgmap v301: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 127 op/s 2024-03-30T04:11:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:37 smithi154 ceph-mon[79161]: pgmap v301: 129 pgs: 129 active+clean; 541 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 127 op/s 2024-03-30T04:11:38.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 964539 10.26 MB/sec execute 375 sec latency 27.678 ms 2024-03-30T04:11:39.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 966687 10.26 MB/sec execute 376 sec latency 17.982 ms 2024-03-30T04:11:39.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:39 smithi193 ceph-mon[79581]: pgmap v302: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-03-30T04:11:39.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:39 smithi023 ceph-mon[38254]: pgmap v302: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-03-30T04:11:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:39 smithi154 ceph-mon[79161]: pgmap v302: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-03-30T04:11:40.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 968070 10.25 MB/sec execute 377 sec latency 35.353 ms 2024-03-30T04:11:41.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 970222 10.26 MB/sec execute 378 sec latency 26.325 ms 2024-03-30T04:11:42.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 971616 10.25 MB/sec execute 379 sec latency 24.458 ms 2024-03-30T04:11:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:42 smithi154 ceph-mon[79161]: pgmap v303: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:42 smithi193 ceph-mon[79581]: pgmap v303: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:42 smithi023 ceph-mon[38254]: pgmap v303: 129 pgs: 129 active+clean; 543 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.9 MiB/s wr, 126 op/s 2024-03-30T04:11:43.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 973838 10.26 MB/sec execute 380 sec latency 22.814 ms 2024-03-30T04:11:44.221 INFO:tasks.workunit.client.0.smithi023.stdout: 1 975127 10.24 MB/sec execute 381 sec latency 29.569 ms 2024-03-30T04:11:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:44 smithi154 ceph-mon[79161]: pgmap v304: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-30T04:11:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:44 smithi193 ceph-mon[79581]: pgmap v304: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-30T04:11:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:44 smithi023 ceph-mon[38254]: pgmap v304: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 185 op/s 2024-03-30T04:11:45.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 977260 10.25 MB/sec execute 382 sec latency 24.005 ms 2024-03-30T04:11:46.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 978513 10.24 MB/sec execute 383 sec latency 27.073 ms 2024-03-30T04:11:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:46 smithi154 ceph-mon[79161]: pgmap v305: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-03-30T04:11:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:46 smithi193 ceph-mon[79581]: pgmap v305: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-03-30T04:11:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:46 smithi023 ceph-mon[38254]: pgmap v305: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 147 op/s 2024-03-30T04:11:47.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 980631 10.25 MB/sec execute 384 sec latency 39.477 ms 2024-03-30T04:11:48.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 981992 10.24 MB/sec execute 385 sec latency 23.441 ms 2024-03-30T04:11:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:48 smithi154 ceph-mon[79161]: pgmap v306: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-03-30T04:11:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:48 smithi193 ceph-mon[79581]: pgmap v306: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-03-30T04:11:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:48 smithi023 ceph-mon[38254]: pgmap v306: 129 pgs: 129 active+clean; 550 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 123 op/s 2024-03-30T04:11:49.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 984123 10.24 MB/sec execute 386 sec latency 33.421 ms 2024-03-30T04:11:50.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 985518 10.23 MB/sec execute 387 sec latency 23.598 ms 2024-03-30T04:11:50.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:50 smithi154 ceph-mon[79161]: pgmap v307: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 200 op/s 2024-03-30T04:11:50.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:50 smithi193 ceph-mon[79581]: pgmap v307: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 200 op/s 2024-03-30T04:11:50.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:50 smithi023 ceph-mon[38254]: pgmap v307: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 200 op/s 2024-03-30T04:11:51.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 987120 10.23 MB/sec execute 388 sec latency 129.165 ms 2024-03-30T04:11:52.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 988390 10.22 MB/sec execute 389 sec latency 25.138 ms 2024-03-30T04:11:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:52 smithi154 ceph-mon[79161]: pgmap v308: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-30T04:11:52.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:52 smithi193 ceph-mon[79581]: pgmap v308: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-30T04:11:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:52 smithi023 ceph-mon[38254]: pgmap v308: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 138 op/s 2024-03-30T04:11:52.534 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:11:52.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.752+0000 7efe45000700 1 -- 172.21.15.23:0/421773345 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40106090 msgr2=0x7efe4010dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.752+0000 7efe45000700 1 --2- 172.21.15.23:0/421773345 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40106090 0x7efe4010dc10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7efe38009f90 tx=0x7efe3800b330 comp rx=0 tx=0).stop 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 -- 172.21.15.23:0/421773345 shutdown_connections 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 --2- 172.21.15.23:0/421773345 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40106090 0x7efe4010dc10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 --2- 172.21.15.23:0/421773345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40100f70 0x7efe40101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 --2- 172.21.15.23:0/421773345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40101920 0x7efe40105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 -- 172.21.15.23:0/421773345 >> 172.21.15.23:0/421773345 conn(0x7efe400fc7f0 msgr2=0x7efe400fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 -- 172.21.15.23:0/421773345 shutdown_connections 2024-03-30T04:11:52.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.753+0000 7efe45000700 1 -- 172.21.15.23:0/421773345 wait complete. 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 Processor -- start 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 -- start start 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40100f70 0x7efe400730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40073b30 0x7efe401a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe401a5ac0 con 0x7efe40073b30 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe401a5c00 con 0x7efe40106090 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe45000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe401a5dc0 con 0x7efe40100f70 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe3d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe3de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40100f70 0x7efe400730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.754+0000 7efe3d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54606/0 (socket says 172.21.15.23:54606) 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 -- 172.21.15.23:0/3620547244 learned_addr learned my addr 172.21.15.23:0/3620547244 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3e800700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40073b30 0x7efe401a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 -- 172.21.15.23:0/3620547244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40100f70 msgr2=0x7efe400730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40100f70 0x7efe400730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 -- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40073b30 msgr2=0x7efe401a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40073b30 0x7efe401a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 -- 172.21.15.23:0/3620547244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c0095f0 con 0x7efe40106090 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe3d400700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00ec70 tx=0x7efe2c00ef80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.755+0000 7efe36a00700 1 -- 172.21.15.23:0/3620547244 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c0099d0 con 0x7efe40106090 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.756+0000 7efe36a00700 1 -- 172.21.15.23:0/3620547244 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe2c0044e0 con 0x7efe40106090 2024-03-30T04:11:52.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.756+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe38009c40 con 0x7efe40106090 2024-03-30T04:11:52.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.756+0000 7efe36a00700 1 -- 172.21.15.23:0/3620547244 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2c010430 con 0x7efe40106090 2024-03-30T04:11:52.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.756+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe401a6420 con 0x7efe40106090 2024-03-30T04:11:52.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.761+0000 7efe36a00700 1 -- 172.21.15.23:0/3620547244 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7efe2c003f10 con 0x7efe40106090 2024-03-30T04:11:52.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.762+0000 7efe58b93b80 1 Processor -- start 2024-03-30T04:11:52.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.762+0000 7efe58b93b80 1 -- start start 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c60458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c60481e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5f93990 con 0x55e6c6045df0 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5f91350 con 0x55e6c603a5d0 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe58b93b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5f8cc40 con 0x55e6c603aef0 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe3e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe3e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54622/0 (socket says 172.21.15.23:54622) 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe3e800700 1 -- 172.21.15.23:0/4147828395 learned_addr learned my addr 172.21.15.23:0/4147828395 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe3d400700 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c60481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.763+0000 7efe3de00700 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c60458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 -- 172.21.15.23:0/4147828395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 msgr2=0x55e6c60458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c60458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 -- 172.21.15.23:0/4147828395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 msgr2=0x55e6c60481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c60481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe36a00700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 0x7efe240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 -- 172.21.15.23:0/4147828395 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6c5e0e3b0 con 0x55e6c603a5d0 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe36a00700 1 -- 172.21.15.23:0/3620547244 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7efe2c014070 con 0x7efe40106090 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efe38009520 tx=0x7efe38009ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=802e464b3b5b01b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.764+0000 7efe3e800700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 0x7efe240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe2b400700 1 -- 172.21.15.23:0/4147828395 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe38030040 con 0x55e6c603a5d0 2024-03-30T04:11:52.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe2b400700 1 -- 172.21.15.23:0/4147828395 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe38007840 con 0x55e6c603a5d0 2024-03-30T04:11:52.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe2b400700 1 -- 172.21.15.23:0/4147828395 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3800e040 con 0x55e6c603a5d0 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe58b93b80 1 -- 172.21.15.23:0/4147828395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 msgr2=0x55e6c603a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe3e800700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 0x7efe240a3750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe3802b000 tx=0x7efe3802c660 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.765+0000 7efe58b93b80 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efe38009520 tx=0x7efe38009ac0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 -- 172.21.15.23:0/4147828395 shutdown_connections 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c60481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c603a9b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 --2- 172.21.15.23:0/4147828395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c60458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 -- 172.21.15.23:0/4147828395 >> 172.21.15.23:0/4147828395 conn(0x55e6c5f9a6a0 msgr2=0x55e6c6036a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 -- 172.21.15.23:0/4147828395 shutdown_connections 2024-03-30T04:11:52.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 -- 172.21.15.23:0/4147828395 wait complete. 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 -- start start 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.766+0000 7efe58b93b80 1 Processor -- start 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c6045790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c6042a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5f28d80 con 0x55e6c603aef0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5f28a20 con 0x55e6c603a5d0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c5e106a0 con 0x55e6c6045df0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe58b93b80 10 client.0 _ll_register_callbacks cb 0x55e6c5f32470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c6042a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c6045790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c6045790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54630/0 (socket says 172.21.15.23:54630) 2024-03-30T04:11:52.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3de00700 1 -- 172.21.15.23:0/3576848992 learned_addr learned my addr 172.21.15.23:0/3576848992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.767+0000 7efe3e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35914/0 (socket says 172.21.15.23:35914) 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 -- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 msgr2=0x55e6c6045790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c6045790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 -- 172.21.15.23:0/3576848992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 msgr2=0x55e6c6042a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c6042a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 -- 172.21.15.23:0/3576848992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4010e480 con 0x55e6c6045df0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe3e800700 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7efe38009520 tx=0x7efe38034000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=689f7868d8b7fb44 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe1fe00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3800e030 con 0x55e6c6045df0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7efe38003ab0 con 0x55e6c6045df0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.768+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6c5f4d380 con 0x55e6c6045df0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe38030040 con 0x55e6c6045df0 2024-03-30T04:11:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 10 client.28156 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e6c5dbab60 con 0x55e6c6045df0 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e6c5f4d380 con 0x55e6c6045df0 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efe380050b0 con 0x55e6c6045df0 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 10 client.28156 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 10 client.28156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 20 client.28156 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c6055530 0x55e6c6057920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 4 client.28156 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6c5f85700 con 0x55e6c6055530 2024-03-30T04:11:52.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.769+0000 7efe3de00700 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c6055530 0x55e6c6057920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.770+0000 7efe3de00700 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c6055530 0x55e6c6057920 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c041dcd74aed0c server_cookie=6bfa5e23d48d5969 in_seq=0 out_seq=0 2024-03-30T04:11:52.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.770+0000 7efe1fe00700 10 client.28156 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:52.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe1fe00700 1 -- 172.21.15.23:0/3576848992 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e6c5f85700 con 0x55e6c6055530 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe1fe00700 10 client.28156 handle_command_reply: tid=0 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 client.28156 shutdown 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 msgr2=0x55e6c6042fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7efe38009520 tx=0x7efe38034000 comp rx=0 tx=0).stop 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 shutdown_connections 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c6055530 0x55e6c6057920 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c6045df0 0x55e6c6042fd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe1fe00700 0 client.28156 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c6045790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3576848992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c6042a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 >> 172.21.15.23:0/3576848992 conn(0x55e6c5f9a6a0 msgr2=0x55e6c603c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 shutdown_connections 2024-03-30T04:11:52.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 1 -- 172.21.15.23:0/3576848992 wait complete. 2024-03-30T04:11:52.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.771+0000 7efe58b93b80 20 client.28156 trim_cache size 0 max 16384 2024-03-30T04:11:52.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 Processor -- start 2024-03-30T04:11:52.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 -- start start 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c60452c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c6045df0 0x55e6c6051160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c603d830 con 0x55e6c603aef0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c604cf50 con 0x55e6c6045df0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe58b93b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c6040f60 con 0x55e6c603a5d0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe3de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe3d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c60452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe3de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35920/0 (socket says 172.21.15.23:35920) 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe3de00700 1 -- 172.21.15.23:0/3401756625 learned_addr learned my addr 172.21.15.23:0/3401756625 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.791+0000 7efe3e800700 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c6045df0 0x55e6c6051160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 -- 172.21.15.23:0/3401756625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c6045df0 msgr2=0x55e6c6051160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c6045df0 0x55e6c6051160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 -- 172.21.15.23:0/3401756625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 msgr2=0x55e6c60452c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c60452c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 -- 172.21.15.23:0/3401756625 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6c5f52ea0 con 0x55e6c603a5d0 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe3de00700 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55e6c60581f0 tx=0x7efe3001fcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7282a771f5a882c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe2be00700 1 -- 172.21.15.23:0/3401756625 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3001f050 con 0x55e6c603a5d0 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe2be00700 1 -- 172.21.15.23:0/3401756625 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe30023070 con 0x55e6c603a5d0 2024-03-30T04:11:52.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.792+0000 7efe2be00700 1 -- 172.21.15.23:0/3401756625 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3001e710 con 0x55e6c603a5d0 2024-03-30T04:11:52.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- 172.21.15.23:0/3401756625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 msgr2=0x55e6c604ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55e6c60581f0 tx=0x7efe3001fcd0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- 172.21.15.23:0/3401756625 shutdown_connections 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c6045df0 0x55e6c6051160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603a5d0 0x55e6c604ca10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 --2- 172.21.15.23:0/3401756625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c603aef0 0x55e6c60452c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- 172.21.15.23:0/3401756625 >> 172.21.15.23:0/3401756625 conn(0x55e6c5f9a6a0 msgr2=0x55e6c602f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- 172.21.15.23:0/3401756625 shutdown_connections 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- 172.21.15.23:0/3401756625 wait complete. 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.793+0000 7efe58b93b80 1 -- start start 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 Processor -- start 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c604abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c604b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c604b690 con 0x55e6c6045df0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c604b800 con 0x55e6c603a5d0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c604b9c0 con 0x55e6c603aef0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe58b93b80 10 client.0 _ll_register_callbacks cb 0x55e6c5f32470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe3de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54648/0 (socket says 172.21.15.23:54648) 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe3de00700 1 -- 172.21.15.23:0/2948444733 learned_addr learned my addr 172.21.15.23:0/2948444733 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe3e800700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c604b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.794+0000 7efe3d400700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c604abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 -- 172.21.15.23:0/2948444733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 msgr2=0x55e6c604abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c604abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 -- 172.21.15.23:0/2948444733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 msgr2=0x55e6c604b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c604b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 -- 172.21.15.23:0/2948444733 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe30009ca0 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe3de00700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efe30010f00 tx=0x7efe300108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7377dcf8b6bb09aa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe1fe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe30010b20 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7efe30010c80 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.795+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe38007840 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe3001f020 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 10 client.28172 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e6c5e12cd0 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe38007840 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7efe30004820 con 0x55e6c603a5d0 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 10 client.28172 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 10 client.28172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 20 client.28172 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:52.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c60aa8f0 0x55e6c60acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 4 client.28172 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e6c5ddbcf0 con 0x55e6c60aa8f0 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.796+0000 7efe3d400700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c60aa8f0 0x55e6c60acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.797+0000 7efe1fe00700 10 client.28172 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.797+0000 7efe3d400700 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c60aa8f0 0x55e6c60acd10 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f06f575f79e8e8de server_cookie=d97e501a8184687 in_seq=0 out_seq=0 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.797+0000 7efe1fe00700 1 -- 172.21.15.23:0/2948444733 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e6c5ddbcf0 con 0x55e6c60aa8f0 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.797+0000 7efe1fe00700 10 client.28172 handle_command_reply: tid=0 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.797+0000 7efe58b93b80 1 client.28172 shutdown 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 msgr2=0x55e6c604c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efe30010f00 tx=0x7efe300108c0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 shutdown_connections 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e6c6045df0 0x55e6c604b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e6c603a5d0 0x55e6c604c6e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e6c60aa8f0 0x55e6c60acd10 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 --2- 172.21.15.23:0/2948444733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6c603aef0 0x55e6c604abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 >> 172.21.15.23:0/2948444733 conn(0x55e6c5f9a6a0 msgr2=0x55e6c603c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe1fe00700 0 client.28172 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 shutdown_connections 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 1 -- 172.21.15.23:0/2948444733 wait complete. 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe58b93b80 20 client.28172 trim_cache size 0 max 16384 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:11:52.801 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 msgr2=0x7efe240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 0x7efe240a3750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe3802b000 tx=0x7efe3802c660 comp rx=0 tx=0).stop 2024-03-30T04:11:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 msgr2=0x7efe400735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.798+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7efe2c00ec70 tx=0x7efe2c00ef80 comp rx=0 tx=0).stop 2024-03-30T04:11:52.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 shutdown_connections 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efe40073b30 0x7efe401a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe40100f70 0x7efe400730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efe240a1290 0x7efe240a3750 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 --2- 172.21.15.23:0/3620547244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efe40106090 0x7efe400735f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 >> 172.21.15.23:0/3620547244 conn(0x7efe400fc7f0 msgr2=0x7efe400fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 shutdown_connections 2024-03-30T04:11:52.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:52.799+0000 7efe45000700 1 -- 172.21.15.23:0/3620547244 wait complete. 2024-03-30T04:11:52.816 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:11:52.816 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:11:53.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.036+0000 7f7d7f200700 1 -- 172.21.15.23:0/522350128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d78073b00 msgr2=0x7f7d7810cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.036+0000 7f7d7f200700 1 --2- 172.21.15.23:0/522350128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d78073b00 0x7f7d7810cef0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009990 tx=0x7f7d64009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 -- 172.21.15.23:0/522350128 shutdown_connections 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 --2- 172.21.15.23:0/522350128 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d7810d430 0x7f7d7810f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 --2- 172.21.15.23:0/522350128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d780735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 --2- 172.21.15.23:0/522350128 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d78073b00 0x7f7d7810cef0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 -- 172.21.15.23:0/522350128 >> 172.21.15.23:0/522350128 conn(0x7f7d780fc700 msgr2=0x7f7d780feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 -- 172.21.15.23:0/522350128 shutdown_connections 2024-03-30T04:11:53.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.037+0000 7f7d7f200700 1 -- 172.21.15.23:0/522350128 wait complete. 2024-03-30T04:11:53.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 Processor -- start 2024-03-30T04:11:53.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 -- start start 2024-03-30T04:11:53.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d78073b00 0x7f7d78199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 0x7f7d7819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7819e300 con 0x7f7d780731e0 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7819e470 con 0x7f7d7810d430 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.038+0000 7f7d7f200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d7819e630 con 0x7f7d78073b00 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d7d200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 0x7f7d7819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49052/0 (socket says 172.21.15.23:49052) 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 -- 172.21.15.23:0/2550313658 learned_addr learned my addr 172.21.15.23:0/2550313658 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77400700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d78073b00 0x7f7d78199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 -- 172.21.15.23:0/2550313658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d78073b00 msgr2=0x7f7d78199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d78073b00 0x7f7d78199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 -- 172.21.15.23:0/2550313658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 msgr2=0x7f7d7819dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 0x7f7d7819dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 -- 172.21.15.23:0/2550313658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640095f0 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d7d200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 0x7f7d7819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.039+0000 7f7d77e00700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9a0 tx=0x7f7d6800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.040+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68009790 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.040+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d68010430 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.040+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d68018680 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.040+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d7819e910 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.040+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d781a6150 con 0x7f7d780731e0 2024-03-30T04:11:53.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.041+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d7819eb30 con 0x7f7d780731e0 2024-03-30T04:11:53.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.043+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f7d680105a0 con 0x7f7d780731e0 2024-03-30T04:11:53.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.044+0000 7f7d74c00700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 0x7f7d5c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.044+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7d68014070 con 0x7f7d780731e0 2024-03-30T04:11:53.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.045+0000 7f7d77400700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 0x7f7d5c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.045+0000 7f7d77400700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 0x7f7d5c0a3750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d7819a930 tx=0x7f7d6400b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.047+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f7d68096350 con 0x7f7d780731e0 2024-03-30T04:11:53.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 990182 10.21 MB/sec execute 390 sec latency 21.220 ms 2024-03-30T04:11:53.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.341+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d78108640 con 0x7f7d780731e0 2024-03-30T04:11:53.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.342+0000 7f7d74c00700 1 -- 172.21.15.23:0/2550313658 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f7d68026090 con 0x7f7d780731e0 2024-03-30T04:11:53.340 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:11:53.340 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:11:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.344+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 msgr2=0x7f7d5c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.344+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 0x7f7d5c0a3750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7d7819a930 tx=0x7f7d6400b260 comp rx=0 tx=0).stop 2024-03-30T04:11:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.344+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 msgr2=0x7f7d781993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.344+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9a0 tx=0x7f7d6800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 shutdown_connections 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7d5c0a1290 0x7f7d5c0a3750 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d7810d430 0x7f7d7819dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7d780731e0 0x7f7d781993e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 --2- 172.21.15.23:0/2550313658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d78073b00 0x7f7d78199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 >> 172.21.15.23:0/2550313658 conn(0x7f7d780fc700 msgr2=0x7f7d780fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 shutdown_connections 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.345+0000 7f7d7f200700 1 -- 172.21.15.23:0/2550313658 wait complete. 2024-03-30T04:11:53.344 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:11:53.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:11:53.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 -- 172.21.15.23:0/4228335513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 msgr2=0x7f8488101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 --2- 172.21.15.23:0/4228335513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f8488101340 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8478009990 tx=0x7f8478009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 -- 172.21.15.23:0/4228335513 shutdown_connections 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 --2- 172.21.15.23:0/4228335513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f848810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 --2- 172.21.15.23:0/4228335513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f8488101340 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 --2- 172.21.15.23:0/4228335513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488101910 0x7f8488105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 -- 172.21.15.23:0/4228335513 >> 172.21.15.23:0/4228335513 conn(0x7f84880fc7c0 msgr2=0x7f84880febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 -- 172.21.15.23:0/4228335513 shutdown_connections 2024-03-30T04:11:53.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.595+0000 7f848e200700 1 -- 172.21.15.23:0/4228335513 wait complete. 2024-03-30T04:11:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.597+0000 7f848e200700 1 Processor -- start 2024-03-30T04:11:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.597+0000 7f848e200700 1 -- start start 2024-03-30T04:11:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.597+0000 7f848e200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f84880730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.597+0000 7f848e200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f84880735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.597+0000 7f848e200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488073b30 0x7f84881a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f848e200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881a59c0 con 0x7f8488073b30 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f848e200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881a5b30 con 0x7f8488100f60 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f848e200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881a5cf0 con 0x7f8488106080 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f84880730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f84880735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f84880730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54692/0 (socket says 172.21.15.23:54692) 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8487400700 1 -- 172.21.15.23:0/3396894452 learned_addr learned my addr 172.21.15.23:0/3396894452 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8487e00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488073b30 0x7f84881a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 -- 172.21.15.23:0/3396894452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 msgr2=0x7f84880730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f84880730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 -- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488073b30 msgr2=0x7f84881a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488073b30 0x7f84881a5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 -- 172.21.15.23:0/3396894452 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84780095f0 con 0x7f8488106080 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.598+0000 7f8486a00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f84880735f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f847400e9c0 tx=0x7f847400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.599+0000 7f847fe00700 1 -- 172.21.15.23:0/3396894452 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847400c9c0 con 0x7f8488106080 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.599+0000 7f847fe00700 1 -- 172.21.15.23:0/3396894452 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8474010430 con 0x7f8488106080 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.599+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84881a5fd0 con 0x7f8488106080 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.599+0000 7f847fe00700 1 -- 172.21.15.23:0/3396894452 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474018680 con 0x7f8488106080 2024-03-30T04:11:53.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.599+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84881a6450 con 0x7f8488106080 2024-03-30T04:11:53.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.602+0000 7f847fe00700 1 -- 172.21.15.23:0/3396894452 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f847400cb20 con 0x7f8488106080 2024-03-30T04:11:53.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.603+0000 7f84a1d21b80 1 Processor -- start 2024-03-30T04:11:53.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.603+0000 7f847fe00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 0x7f846c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.603+0000 7f84a1d21b80 1 -- start start 2024-03-30T04:11:53.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.603+0000 7f847fe00700 1 -- 172.21.15.23:0/3396894452 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8474014070 con 0x7f8488106080 2024-03-30T04:11:53.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487e00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 0x7f846c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b631e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23aae990 con 0x564f23b60df0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23aac350 con 0x564f23b555d0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23aa7c40 con 0x564f23b55ef0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54702/0 (socket says 172.21.15.23:54702) 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 -- 172.21.15.23:0/4224564932 learned_addr learned my addr 172.21.15.23:0/4224564932 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487e00700 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8486a00700 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 -- 172.21.15.23:0/4224564932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 msgr2=0x564f23b608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 -- 172.21.15.23:0/4224564932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 msgr2=0x564f23b631e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b631e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.604+0000 7f8487400700 1 -- 172.21.15.23:0/4224564932 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f239293b0 con 0x564f23b555d0 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.605+0000 7f8487e00700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 0x7f846c0a3750 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f848000f870 tx=0x7f848000fc30 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.605+0000 7f8487400700 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f847800b640 tx=0x7f847800b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2c484c17cf7ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.605+0000 7f8473e00700 1 -- 172.21.15.23:0/4224564932 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847801d070 con 0x564f23b555d0 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.605+0000 7f8473e00700 1 -- 172.21.15.23:0/4224564932 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f847801e040 con 0x564f23b555d0 2024-03-30T04:11:53.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.605+0000 7f8473e00700 1 -- 172.21.15.23:0/4224564932 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478004900 con 0x564f23b555d0 2024-03-30T04:11:53.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- 172.21.15.23:0/4224564932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 msgr2=0x564f23b559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f847800b640 tx=0x7f847800b670 comp rx=0 tx=0).stop 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- 172.21.15.23:0/4224564932 shutdown_connections 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b559b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/4224564932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- 172.21.15.23:0/4224564932 >> 172.21.15.23:0/4224564932 conn(0x564f23ab56a0 msgr2=0x564f23b51a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- 172.21.15.23:0/4224564932 shutdown_connections 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- 172.21.15.23:0/4224564932 wait complete. 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.606+0000 7f84a1d21b80 1 -- start start 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 Processor -- start 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b5df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23a43d80 con 0x564f23b55ef0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23a43a20 con 0x564f23b555d0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f2392b6a0 con 0x564f23b60df0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.607+0000 7f84a1d21b80 10 client.0 _ll_register_callbacks cb 0x564f23a4d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8486a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8487e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b5df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54706/0 (socket says 172.21.15.23:54706) 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8487400700 1 -- 172.21.15.23:0/523167402 learned_addr learned my addr 172.21.15.23:0/523167402 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8486a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49082/0 (socket says 172.21.15.23:49082) 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.608+0000 7f8487400700 1 -- 172.21.15.23:0/523167402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 msgr2=0x564f23b5df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487400700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b5df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487400700 1 -- 172.21.15.23:0/523167402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 msgr2=0x564f23b5da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487400700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487400700 1 -- 172.21.15.23:0/523167402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848810e470 con 0x564f23b555d0 2024-03-30T04:11:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8486a00700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:53.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487e00700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.609+0000 7f8487400700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8478004540 tx=0x7f8478005c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc846e93e2e1150e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f8470c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847801d030 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f23a68380 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f84780045a0 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8478004760 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f84a1d21b80 10 client.28190 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564f238d5b60 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.610+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8478044080 con 0x564f23b555d0 2024-03-30T04:11:53.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f847800f950 con 0x564f23b555d0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 10 client.28190 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 10 client.28190 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 20 client.28190 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23b703f0 0x564f23b727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 4 client.28190 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f23aa0700 con 0x564f23b703f0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.611+0000 7f8486a00700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23b703f0 0x564f23b727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.612+0000 7f8486a00700 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23b703f0 0x564f23b727e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66ce8090e37cccf7 server_cookie=71796a0dee5f8eb0 in_seq=0 out_seq=0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.612+0000 7f8470c00700 10 client.28190 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f8470c00700 1 -- 172.21.15.23:0/523167402 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564f23aa0700 con 0x564f23b703f0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f8470c00700 10 client.28190 handle_command_reply: tid=0 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 client.28190 shutdown 2024-03-30T04:11:53.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 msgr2=0x564f23b60740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f8478004540 tx=0x7f8478005c40 comp rx=0 tx=0).stop 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 shutdown_connections 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23b703f0 0x564f23b727e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b60740 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/523167402 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b55ef0 0x564f23b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f8470c00700 0 client.28190 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 >> 172.21.15.23:0/523167402 conn(0x564f23ab56a0 msgr2=0x564f23b57510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 shutdown_connections 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.613+0000 7f84a1d21b80 1 -- 172.21.15.23:0/523167402 wait complete. 2024-03-30T04:11:53.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.614+0000 7f84a1d21b80 20 client.28190 trim_cache size 0 max 16384 2024-03-30T04:11:53.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.629+0000 7f84a1d21b80 1 Processor -- start 2024-03-30T04:11:53.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.629+0000 7f84a1d21b80 1 -- start start 2024-03-30T04:11:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 0x564f23b67a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b58d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b5e590 con 0x564f23b555d0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b67f50 con 0x564f23b55ef0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b5bf80 con 0x564f23b60df0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54718/0 (socket says 172.21.15.23:54718) 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 -- 172.21.15.23:0/1753455914 learned_addr learned my addr 172.21.15.23:0/1753455914 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8487400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 0x564f23b67a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 -- 172.21.15.23:0/1753455914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 msgr2=0x564f23b58d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8487e00700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b58d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b58d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 -- 172.21.15.23:0/1753455914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 msgr2=0x564f23b67a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 0x564f23b67a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8486a00700 1 -- 172.21.15.23:0/1753455914 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f23a6dea0 con 0x564f23b55ef0 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8487e00700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b58d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.630+0000 7f8487400700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 0x564f23b67a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.631+0000 7f8486a00700 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f847400c5b0 tx=0x7f8474096290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8c64e57a36d1c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.631+0000 7f847d200700 1 -- 172.21.15.23:0/1753455914 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474099ed0 con 0x564f23b55ef0 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.631+0000 7f847d200700 1 -- 172.21.15.23:0/1753455914 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f847409dd00 con 0x564f23b55ef0 2024-03-30T04:11:53.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.631+0000 7f847d200700 1 -- 172.21.15.23:0/1753455914 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8474098300 con 0x564f23b55ef0 2024-03-30T04:11:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- 172.21.15.23:0/1753455914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 msgr2=0x564f23b4a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f847400c5b0 tx=0x7f8474096290 comp rx=0 tx=0).stop 2024-03-30T04:11:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- 172.21.15.23:0/1753455914 shutdown_connections 2024-03-30T04:11:53.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b60df0 0x564f23b58d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b555d0 0x564f23b67a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/1753455914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b55ef0 0x564f23b4a5a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- 172.21.15.23:0/1753455914 >> 172.21.15.23:0/1753455914 conn(0x564f23ab56a0 msgr2=0x564f23b770c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- 172.21.15.23:0/1753455914 shutdown_connections 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- 172.21.15.23:0/1753455914 wait complete. 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.632+0000 7f84a1d21b80 1 -- start start 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 Processor -- start 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b67a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b65bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b662d0 con 0x564f23b60df0 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b66440 con 0x564f23b555d0 2024-03-30T04:11:53.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.633+0000 7f84a1d21b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f23b66600 con 0x564f23b55ef0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f84a1d21b80 10 client.0 _ll_register_callbacks cb 0x564f23a4d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54728/0 (socket says 172.21.15.23:54728) 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 -- 172.21.15.23:0/2880947738 learned_addr learned my addr 172.21.15.23:0/2880947738 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8486a00700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b67a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487e00700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b65bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 -- 172.21.15.23:0/2880947738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 msgr2=0x564f23b67a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b67a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 -- 172.21.15.23:0/2880947738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 msgr2=0x564f23b65bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b65bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 -- 172.21.15.23:0/2880947738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84881a6450 con 0x564f23b555d0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8487400700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f847800b2e0 tx=0x7f847800f710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e029d7f635a1561 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.634+0000 7f8470c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:53.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847801d030 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f847801e040 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8478045030 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8478005350 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f84a1d21b80 10 client.28196 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564f2392dcd0 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f847801e040 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.635+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8478004600 con 0x564f23b555d0 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 10 client.28196 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 10 client.28196 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 20 client.28196 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23bc50a0 0x564f23bc74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 4 client.28196 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:53.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564f238f6cf0 con 0x564f23bc50a0 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f8486a00700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23bc50a0 0x564f23bc74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f8470c00700 10 client.28196 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.636+0000 7f8486a00700 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23bc50a0 0x564f23bc74c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f7dd259a502ebe8b server_cookie=5ab762581056d916 in_seq=0 out_seq=0 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f8470c00700 1 -- 172.21.15.23:0/2880947738 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564f238f6cf0 con 0x564f23bc50a0 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f8470c00700 10 client.28196 handle_command_reply: tid=0 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 client.28196 shutdown 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 msgr2=0x564f23b674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f847800b2e0 tx=0x7f847800f710 comp rx=0 tx=0).stop 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 shutdown_connections 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564f23b60df0 0x564f23b65bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564f23b555d0 0x564f23b674d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564f23bc50a0 0x564f23bc74c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 --2- 172.21.15.23:0/2880947738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564f23b55ef0 0x564f23b67a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f8470c00700 0 client.28196 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:11:53.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 >> 172.21.15.23:0/2880947738 conn(0x564f23ab56a0 msgr2=0x564f23b58d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 shutdown_connections 2024-03-30T04:11:53.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.637+0000 7f84a1d21b80 1 -- 172.21.15.23:0/2880947738 wait complete. 2024-03-30T04:11:53.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f84a1d21b80 20 client.28196 trim_cache size 0 max 16384 2024-03-30T04:11:53.642 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 msgr2=0x7f846c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 0x7f846c0a3750 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f848000f870 tx=0x7f848000fc30 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 msgr2=0x7f84880735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f84880735f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f847400e9c0 tx=0x7f847400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 shutdown_connections 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8488073b30 0x7f84881a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f846c0a1290 0x7f846c0a3750 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8488100f60 0x7f84880730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 --2- 172.21.15.23:0/3396894452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8488106080 0x7f84880735f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 >> 172.21.15.23:0/3396894452 conn(0x7f84880fc7c0 msgr2=0x7f84880fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 shutdown_connections 2024-03-30T04:11:53.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.638+0000 7f848e200700 1 -- 172.21.15.23:0/3396894452 wait complete. 2024-03-30T04:11:53.655 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:11:53.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 -- 172.21.15.23:0/1432348823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0103170 msgr2=0x7f85a0103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 --2- 172.21.15.23:0/1432348823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0103170 0x7f85a0103550 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8590009990 tx=0x7f8590009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 -- 172.21.15.23:0/1432348823 shutdown_connections 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 --2- 172.21.15.23:0/1432348823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a0108290 0x7f85a010fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 --2- 172.21.15.23:0/1432348823 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0103170 0x7f85a0103550 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 --2- 172.21.15.23:0/1432348823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103b20 0x7f85a0107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 -- 172.21.15.23:0/1432348823 >> 172.21.15.23:0/1432348823 conn(0x7f85a00fe990 msgr2=0x7f85a0100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 -- 172.21.15.23:0/1432348823 shutdown_connections 2024-03-30T04:11:53.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.871+0000 7f85a5000700 1 -- 172.21.15.23:0/1432348823 wait complete. 2024-03-30T04:11:53.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.872+0000 7f85a5000700 1 Processor -- start 2024-03-30T04:11:53.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 -- start start 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0108290 0x7f85a01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a019dd60 0x7f85a019e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a019e7e0 con 0x7f85a0108290 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a019e950 con 0x7f85a019dd60 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f85a5000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a019eb10 con 0x7f85a0103170 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f859de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f859de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36012/0 (socket says 172.21.15.23:36012) 2024-03-30T04:11:53.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.873+0000 7f859de00700 1 -- 172.21.15.23:0/2979377970 learned_addr learned my addr 172.21.15.23:0/2979377970 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859d400700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0108290 0x7f85a01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859e800700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a019dd60 0x7f85a019e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 -- 172.21.15.23:0/2979377970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a019dd60 msgr2=0x7f85a019e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a019dd60 0x7f85a019e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 -- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0108290 msgr2=0x7f85a01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0108290 0x7f85a01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 -- 172.21.15.23:0/2979377970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c009520 con 0x7f85a0103170 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f859de00700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8590000c00 tx=0x7f8590005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.874+0000 7f8596a00700 1 -- 172.21.15.23:0/2979377970 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859001d070 con 0x7f85a0103170 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.875+0000 7f8596a00700 1 -- 172.21.15.23:0/2979377970 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8590022470 con 0x7f85a0103170 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.875+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85900095f0 con 0x7f85a0103170 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.875+0000 7f8596a00700 1 -- 172.21.15.23:0/2979377970 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590017650 con 0x7f85a0103170 2024-03-30T04:11:53.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.875+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a01a6300 con 0x7f85a0103170 2024-03-30T04:11:53.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.878+0000 7f8596a00700 1 -- 172.21.15.23:0/2979377970 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f85900225e0 con 0x7f85a0103170 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.879+0000 7f85b8bc7b80 1 Processor -- start 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.879+0000 7f8596a00700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 0x7f85840a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.879+0000 7f85b8bc7b80 1 -- start start 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.879+0000 7f8596a00700 1 -- 172.21.15.23:0/2979377970 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f85900ce3a0 con 0x7f85a0103170 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859e800700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 0x7f85840a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aa11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf629ec990 con 0x55bf62a9edf0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf629ea350 con 0x55bf62a93ef0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf629e5c40 con 0x55bf62a935d0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54762/0 (socket says 172.21.15.23:54762) 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859d400700 1 -- 172.21.15.23:0/103506931 learned_addr learned my addr 172.21.15.23:0/103506931 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859de00700 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.880+0000 7f859e800700 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aa11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 -- 172.21.15.23:0/103506931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 msgr2=0x55bf62a939b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a939b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859e800700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 0x7f85840a37d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f859800f540 tx=0x7f859800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 -- 172.21.15.23:0/103506931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 msgr2=0x55bf62aa11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aa11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 -- 172.21.15.23:0/103506931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf628673b0 con 0x55bf62a93ef0 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f859d400700 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f858c009e80 tx=0x7f858c011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc1b7f93723e2947 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f858b400700 1 -- 172.21.15.23:0/103506931 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c004250 con 0x55bf62a93ef0 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f858b400700 1 -- 172.21.15.23:0/103506931 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f858c0043b0 con 0x55bf62a93ef0 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.881+0000 7f858b400700 1 -- 172.21.15.23:0/103506931 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c019920 con 0x55bf62a93ef0 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.882+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/103506931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 msgr2=0x55bf62a9e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.882+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f858c009e80 tx=0x7f858c011720 comp rx=0 tx=0).stop 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/103506931 shutdown_connections 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aa11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a939b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.882 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/103506931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a9e8b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/103506931 >> 172.21.15.23:0/103506931 conn(0x55bf629f36a0 msgr2=0x55bf62a8fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/103506931 shutdown_connections 2024-03-30T04:11:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/103506931 wait complete. 2024-03-30T04:11:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.883+0000 7f85b8bc7b80 1 -- start start 2024-03-30T04:11:53.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 Processor -- start 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a9bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 0x55bf62a9c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62981d80 con 0x55bf62a93ef0 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62981a20 con 0x55bf62a9edf0 2024-03-30T04:11:53.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf628696a0 con 0x55bf62a935d0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f85b8bc7b80 10 client.0 _ll_register_callbacks cb 0x55bf6298b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f859d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.884+0000 7f859e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 0x55bf62a9c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49136/0 (socket says 172.21.15.23:49136) 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859d400700 1 -- 172.21.15.23:0/3561465947 learned_addr learned my addr 172.21.15.23:0/3561465947 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859de00700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a9bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 -- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 msgr2=0x55bf62a9bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a9bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 -- 172.21.15.23:0/3561465947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 msgr2=0x55bf62a9c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 -- 172.21.15.23:0/3561465947 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c009ab0 con 0x55bf62a9edf0 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859d400700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:53.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f859e800700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 0x55bf62a9c640 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f859800fc90 tx=0x7f8598004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d997627e5d2ca3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f857fe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.885+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859800e040 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f859801c040 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf629a6380 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8598017ae0 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 10 client.28208 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55bf62813b60 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bf629a6380 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8598021400 con 0x55bf62a9edf0 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 10 client.28208 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 10 client.28208 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:11:53.889 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 20 client.28208 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62ab72e0 0x55bf62ab9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 4 client.28208 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.886+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf629de700 con 0x55bf62ab72e0 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.887+0000 7f859de00700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62ab72e0 0x55bf62ab9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.887+0000 7f859de00700 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62ab72e0 0x55bf62ab9700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=40aef0dc37d37c8a server_cookie=c3b4b3dc8a492c79 in_seq=0 out_seq=0 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.887+0000 7f857fe00700 10 client.28208 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f857fe00700 1 -- 172.21.15.23:0/3561465947 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55bf629de700 con 0x55bf62ab72e0 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f857fe00700 10 client.28208 handle_command_reply: tid=0 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 client.28208 shutdown 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 msgr2=0x55bf62a9c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 0x55bf62a9c640 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f859800fc90 tx=0x7f8598004ad0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 shutdown_connections 2024-03-30T04:11:53.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62ab72e0 0x55bf62ab9700 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a9edf0 0x55bf62a9c640 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62a9bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f857fe00700 0 client.28208 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/3561465947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a93ef0 0x55bf62a9c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 >> 172.21.15.23:0/3561465947 conn(0x55bf629f36a0 msgr2=0x55bf62a95530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 shutdown_connections 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.888+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/3561465947 wait complete. 2024-03-30T04:11:53.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.889+0000 7f85b8bc7b80 20 client.28208 trim_cache size 0 max 16384 2024-03-30T04:11:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.903+0000 7f85b8bc7b80 1 Processor -- start 2024-03-30T04:11:53.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 -- start start 2024-03-30T04:11:53.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a96480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aae330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62a94ef0 con 0x55bf62a9edf0 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62a950f0 con 0x55bf62a93ef0 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62aae870 con 0x55bf62a935d0 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f859de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f859e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f859de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36034/0 (socket says 172.21.15.23:36034) 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.904+0000 7f859de00700 1 -- 172.21.15.23:0/1843321553 learned_addr learned my addr 172.21.15.23:0/1843321553 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 -- 172.21.15.23:0/1843321553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 msgr2=0x55bf62a96480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a96480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 -- 172.21.15.23:0/1843321553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 msgr2=0x55bf62aae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 -- 172.21.15.23:0/1843321553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf629abea0 con 0x55bf62a935d0 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859e800700 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f859de00700 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8590022c40 tx=0x7f8590022c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd26f4dd1c92efbe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f858be00700 1 -- 172.21.15.23:0/1843321553 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859009ab80 con 0x55bf62a935d0 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f858be00700 1 -- 172.21.15.23:0/1843321553 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f859009b920 con 0x55bf62a935d0 2024-03-30T04:11:53.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.905+0000 7f858be00700 1 -- 172.21.15.23:0/1843321553 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8590099360 con 0x55bf62a935d0 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/1843321553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 msgr2=0x55bf62ab96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8590022c40 tx=0x7f8590022c90 comp rx=0 tx=0).stop 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/1843321553 shutdown_connections 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62aae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a935d0 0x55bf62ab96d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/1843321553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a93ef0 0x55bf62a96480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/1843321553 >> 172.21.15.23:0/1843321553 conn(0x55bf629f36a0 msgr2=0x55bf62a9b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.906+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/1843321553 shutdown_connections 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.907+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/1843321553 wait complete. 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.907+0000 7f85b8bc7b80 1 -- start start 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.907+0000 7f85b8bc7b80 1 Processor -- start 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a93ef0 0x55bf62aac250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62b0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62aacb60 con 0x55bf62a9edf0 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62b0a650 con 0x55bf62a935d0 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf62b0a790 con 0x55bf62a93ef0 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f85b8bc7b80 10 client.0 _ll_register_callbacks cb 0x55bf6298b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54794/0 (socket says 172.21.15.23:54794) 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 -- 172.21.15.23:0/971528856 learned_addr learned my addr 172.21.15.23:0/971528856 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:53.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859d400700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a93ef0 0x55bf62aac250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859e800700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62b0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 -- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a93ef0 msgr2=0x55bf62aac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a93ef0 0x55bf62aac250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 -- 172.21.15.23:0/971528856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 msgr2=0x55bf62b0a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62b0a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859de00700 1 -- 172.21.15.23:0/971528856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bf62813b60 con 0x55bf62a935d0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.908+0000 7f859e800700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62b0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f859de00700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f85900a30e0 tx=0x7f8590009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43aafddc8f56833f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85900a3610 con 0x55bf62a935d0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f859009b220 con 0x55bf62a935d0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f858c0043b0 con 0x55bf62a935d0 2024-03-30T04:11:53.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85900228c0 con 0x55bf62a935d0 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 10 client.28211 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f85a01a6300 con 0x55bf62a935d0 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f858c0043b0 con 0x55bf62a935d0 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f859009b560 con 0x55bf62a935d0 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 10 client.28211 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 10 client.28211 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 20 client.28211 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.909+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62b0b9d0 0x55bf62b0ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.910+0000 7f85b8bc7b80 4 client.28211 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:53.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.910+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bf62834cf0 con 0x55bf62b0b9d0 2024-03-30T04:11:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.910+0000 7f859d400700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62b0b9d0 0x55bf62b0ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:53.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f859d400700 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62b0b9d0 0x55bf62b0ddf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=82be71c23f4579bb server_cookie=71db532524f3b4ba in_seq=0 out_seq=0 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f857fe00700 10 client.28211 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f857fe00700 1 -- 172.21.15.23:0/971528856 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bf62834cf0 con 0x55bf62b0b9d0 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f857fe00700 10 client.28211 handle_command_reply: tid=0 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 client.28211 shutdown 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 msgr2=0x55bf62aabd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f85900a30e0 tx=0x7f8590009510 comp rx=0 tx=0).stop 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 shutdown_connections 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55bf62b0b9d0 0x55bf62b0ddf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55bf62a9edf0 0x55bf62b0a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f857fe00700 0 client.28211 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55bf62a935d0 0x55bf62aabd10 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 --2- 172.21.15.23:0/971528856 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bf62a93ef0 0x55bf62aac250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.911+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 >> 172.21.15.23:0/971528856 conn(0x55bf629f36a0 msgr2=0x55bf62a8ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 shutdown_connections 2024-03-30T04:11:53.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85b8bc7b80 1 -- 172.21.15.23:0/971528856 wait complete. 2024-03-30T04:11:53.914 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85b8bc7b80 20 client.28211 trim_cache size 0 max 16384 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 msgr2=0x7f85840a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 0x7f85840a37d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f859800f540 tx=0x7f859800b040 comp rx=0 tx=0).stop 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 msgr2=0x7f85a0199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.912+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8590000c00 tx=0x7f8590005740 comp rx=0 tx=0).stop 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 shutdown_connections 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f85840a1310 0x7f85840a37d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85a019dd60 0x7f85a019e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f85a0103170 0x7f85a0199480 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 --2- 172.21.15.23:0/2979377970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f85a0108290 0x7f85a01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 >> 172.21.15.23:0/2979377970 conn(0x7f85a00fe990 msgr2=0x7f85a00ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 shutdown_connections 2024-03-30T04:11:53.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:53.913+0000 7f85a5000700 1 -- 172.21.15.23:0/2979377970 wait complete. 2024-03-30T04:11:53.931 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:11:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.149+0000 7f36af400700 1 -- 172.21.15.23:0/1872503284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a8101910 msgr2=0x7f36a8105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.149+0000 7f36af400700 1 --2- 172.21.15.23:0/1872503284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a8101910 0x7f36a8105950 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3694009990 tx=0x7f3694009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 -- 172.21.15.23:0/1872503284 shutdown_connections 2024-03-30T04:11:54.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 --2- 172.21.15.23:0/1872503284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 --2- 172.21.15.23:0/1872503284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 --2- 172.21.15.23:0/1872503284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a8101910 0x7f36a8105950 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 -- 172.21.15.23:0/1872503284 >> 172.21.15.23:0/1872503284 conn(0x7f36a80fc7c0 msgr2=0x7f36a80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 -- 172.21.15.23:0/1872503284 shutdown_connections 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.150+0000 7f36af400700 1 -- 172.21.15.23:0/1872503284 wait complete. 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.151+0000 7f36af400700 1 Processor -- start 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.151+0000 7f36af400700 1 -- start start 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a81999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 0x7f36a819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819e780 con 0x7f36a819dd00 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819e8f0 con 0x7f36a8106080 2024-03-30T04:11:54.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36af400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a819eab0 con 0x7f36a8100f60 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36a7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a81999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36072/0 (socket says 172.21.15.23:36072) 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36aca00700 1 -- 172.21.15.23:0/3868594975 learned_addr learned my addr 172.21.15.23:0/3868594975 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.152+0000 7f36ad400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 0x7f36a819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 -- 172.21.15.23:0/3868594975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 msgr2=0x7f36a81999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a81999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 -- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 msgr2=0x7f36a819e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 0x7f36a819e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 -- 172.21.15.23:0/3868594975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940095f0 con 0x7f36a8100f60 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36ad400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 0x7f36a819e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36a7e00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a81999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.153+0000 7f36aca00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f369c00e9a0 tx=0x7f369c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.154+0000 7f36a5600700 1 -- 172.21.15.23:0/3868594975 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c009790 con 0x7f36a8100f60 2024-03-30T04:11:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.154+0000 7f36a5600700 1 -- 172.21.15.23:0/3868594975 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f369c00cbd0 con 0x7f36a8100f60 2024-03-30T04:11:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.154+0000 7f36a5600700 1 -- 172.21.15.23:0/3868594975 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c018680 con 0x7f36a8100f60 2024-03-30T04:11:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.154+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a81a5fb0 con 0x7f36a8100f60 2024-03-30T04:11:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.155+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a81a63e0 con 0x7f36a8100f60 2024-03-30T04:11:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.157+0000 7f36a5600700 1 -- 172.21.15.23:0/3868594975 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f369c010430 con 0x7f36a8100f60 2024-03-30T04:11:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.158+0000 7f36a5600700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 0x7f36900a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.159+0000 7f36a5600700 1 -- 172.21.15.23:0/3868594975 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f369c014070 con 0x7f36a8100f60 2024-03-30T04:11:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.159+0000 7f36a7e00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 0x7f36900a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36a7e00700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 0x7f36900a37d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f36a819a970 tx=0x7f3694005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 Processor -- start 2024-03-30T04:11:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 -- start start 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c4559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c4631e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c3ae990 con 0x563d4c455ef0 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c3ac350 con 0x563d4c460df0 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.160+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c3a7c40 con 0x563d4c4555d0 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49190/0 (socket says 172.21.15.23:49190) 2024-03-30T04:11:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 -- 172.21.15.23:0/492203380 learned_addr learned my addr 172.21.15.23:0/492203380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36ad400700 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c4631e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 -- 172.21.15.23:0/492203380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 msgr2=0x563d4c4559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c4559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 -- 172.21.15.23:0/492203380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 msgr2=0x563d4c4631e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c4631e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 -- 172.21.15.23:0/492203380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d4c2293b0 con 0x563d4c455ef0 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36ad400700 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c4631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.161+0000 7f36a7e00700 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3694015000 tx=0x7f3694020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edfa4df9e061323 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f3699c00700 1 -- 172.21.15.23:0/492203380 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369402b070 con 0x563d4c455ef0 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f3699c00700 1 -- 172.21.15.23:0/492203380 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f369401e040 con 0x563d4c455ef0 2024-03-30T04:11:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f3699c00700 1 -- 172.21.15.23:0/492203380 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3694024cc0 con 0x563d4c455ef0 2024-03-30T04:11:54.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 -- 172.21.15.23:0/492203380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 msgr2=0x563d4c4608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3694015000 tx=0x7f3694020000 comp rx=0 tx=0).stop 2024-03-30T04:11:54.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 -- 172.21.15.23:0/492203380 shutdown_connections 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c4631e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c4559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/492203380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c455ef0 0x563d4c4608b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.162+0000 7f36c2e62b80 1 -- 172.21.15.23:0/492203380 >> 172.21.15.23:0/492203380 conn(0x563d4c3b56a0 msgr2=0x563d4c451a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.163+0000 7f36c2e62b80 1 -- 172.21.15.23:0/492203380 shutdown_connections 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.163+0000 7f36c2e62b80 1 -- 172.21.15.23:0/492203380 wait complete. 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.163+0000 7f36c2e62b80 1 -- start start 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.163+0000 7f36c2e62b80 1 Processor -- start 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c460870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c455ef0 0x563d4c45db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c343d80 con 0x563d4c460df0 2024-03-30T04:11:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c343a20 con 0x563d4c455ef0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c22b6a0 con 0x563d4c4555d0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36c2e62b80 10 client.0 _ll_register_callbacks cb 0x563d4c34d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49196/0 (socket says 172.21.15.23:49196) 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 -- 172.21.15.23:0/4239843869 learned_addr learned my addr 172.21.15.23:0/4239843869 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36aca00700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c460870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 -- 172.21.15.23:0/4239843869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 msgr2=0x563d4c460870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c460870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 -- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c455ef0 msgr2=0x563d4c45db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c455ef0 0x563d4c45db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.164+0000 7f36ad400700 1 -- 172.21.15.23:0/4239843869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a0009050 con 0x563d4c460df0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f36ad400700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f36a0007f60 tx=0x7f36a0007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8176c1a1b16bed86 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f368e000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a000e050 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f36a0013bd0 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563d4c368380 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36a0013dd0 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f36c2e62b80 10 client.18240 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.165+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f36a810e470 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563d4c368380 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f36a00154b0 con 0x563d4c460df0 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 10 client.18240 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 10 client.18240 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 20 client.18240 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c470a70 0x563d4c472e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 4 client.18240 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:54.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d4c3a0700 con 0x563d4c470a70 2024-03-30T04:11:54.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.166+0000 7f36aca00700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c470a70 0x563d4c472e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.167+0000 7f368e000700 10 client.18240 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.167+0000 7f36aca00700 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c470a70 0x563d4c472e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b6005ce2cbbcdee server_cookie=ab2850ef1b146ed7 in_seq=0 out_seq=0 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f368e000700 1 -- 172.21.15.23:0/4239843869 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563d4c3a0700 con 0x563d4c470a70 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f368e000700 10 client.18240 handle_command_reply: tid=0 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 client.18240 shutdown 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 msgr2=0x563d4c45e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f36a0007f60 tx=0x7f36a0007f90 comp rx=0 tx=0).stop 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 shutdown_connections 2024-03-30T04:11:54.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c470a70 0x563d4c472e90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c45e250 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c4555d0 0x563d4c460870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/4239843869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c455ef0 0x563d4c45db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f368e000700 0 client.18240 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 >> 172.21.15.23:0/4239843869 conn(0x563d4c3b56a0 msgr2=0x563d4c4576c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 shutdown_connections 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.168+0000 7f36c2e62b80 1 -- 172.21.15.23:0/4239843869 wait complete. 2024-03-30T04:11:54.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.169+0000 7f36c2e62b80 20 client.18240 trim_cache size 0 max 16384 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 Processor -- start 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 -- start start 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c4589d0 con 0x563d4c4555d0 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c467f20 con 0x563d4c460df0 2024-03-30T04:11:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.184+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c451fa0 con 0x563d4c455ef0 2024-03-30T04:11:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36ad400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49198/0 (socket says 172.21.15.23:49198) 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36ad400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54842/0 (socket says 172.21.15.23:54842) 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 -- 172.21.15.23:0/3013228786 learned_addr learned my addr 172.21.15.23:0/3013228786 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36a7e00700 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 -- 172.21.15.23:0/3013228786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 msgr2=0x563d4c4679e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 -- 172.21.15.23:0/3013228786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 msgr2=0x563d4c458e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36aca00700 1 -- 172.21.15.23:0/3013228786 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d4c1d5b60 con 0x563d4c4555d0 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.185+0000 7f36ad400700 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.186+0000 7f36aca00700 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f369c096900 tx=0x7f369c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05dd92c01450cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.186+0000 7f369a600700 1 -- 172.21.15.23:0/3013228786 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c09aea0 con 0x563d4c4555d0 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.186+0000 7f369a600700 1 -- 172.21.15.23:0/3013228786 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f369c09a590 con 0x563d4c4555d0 2024-03-30T04:11:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.186+0000 7f369a600700 1 -- 172.21.15.23:0/3013228786 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c098350 con 0x563d4c4555d0 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- 172.21.15.23:0/3013228786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 msgr2=0x563d4c45e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f369c096900 tx=0x7f369c00eef0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- 172.21.15.23:0/3013228786 shutdown_connections 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c460df0 0x563d4c458e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c4555d0 0x563d4c45e160 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/3013228786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- 172.21.15.23:0/3013228786 >> 172.21.15.23:0/3013228786 conn(0x563d4c3b56a0 msgr2=0x563d4c44a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- 172.21.15.23:0/3013228786 shutdown_connections 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- 172.21.15.23:0/3013228786 wait complete. 2024-03-30T04:11:54.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 -- start start 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.187+0000 7f36c2e62b80 1 Processor -- start 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 0x563d4c467470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c465b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c4661f0 con 0x563d4c460df0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c466330 con 0x563d4c4555d0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d4c4664f0 con 0x563d4c455ef0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36c2e62b80 10 client.0 _ll_register_callbacks cb 0x563d4c34d470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 0x563d4c467470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 0x563d4c467470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54848/0 (socket says 172.21.15.23:54848) 2024-03-30T04:11:54.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36aca00700 1 -- 172.21.15.23:0/2537613107 learned_addr learned my addr 172.21.15.23:0/2537613107 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.188+0000 7f36ad400700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c465b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 -- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 msgr2=0x563d4c467470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 0x563d4c467470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 -- 172.21.15.23:0/2537613107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 msgr2=0x563d4c465b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c465b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 -- 172.21.15.23:0/2537613107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a810e470 con 0x563d4c455ef0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36ad400700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c465b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36a7e00700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f369400fa20 tx=0x7f369400bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3fc8af4421f1495 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f368e000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369402b040 con 0x563d4c455ef0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f369401e040 con 0x563d4c455ef0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f36940249e0 con 0x563d4c455ef0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.189+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3694024ba0 con 0x563d4c455ef0 2024-03-30T04:11:54.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 10 client.28213 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f36a81a63e0 con 0x563d4c455ef0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f369401e040 con 0x563d4c455ef0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f369402f400 con 0x563d4c455ef0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 10 client.28213 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 10 client.28213 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 20 client.28213 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c4c57a0 0x563d4c4c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 4 client.28213 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563d4c1f6cf0 con 0x563d4c4c57a0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.190+0000 7f36aca00700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c4c57a0 0x563d4c4c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.191+0000 7f36aca00700 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c4c57a0 0x563d4c4c7bc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=720286d8bfbdc2eb server_cookie=6209b6354f0d7720 in_seq=0 out_seq=0 2024-03-30T04:11:54.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.191+0000 7f368e000700 10 client.28213 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f368e000700 1 -- 172.21.15.23:0/2537613107 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563d4c1f6cf0 con 0x563d4c4c57a0 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f368e000700 10 client.28213 handle_command_reply: tid=0 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 client.28213 shutdown 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 msgr2=0x563d4c4679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f369400fa20 tx=0x7f369400bf30 comp rx=0 tx=0).stop 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 shutdown_connections 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x563d4c4c57a0 0x563d4c4c7bc0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563d4c460df0 0x563d4c465b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563d4c4555d0 0x563d4c467470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 --2- 172.21.15.23:0/2537613107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563d4c455ef0 0x563d4c4679b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f368e000700 0 client.28213 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 >> 172.21.15.23:0/2537613107 conn(0x563d4c3b56a0 msgr2=0x563d4c458e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 shutdown_connections 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.192+0000 7f36c2e62b80 1 -- 172.21.15.23:0/2537613107 wait complete. 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36c2e62b80 20 client.28213 trim_cache size 0 max 16384 2024-03-30T04:11:54.195 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 msgr2=0x7f36900a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 0x7f36900a37d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f36a819a970 tx=0x7f3694005fd0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 msgr2=0x7f36a8199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f369c00e9a0 tx=0x7f369c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 shutdown_connections 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f36a819dd00 0x7f36a819e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36a8100f60 0x7f36a8199470 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f36900a1310 0x7f36900a37d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 --2- 172.21.15.23:0/3868594975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f36a8106080 0x7f36a81999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 >> 172.21.15.23:0/3868594975 conn(0x7f36a80fc7c0 msgr2=0x7f36a80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.193+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 shutdown_connections 2024-03-30T04:11:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.194+0000 7f36af400700 1 -- 172.21.15.23:0/3868594975 wait complete. 2024-03-30T04:11:54.208 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:54 smithi023 ceph-mon[38254]: pgmap v309: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 207 op/s 2024-03-30T04:11:54.208 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:54 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2550313658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:54.211 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:11:54.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 991824 10.21 MB/sec execute 391 sec latency 9.384 ms 2024-03-30T04:11:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:54 smithi154 ceph-mon[79161]: pgmap v309: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 207 op/s 2024-03-30T04:11:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:54 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2550313658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 -- 172.21.15.23:0/1715390139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073aa0 msgr2=0x7f035410d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 --2- 172.21.15.23:0/1715390139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073aa0 0x7f035410d040 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0340009990 tx=0x7f0340009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 -- 172.21.15.23:0/1715390139 shutdown_connections 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 --2- 172.21.15.23:0/1715390139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 0x7f035410f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 --2- 172.21.15.23:0/1715390139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073180 0x7f0354073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 --2- 172.21.15.23:0/1715390139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073aa0 0x7f035410d040 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.450+0000 7f035b200700 1 -- 172.21.15.23:0/1715390139 >> 172.21.15.23:0/1715390139 conn(0x7f03540fc7f0 msgr2=0x7f03540fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.451+0000 7f035b200700 1 -- 172.21.15.23:0/1715390139 shutdown_connections 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.451+0000 7f035b200700 1 -- 172.21.15.23:0/1715390139 wait complete. 2024-03-30T04:11:54.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.452+0000 7f035b200700 1 Processor -- start 2024-03-30T04:11:54.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.452+0000 7f035b200700 1 -- start start 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 0x7f0354199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 0x7f035419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035419e350 con 0x7f035410d580 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035419e4c0 con 0x7f0354073180 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f035b200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035419e680 con 0x7f0354073aa0 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54870/0 (socket says 172.21.15.23:54870) 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 -- 172.21.15.23:0/1913947430 learned_addr learned my addr 172.21.15.23:0/1913947430 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0359200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 0x7f035419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353400700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 0x7f0354199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 -- 172.21.15.23:0/1913947430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 msgr2=0x7f0354199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 0x7f0354199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 -- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 msgr2=0x7f035419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 0x7f035419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353e00700 1 -- 172.21.15.23:0/1913947430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03400095f0 con 0x7f0354073180 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.453+0000 7f0353400700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 0x7f0354199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f0353e00700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f034400e9a0 tx=0x7f034400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f0350c00700 1 -- 172.21.15.23:0/1913947430 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344009790 con 0x7f0354073180 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035419e960 con 0x7f0354073180 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f0350c00700 1 -- 172.21.15.23:0/1913947430 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f034400cbd0 con 0x7f0354073180 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f0350c00700 1 -- 172.21.15.23:0/1913947430 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344018680 con 0x7f0354073180 2024-03-30T04:11:54.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.454+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03541a61a0 con 0x7f0354073180 2024-03-30T04:11:54.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.458+0000 7f0350c00700 1 -- 172.21.15.23:0/1913947430 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f034400cd40 con 0x7f0354073180 2024-03-30T04:11:54.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.459+0000 7f036ecbeb80 1 Processor -- start 2024-03-30T04:11:54.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.459+0000 7f036ecbeb80 1 -- start start 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.459+0000 7f0350c00700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 0x7f03380a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0350c00700 1 -- 172.21.15.23:0/1913947430 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0344014070 con 0x7f0354073180 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f891e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923ed4990 con 0x561923f7b5d0 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923ed2350 con 0x561923f86df0 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923ecdc40 con 0x561923f7bef0 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0359200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 0x7f03380a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36142/0 (socket says 172.21.15.23:36142) 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49218/0 (socket says 172.21.15.23:49218) 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 -- 172.21.15.23:0/1333104555 learned_addr learned my addr 172.21.15.23:0/1333104555 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0359200700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f891e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 -- 172.21.15.23:0/1333104555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 msgr2=0x561923f891e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f891e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 -- 172.21.15.23:0/1333104555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 msgr2=0x561923f7b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353400700 1 -- 172.21.15.23:0/1333104555 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561923d4f3b0 con 0x561923f7bef0 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0353e00700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.460+0000 7f0359200700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f891e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.461+0000 7f0359200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 0x7f03380a37d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f03540679c0 tx=0x7f034c003a20 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.461+0000 7f0353400700 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0340006010 tx=0x7f0340005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebb853dc7635d72a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.461+0000 7f0349600700 1 -- 172.21.15.23:0/1333104555 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03400056a0 con 0x561923f7bef0 2024-03-30T04:11:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.461+0000 7f0349600700 1 -- 172.21.15.23:0/1333104555 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f034001e040 con 0x561923f7bef0 2024-03-30T04:11:54.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.461+0000 7f0349600700 1 -- 172.21.15.23:0/1333104555 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0340004a80 con 0x561923f7bef0 2024-03-30T04:11:54.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1333104555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 msgr2=0x561923f868b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0340006010 tx=0x7f0340005c00 comp rx=0 tx=0).stop 2024-03-30T04:11:54.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1333104555 shutdown_connections 2024-03-30T04:11:54.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f891e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7b5d0 0x561923f7b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1333104555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7bef0 0x561923f868b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1333104555 >> 172.21.15.23:0/1333104555 conn(0x561923edb6a0 msgr2=0x561923f77a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1333104555 shutdown_connections 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.462+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1333104555 wait complete. 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 -- start start 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 Processor -- start 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f86790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923f84040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923e69d80 con 0x561923f86df0 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923e69a20 con 0x561923f7bef0 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923d516a0 con 0x561923f7b5d0 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.463+0000 7f036ecbeb80 10 client.0 _ll_register_callbacks cb 0x561923e73470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54886/0 (socket says 172.21.15.23:54886) 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 -- 172.21.15.23:0/2337513603 learned_addr learned my addr 172.21.15.23:0/2337513603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353e00700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f86790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0359200700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923f84040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 -- 172.21.15.23:0/2337513603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 msgr2=0x561923f86790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f86790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 -- 172.21.15.23:0/2337513603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 msgr2=0x561923f84040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923f84040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 -- 172.21.15.23:0/2337513603 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03541a61a0 con 0x561923f7bef0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353e00700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f86790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0359200700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923f84040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f0353400700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0340009350 tx=0x7f034000ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258ee73afba4bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.464+0000 7f033e000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03400048e0 con 0x561923f7bef0 2024-03-30T04:11:54.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561923e8e380 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0340004a40 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0340004c00 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 10 client.28238 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f03541106c0 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561923e8e380 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03400223f0 con 0x561923f7bef0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 10 client.28238 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 10 client.28238 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 20 client.28238 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923f964c0 0x561923f988e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 4 client.28238 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.465+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561923ec6700 con 0x561923f964c0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.466+0000 7f0353e00700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923f964c0 0x561923f988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.466+0000 7f0353e00700 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923f964c0 0x561923f988e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1def7584305b2629 server_cookie=9f2d0b2f0fc0f90d in_seq=0 out_seq=0 2024-03-30T04:11:54.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.466+0000 7f033e000700 10 client.28238 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f033e000700 1 -- 172.21.15.23:0/2337513603 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561923ec6700 con 0x561923f964c0 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f033e000700 10 client.28238 handle_command_reply: tid=0 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 client.28238 shutdown 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 msgr2=0x561923f83a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f0340009350 tx=0x7f034000ba30 comp rx=0 tx=0).stop 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 shutdown_connections 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923f84040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923f964c0 0x561923f988e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f86790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/2337513603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f83a80 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f033e000700 0 client.28238 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:54.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 >> 172.21.15.23:0/2337513603 conn(0x561923edb6a0 msgr2=0x561923f7d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 shutdown_connections 2024-03-30T04:11:54.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.467+0000 7f036ecbeb80 1 -- 172.21.15.23:0/2337513603 wait complete. 2024-03-30T04:11:54.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.468+0000 7f036ecbeb80 20 client.28238 trim_cache size 0 max 16384 2024-03-30T04:11:54.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.482+0000 7f036ecbeb80 1 Processor -- start 2024-03-30T04:11:54.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.482+0000 7f036ecbeb80 1 -- start start 2024-03-30T04:11:54.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.482+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923f843c0 con 0x561923f7bef0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923f8df80 con 0x561923f86df0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923f79160 con 0x561923f7b5d0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49228/0 (socket says 172.21.15.23:49228) 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 -- 172.21.15.23:0/3905047508 learned_addr learned my addr 172.21.15.23:0/3905047508 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0359200700 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36164/0 (socket says 172.21.15.23:36164) 2024-03-30T04:11:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 -- 172.21.15.23:0/3905047508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 msgr2=0x561923f8da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 -- 172.21.15.23:0/3905047508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 msgr2=0x561923f862c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f862c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0353400700 1 -- 172.21.15.23:0/3905047508 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561923cfbb60 con 0x561923f7bef0 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.483+0000 7f0359200700 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.484+0000 7f0353400700 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f03400057d0 tx=0x7f0340001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfa9e8f95995a3b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.484+0000 7f034a000700 1 -- 172.21.15.23:0/3905047508 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03400225f0 con 0x561923f7bef0 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.484+0000 7f034a000700 1 -- 172.21.15.23:0/3905047508 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0340005a00 con 0x561923f7bef0 2024-03-30T04:11:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.484+0000 7f034a000700 1 -- 172.21.15.23:0/3905047508 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034002ad80 con 0x561923f7bef0 2024-03-30T04:11:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- 172.21.15.23:0/3905047508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 msgr2=0x561923f705a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f03400057d0 tx=0x7f0340001e50 comp rx=0 tx=0).stop 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- 172.21.15.23:0/3905047508 shutdown_connections 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f86df0 0x561923f862c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/3905047508 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f7bef0 0x561923f705a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- 172.21.15.23:0/3905047508 >> 172.21.15.23:0/3905047508 conn(0x561923edb6a0 msgr2=0x561923f9d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- 172.21.15.23:0/3905047508 shutdown_connections 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- 172.21.15.23:0/3905047508 wait complete. 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.485+0000 7f036ecbeb80 1 -- start start 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 Processor -- start 2024-03-30T04:11:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f8bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923f8c480 con 0x561923f86df0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923fe9830 con 0x561923f7bef0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.486+0000 7f036ecbeb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561923fe99a0 con 0x561923f7b5d0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.487+0000 7f036ecbeb80 10 client.0 _ll_register_callbacks cb 0x561923e73470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.487+0000 7f0359200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.487+0000 7f0359200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49238/0 (socket says 172.21.15.23:49238) 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.487+0000 7f0353e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.487+0000 7f0359200700 1 -- 172.21.15.23:0/1737157984 learned_addr learned my addr 172.21.15.23:0/1737157984 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0353400700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f8bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0359200700 1 -- 172.21.15.23:0/1737157984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 msgr2=0x561923f8b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0359200700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0359200700 1 -- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 msgr2=0x561923f8bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0359200700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f8bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0359200700 1 -- 172.21.15.23:0/1737157984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c021040 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0353e00700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.488+0000 7f0353400700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f8bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f0359200700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f034c009c90 tx=0x7f034c009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d90117fe4af484 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f033e000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034c019890 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f034c0199f0 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f034001e040 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f034c019bb0 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f036ecbeb80 10 client.18264 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.489+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f03541106c0 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f034001e040 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f034c024070 con 0x561923f86df0 2024-03-30T04:11:54.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 10 client.18264 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 10 client.18264 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 20 client.18264 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923feb290 0x561923fed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 4 client.18264 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561923d1ccf0 con 0x561923feb290 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.490+0000 7f0353e00700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923feb290 0x561923fed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f0353e00700 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923feb290 0x561923fed6b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bd3ef6a7ec7127b server_cookie=ca6ff3f44e35c0ab in_seq=0 out_seq=0 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f033e000700 10 client.18264 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f033e000700 1 -- 172.21.15.23:0/1737157984 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561923d1ccf0 con 0x561923feb290 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f033e000700 10 client.18264 handle_command_reply: tid=0 2024-03-30T04:11:54.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f036ecbeb80 1 client.18264 shutdown 2024-03-30T04:11:54.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 msgr2=0x561923fe92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.491+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f034c009c90 tx=0x7f034c009100 comp rx=0 tx=0).stop 2024-03-30T04:11:54.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 shutdown_connections 2024-03-30T04:11:54.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561923f86df0 0x561923fe92f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x561923feb290 0x561923fed6b0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561923f7b5d0 0x561923f8b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 --2- 172.21.15.23:0/1737157984 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561923f7bef0 0x561923f8bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 >> 172.21.15.23:0/1737157984 conn(0x561923edb6a0 msgr2=0x561923f705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f033e000700 0 client.18264 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 shutdown_connections 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 1 -- 172.21.15.23:0/1737157984 wait complete. 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.492+0000 7f036ecbeb80 20 client.18264 trim_cache size 0 max 16384 2024-03-30T04:11:54.493 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:54 smithi193 ceph-mon[79581]: pgmap v309: 129 pgs: 129 active+clean; 554 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 207 op/s 2024-03-30T04:11:54.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:54 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2550313658' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:11:54.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.493+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 msgr2=0x7f03380a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.493+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 0x7f03380a37d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f03540679c0 tx=0x7f034c003a20 comp rx=0 tx=0).stop 2024-03-30T04:11:54.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.493+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 msgr2=0x7f0354199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.493+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f034400e9a0 tx=0x7f034400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 shutdown_connections 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f035410d580 0x7f035419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03380a1310 0x7f03380a37d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0354073180 0x7f0354199430 secure :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f034400e9a0 tx=0x7f034400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 --2- 172.21.15.23:0/1913947430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0354073aa0 0x7f0354199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.494+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 >> 172.21.15.23:0/1913947430 conn(0x7f03540fc7f0 msgr2=0x7f03540fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.497+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 shutdown_connections 2024-03-30T04:11:54.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.497+0000 7f035b200700 1 -- 172.21.15.23:0/1913947430 wait complete. 2024-03-30T04:11:54.531 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:11:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 -- 172.21.15.23:0/2831650471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc4101910 msgr2=0x7fccc4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 --2- 172.21.15.23:0/2831650471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc4101910 0x7fccc4105950 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fccb00098f0 tx=0x7fccb0009c00 comp rx=0 tx=0).stop 2024-03-30T04:11:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 -- 172.21.15.23:0/2831650471 shutdown_connections 2024-03-30T04:11:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 --2- 172.21.15.23:0/2831650471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4106080 0x7fccc410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 --2- 172.21.15.23:0/2831650471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4100f60 0x7fccc4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 --2- 172.21.15.23:0/2831650471 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc4101910 0x7fccc4105950 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 -- 172.21.15.23:0/2831650471 >> 172.21.15.23:0/2831650471 conn(0x7fccc40fc7c0 msgr2=0x7fccc40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 -- 172.21.15.23:0/2831650471 shutdown_connections 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.792+0000 7fccc8c00700 1 -- 172.21.15.23:0/2831650471 wait complete. 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.793+0000 7fccc8c00700 1 Processor -- start 2024-03-30T04:11:54.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.794+0000 7fccc8c00700 1 -- start start 2024-03-30T04:11:54.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.794+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 0x7fccc41994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 0x7fccc4199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc8c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc419ddc0 0x7fccc419e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc8c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc419e840 con 0x7fccc419ddc0 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc8c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc419e9b0 con 0x7fccc4106080 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc8c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc419eb70 con 0x7fccc4100f60 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 0x7fccc4199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 0x7fccc4199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54906/0 (socket says 172.21.15.23:54906) 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1400700 1 -- 172.21.15.23:0/4235829787 learned_addr learned my addr 172.21.15.23:0/4235829787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 0x7fccc41994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc2800700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc419ddc0 0x7fccc419e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 -- 172.21.15.23:0/4235829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 msgr2=0x7fccc4199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 0x7fccc4199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 -- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc419ddc0 msgr2=0x7fccc419e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc419ddc0 0x7fccc419e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.795+0000 7fccc1e00700 1 -- 172.21.15.23:0/4235829787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb40095f0 con 0x7fccc4100f60 2024-03-30T04:11:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 0x7fccc41994e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fccb400eb90 tx=0x7fccb400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccbaa00700 1 -- 172.21.15.23:0/4235829787 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb400ccf0 con 0x7fccc4100f60 2024-03-30T04:11:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccbaa00700 1 -- 172.21.15.23:0/4235829787 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccb400ce50 con 0x7fccc4100f60 2024-03-30T04:11:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccbaa00700 1 -- 172.21.15.23:0/4235829787 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4010430 con 0x7fccc4100f60 2024-03-30T04:11:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb0009520 con 0x7fccc4100f60 2024-03-30T04:11:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.796+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc41a63d0 con 0x7fccc4100f60 2024-03-30T04:11:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.800+0000 7fccbaa00700 1 -- 172.21.15.23:0/4235829787 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fccb4003f30 con 0x7fccc4100f60 2024-03-30T04:11:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.801+0000 7fccbaa00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 0x7fcca80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.801+0000 7fccbaa00700 1 -- 172.21.15.23:0/4235829787 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fccb4014070 con 0x7fccc4100f60 2024-03-30T04:11:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccc1400700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 0x7fcca80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 Processor -- start 2024-03-30T04:11:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 -- start start 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bca3990 con 0x55833bd4a5d0 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bca1350 con 0x55833bd4aef0 2024-03-30T04:11:54.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.802+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bc9cc40 con 0x55833bd55df0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1400700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 0x7fcca80a37d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c010 tx=0x7fccb0019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49262/0 (socket says 172.21.15.23:49262) 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 -- 172.21.15.23:0/4196792583 learned_addr learned my addr 172.21.15.23:0/4196792583 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1400700 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc2800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36214/0 (socket says 172.21.15.23:36214) 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 -- 172.21.15.23:0/4196792583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 msgr2=0x55833bd581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 -- 172.21.15.23:0/4196792583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 msgr2=0x55833bd558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 -- 172.21.15.23:0/4196792583 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833bb1e3b0 con 0x55833bd4a5d0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.803+0000 7fccc1e00700 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fccb409a260 tx=0x7fccb4099e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b63d5ffa15e4a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccaf000700 1 -- 172.21.15.23:0/4196792583 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb409a9d0 con 0x55833bd4a5d0 2024-03-30T04:11:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccaf000700 1 -- 172.21.15.23:0/4196792583 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccb409a2d0 con 0x55833bd4a5d0 2024-03-30T04:11:54.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccaf000700 1 -- 172.21.15.23:0/4196792583 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4004440 con 0x55833bd4a5d0 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/4196792583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 msgr2=0x55833bd4a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fccb409a260 tx=0x7fccb4099e90 comp rx=0 tx=0).stop 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/4196792583 shutdown_connections 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd4a9b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.804+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/4196792583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.805+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/4196792583 >> 172.21.15.23:0/4196792583 conn(0x55833bcaa6a0 msgr2=0x55833bd46a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.805+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/4196792583 shutdown_connections 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.805+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/4196792583 wait complete. 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.805+0000 7fccdc6b4b80 1 -- start start 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.805+0000 7fccdc6b4b80 1 Processor -- start 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd55df0 0x55833bd53130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bc38d80 con 0x55833bd55df0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bc38a20 con 0x55833bd4aef0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bb206a0 con 0x55833bd4a5d0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccdc6b4b80 10 client.0 _ll_register_callbacks cb 0x55833bc42470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc1400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54918/0 (socket says 172.21.15.23:54918) 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc1400700 1 -- 172.21.15.23:0/1184943217 learned_addr learned my addr 172.21.15.23:0/1184943217 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36228/0 (socket says 172.21.15.23:36228) 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.806+0000 7fccc2800700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd55df0 0x55833bd53130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 -- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 msgr2=0x55833bd52b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 -- 172.21.15.23:0/1184943217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd55df0 msgr2=0x55833bd53130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd55df0 0x55833bd53130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 -- 172.21.15.23:0/1184943217 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc41a63d0 con 0x55833bd4a5d0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1400700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccc1e00700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fccb40a2910 tx=0x7fccb4096c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45bec709c900b362 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fcca3400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40a2750 con 0x55833bd4a5d0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55833bc5d380 con 0x55833bd4a5d0 2024-03-30T04:11:54.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fccb40043f0 con 0x55833bd4a5d0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb409a530 con 0x55833bd4a5d0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccdc6b4b80 10 client.28240 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.807+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55833bacab60 con 0x55833bd4a5d0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55833bc5d380 con 0x55833bd4a5d0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccb40a23e0 con 0x55833bd4a5d0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 10 client.28240 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 10 client.28240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 20 client.28240 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bd656a0 0x55833bd67ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 4 client.28240 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55833bc95700 con 0x55833bd656a0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.808+0000 7fccc1400700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bd656a0 0x55833bd67ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.809+0000 7fccc1400700 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bd656a0 0x55833bd67ac0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b630ca07a191bd2 server_cookie=1251e3f7b353e07c in_seq=0 out_seq=0 2024-03-30T04:11:54.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.809+0000 7fcca3400700 10 client.28240 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fcca3400700 1 -- 172.21.15.23:0/1184943217 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55833bc95700 con 0x55833bd656a0 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fcca3400700 10 client.28240 handle_command_reply: tid=0 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 client.28240 shutdown 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 msgr2=0x55833bd55810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fccb40a2910 tx=0x7fccb4096c70 comp rx=0 tx=0).stop 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 shutdown_connections 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd55df0 0x55833bd53130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bd656a0 0x55833bd67ac0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd55810 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fcca3400700 0 client.28240 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/1184943217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd52b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 >> 172.21.15.23:0/1184943217 conn(0x55833bcaa6a0 msgr2=0x55833bd4c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 shutdown_connections 2024-03-30T04:11:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/1184943217 wait complete. 2024-03-30T04:11:54.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.810+0000 7fccdc6b4b80 20 client.28240 trim_cache size 0 max 16384 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.824+0000 7fccdc6b4b80 1 Processor -- start 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.824+0000 7fccdc6b4b80 1 -- start start 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4aef0 0x55833bd4d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd55df0 0x55833bd5ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bd54100 con 0x55833bd4aef0 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bd53710 con 0x55833bd55df0 2024-03-30T04:11:54.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bd4c130 con 0x55833bd4a5d0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36240/0 (socket says 172.21.15.23:36240) 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 -- 172.21.15.23:0/313197779 learned_addr learned my addr 172.21.15.23:0/313197779 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc2800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd55df0 0x55833bd5ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1400700 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4aef0 0x55833bd4d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 -- 172.21.15.23:0/313197779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd55df0 msgr2=0x55833bd5ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd55df0 0x55833bd5ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 -- 172.21.15.23:0/313197779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4aef0 msgr2=0x55833bd4d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4aef0 0x55833bd4d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.825+0000 7fccc1e00700 1 -- 172.21.15.23:0/313197779 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833bc62ea0 con 0x55833bd4a5d0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.826+0000 7fccc1e00700 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cae90 tx=0x7fccb4009a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e15c3aa756c87f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.826+0000 7fccafa00700 1 -- 172.21.15.23:0/313197779 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb40b2e80 con 0x55833bd4a5d0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.826+0000 7fccafa00700 1 -- 172.21.15.23:0/313197779 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccb40ccec0 con 0x55833bd4a5d0 2024-03-30T04:11:54.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.826+0000 7fccafa00700 1 -- 172.21.15.23:0/313197779 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4099a00 con 0x55833bd4a5d0 2024-03-30T04:11:54.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/313197779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 msgr2=0x55833bd530f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccb40cae90 tx=0x7fccb4009a30 comp rx=0 tx=0).stop 2024-03-30T04:11:54.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/313197779 shutdown_connections 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd55df0 0x55833bd5ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd4a5d0 0x55833bd530f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/313197779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4aef0 0x55833bd4d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/313197779 >> 172.21.15.23:0/313197779 conn(0x55833bcaa6a0 msgr2=0x55833bd3ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/313197779 shutdown_connections 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/313197779 wait complete. 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 -- start start 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.827+0000 7fccdc6b4b80 1 Processor -- start 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd5af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd5b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bdb8510 con 0x55833bd4a5d0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bdb8650 con 0x55833bd4aef0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833bdb8810 con 0x55833bd55df0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccdc6b4b80 10 client.0 _ll_register_callbacks cb 0x55833bc42470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49290/0 (socket says 172.21.15.23:49290) 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 -- 172.21.15.23:0/335030191 learned_addr learned my addr 172.21.15.23:0/335030191 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1400700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd5af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc2800700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd5b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 -- 172.21.15.23:0/335030191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 msgr2=0x55833bd5b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd5b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 -- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 msgr2=0x55833bd5af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.828+0000 7fccc1e00700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd5af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccc1e00700 1 -- 172.21.15.23:0/335030191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55833bacab60 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccc2800700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd5b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccc1e00700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fccb40caf10 tx=0x7fccb409a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cad92125c05525 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fcca3400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb4099c30 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fccb40043f0 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fccb409a2d0 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccb40a2750 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccdc6b4b80 10 client.18282 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55833bb22cd0 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.829+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fccb408b020 con 0x55833bd4a5d0 2024-03-30T04:11:54.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fccb4083ad0 con 0x55833bd4a5d0 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 10 client.18282 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 10 client.18282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 20 client.18282 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bdb9ae0 0x55833bdbbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 4 client.18282 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55833baebcf0 con 0x55833bdb9ae0 2024-03-30T04:11:54.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.830+0000 7fccc1400700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bdb9ae0 0x55833bdbbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:54.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccc1400700 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bdb9ae0 0x55833bdbbed0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2b36a5f774e6299 server_cookie=90b491febae9cf0a in_seq=0 out_seq=0 2024-03-30T04:11:54.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fcca3400700 10 client.18282 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:54.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fcca3400700 1 -- 172.21.15.23:0/335030191 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55833baebcf0 con 0x55833bdb9ae0 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fcca3400700 10 client.18282 handle_command_reply: tid=0 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 client.18282 shutdown 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 msgr2=0x55833bd5aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fccb40caf10 tx=0x7fccb409a9d0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 shutdown_connections 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55833bdb9ae0 0x55833bdbbed0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55833bd55df0 0x55833bd5b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55833bd4a5d0 0x55833bd5aa50 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 --2- 172.21.15.23:0/335030191 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55833bd4aef0 0x55833bd5af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 >> 172.21.15.23:0/335030191 conn(0x55833bcaa6a0 msgr2=0x55833bd52360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.831+0000 7fcca3400700 0 client.18282 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 shutdown_connections 2024-03-30T04:11:54.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccdc6b4b80 1 -- 172.21.15.23:0/335030191 wait complete. 2024-03-30T04:11:54.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccdc6b4b80 20 client.18282 trim_cache size 0 max 16384 2024-03-30T04:11:54.835 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 msgr2=0x7fcca80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 0x7fcca80a37d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fccb000c010 tx=0x7fccb0019040 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 msgr2=0x7fccc41994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 0x7fccc41994e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fccb400eb90 tx=0x7fccb400eea0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 shutdown_connections 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fccc419ddc0 0x7fccc419e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccc4100f60 0x7fccc41994e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcca80a1310 0x7fcca80a37d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 --2- 172.21.15.23:0/4235829787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fccc4106080 0x7fccc4199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.832+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 >> 172.21.15.23:0/4235829787 conn(0x7fccc40fc7c0 msgr2=0x7fccc40fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.833+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 shutdown_connections 2024-03-30T04:11:54.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:54.833+0000 7fccc8c00700 1 -- 172.21.15.23:0/4235829787 wait complete. 2024-03-30T04:11:55.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 993435 10.20 MB/sec execute 392 sec latency 22.555 ms 2024-03-30T04:11:55.850 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:11:55.851 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:11:56.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 -- 172.21.15.23:0/2480584263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0101910 msgr2=0x7f9ee0105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 --2- 172.21.15.23:0/2480584263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0101910 0x7f9ee0105950 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009990 tx=0x7f9ecc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 -- 172.21.15.23:0/2480584263 shutdown_connections 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 --2- 172.21.15.23:0/2480584263 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee0106080 0x7f9ee010dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 --2- 172.21.15.23:0/2480584263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 --2- 172.21.15.23:0/2480584263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0101910 0x7f9ee0105950 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 -- 172.21.15.23:0/2480584263 >> 172.21.15.23:0/2480584263 conn(0x7f9ee00fc7c0 msgr2=0x7f9ee00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 -- 172.21.15.23:0/2480584263 shutdown_connections 2024-03-30T04:11:56.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.114+0000 7f9ee6800700 1 -- 172.21.15.23:0/2480584263 wait complete. 2024-03-30T04:11:56.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.115+0000 7f9ee6800700 1 Processor -- start 2024-03-30T04:11:56.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.115+0000 7f9ee6800700 1 -- start start 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 0x7f9ee0199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee019dd00 0x7f9ee019e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee019e780 con 0x7f9ee019dd00 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee019e8f0 con 0x7f9ee0106080 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9ee6800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee019eab0 con 0x7f9ee0100f60 2024-03-30T04:11:56.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36278/0 (socket says 172.21.15.23:36278) 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edf400700 1 -- 172.21.15.23:0/3741801509 learned_addr learned my addr 172.21.15.23:0/3741801509 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 0x7f9ee0199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edfe00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee019dd00 0x7f9ee019e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 -- 172.21.15.23:0/3741801509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 msgr2=0x7f9ee0199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 -- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee019dd00 msgr2=0x7f9ee019e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee019dd00 0x7f9ee019e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.116+0000 7f9edea00700 1 -- 172.21.15.23:0/3741801509 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0095f0 con 0x7f9ee0106080 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9edf400700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:56.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9edea00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 0x7f9ee0199960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0098e0 tx=0x7f9ecc005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9ed7e00700 1 -- 172.21.15.23:0/3741801509 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc01d070 con 0x7f9ee0106080 2024-03-30T04:11:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9ed7e00700 1 -- 172.21.15.23:0/3741801509 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ecc022470 con 0x7f9ee0106080 2024-03-30T04:11:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9ed7e00700 1 -- 172.21.15.23:0/3741801509 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc017650 con 0x7f9ee0106080 2024-03-30T04:11:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee01a5fb0 con 0x7f9ee0106080 2024-03-30T04:11:56.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.117+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01a6380 con 0x7f9ee0106080 2024-03-30T04:11:56.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.121+0000 7f9ed7e00700 1 -- 172.21.15.23:0/3741801509 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9ecc0225e0 con 0x7f9ee0106080 2024-03-30T04:11:56.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.122+0000 7f9efa223b80 1 Processor -- start 2024-03-30T04:11:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.122+0000 7f9efa223b80 1 -- start start 2024-03-30T04:11:56.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582effa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582e42a10 con 0x557582efd660 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582e403d0 con 0x557582ef2780 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9efa223b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582e3bcc0 con 0x557582ef1e60 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9ed7e00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 0x7f9ec40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54948/0 (socket says 172.21.15.23:54948) 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 -- 172.21.15.23:0/4054060222 learned_addr learned my addr 172.21.15.23:0/4054060222 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edfe00700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edfe00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 0x7f9ec40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9ed7e00700 1 -- 172.21.15.23:0/3741801509 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9ecc0ce350 con 0x7f9ee0106080 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edea00700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582effa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 -- 172.21.15.23:0/4054060222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 msgr2=0x557582ef2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 -- 172.21.15.23:0/4054060222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 msgr2=0x557582effa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582effa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edf400700 1 -- 172.21.15.23:0/4054060222 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557582cbd430 con 0x557582ef2780 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.123+0000 7f9edea00700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582effa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.124+0000 7f9edf400700 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000f540 tx=0x7f9ed000f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=571e5073b1effd7e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.124+0000 7f9edfe00700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 0x7f9ec40a37d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009ec0 tx=0x7f9ed800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.124+0000 7f9ed4c00700 1 -- 172.21.15.23:0/4054060222 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0010840 con 0x557582ef2780 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.124+0000 7f9ed4c00700 1 -- 172.21.15.23:0/4054060222 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ed0010e60 con 0x557582ef2780 2024-03-30T04:11:56.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.124+0000 7f9ed4c00700 1 -- 172.21.15.23:0/4054060222 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0018db0 con 0x557582ef2780 2024-03-30T04:11:56.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 -- 172.21.15.23:0/4054060222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 msgr2=0x557582efd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000f540 tx=0x7f9ed000f850 comp rx=0 tx=0).stop 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 -- 172.21.15.23:0/4054060222 shutdown_connections 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582effa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 --2- 172.21.15.23:0/4054060222 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efd120 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 -- 172.21.15.23:0/4054060222 >> 172.21.15.23:0/4054060222 conn(0x557582e49720 msgr2=0x557582ee5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 -- 172.21.15.23:0/4054060222 shutdown_connections 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.125+0000 7f9efa223b80 1 -- 172.21.15.23:0/4054060222 wait complete. 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 -- start start 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 Processor -- start 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582efd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582efab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582dd7e00 con 0x557582efd660 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582dd7aa0 con 0x557582ef2780 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582cbf720 con 0x557582ef1e60 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.126+0000 7f9efa223b80 10 client.0 _ll_register_callbacks cb 0x557582de14f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edf400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582efd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54956/0 (socket says 172.21.15.23:54956) 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 -- 172.21.15.23:0/1687915492 learned_addr learned my addr 172.21.15.23:0/1687915492 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edfe00700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582efab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 -- 172.21.15.23:0/1687915492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 msgr2=0x557582efd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582efd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 -- 172.21.15.23:0/1687915492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 msgr2=0x557582efab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582efab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 -- 172.21.15.23:0/1687915492 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed000ae10 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9edea00700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00b520 tx=0x7f9ecc022bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb0c7e0f7645548 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9ec9600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc09aa10 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9ecc09b7b0 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.127+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557582dfc400 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ecc09b0b0 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 10 client.28271 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557582c69be0 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557582dfc400 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9ecc07b020 con 0x557582ef2780 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 10 client.28271 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 10 client.28271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 20 client.28271 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:56.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.128+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582f0cdd0 0x557582f0f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.129+0000 7f9efa223b80 4 client.28271 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.129+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557582e34780 con 0x557582f0cdd0 2024-03-30T04:11:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.129+0000 7f9edf400700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582f0cdd0 0x557582f0f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.129+0000 7f9ec9600700 10 client.28271 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.129+0000 7f9edf400700 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582f0cdd0 0x557582f0f1f0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e0aa419e10e870c server_cookie=87ee2ec0acc85c1e in_seq=0 out_seq=0 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.130+0000 7f9ec9600700 1 -- 172.21.15.23:0/1687915492 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x557582e34780 con 0x557582f0cdd0 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.130+0000 7f9ec9600700 10 client.28271 handle_command_reply: tid=0 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.130+0000 7f9efa223b80 1 client.28271 shutdown 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.130+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 msgr2=0x557582efa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.130+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc00b520 tx=0x7f9ecc022bd0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 shutdown_connections 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582efd660 0x557582efab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582f0cdd0 0x557582f0f1f0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582efd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1687915492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efa3f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 >> 172.21.15.23:0/1687915492 conn(0x557582e49720 msgr2=0x557582ef3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9ec9600700 0 client.28271 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 shutdown_connections 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 1 -- 172.21.15.23:0/1687915492 wait complete. 2024-03-30T04:11:56.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.131+0000 7f9efa223b80 20 client.28271 trim_cache size 0 max 16384 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.149+0000 7f9efa223b80 1 Processor -- start 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.149+0000 7f9efa223b80 1 -- start start 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f04280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582e34780 con 0x557582e49720 2024-03-30T04:11:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582dfc400 con 0x557582ef2780 2024-03-30T04:11:56.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9efa223b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582efb370 con 0x557582ef1e60 2024-03-30T04:11:56.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:36298/0 (socket says 172.21.15.23:36298) 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 -- 172.21.15.23:0/1271682959 learned_addr learned my addr 172.21.15.23:0/1271682959 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edfe00700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edf400700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f04280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 -- 172.21.15.23:0/1271682959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 msgr2=0x557582efcbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efcbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 -- 172.21.15.23:0/1271682959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 msgr2=0x557582f04280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f04280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edea00700 1 -- 172.21.15.23:0/1271682959 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557582e01f20 con 0x557582ef1e60 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.150+0000 7f9edf400700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f04280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.151+0000 7f9edea00700 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0975b0 tx=0x7f9ecc09a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb38b47e02edb34 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.151+0000 7f9ed5600700 1 -- 172.21.15.23:0/1271682959 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc09aa10 con 0x557582ef1e60 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.151+0000 7f9ed5600700 1 -- 172.21.15.23:0/1271682959 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ecc09b7b0 con 0x557582ef1e60 2024-03-30T04:11:56.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.151+0000 7f9ed5600700 1 -- 172.21.15.23:0/1271682959 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc0a34a0 con 0x557582ef1e60 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- 172.21.15.23:0/1271682959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 msgr2=0x557582ef45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0975b0 tx=0x7f9ecc09a5f0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- 172.21.15.23:0/1271682959 shutdown_connections 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef2780 0x557582efcbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f04280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1271682959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef1e60 0x557582ef45a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- 172.21.15.23:0/1271682959 >> 172.21.15.23:0/1271682959 conn(0x557582ede980 msgr2=0x557582ef3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- 172.21.15.23:0/1271682959 shutdown_connections 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- 172.21.15.23:0/1271682959 wait complete. 2024-03-30T04:11:56.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.152+0000 7f9efa223b80 1 -- start start 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.153+0000 7f9efa223b80 1 Processor -- start 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.153+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f5b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.153+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 0x557582f5b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9efa223b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef2780 0x557582f03710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9efa223b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582f03ce0 con 0x557582e49720 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9efa223b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582f03e50 con 0x557582ef1e60 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9efa223b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557582f04010 con 0x557582ef2780 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9efa223b80 10 client.0 _ll_register_callbacks cb 0x557582de14f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edf400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f5b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 0x557582f5b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edfe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef2780 0x557582f03710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edf400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f5b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49338/0 (socket says 172.21.15.23:49338) 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edf400700 1 -- 172.21.15.23:0/1403315478 learned_addr learned my addr 172.21.15.23:0/1403315478 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 -- 172.21.15.23:0/1403315478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef2780 msgr2=0x557582f03710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef2780 0x557582f03710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 -- 172.21.15.23:0/1403315478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 msgr2=0x557582f5b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f5b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.154+0000 7f9edea00700 1 -- 172.21.15.23:0/1403315478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557582c69be0 con 0x557582ef1e60 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.155+0000 7f9edea00700 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 0x557582f5b8d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x557582ee61a0 tx=0x7f9ecc009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1aec2cdd233aadcc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.155+0000 7f9ec9600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.158+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc09aa10 con 0x557582ef1e60 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.158+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9ecc09a710 con 0x557582ef1e60 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.158+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ed0010e60 con 0x557582ef1e60 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.158+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ecc0a34a0 con 0x557582ef1e60 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.158+0000 7f9efa223b80 10 client.28277 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:56.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557582cc1d50 con 0x557582ef1e60 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ed0010e60 con 0x557582ef1e60 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9ecc07b040 con 0x557582ef1e60 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 10 client.28277 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 10 client.28277 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 20 client.28277 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582efd660 0x557582f64dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 4 client.28277 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:11:56.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557582c8ad70 con 0x557582efd660 2024-03-30T04:11:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.159+0000 7f9edf400700 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582efd660 0x557582f64dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.160+0000 7f9edf400700 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582efd660 0x557582f64dc0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a548f1461ab4346 server_cookie=5523054a87d604f6 in_seq=0 out_seq=0 2024-03-30T04:11:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.160+0000 7f9ec9600700 10 client.28277 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9ec9600700 1 -- 172.21.15.23:0/1403315478 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557582c8ad70 con 0x557582efd660 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9ec9600700 10 client.28277 handle_command_reply: tid=0 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 client.28277 shutdown 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 msgr2=0x557582f5b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 0x557582f5b8d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x557582ee61a0 tx=0x7f9ecc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 shutdown_connections 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x557582efd660 0x557582f64dc0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557582ef2780 0x557582f03710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x557582e49720 0x557582f5b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 --2- 172.21.15.23:0/1403315478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x557582ef1e60 0x557582f5b8d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9ec9600700 0 client.28277 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.161+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 >> 172.21.15.23:0/1403315478 conn(0x557582ede980 msgr2=0x557582ef46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 shutdown_connections 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9efa223b80 1 -- 172.21.15.23:0/1403315478 wait complete. 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9efa223b80 20 client.28277 trim_cache size 0 max 16384 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:11:56.161 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "74dad0e3-83c3-4b2d-ad62-431ff9d3daf2", 2024-03-30T04:11:56.162 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:11:56.162 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 msgr2=0x7f9ec40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 0x7f9ec40a37d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009ec0 tx=0x7f9ed800b040 comp rx=0 tx=0).stop 2024-03-30T04:11:56.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 msgr2=0x7f9ee0199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.162+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 0x7f9ee0199960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0098e0 tx=0x7f9ecc005740 comp rx=0 tx=0).stop 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 shutdown_connections 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ee019dd00 0x7f9ee019e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ee0100f60 0x7f9ee0199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ec40a1310 0x7f9ec40a37d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 --2- 172.21.15.23:0/3741801509 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee0106080 0x7f9ee0199960 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 >> 172.21.15.23:0/3741801509 conn(0x7f9ee00fc7c0 msgr2=0x7f9ee00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 shutdown_connections 2024-03-30T04:11:56.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.163+0000 7f9ee6800700 1 -- 172.21.15.23:0/3741801509 wait complete. 2024-03-30T04:11:56.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:56 smithi023 ceph-mon[38254]: pgmap v310: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-03-30T04:11:56.180 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:11:56.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 995138 10.21 MB/sec execute 393 sec latency 69.379 ms 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 -- 172.21.15.23:0/914662829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4924101910 msgr2=0x7f4924105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 --2- 172.21.15.23:0/914662829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4924101910 0x7f4924105950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f4910009990 tx=0x7f4910009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 -- 172.21.15.23:0/914662829 shutdown_connections 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 --2- 172.21.15.23:0/914662829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f492410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 --2- 172.21.15.23:0/914662829 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f4924101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 --2- 172.21.15.23:0/914662829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4924101910 0x7f4924105950 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 -- 172.21.15.23:0/914662829 >> 172.21.15.23:0/914662829 conn(0x7f49240fc7c0 msgr2=0x7f49240febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 -- 172.21.15.23:0/914662829 shutdown_connections 2024-03-30T04:11:56.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.391+0000 7f492a200700 1 -- 172.21.15.23:0/914662829 wait complete. 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.392+0000 7f492a200700 1 Processor -- start 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.392+0000 7f492a200700 1 -- start start 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.392+0000 7f492a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.392+0000 7f492a200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f4924199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f492a200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f492419dcc0 0x7f492419e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f492a200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419e740 con 0x7f4924100f60 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f492a200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419e8b0 con 0x7f492419dcc0 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f492a200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492419ea70 con 0x7f4924106080 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49360/0 (socket says 172.21.15.23:49360) 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 -- 172.21.15.23:0/2255528623 learned_addr learned my addr 172.21.15.23:0/2255528623 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4922a00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f4924199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923e00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f492419dcc0 0x7f492419e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 -- 172.21.15.23:0/2255528623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 msgr2=0x7f4924199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f4924199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 -- 172.21.15.23:0/2255528623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f492419dcc0 msgr2=0x7f492419e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f492419dcc0 0x7f492419e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4923400700 1 -- 172.21.15.23:0/2255528623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100095f0 con 0x7f4924100f60 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.393+0000 7f4922a00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f4924199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f4923400700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f491400e9a0 tx=0x7f491400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f491be00700 1 -- 172.21.15.23:0/2255528623 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914009790 con 0x7f4924100f60 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f491be00700 1 -- 172.21.15.23:0/2255528623 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4914010430 con 0x7f4924100f60 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49241a5fd0 con 0x7f4924100f60 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f491be00700 1 -- 172.21.15.23:0/2255528623 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914018680 con 0x7f4924100f60 2024-03-30T04:11:56.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.394+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49241a6450 con 0x7f4924100f60 2024-03-30T04:11:56.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.397+0000 7f491be00700 1 -- 172.21.15.23:0/2255528623 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f49140105a0 con 0x7f4924100f60 2024-03-30T04:11:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.399+0000 7f493dc56b80 1 Processor -- start 2024-03-30T04:11:56.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.399+0000 7f493dc56b80 1 -- start start 2024-03-30T04:11:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.399+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e56fff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.399+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e5700a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.399+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f493dc56b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56f58990 con 0x564e5700adf0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f493dc56b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56f56350 con 0x564e56fffef0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f493dc56b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56f51c40 con 0x564e56fff5d0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f491be00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 0x7f49080a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4923400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e5700a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49362/0 (socket says 172.21.15.23:49362) 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 -- 172.21.15.23:0/2235764909 learned_addr learned my addr 172.21.15.23:0/2235764909 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f491be00700 1 -- 172.21.15.23:0/2255528623 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4914014070 con 0x7f4924100f60 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4923e00700 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e56fff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4923e00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 0x7f49080a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 -- 172.21.15.23:0/2235764909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 msgr2=0x564e56fff9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e56fff9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 -- 172.21.15.23:0/2235764909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 msgr2=0x564e5700a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e5700a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4922a00700 1 -- 172.21.15.23:0/2235764909 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e56dd33b0 con 0x564e5700adf0 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.400+0000 7f4923400700 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e5700a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.401+0000 7f4922a00700 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4910009990 tx=0x7f4910005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5471b688997d0f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.401+0000 7f4923e00700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 0x7f49080a3750 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f491c009ec0 tx=0x7f491c00b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.401+0000 7f490fe00700 1 -- 172.21.15.23:0/2235764909 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491001d070 con 0x564e5700adf0 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.401+0000 7f490fe00700 1 -- 172.21.15.23:0/2235764909 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f491001e040 con 0x564e5700adf0 2024-03-30T04:11:56.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.401+0000 7f490fe00700 1 -- 172.21.15.23:0/2235764909 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4910004c20 con 0x564e5700adf0 2024-03-30T04:11:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- 172.21.15.23:0/2235764909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 msgr2=0x564e5700d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4910009990 tx=0x7f4910005f00 comp rx=0 tx=0).stop 2024-03-30T04:11:56.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- 172.21.15.23:0/2235764909 shutdown_connections 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e5700d1e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e56fff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 --2- 172.21.15.23:0/2235764909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e5700a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- 172.21.15.23:0/2235764909 >> 172.21.15.23:0/2235764909 conn(0x564e56f5f6a0 msgr2=0x564e56ffba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- 172.21.15.23:0/2235764909 shutdown_connections 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- 172.21.15.23:0/2235764909 wait complete. 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.402+0000 7f493dc56b80 1 -- start start 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 Processor -- start 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e57008020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56eedd80 con 0x564e5700adf0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56eeda20 con 0x564e56fffef0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56dd56a0 con 0x564e56fff5d0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f493dc56b80 10 client.0 _ll_register_callbacks cb 0x564e56ef7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f4922a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f4922a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54988/0 (socket says 172.21.15.23:54988) 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.403+0000 7f4922a00700 1 -- 172.21.15.23:0/275000079 learned_addr learned my addr 172.21.15.23:0/275000079 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4923e00700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e57008020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4923400700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 -- 172.21.15.23:0/275000079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 msgr2=0x564e5700a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 -- 172.21.15.23:0/275000079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 msgr2=0x564e57008020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e57008020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 -- 172.21.15.23:0/275000079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49241a6450 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4923400700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f4922a00700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4910022a30 tx=0x7f4910004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2ca6642a11594e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f490cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491001d030 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.404+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e56f12380 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f491000f460 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4910005bc0 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 10 client.28289 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f492410e470 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e56f12380 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f491002c620 con 0x564e56fffef0 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 10 client.28289 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:56.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 10 client.28289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 20 client.28289 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5701a390 0x564e5701c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.405+0000 7f493dc56b80 4 client.28289 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.406+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e56f4a700 con 0x564e5701a390 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.406+0000 7f4923400700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5701a390 0x564e5701c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.406+0000 7f4923400700 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5701a390 0x564e5701c780 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a2540caff0d7a server_cookie=6c42d2d7fd3731bb in_seq=0 out_seq=0 2024-03-30T04:11:56.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.406+0000 7f490cc00700 10 client.28289 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.407+0000 7f490cc00700 1 -- 172.21.15.23:0/275000079 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564e56f4a700 con 0x564e5701a390 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.407+0000 7f490cc00700 10 client.28289 handle_command_reply: tid=0 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.407+0000 7f493dc56b80 1 client.28289 shutdown 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 msgr2=0x564e57007ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4910022a30 tx=0x7f4910004500 comp rx=0 tx=0).stop 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 shutdown_connections 2024-03-30T04:11:56.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e5700adf0 0x564e57008020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5701a390 0x564e5701c780 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 --2- 172.21.15.23:0/275000079 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e56fffef0 0x564e57007ac0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 >> 172.21.15.23:0/275000079 conn(0x564e56f5f6a0 msgr2=0x564e570016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f490cc00700 0 client.28289 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 shutdown_connections 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 1 -- 172.21.15.23:0/275000079 wait complete. 2024-03-30T04:11:56.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.408+0000 7f493dc56b80 20 client.28289 trim_cache size 0 max 16384 2024-03-30T04:11:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.427+0000 7f493dc56b80 1 Processor -- start 2024-03-30T04:11:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.427+0000 7f493dc56b80 1 -- start start 2024-03-30T04:11:56.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 0x564e57011a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e56ff45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e570086a0 con 0x564e56fff5d0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e57011f80 con 0x564e5700adf0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f493dc56b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e56ffc660 con 0x564e56fffef0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4922a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 0x564e57011a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49376/0 (socket says 172.21.15.23:49376) 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e56ff45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 -- 172.21.15.23:0/1638149185 learned_addr learned my addr 172.21.15.23:0/1638149185 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 -- 172.21.15.23:0/1638149185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 msgr2=0x564e57011a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 0x564e57011a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 -- 172.21.15.23:0/1638149185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 msgr2=0x564e56ff45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e56ff45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4923400700 1 -- 172.21.15.23:0/1638149185 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e56d7fb60 con 0x564e56fff5d0 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.428+0000 7f4922a00700 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 0x564e57011a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f4923400700 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f491400e970 tx=0x7f4914025310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524d7d287173423a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f4919200700 1 -- 172.21.15.23:0/1638149185 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914025170 con 0x564e56fff5d0 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f4919200700 1 -- 172.21.15.23:0/1638149185 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f491409a6b0 con 0x564e56fff5d0 2024-03-30T04:11:56.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f4919200700 1 -- 172.21.15.23:0/1638149185 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914098470 con 0x564e56fff5d0 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f493dc56b80 1 -- 172.21.15.23:0/1638149185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 msgr2=0x564e57002570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.429+0000 7f493dc56b80 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f491400e970 tx=0x7f4914025310 comp rx=0 tx=0).stop 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 -- 172.21.15.23:0/1638149185 shutdown_connections 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e56ff45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fff5d0 0x564e57002570 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 --2- 172.21.15.23:0/1638149185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fffef0 0x564e57011a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 -- 172.21.15.23:0/1638149185 >> 172.21.15.23:0/1638149185 conn(0x564e56f5f6a0 msgr2=0x564e57021070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 -- 172.21.15.23:0/1638149185 shutdown_connections 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.430+0000 7f493dc56b80 1 -- 172.21.15.23:0/1638149185 wait complete. 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 -- start start 2024-03-30T04:11:56.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 Processor -- start 2024-03-30T04:11:56.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:56 smithi154 ceph-mon[79161]: pgmap v310: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e57010a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e5706e130 con 0x564e56fffef0 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e5706e270 con 0x564e5700adf0 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e5706e3b0 con 0x564e56fff5d0 2024-03-30T04:11:56.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.431+0000 7f493dc56b80 10 client.0 _ll_register_callbacks cb 0x564e56ef7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:49390/0 (socket says 172.21.15.23:49390) 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 -- 172.21.15.23:0/481563446 learned_addr learned my addr 172.21.15.23:0/481563446 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 -- 172.21.15.23:0/481563446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 msgr2=0x564e5700ffb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700ffb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 -- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 msgr2=0x564e57010a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e57010a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 -- 172.21.15.23:0/481563446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492410e470 con 0x564e56fffef0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f4922a00700 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4910004570 tx=0x7f4910009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36e129b315a7da3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f490cc00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491001d040 con 0x564e56fffef0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f491001e040 con 0x564e56fffef0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.432+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f491000be60 con 0x564e56fffef0 2024-03-30T04:11:56.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4910005bc0 con 0x564e56fffef0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 10 client.18315 fetch_fsmap learned FSMap version 30 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f49241a5fd0 con 0x564e56fffef0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f491001e040 con 0x564e56fffef0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4910004850 con 0x564e56fffef0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 10 client.18315 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 10 client.18315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 20 client.18315 populate_metadata read hostname 'smithi023' 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5706f660 0x564e57071a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 4 client.18315 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.433+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564e56da0cf0 con 0x564e5706f660 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.434+0000 7f4923400700 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5706f660 0x564e57071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f4923400700 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5706f660 0x564e57071a50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb595ecde35b9ee9 server_cookie=264e8114bebf7804 in_seq=0 out_seq=0 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f490cc00700 10 client.18315 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f490cc00700 1 -- 172.21.15.23:0/481563446 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564e56da0cf0 con 0x564e5706f660 2024-03-30T04:11:56.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f490cc00700 10 client.18315 handle_command_reply: tid=0 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 client.18315 shutdown 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 msgr2=0x564e570104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4910004570 tx=0x7f4910009ca0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 shutdown_connections 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e5700adf0 0x564e57010a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564e5706f660 0x564e57071a50 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e56fff5d0 0x564e5700ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 --2- 172.21.15.23:0/481563446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e56fffef0 0x564e570104f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f490cc00700 0 client.18315 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 >> 172.21.15.23:0/481563446 conn(0x564e56f5f6a0 msgr2=0x564e57007320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 shutdown_connections 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.435+0000 7f493dc56b80 1 -- 172.21.15.23:0/481563446 wait complete. 2024-03-30T04:11:56.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f493dc56b80 20 client.18315 trim_cache size 0 max 16384 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "74dad0e3-83c3-4b2d-ad62-431ff9d3daf2": { 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "74dad0e3-83c3-4b2d-ad62-431ff9d3daf2", 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:11:56.438 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 msgr2=0x7f49080a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 0x7f49080a3750 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f491c009ec0 tx=0x7f491c00b040 comp rx=0 tx=0).stop 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 msgr2=0x7f49241993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f491400e9a0 tx=0x7f491400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 shutdown_connections 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f49080a1290 0x7f49080a3750 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f492419dcc0 0x7f492419e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4924100f60 0x7f49241993e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 --2- 172.21.15.23:0/2255528623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4924106080 0x7f4924199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:11:56.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 >> 172.21.15.23:0/2255528623 conn(0x7f49240fc7c0 msgr2=0x7f49240feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:11:56.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 shutdown_connections 2024-03-30T04:11:56.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:11:56.436+0000 7f492a200700 1 -- 172.21.15.23:0/2255528623 wait complete. 2024-03-30T04:11:56.452 INFO:tasks.cephfs.filesystem:scrub status for tag:74dad0e3-83c3-4b2d-ad62-431ff9d3daf2 - {'path': '/', 'tag': '74dad0e3-83c3-4b2d-ad62-431ff9d3daf2', 'options': 'recursive,force'} 2024-03-30T04:11:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:56 smithi193 ceph-mon[79581]: pgmap v310: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 161 op/s 2024-03-30T04:11:57.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 996333 10.20 MB/sec execute 394 sec latency 43.044 ms 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:57 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:11:57.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:11:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:11:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:57 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:11:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:11:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:11:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:57 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:11:58.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 998542 10.21 MB/sec execute 395 sec latency 28.246 ms 2024-03-30T04:11:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:11:58 smithi154 ceph-mon[79161]: pgmap v311: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:11:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:11:58 smithi193 ceph-mon[79581]: pgmap v311: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:11:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:11:58 smithi023 ceph-mon[38254]: pgmap v311: 129 pgs: 129 active+clean; 556 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:11:59.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 999932 10.19 MB/sec execute 396 sec latency 24.905 ms 2024-03-30T04:12:00.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1002089 10.20 MB/sec execute 397 sec latency 27.975 ms 2024-03-30T04:12:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:00 smithi154 ceph-mon[79161]: pgmap v312: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 238 op/s 2024-03-30T04:12:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:00 smithi193 ceph-mon[79581]: pgmap v312: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 238 op/s 2024-03-30T04:12:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:00 smithi023 ceph-mon[38254]: pgmap v312: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.5 MiB/s wr, 238 op/s 2024-03-30T04:12:01.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1002880 10.18 MB/sec execute 398 sec latency 152.256 ms 2024-03-30T04:12:02.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1003317 10.17 MB/sec execute 399 sec latency 81.367 ms 2024-03-30T04:12:02.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:02 smithi154 ceph-mon[79161]: pgmap v313: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:12:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:02 smithi193 ceph-mon[79581]: pgmap v313: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:12:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:02 smithi023 ceph-mon[38254]: pgmap v313: 129 pgs: 129 active+clean; 560 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.1 MiB/s wr, 160 op/s 2024-03-30T04:12:03.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1005202 10.16 MB/sec execute 400 sec latency 44.532 ms 2024-03-30T04:12:04.222 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1006550 10.15 MB/sec execute 401 sec latency 27.828 ms 2024-03-30T04:12:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:04 smithi154 ceph-mon[79161]: pgmap v314: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-03-30T04:12:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:04 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:12:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:04 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:12:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:04 smithi193 ceph-mon[79581]: pgmap v314: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-03-30T04:12:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:04 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:12:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:04 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:12:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:04 smithi023 ceph-mon[38254]: pgmap v314: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.6 MiB/s wr, 230 op/s 2024-03-30T04:12:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:04 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:12:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:04 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:12:05.223 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1008726 10.16 MB/sec execute 402 sec latency 46.683 ms 2024-03-30T04:12:06.224 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1010112 10.15 MB/sec execute 403 sec latency 23.511 ms 2024-03-30T04:12:06.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:06 smithi154 ceph-mon[79161]: pgmap v315: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 189 op/s 2024-03-30T04:12:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:06 smithi193 ceph-mon[79581]: pgmap v315: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 189 op/s 2024-03-30T04:12:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:06 smithi023 ceph-mon[38254]: pgmap v315: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 189 op/s 2024-03-30T04:12:07.224 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1011877 10.15 MB/sec execute 404 sec latency 56.106 ms 2024-03-30T04:12:08.224 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1013391 10.15 MB/sec execute 405 sec latency 22.395 ms 2024-03-30T04:12:08.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:08 smithi154 ceph-mon[79161]: pgmap v316: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 175 op/s 2024-03-30T04:12:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:08 smithi193 ceph-mon[79581]: pgmap v316: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 175 op/s 2024-03-30T04:12:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:08 smithi023 ceph-mon[38254]: pgmap v316: 129 pgs: 129 active+clean; 573 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 175 op/s 2024-03-30T04:12:09.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1015405 10.15 MB/sec execute 406 sec latency 31.094 ms 2024-03-30T04:12:10.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1016515 10.14 MB/sec execute 407 sec latency 18.611 ms 2024-03-30T04:12:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:10 smithi154 ceph-mon[79161]: pgmap v317: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-30T04:12:10.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:10 smithi154 ceph-mon[79161]: osdmap e98: 12 total, 12 up, 12 in 2024-03-30T04:12:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:10 smithi193 ceph-mon[79581]: pgmap v317: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-30T04:12:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:10 smithi193 ceph-mon[79581]: osdmap e98: 12 total, 12 up, 12 in 2024-03-30T04:12:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:10 smithi023 ceph-mon[38254]: pgmap v317: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 267 op/s 2024-03-30T04:12:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:10 smithi023 ceph-mon[38254]: osdmap e98: 12 total, 12 up, 12 in 2024-03-30T04:12:11.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1017875 10.13 MB/sec execute 408 sec latency 31.737 ms 2024-03-30T04:12:12.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1019398 10.13 MB/sec execute 409 sec latency 202.124 ms 2024-03-30T04:12:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:12 smithi193 ceph-mon[79581]: pgmap v319: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-30T04:12:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:12 smithi023 ceph-mon[38254]: pgmap v319: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-30T04:12:12.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:12 smithi154 ceph-mon[79161]: pgmap v319: 129 pgs: 129 active+clean; 566 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.7 MiB/s wr, 227 op/s 2024-03-30T04:12:13.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1020330 10.11 MB/sec execute 410 sec latency 210.430 ms 2024-03-30T04:12:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:13 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:12:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:13 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:12:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:13 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:12:14.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1021211 10.10 MB/sec execute 411 sec latency 21.092 ms 2024-03-30T04:12:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:14 smithi193 ceph-mon[79581]: pgmap v320: 129 pgs: 129 active+clean; 551 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 231 op/s 2024-03-30T04:12:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:14 smithi023 ceph-mon[38254]: pgmap v320: 129 pgs: 129 active+clean; 551 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 231 op/s 2024-03-30T04:12:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:14 smithi154 ceph-mon[79161]: pgmap v320: 129 pgs: 129 active+clean; 551 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 231 op/s 2024-03-30T04:12:15.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1023105 10.11 MB/sec execute 412 sec latency 49.138 ms 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: pgmap v321: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:12:15.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:15 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:12:15.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: pgmap v321: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:12:15.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:12:15.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:15 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:12:15.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: pgmap v321: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 202 op/s 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi154 to 182.5M 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi154 to 191374848: error parsing value: Value '191374848' is below minimum 939524096 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:15.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi193 to 182.5M 2024-03-30T04:12:15.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:15 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi193 to 191378432: error parsing value: Value '191378432' is below minimum 939524096 2024-03-30T04:12:16.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1023898 10.09 MB/sec execute 413 sec latency 30.424 ms 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:16 smithi154 ceph-mon[79161]: osdmap e99: 12 total, 12 up, 12 in 2024-03-30T04:12:16.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:16.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:16 smithi193 ceph-mon[79581]: osdmap e99: 12 total, 12 up, 12 in 2024-03-30T04:12:16.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: Adjusting osd_memory_target on smithi023 to 221.5M 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: Unable to set osd_memory_target on smithi023 to 232359321: error parsing value: Value '232359321' is below minimum 939524096 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:12:16.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:16 smithi023 ceph-mon[38254]: osdmap e99: 12 total, 12 up, 12 in 2024-03-30T04:12:17.225 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1025534 10.08 MB/sec execute 414 sec latency 57.262 ms 2024-03-30T04:12:18.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1026924 10.08 MB/sec execute 415 sec latency 25.332 ms 2024-03-30T04:12:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:18 smithi154 ceph-mon[79161]: pgmap v323: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 115 op/s 2024-03-30T04:12:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:18 smithi193 ceph-mon[79581]: pgmap v323: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 115 op/s 2024-03-30T04:12:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:18 smithi023 ceph-mon[38254]: pgmap v323: 129 pgs: 129 active+clean; 546 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 115 op/s 2024-03-30T04:12:19.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1028326 10.07 MB/sec execute 416 sec latency 30.335 ms 2024-03-30T04:12:20.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1030308 10.08 MB/sec execute 417 sec latency 18.914 ms 2024-03-30T04:12:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:20 smithi154 ceph-mon[79161]: pgmap v324: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-03-30T04:12:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:20 smithi193 ceph-mon[79581]: pgmap v324: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-03-30T04:12:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:20 smithi023 ceph-mon[38254]: pgmap v324: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 180 op/s 2024-03-30T04:12:21.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1031221 10.06 MB/sec execute 418 sec latency 32.345 ms 2024-03-30T04:12:22.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1032827 10.05 MB/sec execute 419 sec latency 152.033 ms 2024-03-30T04:12:22.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:22 smithi154 ceph-mon[79161]: pgmap v325: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-03-30T04:12:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:22 smithi193 ceph-mon[79581]: pgmap v325: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-03-30T04:12:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:22 smithi023 ceph-mon[38254]: pgmap v325: 129 pgs: 129 active+clean; 555 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 163 op/s 2024-03-30T04:12:23.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1034138 10.05 MB/sec execute 420 sec latency 28.241 ms 2024-03-30T04:12:24.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1035554 10.04 MB/sec execute 421 sec latency 27.743 ms 2024-03-30T04:12:24.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:24 smithi154 ceph-mon[79161]: pgmap v326: 129 pgs: 129 active+clean; 558 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-03-30T04:12:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:24 smithi193 ceph-mon[79581]: pgmap v326: 129 pgs: 129 active+clean; 558 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-03-30T04:12:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:24 smithi023 ceph-mon[38254]: pgmap v326: 129 pgs: 129 active+clean; 558 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 150 op/s 2024-03-30T04:12:25.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1037440 10.05 MB/sec execute 422 sec latency 23.618 ms 2024-03-30T04:12:26.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1038543 10.03 MB/sec execute 423 sec latency 30.661 ms 2024-03-30T04:12:26.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:26 smithi154 ceph-mon[79161]: pgmap v327: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 158 op/s 2024-03-30T04:12:26.453 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:12:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:26 smithi193 ceph-mon[79581]: pgmap v327: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 158 op/s 2024-03-30T04:12:26.517 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:26 smithi023 ceph-mon[38254]: pgmap v327: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.5 MiB/s wr, 158 op/s 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 -- 172.21.15.23:0/3569067126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba0069c10 msgr2=0x7ffba0105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 --2- 172.21.15.23:0/3569067126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba0069c10 0x7ffba0105980 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009990 tx=0x7ffb90009ca0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 -- 172.21.15.23:0/3569067126 shutdown_connections 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 --2- 172.21.15.23:0/3569067126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 0x7ffba010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 --2- 172.21.15.23:0/3569067126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba0069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 --2- 172.21.15.23:0/3569067126 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba0069c10 0x7ffba0105980 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 -- 172.21.15.23:0/3569067126 >> 172.21.15.23:0/3569067126 conn(0x7ffba0076d80 msgr2=0x7ffba0077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 -- 172.21.15.23:0/3569067126 shutdown_connections 2024-03-30T04:12:26.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.670+0000 7ffba8800700 1 -- 172.21.15.23:0/3569067126 wait complete. 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.671+0000 7ffba8800700 1 Processor -- start 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.671+0000 7ffba8800700 1 -- start start 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 0x7ffba01999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba019dcf0 0x7ffba019e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019e770 con 0x7ffba0069260 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019e8e0 con 0x7ffba019dcf0 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba8800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019eaa0 con 0x7ffba01060b0 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45602/0 (socket says 172.21.15.23:45602) 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba5e00700 1 -- 172.21.15.23:0/3553033096 learned_addr learned my addr 172.21.15.23:0/3553033096 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:26.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba6800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba019dcf0 0x7ffba019e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.672+0000 7ffba5400700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 0x7ffba01999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 -- 172.21.15.23:0/3553033096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 msgr2=0x7ffba01999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 0x7ffba01999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 -- 172.21.15.23:0/3553033096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba019dcf0 msgr2=0x7ffba019e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba019dcf0 0x7ffba019e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 -- 172.21.15.23:0/3553033096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c009c40 con 0x7ffba0069260 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400e9a0 tx=0x7ffb9400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffb9aa00700 1 -- 172.21.15.23:0/3553033096 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94009790 con 0x7ffba0069260 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb900095f0 con 0x7ffba0069260 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffb9aa00700 1 -- 172.21.15.23:0/3553033096 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb9400cbd0 con 0x7ffba0069260 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffb9aa00700 1 -- 172.21.15.23:0/3553033096 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9400eee0 con 0x7ffba0069260 2024-03-30T04:12:26.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.673+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba01a62f0 con 0x7ffba0069260 2024-03-30T04:12:26.675 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.676+0000 7ffb9aa00700 1 -- 172.21.15.23:0/3553033096 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ffb9400cd40 con 0x7ffba0069260 2024-03-30T04:12:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.678+0000 7ffbbc2ddb80 1 Processor -- start 2024-03-30T04:12:26.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 -- start start 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffb9aa00700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 0x7ffb880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffb9aa00700 1 -- 172.21.15.23:0/3553033096 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ffb94014070 con 0x7ffba0069260 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b9a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffba6800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 0x7ffb880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b9a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b8ef990 con 0x56265b996ef0 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b8ed350 con 0x56265b9a1df0 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.679+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b8e8c40 con 0x56265b9965d0 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b9a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b9a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b9a41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40000/0 (socket says 172.21.15.23:40000) 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba6800700 1 -- 172.21.15.23:0/3232297397 learned_addr learned my addr 172.21.15.23:0/3232297397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:26.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 -- 172.21.15.23:0/3232297397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 msgr2=0x56265b9969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 -- 172.21.15.23:0/3232297397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 msgr2=0x56265b9a41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b9a41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.680+0000 7ffba5400700 1 -- 172.21.15.23:0/3232297397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56265b76a3b0 con 0x56265b996ef0 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.681+0000 7ffba5400700 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b9a18b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffb90000c00 tx=0x7ffb90005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc33624e722c088d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.681+0000 7ffba6800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 0x7ffb880a3750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00fc70 tx=0x7ffb9c00f630 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.682+0000 7ffb8f000700 1 -- 172.21.15.23:0/3232297397 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb900056a0 con 0x56265b996ef0 2024-03-30T04:12:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.682+0000 7ffb8f000700 1 -- 172.21.15.23:0/3232297397 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb9001e040 con 0x56265b996ef0 2024-03-30T04:12:26.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.682+0000 7ffb8f000700 1 -- 172.21.15.23:0/3232297397 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90004900 con 0x56265b996ef0 2024-03-30T04:12:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.683+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3232297397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 msgr2=0x56265b9a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.683+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b9a18b0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ffb90000c00 tx=0x7ffb90005ea0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.686+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3232297397 shutdown_connections 2024-03-30T04:12:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.686+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b9a41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.686+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.686+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3232297397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b9a18b0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.686+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3232297397 >> 172.21.15.23:0/3232297397 conn(0x56265b8f66a0 msgr2=0x56265b992a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.687+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3232297397 shutdown_connections 2024-03-30T04:12:26.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.688+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3232297397 wait complete. 2024-03-30T04:12:26.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.690+0000 7ffbbc2ddb80 1 -- start start 2024-03-30T04:12:26.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.691+0000 7ffbbc2ddb80 1 Processor -- start 2024-03-30T04:12:26.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9a1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b99f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b884d80 con 0x56265b996ef0 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b884a20 con 0x56265b9a1df0 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b76c6a0 con 0x56265b9965d0 2024-03-30T04:12:26.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.692+0000 7ffbbc2ddb80 10 client.0 _ll_register_callbacks cb 0x56265b88e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:26.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.694+0000 7ffba5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.694+0000 7ffba5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45622/0 (socket says 172.21.15.23:45622) 2024-03-30T04:12:26.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.694+0000 7ffba5400700 1 -- 172.21.15.23:0/3916014300 learned_addr learned my addr 172.21.15.23:0/3916014300 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:26.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.694+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.696+0000 7ffba5400700 1 -- 172.21.15.23:0/3916014300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 msgr2=0x56265b9a1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.696+0000 7ffba5400700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9a1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.696+0000 7ffba5400700 1 -- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 msgr2=0x56265b99f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.696+0000 7ffba5400700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b99f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.696+0000 7ffba5400700 1 -- 172.21.15.23:0/3916014300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba010e470 con 0x56265b996ef0 2024-03-30T04:12:26.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.697+0000 7ffba5400700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ffb90005f50 tx=0x7ffb9000b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cdcfc374cfc210 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.697+0000 7ffb83400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:12:26.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.701+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90004700 con 0x56265b996ef0 2024-03-30T04:12:26.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.702+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ffb90004860 con 0x56265b996ef0 2024-03-30T04:12:26.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.703+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56265b8a9380 con 0x56265b996ef0 2024-03-30T04:12:26.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.703+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb90004a20 con 0x56265b996ef0 2024-03-30T04:12:26.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.706+0000 7ffbbc2ddb80 10 client.18327 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:26.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.706+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56265b716b60 con 0x56265b996ef0 2024-03-30T04:12:26.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.707+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56265b8a9380 con 0x56265b996ef0 2024-03-30T04:12:26.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb90034070 con 0x56265b996ef0 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 10 client.18327 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 10 client.18327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 20 client.18327 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265b9a8490 0x56265b9aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 4 client.18327 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:26.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.708+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56265b8e1700 con 0x56265b9a8490 2024-03-30T04:12:26.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.709+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265b9a8490 0x56265b9aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.713+0000 7ffb83400700 10 client.18327 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:26.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.713+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265b9a8490 0x56265b9aa8b0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e174579534764977 server_cookie=aa9635fbad0e095e in_seq=0 out_seq=0 2024-03-30T04:12:26.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.718+0000 7ffb83400700 1 -- 172.21.15.23:0/3916014300 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56265b8e1700 con 0x56265b9a8490 2024-03-30T04:12:26.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.718+0000 7ffb83400700 10 client.18327 handle_command_reply: tid=0 2024-03-30T04:12:26.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.718+0000 7ffbbc2ddb80 1 client.18327 shutdown 2024-03-30T04:12:26.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 msgr2=0x56265b99eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ffb90005f50 tx=0x7ffb9000b750 comp rx=0 tx=0).stop 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 shutdown_connections 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b9a1df0 0x56265b99f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265b9a8490 0x56265b9aa8b0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9a1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3916014300 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b996ef0 0x56265b99eb40 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 >> 172.21.15.23:0/3916014300 conn(0x56265b8f66a0 msgr2=0x56265b9986c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.719+0000 7ffb83400700 0 client.18327 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:26.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.720+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 shutdown_connections 2024-03-30T04:12:26.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.720+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3916014300 wait complete. 2024-03-30T04:12:26.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.721+0000 7ffbbc2ddb80 20 client.18327 trim_cache size 0 max 16384 2024-03-30T04:12:26.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.767+0000 7ffbbc2ddb80 1 Processor -- start 2024-03-30T04:12:26.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 -- start start 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9965d0 0x56265b991f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9af070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b9984d0 con 0x56265b9965d0 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b992de0 con 0x56265b996ef0 2024-03-30T04:12:26.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.768+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b992fa0 con 0x56265b9a1df0 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.769+0000 7ffba6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.769+0000 7ffba6800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:40336/0 (socket says 172.21.15.23:40336) 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.769+0000 7ffba5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.769+0000 7ffba5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9965d0 0x56265b991f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.769+0000 7ffba6800700 1 -- 172.21.15.23:0/3736300087 learned_addr learned my addr 172.21.15.23:0/3736300087 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:26.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 -- 172.21.15.23:0/3736300087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 msgr2=0x56265b9af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 -- 172.21.15.23:0/3736300087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9965d0 msgr2=0x56265b991f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9965d0 0x56265b991f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 -- 172.21.15.23:0/3736300087 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56265b8aeea0 con 0x56265b9a1df0 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.770+0000 7ffba6800700 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c006040 tx=0x7ffb9c01b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133b06044324602e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.771+0000 7ffb8fa00700 1 -- 172.21.15.23:0/3736300087 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c00e040 con 0x56265b9a1df0 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.771+0000 7ffb8fa00700 1 -- 172.21.15.23:0/3736300087 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb9c00f040 con 0x56265b9a1df0 2024-03-30T04:12:26.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.771+0000 7ffb8fa00700 1 -- 172.21.15.23:0/3736300087 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9c01a7b0 con 0x56265b9a1df0 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3736300087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 msgr2=0x56265b98b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c006040 tx=0x7ffb9c01b000 comp rx=0 tx=0).stop 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3736300087 shutdown_connections 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9a1df0 0x56265b98b5a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9965d0 0x56265b991f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3736300087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3736300087 >> 172.21.15.23:0/3736300087 conn(0x56265b8f66a0 msgr2=0x56265b9aa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3736300087 shutdown_connections 2024-03-30T04:12:26.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.772+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3736300087 wait complete. 2024-03-30T04:12:26.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.773+0000 7ffbbc2ddb80 1 -- start start 2024-03-30T04:12:26.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.773+0000 7ffbbc2ddb80 1 Processor -- start 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9a1df0 0x56265b9ad050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b9ad650 con 0x56265b9a1df0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b9ad7c0 con 0x56265b996ef0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56265b9ad980 con 0x56265b9965d0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffbbc2ddb80 10 client.0 _ll_register_callbacks cb 0x56265b88e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:40340/0 (socket says 172.21.15.23:40340) 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba5e00700 1 -- 172.21.15.23:0/3743723576 learned_addr learned my addr 172.21.15.23:0/3743723576 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:26.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba5400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40030/0 (socket says 172.21.15.23:40030) 2024-03-30T04:12:26.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.774+0000 7ffba6800700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9a1df0 0x56265b9ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 -- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 msgr2=0x56265b9aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 -- 172.21.15.23:0/3743723576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9a1df0 msgr2=0x56265b9ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9a1df0 0x56265b9ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 -- 172.21.15.23:0/3743723576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c00fdb0 con 0x56265b9965d0 2024-03-30T04:12:26.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffba5e00700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400e970 tx=0x7ffb940cc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85564be533ca7382 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffb83400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9409de40 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ffb940b6030 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.775+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb9001e040 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb940a2490 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 10 client.28294 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56265b76ecd0 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb9407b030 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb940a2000 con 0x56265b9965d0 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 10 client.28294 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 10 client.28294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 20 client.28294 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265ba17710 0x56265ba19b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:26.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 4 client.28294 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56265b737cf0 con 0x56265ba17710 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.776+0000 7ffba5400700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265ba17710 0x56265ba19b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffba5400700 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265ba17710 0x56265ba19b00 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=931051705c667e6a server_cookie=7025612d0b89d949 in_seq=0 out_seq=0 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffb83400700 10 client.28294 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffb83400700 1 -- 172.21.15.23:0/3743723576 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56265b737cf0 con 0x56265ba17710 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffb83400700 10 client.28294 handle_command_reply: tid=0 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 client.28294 shutdown 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 msgr2=0x56265b9ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400e970 tx=0x7ffb940cc040 comp rx=0 tx=0).stop 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 shutdown_connections 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56265b9a1df0 0x56265b9ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56265ba17710 0x56265ba19b00 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56265b9965d0 0x56265b9ae860 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 --2- 172.21.15.23:0/3743723576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56265b996ef0 0x56265b9aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 >> 172.21.15.23:0/3743723576 conn(0x56265b8f66a0 msgr2=0x56265b98b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffb83400700 0 client.28294 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.777+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 shutdown_connections 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffbbc2ddb80 1 -- 172.21.15.23:0/3743723576 wait complete. 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffbbc2ddb80 20 client.28294 trim_cache size 0 max 16384 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:12:26.779 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 msgr2=0x7ffb880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 0x7ffb880a3750 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c00fc70 tx=0x7ffb9c00f630 comp rx=0 tx=0).stop 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 msgr2=0x7ffba01994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.778+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7ffb9400e9a0 tx=0x7ffb9400ed60 comp rx=0 tx=0).stop 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 shutdown_connections 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb880a1290 0x7ffb880a3750 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffba019dcf0 0x7ffba019e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffba0069260 0x7ffba01994b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 --2- 172.21.15.23:0/3553033096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffba01060b0 0x7ffba01999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:26.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 >> 172.21.15.23:0/3553033096 conn(0x7ffba0076d80 msgr2=0x7ffba0107240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:26.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 shutdown_connections 2024-03-30T04:12:26.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:26.779+0000 7ffba8800700 1 -- 172.21.15.23:0/3553033096 wait complete. 2024-03-30T04:12:26.797 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:12:26.797 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:12:27.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 -- 172.21.15.23:0/202089040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c101810 msgr2=0x7f978c105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 --2- 172.21.15.23:0/202089040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c101810 0x7f978c105850 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9778009990 tx=0x7f9778009ca0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 -- 172.21.15.23:0/202089040 shutdown_connections 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 --2- 172.21.15.23:0/202089040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c105f80 0x7f978c10db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 --2- 172.21.15.23:0/202089040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c100e60 0x7f978c101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 --2- 172.21.15.23:0/202089040 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c101810 0x7f978c105850 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 -- 172.21.15.23:0/202089040 >> 172.21.15.23:0/202089040 conn(0x7f978c0fc760 msgr2=0x7f978c0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 -- 172.21.15.23:0/202089040 shutdown_connections 2024-03-30T04:12:27.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.008+0000 7f9792800700 1 -- 172.21.15.23:0/202089040 wait complete. 2024-03-30T04:12:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.009+0000 7f9792800700 1 Processor -- start 2024-03-30T04:12:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.009+0000 7f9792800700 1 -- start start 2024-03-30T04:12:27.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.009+0000 7f9792800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.009+0000 7f9792800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 0x7f978c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f9792800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c073b30 0x7f978c1a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f9792800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a5960 con 0x7f978c100e60 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f9792800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a5aa0 con 0x7f978c105f80 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f9792800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978c1a5c60 con 0x7f978c073b30 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45668/0 (socket says 172.21.15.23:45668) 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 -- 172.21.15.23:0/3538307933 learned_addr learned my addr 172.21.15.23:0/3538307933 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978aa00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 0x7f978c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978be00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c073b30 0x7f978c1a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 -- 172.21.15.23:0/3538307933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c073b30 msgr2=0x7f978c1a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c073b30 0x7f978c1a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 -- 172.21.15.23:0/3538307933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 msgr2=0x7f978c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 0x7f978c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978b400700 1 -- 172.21.15.23:0/3538307933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97780095f0 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.010+0000 7f978aa00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 0x7f978c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f978b400700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e9a0 tx=0x7f977c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c009790 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f977c010430 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978c1a5f40 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977c018680 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.011+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c1a6410 con 0x7f978c100e60 2024-03-30T04:12:27.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.012+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f978c109250 con 0x7f978c100e60 2024-03-30T04:12:27.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.014+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f977c0105a0 con 0x7f978c100e60 2024-03-30T04:12:27.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.015+0000 7f9783e00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 0x7f97700a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.015+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f977c014070 con 0x7f978c100e60 2024-03-30T04:12:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.016+0000 7f978aa00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 0x7f97700a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.016+0000 7f978aa00700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 0x7f97700a37d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f978c074790 tx=0x7f9778005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.021+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f977c0529f0 con 0x7f978c100e60 2024-03-30T04:12:27.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1040515 10.04 MB/sec execute 424 sec latency 96.551 ms 2024-03-30T04:12:27.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.325+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f978c0745e0 con 0x7f978c100e60 2024-03-30T04:12:27.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.325+0000 7f9783e00700 1 -- 172.21.15.23:0/3538307933 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f977c096270 con 0x7f978c100e60 2024-03-30T04:12:27.324 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:12:27.324 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 msgr2=0x7f97700a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 0x7f97700a37d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f978c074790 tx=0x7f9778005fd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 msgr2=0x7f978c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f977c00e9a0 tx=0x7f977c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 shutdown_connections 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f978c073b30 0x7f978c1a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f978c100e60 0x7f978c0730b0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f97700a1310 0x7f97700a37d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 --2- 172.21.15.23:0/3538307933 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f978c105f80 0x7f978c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 >> 172.21.15.23:0/3538307933 conn(0x7f978c0fc760 msgr2=0x7f978c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 shutdown_connections 2024-03-30T04:12:27.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.328+0000 7f9792800700 1 -- 172.21.15.23:0/3538307933 wait complete. 2024-03-30T04:12:27.328 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:12:27.346 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 -- 172.21.15.23:0/381894305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5948069540 msgr2=0x7f5948069920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 --2- 172.21.15.23:0/381894305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5948069540 0x7f5948069920 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f59380099c0 tx=0x7f5938009cd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 -- 172.21.15.23:0/381894305 shutdown_connections 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 --2- 172.21.15.23:0/381894305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f59481060a0 0x7f594810dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 --2- 172.21.15.23:0/381894305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5948069540 0x7f5948069920 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 --2- 172.21.15.23:0/381894305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5948069ef0 0x7f5948105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 -- 172.21.15.23:0/381894305 >> 172.21.15.23:0/381894305 conn(0x7f5948076d50 msgr2=0x7f5948077160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 -- 172.21.15.23:0/381894305 shutdown_connections 2024-03-30T04:12:27.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.557+0000 7f594da00700 1 -- 172.21.15.23:0/381894305 wait complete. 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.558+0000 7f594da00700 1 Processor -- start 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.558+0000 7f594da00700 1 -- start start 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 0x7f5948199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594819e7b0 con 0x7f5948069540 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594819e920 con 0x7f594819dd30 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f594da00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594819eae0 con 0x7f59481060a0 2024-03-30T04:12:27.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5947400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5947400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40066/0 (socket says 172.21.15.23:40066) 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45686/0 (socket says 172.21.15.23:45686) 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5947400700 1 -- 172.21.15.23:0/1756469977 learned_addr learned my addr 172.21.15.23:0/1756469977 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 0x7f5948199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 -- 172.21.15.23:0/1756469977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 msgr2=0x7f594819e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 -- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 msgr2=0x7f5948199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946000700 1 -- 172.21.15.23:0/1756469977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59380095f0 con 0x7f59481060a0 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5947400700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.559+0000 7f5946a00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f5946000700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 0x7f5948199990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9c0 tx=0x7f593400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f593f400700 1 -- 172.21.15.23:0/1756469977 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593400c9c0 con 0x7f59481060a0 2024-03-30T04:12:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f593f400700 1 -- 172.21.15.23:0/1756469977 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5934010430 con 0x7f59481060a0 2024-03-30T04:12:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59481a5fe0 con 0x7f59481060a0 2024-03-30T04:12:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f593f400700 1 -- 172.21.15.23:0/1756469977 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934018680 con 0x7f59481060a0 2024-03-30T04:12:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.560+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59481a64c0 con 0x7f59481060a0 2024-03-30T04:12:27.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.563+0000 7f593f400700 1 -- 172.21.15.23:0/1756469977 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f593400cb20 con 0x7f59481060a0 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.564+0000 7f593f400700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 0x7f592c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.564+0000 7f59614c5b80 1 Processor -- start 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f593f400700 1 -- 172.21.15.23:0/1756469977 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5934014070 con 0x7f59481060a0 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 -- start start 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f5946a00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 0x7f592c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611304009b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x56113040e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130359990 con 0x56113040bdf0 2024-03-30T04:12:27.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130357350 con 0x5611304005d0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f59614c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130352c40 con 0x561130400ef0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.565+0000 7f5946000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611304009b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:40390/0 (socket says 172.21.15.23:40390) 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 -- 172.21.15.23:0/804488161 learned_addr learned my addr 172.21.15.23:0/804488161 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5947400700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x56113040e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946a00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 0x7f592c0a3750 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5938009990 tx=0x7f5938005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 -- 172.21.15.23:0/804488161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 msgr2=0x5611304009b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611304009b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 -- 172.21.15.23:0/804488161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 msgr2=0x56113040e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x56113040e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 -- 172.21.15.23:0/804488161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611301d43b0 con 0x561130400ef0 2024-03-30T04:12:27.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946a00700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611304009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5947400700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x56113040e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5946000700 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f59340a1e20 tx=0x7f5934099db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a794733167f00bb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5933e00700 1 -- 172.21.15.23:0/804488161 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593409dd00 con 0x561130400ef0 2024-03-30T04:12:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5933e00700 1 -- 172.21.15.23:0/804488161 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59340a2350 con 0x561130400ef0 2024-03-30T04:12:27.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.566+0000 7f5933e00700 1 -- 172.21.15.23:0/804488161 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593409a130 con 0x561130400ef0 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 -- 172.21.15.23:0/804488161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 msgr2=0x56113040b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f59340a1e20 tx=0x7f5934099db0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 -- 172.21.15.23:0/804488161 shutdown_connections 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x56113040e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611304009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 --2- 172.21.15.23:0/804488161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x56113040b8b0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 -- 172.21.15.23:0/804488161 >> 172.21.15.23:0/804488161 conn(0x5611303606a0 msgr2=0x5611303fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 -- 172.21.15.23:0/804488161 shutdown_connections 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.567+0000 7f59614c5b80 1 -- 172.21.15.23:0/804488161 wait complete. 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 -- start start 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 Processor -- start 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561130400ef0 0x561130408bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x5611304092c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611302eed80 con 0x561130400ef0 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611302eea20 con 0x5611304005d0 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.568+0000 7f59614c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611301d66a0 con 0x56113040bdf0 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f59614c5b80 10 client.0 _ll_register_callbacks cb 0x5611302f8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:27.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40076/0 (socket says 172.21.15.23:40076) 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 -- 172.21.15.23:0/3876760990 learned_addr learned my addr 172.21.15.23:0/3876760990 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5947400700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x5611304092c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946000700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561130400ef0 0x561130408bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 -- 172.21.15.23:0/3876760990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 msgr2=0x5611304092c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x5611304092c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 -- 172.21.15.23:0/3876760990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561130400ef0 msgr2=0x561130408bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561130400ef0 0x561130408bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 -- 172.21.15.23:0/3876760990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59481a64c0 con 0x5611304005d0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5946a00700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f59380099c0 tx=0x7f5938009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25c073c464e0f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.569+0000 7f5930c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593800bee0 con 0x5611304005d0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561130313380 con 0x5611304005d0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5938025450 con 0x5611304005d0 2024-03-30T04:12:27.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5938025610 con 0x5611304005d0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f59614c5b80 10 client.28325 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561130180b60 con 0x5611304005d0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561130313380 con 0x5611304005d0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5938010870 con 0x5611304005d0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f59614c5b80 10 client.28325 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.570+0000 7f59614c5b80 10 client.28325 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f59614c5b80 20 client.28325 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56113041b990 0x56113041ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f59614c5b80 4 client.28325 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56113034b700 con 0x56113041b990 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f5946000700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56113041b990 0x56113041ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f5946000700 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56113041b990 0x56113041ddb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1acd4236c837a60b server_cookie=6e688cb96a3d539e in_seq=0 out_seq=0 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.571+0000 7f5930c00700 10 client.28325 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:12:27.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f5930c00700 1 -- 172.21.15.23:0/3876760990 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56113034b700 con 0x56113041b990 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f5930c00700 10 client.28325 handle_command_reply: tid=0 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 client.28325 shutdown 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 msgr2=0x56113040b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f59380099c0 tx=0x7f5938009cd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 shutdown_connections 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x56113041b990 0x56113041ddb0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x5611304092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x56113040b870 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3876760990 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561130400ef0 0x561130408bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f5930c00700 0 client.28325 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.572+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 >> 172.21.15.23:0/3876760990 conn(0x5611303606a0 msgr2=0x5611304026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.573+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 shutdown_connections 2024-03-30T04:12:27.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.573+0000 7f59614c5b80 1 -- 172.21.15.23:0/3876760990 wait complete. 2024-03-30T04:12:27.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.573+0000 7f59614c5b80 20 client.28325 trim_cache size 0 max 16384 2024-03-30T04:12:27.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.588+0000 7f59614c5b80 1 Processor -- start 2024-03-30T04:12:27.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.588+0000 7f59614c5b80 1 -- start start 2024-03-30T04:12:27.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.588+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611303fbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.588+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x5611303fc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f59614c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130401f60 con 0x56113040bdf0 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f59614c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611304035b0 con 0x5611304005d0 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f59614c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611304024d0 con 0x561130400ef0 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f5946000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f5946000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:40408/0 (socket says 172.21.15.23:40408) 2024-03-30T04:12:27.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f5946000700 1 -- 172.21.15.23:0/3458366608 learned_addr learned my addr 172.21.15.23:0/3458366608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f5947400700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x5611303fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.589+0000 7f5946a00700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611303fbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946000700 1 -- 172.21.15.23:0/3458366608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 msgr2=0x5611303fbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946000700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611303fbf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946000700 1 -- 172.21.15.23:0/3458366608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 msgr2=0x5611303fc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946000700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x5611303fc530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946000700 1 -- 172.21.15.23:0/3458366608 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561130318ea0 con 0x561130400ef0 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5946a00700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611303fbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.590+0000 7f5947400700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x5611303fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.591+0000 7f5946000700 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f59340963d0 tx=0x7f5934096340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fea5443a2dc90a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.591+0000 7f593cc00700 1 -- 172.21.15.23:0/3458366608 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340106a0 con 0x561130400ef0 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.591+0000 7f593cc00700 1 -- 172.21.15.23:0/3458366608 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f593409dd00 con 0x561130400ef0 2024-03-30T04:12:27.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.591+0000 7f593cc00700 1 -- 172.21.15.23:0/3458366608 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593409a130 con 0x561130400ef0 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- 172.21.15.23:0/3458366608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 msgr2=0x561130412a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f59340963d0 tx=0x7f5934096340 comp rx=0 tx=0).stop 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- 172.21.15.23:0/3458366608 shutdown_connections 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56113040bdf0 0x5611303fc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5611304005d0 0x5611303fbf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3458366608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561130400ef0 0x561130412a70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- 172.21.15.23:0/3458366608 >> 172.21.15.23:0/3458366608 conn(0x5611303606a0 msgr2=0x5611303f55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- 172.21.15.23:0/3458366608 shutdown_connections 2024-03-30T04:12:27.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- 172.21.15.23:0/3458366608 wait complete. 2024-03-30T04:12:27.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.592+0000 7f59614c5b80 1 -- start start 2024-03-30T04:12:27.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 Processor -- start 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611304005d0 0x5611304126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x561130410cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130411400 con 0x5611304005d0 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130411570 con 0x561130400ef0 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561130411730 con 0x56113040bdf0 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.593+0000 7f59614c5b80 10 client.0 _ll_register_callbacks cb 0x7f593800baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.594+0000 7f5946000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.594+0000 7f5946000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40086/0 (socket says 172.21.15.23:40086) 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.594+0000 7f5946000700 1 -- 172.21.15.23:0/3329310379 learned_addr learned my addr 172.21.15.23:0/3329310379 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.594+0000 7f5947400700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x561130410cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 -- 172.21.15.23:0/3329310379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 msgr2=0x561130410cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x561130410cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 -- 172.21.15.23:0/3329310379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611304005d0 msgr2=0x5611304126b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611304005d0 0x5611304126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 -- 172.21.15.23:0/3329310379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5940009260 con 0x561130400ef0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5946000700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5934099bd0 tx=0x7f59340caa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b179d80726c6ca42 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5947400700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x561130410cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.595+0000 7f5930c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340106a0 con 0x561130400ef0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5934010860 con 0x561130400ef0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59340a2350 con 0x561130400ef0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59340b8170 con 0x561130400ef0 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f59614c5b80 10 client.28331 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:27.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.596+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5611301d43b0 con 0x561130400ef0 2024-03-30T04:12:27.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59340a2350 con 0x561130400ef0 2024-03-30T04:12:27.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f59340a1ec0 con 0x561130400ef0 2024-03-30T04:12:27.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 10 client.28331 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:27.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 10 client.28331 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:12:27.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 20 client.28331 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:27.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561130478e20 0x56113047b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 4 client.28331 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:12:27.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.597+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5611301a1cf0 con 0x561130478e20 2024-03-30T04:12:27.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.598+0000 7f5946a00700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561130478e20 0x56113047b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.600+0000 7f5946a00700 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561130478e20 0x56113047b210 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f2e6b07042fc5d50 server_cookie=42fe737812a51401 in_seq=0 out_seq=0 2024-03-30T04:12:27.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.600+0000 7f5930c00700 10 client.28331 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:12:27.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.600+0000 7f5930c00700 1 -- 172.21.15.23:0/3329310379 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5611301a1cf0 con 0x561130478e20 2024-03-30T04:12:27.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.600+0000 7f5930c00700 10 client.28331 handle_command_reply: tid=0 2024-03-30T04:12:27.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.600+0000 7f59614c5b80 1 client.28331 shutdown 2024-03-30T04:12:27.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 msgr2=0x561130412bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5934099bd0 tx=0x7f59340caa70 comp rx=0 tx=0).stop 2024-03-30T04:12:27.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 shutdown_connections 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x561130478e20 0x56113047b210 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56113040bdf0 0x561130410cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5611304005d0 0x5611304126b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 --2- 172.21.15.23:0/3329310379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561130400ef0 0x561130412bf0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 >> 172.21.15.23:0/3329310379 conn(0x5611303606a0 msgr2=0x561130403ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f5930c00700 0 client.28331 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 shutdown_connections 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.601+0000 7f59614c5b80 1 -- 172.21.15.23:0/3329310379 wait complete. 2024-03-30T04:12:27.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.602+0000 7f59614c5b80 20 client.28331 trim_cache size 0 max 16384 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 msgr2=0x7f592c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 0x7f592c0a3750 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5938009990 tx=0x7f5938005fd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 msgr2=0x7f5948199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 0x7f5948199990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f593400e9c0 tx=0x7f593400ecd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 shutdown_connections 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f592c0a1290 0x7f592c0a3750 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f594819dd30 0x7f594819e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5948069540 0x7f5948199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 --2- 172.21.15.23:0/1756469977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59481060a0 0x7f5948199990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 >> 172.21.15.23:0/1756469977 conn(0x7f5948076d50 msgr2=0x7f594810c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 shutdown_connections 2024-03-30T04:12:27.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.603+0000 7f594da00700 1 -- 172.21.15.23:0/1756469977 wait complete. 2024-03-30T04:12:27.633 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:12:27.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 -- 172.21.15.23:0/463116151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf810d460 msgr2=0x7f0bf810f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/463116151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf810d460 0x7f0bf810f8d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000b0a0 tx=0x7f0bf000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 -- 172.21.15.23:0/463116151 shutdown_connections 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/463116151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf810d460 0x7f0bf810f8d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/463116151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf80731e0 0x7f0bf80735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/463116151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf8073b00 0x7f0bf810cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 -- 172.21.15.23:0/463116151 >> 172.21.15.23:0/463116151 conn(0x7f0bf80fc710 msgr2=0x7f0bf80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 -- 172.21.15.23:0/463116151 shutdown_connections 2024-03-30T04:12:27.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.905+0000 7f0bfcc00700 1 -- 172.21.15.23:0/463116151 wait complete. 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 Processor -- start 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 -- start start 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf80731e0 0x7f0bf8199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 0x7f0bf81999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf819e330 con 0x7f0bf810d460 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf819e4a0 con 0x7f0bf8073b00 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.906+0000 7f0bfcc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf819e660 con 0x7f0bf80731e0 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45722/0 (socket says 172.21.15.23:45722) 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 -- 172.21.15.23:0/4201437358 learned_addr learned my addr 172.21.15.23:0/4201437358 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf5400700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 0x7f0bf81999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf80731e0 0x7f0bf8199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 -- 172.21.15.23:0/4201437358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf80731e0 msgr2=0x7f0bf8199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf80731e0 0x7f0bf8199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 -- 172.21.15.23:0/4201437358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 msgr2=0x7f0bf81999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 0x7f0bf81999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 -- 172.21.15.23:0/4201437358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8009520 con 0x7f0bf810d460 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf5400700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 0x7f0bf81999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.907+0000 7f0bf6800700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000b070 tx=0x7f0bf0007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.908+0000 7f0beea00700 1 -- 172.21.15.23:0/4201437358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf000e040 con 0x7f0bf810d460 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.908+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf0009d00 con 0x7f0bf810d460 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.908+0000 7f0beea00700 1 -- 172.21.15.23:0/4201437358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bf000be40 con 0x7f0bf810d460 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.908+0000 7f0beea00700 1 -- 172.21.15.23:0/4201437358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00052a0 con 0x7f0bf810d460 2024-03-30T04:12:27.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.908+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf81a5fa0 con 0x7f0bf810d460 2024-03-30T04:12:27.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.911+0000 7f0beea00700 1 -- 172.21.15.23:0/4201437358 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f0bf0003c00 con 0x7f0bf810d460 2024-03-30T04:12:27.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.912+0000 7f0beea00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 0x7f0bdc0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.912+0000 7f0beea00700 1 -- 172.21.15.23:0/4201437358 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0bf00ce580 con 0x7f0bf810d460 2024-03-30T04:12:27.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.912+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 0x7f0bdc0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 Processor -- start 2024-03-30T04:12:27.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 -- start start 2024-03-30T04:12:27.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3969b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a3a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a3a41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a2ef990 con 0x55b26a3965d0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a2ed350 con 0x55b26a3a1df0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a2e8c40 con 0x55b26a396ef0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 0x7f0bdc0a3740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009d80 tx=0x7f0be8009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.913+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a3a41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45732/0 (socket says 172.21.15.23:45732) 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf5e00700 1 -- 172.21.15.23:0/262339025 learned_addr learned my addr 172.21.15.23:0/262339025 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf5400700 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a3a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 -- 172.21.15.23:0/262339025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 msgr2=0x55b26a3a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a3a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 -- 172.21.15.23:0/262339025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 msgr2=0x55b26a3969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 -- 172.21.15.23:0/262339025 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b26a16a3b0 con 0x55b26a3a1df0 2024-03-30T04:12:27.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0bf6800700 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a3a41e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00cb890 tx=0x7f0bf0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f65f7de21529ece server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0be3000700 1 -- 172.21.15.23:0/262339025 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf009b9e0 con 0x55b26a3a1df0 2024-03-30T04:12:27.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0be3000700 1 -- 172.21.15.23:0/262339025 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bf009b2e0 con 0x55b26a3a1df0 2024-03-30T04:12:27.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.914+0000 7f0be3000700 1 -- 172.21.15.23:0/262339025 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf009b540 con 0x55b26a3a1df0 2024-03-30T04:12:27.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 -- 172.21.15.23:0/262339025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 msgr2=0x55b26a3a41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a3a41e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00cb890 tx=0x7f0bf0016000 comp rx=0 tx=0).stop 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 -- 172.21.15.23:0/262339025 shutdown_connections 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a3a41e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/262339025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a3a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 -- 172.21.15.23:0/262339025 >> 172.21.15.23:0/262339025 conn(0x55b26a2f66a0 msgr2=0x55b26a392a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 -- 172.21.15.23:0/262339025 shutdown_connections 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.915+0000 7f0c107cfb80 1 -- 172.21.15.23:0/262339025 wait complete. 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.916+0000 7f0c107cfb80 1 -- start start 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.916+0000 7f0c107cfb80 1 Processor -- start 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a39eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a39f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a284d80 con 0x55b26a3965d0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a284a20 con 0x55b26a3a1df0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a16c6a0 con 0x55b26a396ef0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0c107cfb80 10 client.0 _ll_register_callbacks cb 0x55b26a28e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45744/0 (socket says 172.21.15.23:45744) 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 -- 172.21.15.23:0/616284056 learned_addr learned my addr 172.21.15.23:0/616284056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf6800700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a39f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5400700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a39eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 -- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 msgr2=0x55b26a39eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a39eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 -- 172.21.15.23:0/616284056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 msgr2=0x55b26a39f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a39f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5e00700 1 -- 172.21.15.23:0/616284056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf81a5fa0 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf5400700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a39eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.917+0000 7f0bf6800700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a39f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0be8012c40 tx=0x7f0be80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de916d5b4102e3bd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0bd7400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8020cd0 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0be8018070 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b26a2a9380 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0be8011e90 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0c107cfb80 10 client.18366 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.918+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b26a116b60 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b26a2a9380 con 0x55b26a3965d0 2024-03-30T04:12:27.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0be8028470 con 0x55b26a3965d0 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 10 client.18366 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 10 client.18366 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 20 client.18366 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a3b1680 0x55b26a3b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 4 client.18366 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b26a2e1700 con 0x55b26a3b1680 2024-03-30T04:12:27.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.919+0000 7f0bf5400700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a3b1680 0x55b26a3b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.920+0000 7f0bf5400700 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a3b1680 0x55b26a3b3aa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2cc1a8208903ec1a server_cookie=f60c6622f7dd3da3 in_seq=0 out_seq=0 2024-03-30T04:12:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.920+0000 7f0bd7400700 10 client.18366 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:12:27.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.920+0000 7f0bd7400700 1 -- 172.21.15.23:0/616284056 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b26a2e1700 con 0x55b26a3b1680 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.920+0000 7f0bd7400700 10 client.18366 handle_command_reply: tid=0 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 client.18366 shutdown 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 msgr2=0x55b26a3a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f0be8012c40 tx=0x7f0be80047c0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 shutdown_connections 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a3b1680 0x55b26a3b3aa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3a1df0 0x55b26a39f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a3965d0 0x55b26a3a1750 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/616284056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a396ef0 0x55b26a39eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 >> 172.21.15.23:0/616284056 conn(0x55b26a2f66a0 msgr2=0x55b26a398530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0bd7400700 0 client.18366 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 shutdown_connections 2024-03-30T04:12:27.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 1 -- 172.21.15.23:0/616284056 wait complete. 2024-03-30T04:12:27.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.921+0000 7f0c107cfb80 20 client.18366 trim_cache size 0 max 16384 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 Processor -- start 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 -- start start 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a38b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a3a8d50 con 0x55b26a396ef0 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a39f700 con 0x55b26a3965d0 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.935+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a39f3f0 con 0x55b26a3a1df0 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40114/0 (socket says 172.21.15.23:40114) 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5e00700 1 -- 172.21.15.23:0/303844629 learned_addr learned my addr 172.21.15.23:0/303844629 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf6800700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a38b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 -- 172.21.15.23:0/303844629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 msgr2=0x55b26a38b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a38b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 -- 172.21.15.23:0/303844629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 msgr2=0x55b26a391f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 -- 172.21.15.23:0/303844629 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b26a2aeea0 con 0x55b26a396ef0 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf6800700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a38b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.936+0000 7f0bf5400700 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a87e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x55b26a3b4350 tx=0x7f0be4009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=450da384709f5f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0be3a00700 1 -- 172.21.15.23:0/303844629 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be401e070 con 0x55b26a396ef0 2024-03-30T04:12:27.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0be3a00700 1 -- 172.21.15.23:0/303844629 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0be4017040 con 0x55b26a396ef0 2024-03-30T04:12:27.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0be3a00700 1 -- 172.21.15.23:0/303844629 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be4003f80 con 0x55b26a396ef0 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 -- 172.21.15.23:0/303844629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 msgr2=0x55b26a3a87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a87e0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x55b26a3b4350 tx=0x7f0be4009260 comp rx=0 tx=0).stop 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 -- 172.21.15.23:0/303844629 shutdown_connections 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a38b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a391f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/303844629 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a87e0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 -- 172.21.15.23:0/303844629 >> 172.21.15.23:0/303844629 conn(0x55b26a2f66a0 msgr2=0x55b26a3b8400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 -- 172.21.15.23:0/303844629 shutdown_connections 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.937+0000 7f0c107cfb80 1 -- 172.21.15.23:0/303844629 wait complete. 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.938+0000 7f0c107cfb80 1 -- start start 2024-03-30T04:12:27.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.938+0000 7f0c107cfb80 1 Processor -- start 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a4044b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a3a75c0 con 0x55b26a396ef0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a4049f0 con 0x55b26a3965d0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b26a404b30 con 0x55b26a3a1df0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0c107cfb80 10 client.0 _ll_register_callbacks cb 0x55b26a28e470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:40130/0 (socket says 172.21.15.23:40130) 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 -- 172.21.15.23:0/556506226 learned_addr learned my addr 172.21.15.23:0/556506226 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf6800700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a4044b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 -- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 msgr2=0x55b26a4044b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a4044b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 -- 172.21.15.23:0/556506226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 msgr2=0x55b26a3a6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5e00700 1 -- 172.21.15.23:0/556506226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b26a116b60 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.939+0000 7f0bf5400700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bf5e00700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0be8020d30 tx=0x7f0be8011fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ee6782c9e1fd02 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bd7400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8028df0 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0bf009b2e0 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f0be8018040 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0be8011a50 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0c107cfb80 10 client.28349 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b26a16a3b0 con 0x55b26a3965d0 2024-03-30T04:12:27.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.940+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0bf009b2e0 con 0x55b26a3965d0 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0be803d020 con 0x55b26a3965d0 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 10 client.28349 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 10 client.28349 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 20 client.28349 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a406140 0x55b26a408560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 4 client.28349 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b26a137cf0 con 0x55b26a406140 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.941+0000 7f0bf5400700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a406140 0x55b26a408560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0bf5400700 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a406140 0x55b26a408560 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e37933954671e389 server_cookie=22297bd51a5c70e9 in_seq=0 out_seq=0 2024-03-30T04:12:27.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0bd7400700 10 client.28349 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:12:27.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0bd7400700 1 -- 172.21.15.23:0/556506226 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b26a137cf0 con 0x55b26a406140 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0bd7400700 10 client.28349 handle_command_reply: tid=0 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 client.28349 shutdown 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 msgr2=0x55b26a3a67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f0be8020d30 tx=0x7f0be8011fd0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 shutdown_connections 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55b26a406140 0x55b26a408560 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b26a3a1df0 0x55b26a4044b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b26a3965d0 0x55b26a3a67c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 --2- 172.21.15.23:0/556506226 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b26a396ef0 0x55b26a3a6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 >> 172.21.15.23:0/556506226 conn(0x55b26a2f66a0 msgr2=0x55b26a3924d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0bd7400700 0 client.28349 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 shutdown_connections 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 1 -- 172.21.15.23:0/556506226 wait complete. 2024-03-30T04:12:27.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.942+0000 7f0c107cfb80 20 client.28349 trim_cache size 0 max 16384 2024-03-30T04:12:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:12:27.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 msgr2=0x7f0bdc0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 0x7f0bdc0a3740 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009d80 tx=0x7f0be8009260 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 msgr2=0x7f0bf819dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000b070 tx=0x7f0bf0007970 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 shutdown_connections 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bf810d460 0x7f0bf819dcf0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0bf80731e0 0x7f0bf8199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f0bdc0a1280 0x7f0bdc0a3740 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 --2- 172.21.15.23:0/4201437358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf8073b00 0x7f0bf81999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 >> 172.21.15.23:0/4201437358 conn(0x7f0bf80fc710 msgr2=0x7f0bf80fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 shutdown_connections 2024-03-30T04:12:27.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:27.943+0000 7f0bfcc00700 1 -- 172.21.15.23:0/4201437358 wait complete. 2024-03-30T04:12:27.962 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:12:28.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 -- 172.21.15.23:0/3992196387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154105ce0 msgr2=0x7f31541060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 --2- 172.21.15.23:0/3992196387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154105ce0 0x7f31541060c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f3144009920 tx=0x7f3144009c30 comp rx=0 tx=0).stop 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 -- 172.21.15.23:0/3992196387 shutdown_connections 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 --2- 172.21.15.23:0/3992196387 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31540687f0 0x7f315410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 --2- 172.21.15.23:0/3992196387 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154105ce0 0x7f31541060c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 --2- 172.21.15.23:0/3992196387 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3154067e30 0x7f31540682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.180+0000 7f3159600700 1 -- 172.21.15.23:0/3992196387 >> 172.21.15.23:0/3992196387 conn(0x7f3154076d40 msgr2=0x7f3154077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.181+0000 7f3159600700 1 -- 172.21.15.23:0/3992196387 shutdown_connections 2024-03-30T04:12:28.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.181+0000 7f3159600700 1 -- 172.21.15.23:0/3992196387 wait complete. 2024-03-30T04:12:28.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.181+0000 7f3159600700 1 Processor -- start 2024-03-30T04:12:28.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 -- start start 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31540687f0 0x7f31541001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31541a5660 con 0x7f31540687f0 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31541a57d0 con 0x7f3154105ce0 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3159600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31541a5990 con 0x7f3154067e30 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3152a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3153400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3152a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60418/0 (socket says 172.21.15.23:60418) 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3153400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42774/0 (socket says 172.21.15.23:42774) 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3152000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31540687f0 0x7f31541001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.182+0000 7f3153400700 1 -- 172.21.15.23:0/2047247742 learned_addr learned my addr 172.21.15.23:0/2047247742 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 -- 172.21.15.23:0/2047247742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 msgr2=0x7f31541006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 -- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31540687f0 msgr2=0x7f31541001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31540687f0 0x7f31541001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 -- 172.21.15.23:0/2047247742 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400095f0 con 0x7f3154067e30 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3153400700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3152a00700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3144000c00 tx=0x7f314400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f314b400700 1 -- 172.21.15.23:0/2047247742 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314401c070 con 0x7f3154067e30 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f314b400700 1 -- 172.21.15.23:0/2047247742 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f314400ba20 con 0x7f3154067e30 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f314b400700 1 -- 172.21.15.23:0/2047247742 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31440038e0 con 0x7f3154067e30 2024-03-30T04:12:28.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.183+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3144009520 con 0x7f3154067e30 2024-03-30T04:12:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.184+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31541a5e50 con 0x7f3154067e30 2024-03-30T04:12:28.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.186+0000 7f314b400700 1 -- 172.21.15.23:0/2047247742 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3144020030 con 0x7f3154067e30 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 Processor -- start 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f314b400700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 0x7f31380a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 -- start start 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f314b400700 1 -- 172.21.15.23:0/2047247742 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f31440cddb0 con 0x7f3154067e30 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f3153400700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 0x7f31380a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fbac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbaf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fafa990 con 0x55a98fba15d0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98faf8350 con 0x55a98fba1ef0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.189+0000 7f316d14ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98faf3c40 con 0x55a98fbacdf0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3152a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3152a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38976/0 (socket says 172.21.15.23:38976) 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3152a00700 1 -- 172.21.15.23:0/603657063 learned_addr learned my addr 172.21.15.23:0/603657063 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3152000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fbac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbaf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 0x7f31380a3750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f314c009c10 tx=0x7f314c00b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 -- 172.21.15.23:0/603657063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 msgr2=0x55a98fbac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fbac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 -- 172.21.15.23:0/603657063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 msgr2=0x55a98fba19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 -- 172.21.15.23:0/603657063 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a98f9753b0 con 0x55a98fbacdf0 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3152a00700 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.190+0000 7f3153400700 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbaf1e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f314c0090c0 tx=0x7f314c01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5d0fc425f9e62a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.191+0000 7f313f400700 1 -- 172.21.15.23:0/603657063 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00eee0 con 0x55a98fbacdf0 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.191+0000 7f313f400700 1 -- 172.21.15.23:0/603657063 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f314c00f040 con 0x55a98fbacdf0 2024-03-30T04:12:28.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.191+0000 7f313f400700 1 -- 172.21.15.23:0/603657063 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c015be0 con 0x55a98fbacdf0 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.191+0000 7f316d14ab80 1 -- 172.21.15.23:0/603657063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 msgr2=0x55a98fbaf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.191+0000 7f316d14ab80 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbaf1e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f314c0090c0 tx=0x7f314c01c000 comp rx=0 tx=0).stop 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 -- 172.21.15.23:0/603657063 shutdown_connections 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbaf1e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fba19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 --2- 172.21.15.23:0/603657063 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fbac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 -- 172.21.15.23:0/603657063 >> 172.21.15.23:0/603657063 conn(0x55a98fb016a0 msgr2=0x55a98fb9da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 -- 172.21.15.23:0/603657063 shutdown_connections 2024-03-30T04:12:28.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 -- 172.21.15.23:0/603657063 wait complete. 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 -- start start 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.192+0000 7f316d14ab80 1 Processor -- start 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba15d0 0x55a98fbac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fbacdf0 0x55a98fbaa110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fa8fd80 con 0x55a98fbacdf0 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fa8fa20 con 0x55a98fba1ef0 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98f9776a0 con 0x55a98fba15d0 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f316d14ab80 10 client.0 _ll_register_callbacks cb 0x55a98fa99470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f3152000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f3152000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42796/0 (socket says 172.21.15.23:42796) 2024-03-30T04:12:28.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f3152000700 1 -- 172.21.15.23:0/2254408024 learned_addr learned my addr 172.21.15.23:0/2254408024 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f3153400700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fbacdf0 0x55a98fbaa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.193+0000 7f3152a00700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba15d0 0x55a98fbac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 -- 172.21.15.23:0/2254408024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba15d0 msgr2=0x55a98fbac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba15d0 0x55a98fbac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 -- 172.21.15.23:0/2254408024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fbacdf0 msgr2=0x55a98fbaa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fbacdf0 0x55a98fbaa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 -- 172.21.15.23:0/2254408024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31541a5e50 con 0x55a98fba1ef0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3152000700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3140012a10 tx=0x7f3140012d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df82fb9c22a1467f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3133e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3140012ea0 con 0x55a98fba1ef0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a98fab4380 con 0x55a98fba1ef0 2024-03-30T04:12:28.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f314000f830 con 0x55a98fba1ef0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3140017070 con 0x55a98fba1ef0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f316d14ab80 10 client.28361 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.194+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f314c00e4f0 con 0x55a98fba1ef0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a98fab4380 con 0x55a98fba1ef0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3140010980 con 0x55a98fba1ef0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 10 client.28361 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 10 client.28361 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 20 client.28361 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fbbca70 0x55a98fbbee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 4 client.28361 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a98faec700 con 0x55a98fbbca70 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.195+0000 7f3152a00700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fbbca70 0x55a98fbbee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.196+0000 7f3133e00700 10 client.28361 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.196+0000 7f3152a00700 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fbbca70 0x55a98fbbee90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6242a5ac820cd666 server_cookie=7c6935f7cd09498c in_seq=0 out_seq=0 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.196+0000 7f3133e00700 1 -- 172.21.15.23:0/2254408024 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a98faec700 con 0x55a98fbbca70 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.196+0000 7f3133e00700 10 client.28361 handle_command_reply: tid=0 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.196+0000 7f316d14ab80 1 client.28361 shutdown 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 msgr2=0x55a98fba9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3140012a10 tx=0x7f3140012d20 comp rx=0 tx=0).stop 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 shutdown_connections 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fbbca70 0x55a98fbbee90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fbacdf0 0x55a98fbaa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba15d0 0x55a98fbac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2254408024 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba1ef0 0x55a98fba9ae0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f3133e00700 0 client.28361 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:12:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 >> 172.21.15.23:0/2254408024 conn(0x55a98fb016a0 msgr2=0x55a98fba36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 shutdown_connections 2024-03-30T04:12:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 1 -- 172.21.15.23:0/2254408024 wait complete. 2024-03-30T04:12:28.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.197+0000 7f316d14ab80 20 client.28361 trim_cache size 0 max 16384 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 Processor -- start 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 -- start start 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fbbee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 0x55a98fbaa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbaaa80 con 0x55a98fba15d0 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbb3fb0 con 0x55a98fbacdf0 2024-03-30T04:12:28.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f316d14ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbaf1e0 con 0x55a98fba1ef0 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f3152a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fbbee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f3153400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f3153400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42804/0 (socket says 172.21.15.23:42804) 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.211+0000 7f3153400700 1 -- 172.21.15.23:0/2194577240 learned_addr learned my addr 172.21.15.23:0/2194577240 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3152000700 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 0x55a98fbaa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 -- 172.21.15.23:0/2194577240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 msgr2=0x55a98fbaa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 0x55a98fbaa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 -- 172.21.15.23:0/2194577240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 msgr2=0x55a98fbbee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fbbee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 -- 172.21.15.23:0/2194577240 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a98f921b60 con 0x55a98fbacdf0 2024-03-30T04:12:28.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3152000700 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 0x55a98fbaa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f3153400700 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f314c015710 tx=0x7f314c0156a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd05432694e5d17a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f313fe00700 1 -- 172.21.15.23:0/2194577240 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c00eee0 con 0x55a98fbacdf0 2024-03-30T04:12:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f313fe00700 1 -- 172.21.15.23:0/2194577240 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f314c015e70 con 0x55a98fbacdf0 2024-03-30T04:12:28.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.212+0000 7f313fe00700 1 -- 172.21.15.23:0/2194577240 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c009420 con 0x55a98fbacdf0 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 -- 172.21.15.23:0/2194577240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 msgr2=0x55a98fbb3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f314c015710 tx=0x7f314c0156a0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 -- 172.21.15.23:0/2194577240 shutdown_connections 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fbacdf0 0x55a98fbb3a70 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba15d0 0x55a98fbbee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 --2- 172.21.15.23:0/2194577240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fba1ef0 0x55a98fbaa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 -- 172.21.15.23:0/2194577240 >> 172.21.15.23:0/2194577240 conn(0x55a98fb016a0 msgr2=0x55a98fb965a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 -- 172.21.15.23:0/2194577240 shutdown_connections 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.213+0000 7f316d14ab80 1 -- 172.21.15.23:0/2194577240 wait complete. 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 -- start start 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 Processor -- start 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 0x55a98fbb36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 0x55a98fbb1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbb1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbb2510 con 0x55a98fba1ef0 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbb2680 con 0x55a98fba15d0 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98fbb2840 con 0x55a98fbacdf0 2024-03-30T04:12:28.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.214+0000 7f316d14ab80 10 client.0 _ll_register_callbacks cb 0x55a98fa99470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 0x55a98fbb36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 0x55a98fbb1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 0x55a98fbb36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42812/0 (socket says 172.21.15.23:42812) 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152a00700 1 -- 172.21.15.23:0/3115815184 learned_addr learned my addr 172.21.15.23:0/3115815184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3153400700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbb1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 -- 172.21.15.23:0/3115815184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 msgr2=0x55a98fbb1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbb1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 -- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 msgr2=0x55a98fbb36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 0x55a98fbb36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 -- 172.21.15.23:0/3115815184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a98f9753b0 con 0x55a98fba1ef0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3153400700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbb1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3152000700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 0x55a98fbb1a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3140010d30 tx=0x7f314001c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e955bc5cd22f89b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3133e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.215+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314002e070 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3140000e40 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f314c00f040 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3140017070 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 10 client.18390 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55a98fab9ea0 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f314c00f040 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3140031030 con 0x55a98fba1ef0 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 10 client.18390 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 10 client.18390 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 20 client.18390 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fc11cc0 0x55a98fc140b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 4 client.18390 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.216+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a98f942cf0 con 0x55a98fc11cc0 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.217+0000 7f3152a00700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fc11cc0 0x55a98fc140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.217+0000 7f3133e00700 10 client.18390 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.217+0000 7f3152a00700 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fc11cc0 0x55a98fc140b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45b8f3eb4c49110b server_cookie=4985fe3c51d5835b in_seq=0 out_seq=0 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f3133e00700 1 -- 172.21.15.23:0/3115815184 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a98f942cf0 con 0x55a98fc11cc0 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f3133e00700 10 client.18390 handle_command_reply: tid=0 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 client.18390 shutdown 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 msgr2=0x55a98fbb1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 0x55a98fbb1a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3140010d30 tx=0x7f314001c6b0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 shutdown_connections 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55a98fc11cc0 0x55a98fc140b0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a98fbacdf0 0x55a98fbb1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55a98fba15d0 0x55a98fbb36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 --2- 172.21.15.23:0/3115815184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a98fba1ef0 0x55a98fbb1a90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f3133e00700 0 client.18390 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 >> 172.21.15.23:0/3115815184 conn(0x55a98fb016a0 msgr2=0x55a98fbac450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 shutdown_connections 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 1 -- 172.21.15.23:0/3115815184 wait complete. 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.218+0000 7f316d14ab80 20 client.18390 trim_cache size 0 max 16384 2024-03-30T04:12:28.222 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:12:28.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 msgr2=0x7f31380a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 0x7f31380a3750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f314c009c10 tx=0x7f314c00b040 comp rx=0 tx=0).stop 2024-03-30T04:12:28.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 msgr2=0x7f31540ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f3144000c00 tx=0x7f314400fa00 comp rx=0 tx=0).stop 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 shutdown_connections 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31380a1290 0x7f31380a3750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3154105ce0 0x7f31541006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3154067e30 0x7f31540ffc70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 --2- 172.21.15.23:0/2047247742 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31540687f0 0x7f31541001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 >> 172.21.15.23:0/2047247742 conn(0x7f3154076d40 msgr2=0x7f31540fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 shutdown_connections 2024-03-30T04:12:28.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.219+0000 7f3159600700 1 -- 172.21.15.23:0/2047247742 wait complete. 2024-03-30T04:12:28.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1041427 10.03 MB/sec execute 425 sec latency 38.120 ms 2024-03-30T04:12:28.234 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:28 smithi023 ceph-mon[38254]: pgmap v328: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-03-30T04:12:28.235 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:28 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3538307933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:12:28.237 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:12:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:28 smithi154 ceph-mon[79161]: pgmap v328: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-03-30T04:12:28.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:28 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3538307933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 -- 172.21.15.23:0/3511085399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9801060b0 msgr2=0x7fc98010dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 --2- 172.21.15.23:0/3511085399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9801060b0 0x7fc98010dc30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc97800b0a0 tx=0x7fc97800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 -- 172.21.15.23:0/3511085399 shutdown_connections 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 --2- 172.21.15.23:0/3511085399 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9801060b0 0x7fc98010dc30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 --2- 172.21.15.23:0/3511085399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc980069260 0x7fc980069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 --2- 172.21.15.23:0/3511085399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc980069c10 0x7fc980105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 -- 172.21.15.23:0/3511085399 >> 172.21.15.23:0/3511085399 conn(0x7fc980076d80 msgr2=0x7fc980077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 -- 172.21.15.23:0/3511085399 shutdown_connections 2024-03-30T04:12:28.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.459+0000 7fc987000700 1 -- 172.21.15.23:0/3511085399 wait complete. 2024-03-30T04:12:28.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.460+0000 7fc987000700 1 Processor -- start 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.460+0000 7fc987000700 1 -- start start 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9801060b0 0x7fc9801999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98019dd60 0x7fc98019e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019e7e0 con 0x7fc9801060b0 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019e950 con 0x7fc980069260 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc987000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc98019eb10 con 0x7fc98019dd60 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42816/0 (socket says 172.21.15.23:42816) 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 -- 172.21.15.23:0/3466174414 learned_addr learned my addr 172.21.15.23:0/3466174414 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97f400700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9801060b0 0x7fc9801999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc985000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98019dd60 0x7fc98019e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 -- 172.21.15.23:0/3466174414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98019dd60 msgr2=0x7fc98019e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98019dd60 0x7fc98019e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 -- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9801060b0 msgr2=0x7fc9801999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9801060b0 0x7fc9801999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.461+0000 7fc97fe00700 1 -- 172.21.15.23:0/3466174414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978009d00 con 0x7fc980069260 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9a0 tx=0x7fc97000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc97cc00700 1 -- 172.21.15.23:0/3466174414 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970009790 con 0x7fc980069260 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc97cc00700 1 -- 172.21.15.23:0/3466174414 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc970010430 con 0x7fc980069260 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc97cc00700 1 -- 172.21.15.23:0/3466174414 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970018680 con 0x7fc980069260 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9801a6070 con 0x7fc980069260 2024-03-30T04:12:28.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.462+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9801a6550 con 0x7fc980069260 2024-03-30T04:12:28.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.466+0000 7fc97cc00700 1 -- 172.21.15.23:0/3466174414 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc9700105a0 con 0x7fc980069260 2024-03-30T04:12:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.467+0000 7fc97cc00700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 0x7fc9640a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.467+0000 7fc99abe0b80 1 Processor -- start 2024-03-30T04:12:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.467+0000 7fc97cc00700 1 -- 172.21.15.23:0/3466174414 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc970014070 con 0x7fc980069260 2024-03-30T04:12:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc97f400700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 0x7fc9640a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 -- start start 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f535d0 0x55d187f539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 0x55d187f5e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f611e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187eac990 con 0x55d187f535d0 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187eaa350 con 0x55d187f53ef0 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187ea5c40 con 0x55d187f5edf0 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc97f400700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 0x7fc9640a37d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc98019a9d0 tx=0x7fc96c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc97f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 0x55d187f5e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc97f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 0x55d187f5e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42824/0 (socket says 172.21.15.23:42824) 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc97f400700 1 -- 172.21.15.23:0/499275702 learned_addr learned my addr 172.21.15.23:0/499275702 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.468+0000 7fc985000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc97fe00700 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f535d0 0x55d187f539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 -- 172.21.15.23:0/499275702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 msgr2=0x55d187f5e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 0x55d187f5e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 -- 172.21.15.23:0/499275702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f535d0 msgr2=0x55d187f539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f535d0 0x55d187f539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 -- 172.21.15.23:0/499275702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d187d273b0 con 0x55d187f5edf0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc985000700 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f611e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc97800b070 tx=0x7fc978009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343f388ff438c284 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc975600700 1 -- 172.21.15.23:0/499275702 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97800e040 con 0x55d187f5edf0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc975600700 1 -- 172.21.15.23:0/499275702 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc978009460 con 0x55d187f5edf0 2024-03-30T04:12:28.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.469+0000 7fc975600700 1 -- 172.21.15.23:0/499275702 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978003ea0 con 0x55d187f5edf0 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 -- 172.21.15.23:0/499275702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 msgr2=0x55d187f611e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f611e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc97800b070 tx=0x7fc978009c60 comp rx=0 tx=0).stop 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 -- 172.21.15.23:0/499275702 shutdown_connections 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f611e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f535d0 0x55d187f539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/499275702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f53ef0 0x55d187f5e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 -- 172.21.15.23:0/499275702 >> 172.21.15.23:0/499275702 conn(0x55d187eb36a0 msgr2=0x55d187f4fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 -- 172.21.15.23:0/499275702 shutdown_connections 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.471+0000 7fc99abe0b80 1 -- 172.21.15.23:0/499275702 wait complete. 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.472+0000 7fc99abe0b80 1 -- start start 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.472+0000 7fc99abe0b80 1 Processor -- start 2024-03-30T04:12:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f5bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f5c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187e41d80 con 0x55d187f53ef0 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187e41a20 con 0x55d187f535d0 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187d296a0 con 0x55d187f5edf0 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc99abe0b80 10 client.0 _ll_register_callbacks cb 0x55d187e4b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42828/0 (socket says 172.21.15.23:42828) 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc97fe00700 1 -- 172.21.15.23:0/3974975499 learned_addr learned my addr 172.21.15.23:0/3974975499 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.473+0000 7fc985000700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f5c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97f400700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f5bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97fe00700 1 -- 172.21.15.23:0/3974975499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 msgr2=0x55d187f5c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f5c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97fe00700 1 -- 172.21.15.23:0/3974975499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 msgr2=0x55d187f5bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f5bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97fe00700 1 -- 172.21.15.23:0/3974975499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc98010e470 con 0x55d187f535d0 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc985000700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.474+0000 7fc97f400700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f5bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc970096430 tx=0x7fc97009a630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77c3661b9bbe603 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc96a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97009dd60 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc9700a23b0 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d187e66380 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.475+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc970099420 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 10 client.28379 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d187cd3b60 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9700b8020 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc9700a2040 con 0x55d187f535d0 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 10 client.28379 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 10 client.28379 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 20 client.28379 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187f6e910 0x55d187f70d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 4 client.28379 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.476+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d187e9e700 con 0x55d187f6e910 2024-03-30T04:12:28.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.477+0000 7fc97f400700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187f6e910 0x55d187f70d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.478+0000 7fc97f400700 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187f6e910 0x55d187f70d30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf3b1b46060dc73a server_cookie=70c68e76344d4c08 in_seq=0 out_seq=0 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.478+0000 7fc96a000700 10 client.28379 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.478+0000 7fc96a000700 1 -- 172.21.15.23:0/3974975499 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d187e9e700 con 0x55d187f6e910 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.478+0000 7fc96a000700 10 client.28379 handle_command_reply: tid=0 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 client.28379 shutdown 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 msgr2=0x55d187f5e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc970096430 tx=0x7fc97009a630 comp rx=0 tx=0).stop 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 shutdown_connections 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187f6e910 0x55d187f70d30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187f5c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f5e730 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3974975499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f5bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc96a000700 0 client.28379 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 >> 172.21.15.23:0/3974975499 conn(0x55d187eb36a0 msgr2=0x55d187f555d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 shutdown_connections 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3974975499 wait complete. 2024-03-30T04:12:28.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.479+0000 7fc99abe0b80 20 client.28379 trim_cache size 0 max 16384 2024-03-30T04:12:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:28 smithi193 ceph-mon[79581]: pgmap v328: 129 pgs: 129 active+clean; 561 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.5 MiB/s wr, 156 op/s 2024-03-30T04:12:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:28 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3538307933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:12:28.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 Processor -- start 2024-03-30T04:12:28.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 -- start start 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 0x55d187f5a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187f5c620 con 0x55d187f5edf0 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187f54640 con 0x55d187f535d0 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.499+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187f5d050 con 0x55d187f53ef0 2024-03-30T04:12:28.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc985000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 0x55d187f5a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42834/0 (socket says 172.21.15.23:42834) 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60490/0 (socket says 172.21.15.23:60490) 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 -- 172.21.15.23:0/3472130365 learned_addr learned my addr 172.21.15.23:0/3472130365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 -- 172.21.15.23:0/3472130365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 msgr2=0x55d187f5a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 -- 172.21.15.23:0/3472130365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 msgr2=0x55d187f5a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 0x55d187f5a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 -- 172.21.15.23:0/3472130365 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d187e6bea0 con 0x55d187f535d0 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97f400700 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc985000700 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 0x55d187f5a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.500+0000 7fc97fe00700 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc970000c00 tx=0x7fc970010a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f4983122952bc98 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc976000700 1 -- 172.21.15.23:0/3472130365 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9700a23b0 con 0x55d187f535d0 2024-03-30T04:12:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc976000700 1 -- 172.21.15.23:0/3472130365 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9700b8040 con 0x55d187f535d0 2024-03-30T04:12:28.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc976000700 1 -- 172.21.15.23:0/3472130365 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970097620 con 0x55d187f535d0 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3472130365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 msgr2=0x55d187f70d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fc970000c00 tx=0x7fc970010a90 comp rx=0 tx=0).stop 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3472130365 shutdown_connections 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f5edf0 0x55d187f5a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f70d00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/3472130365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f53ef0 0x55d187f5a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3472130365 >> 172.21.15.23:0/3472130365 conn(0x55d187eb36a0 msgr2=0x55d187f485a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.501+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3472130365 shutdown_connections 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.502+0000 7fc99abe0b80 1 -- 172.21.15.23:0/3472130365 wait complete. 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.502+0000 7fc99abe0b80 1 -- start start 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.502+0000 7fc99abe0b80 1 Processor -- start 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f64120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187fbb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187fbbce0 con 0x55d187f53ef0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187fbbe50 con 0x55d187f535d0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d187fbc010 con 0x55d187f5edf0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc99abe0b80 10 client.0 _ll_register_callbacks cb 0x55d187e4b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f64120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42836/0 (socket says 172.21.15.23:42836) 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 -- 172.21.15.23:0/2274825494 learned_addr learned my addr 172.21.15.23:0/2274825494 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc985000700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187fbb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 -- 172.21.15.23:0/2274825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 msgr2=0x55d187fbb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187fbb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 -- 172.21.15.23:0/2274825494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 msgr2=0x55d187f64120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f64120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97fe00700 1 -- 172.21.15.23:0/2274825494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c017070 con 0x55d187f535d0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.503+0000 7fc97f400700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f64120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc97fe00700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc970099c30 tx=0x7fc970099e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b858566afdc48636 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc970095d20 con 0x55d187f535d0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc978009460 con 0x55d187f535d0 2024-03-30T04:12:28.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc9700a23b0 con 0x55d187f535d0 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc97009dd60 con 0x55d187f535d0 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc99abe0b80 10 client.28385 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d187d273b0 con 0x55d187f535d0 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc978009460 con 0x55d187f535d0 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.504+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc97007f070 con 0x55d187f535d0 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 10 client.28385 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 10 client.28385 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 20 client.28385 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187fc3330 0x55d187fc5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 4 client.28385 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d187cf4cf0 con 0x55d187fc3330 2024-03-30T04:12:28.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc97f400700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187fc3330 0x55d187fc5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc96a000700 10 client.28385 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.505+0000 7fc97f400700 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187fc3330 0x55d187fc5720 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=de36a8f93447f90c server_cookie=5d61f92620d21f04 in_seq=0 out_seq=0 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc96a000700 1 -- 172.21.15.23:0/2274825494 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d187cf4cf0 con 0x55d187fc3330 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc96a000700 10 client.28385 handle_command_reply: tid=0 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 client.28385 shutdown 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 msgr2=0x55d187f63be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc970099c30 tx=0x7fc970099e30 comp rx=0 tx=0).stop 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 shutdown_connections 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55d187fc3330 0x55d187fc5720 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d187f5edf0 0x55d187fbb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d187f535d0 0x55d187f63be0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 --2- 172.21.15.23:0/2274825494 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d187f53ef0 0x55d187f64120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc96a000700 0 client.28385 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:12:28.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 >> 172.21.15.23:0/2274825494 conn(0x55d187eb36a0 msgr2=0x55d187f5a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 shutdown_connections 2024-03-30T04:12:28.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 1 -- 172.21.15.23:0/2274825494 wait complete. 2024-03-30T04:12:28.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.506+0000 7fc99abe0b80 20 client.28385 trim_cache size 0 max 16384 2024-03-30T04:12:28.510 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 msgr2=0x7fc9640a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 0x7fc9640a37d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc98019a9d0 tx=0x7fc96c009260 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 msgr2=0x7fc980199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fc97000e9a0 tx=0x7fc97000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 shutdown_connections 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98019dd60 0x7fc98019e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9640a1310 0x7fc9640a37d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc980069260 0x7fc980199480 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 --2- 172.21.15.23:0/3466174414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9801060b0 0x7fc9801999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 >> 172.21.15.23:0/3466174414 conn(0x7fc980076d80 msgr2=0x7fc9800ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 shutdown_connections 2024-03-30T04:12:28.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.507+0000 7fc987000700 1 -- 172.21.15.23:0/3466174414 wait complete. 2024-03-30T04:12:28.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 -- 172.21.15.23:0/3148070812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060101840 msgr2=0x7fe060105880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 --2- 172.21.15.23:0/3148070812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060101840 0x7fe060105880 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0098f0 tx=0x7fe04c009c00 comp rx=0 tx=0).stop 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 -- 172.21.15.23:0/3148070812 shutdown_connections 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 --2- 172.21.15.23:0/3148070812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060105fb0 0x7fe06010db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 --2- 172.21.15.23:0/3148070812 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe060100e90 0x7fe060101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 --2- 172.21.15.23:0/3148070812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060101840 0x7fe060105880 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 -- 172.21.15.23:0/3148070812 >> 172.21.15.23:0/3148070812 conn(0x7fe0600fc710 msgr2=0x7fe0600feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 -- 172.21.15.23:0/3148070812 shutdown_connections 2024-03-30T04:12:28.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.774+0000 7fe064e00700 1 -- 172.21.15.23:0/3148070812 wait complete. 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.775+0000 7fe064e00700 1 Processor -- start 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.775+0000 7fe064e00700 1 -- start start 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 0x7fe0601998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 0x7fe06019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06019e6f0 con 0x7fe06019dc70 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06019e860 con 0x7fe060100e90 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe064e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06019ea20 con 0x7fe060105fb0 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42854/0 (socket says 172.21.15.23:42854) 2024-03-30T04:12:28.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 -- 172.21.15.23:0/1943048830 learned_addr learned my addr 172.21.15.23:0/1943048830 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05d400700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 0x7fe0601998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05e800700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 0x7fe06019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 -- 172.21.15.23:0/1943048830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 msgr2=0x7fe0601998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 0x7fe0601998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 -- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 msgr2=0x7fe06019e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 0x7fe06019e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05de00700 1 -- 172.21.15.23:0/1943048830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c009520 con 0x7fe060100e90 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05d400700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 0x7fe0601998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.776+0000 7fe05e800700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 0x7fe06019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe05de00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe05000e9c0 tx=0x7fe05000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe056a00700 1 -- 172.21.15.23:0/1943048830 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05000cbd0 con 0x7fe060100e90 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe056a00700 1 -- 172.21.15.23:0/1943048830 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe05000eeb0 con 0x7fe060100e90 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0601a5f80 con 0x7fe060100e90 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe056a00700 1 -- 172.21.15.23:0/1943048830 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050005580 con 0x7fe060100e90 2024-03-30T04:12:28.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.777+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0601a63a0 con 0x7fe060100e90 2024-03-30T04:12:28.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.780+0000 7fe056a00700 1 -- 172.21.15.23:0/1943048830 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe050004750 con 0x7fe060100e90 2024-03-30T04:12:28.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.781+0000 7fe07892eb80 1 Processor -- start 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 -- start start 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe056a00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 0x7fe0440a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe056a00700 1 -- 172.21.15.23:0/1943048830 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe050014070 con 0x7fe060100e90 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715bf2df0 0x556715bf51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715b40990 con 0x556715be75d0 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715b3e350 con 0x556715bf2df0 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe07892eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715b39c40 con 0x556715be7ef0 2024-03-30T04:12:28.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe05e800700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 0x7fe0440a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe05de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe05e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715bf2df0 0x556715bf51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe05de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39056/0 (socket says 172.21.15.23:39056) 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.782+0000 7fe05de00700 1 -- 172.21.15.23:0/3617298136 learned_addr learned my addr 172.21.15.23:0/3617298136 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05d400700 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 -- 172.21.15.23:0/3617298136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 msgr2=0x556715bf28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 -- 172.21.15.23:0/3617298136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715bf2df0 msgr2=0x556715bf51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715bf2df0 0x556715bf51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 -- 172.21.15.23:0/3617298136 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567159bb3b0 con 0x556715be75d0 2024-03-30T04:12:28.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05de00700 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe0500a2030 tx=0x7fe05000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21848b6e83aa7f1f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe05e800700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 0x7fe0440a3740 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0580094f0 tx=0x7fe058009240 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe04b400700 1 -- 172.21.15.23:0/3617298136 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05009a0e0 con 0x556715be75d0 2024-03-30T04:12:28.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe04b400700 1 -- 172.21.15.23:0/3617298136 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe05009df10 con 0x556715be75d0 2024-03-30T04:12:28.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.783+0000 7fe04b400700 1 -- 172.21.15.23:0/3617298136 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05009a340 con 0x556715be75d0 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 -- 172.21.15.23:0/3617298136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 msgr2=0x556715be79b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe0500a2030 tx=0x7fe05000cdb0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 -- 172.21.15.23:0/3617298136 shutdown_connections 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715bf2df0 0x556715bf51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715be79b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 --2- 172.21.15.23:0/3617298136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 -- 172.21.15.23:0/3617298136 >> 172.21.15.23:0/3617298136 conn(0x556715b476a0 msgr2=0x556715be3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 -- 172.21.15.23:0/3617298136 shutdown_connections 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.784+0000 7fe07892eb80 1 -- 172.21.15.23:0/3617298136 wait complete. 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.785+0000 7fe07892eb80 1 -- start start 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.785+0000 7fe07892eb80 1 Processor -- start 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.785+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.785+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715befae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.785+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe07892eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715ad5d80 con 0x556715bf2df0 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe07892eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715ad5a20 con 0x556715be75d0 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe07892eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567159bd6a0 con 0x556715be7ef0 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe07892eb80 10 client.0 _ll_register_callbacks cb 0x556715adf470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715befae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42870/0 (socket says 172.21.15.23:42870) 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 -- 172.21.15.23:0/2608349688 learned_addr learned my addr 172.21.15.23:0/2608349688 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05e800700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 -- 172.21.15.23:0/2608349688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 msgr2=0x556715befae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715befae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 -- 172.21.15.23:0/2608349688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 msgr2=0x556715bf0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05de00700 1 -- 172.21.15.23:0/2608349688 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0601a63a0 con 0x556715be75d0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.786+0000 7fe05d400700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715befae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe05de00700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe05000cec0 tx=0x7fe050096aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3dac1898e6a3fa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe03fe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05009a0e0 con 0x556715be75d0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe05009d030 con 0x556715be75d0 2024-03-30T04:12:28.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556715afa380 con 0x556715be75d0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0500997e0 con 0x556715be75d0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe07892eb80 10 client.28397 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.787+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556715967b60 con 0x556715be75d0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556715afa380 con 0x556715be75d0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe050061d20 con 0x556715be75d0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 10 client.28397 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 10 client.28397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 20 client.28397 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c02520 0x556715c04910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 4 client.28397 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556715b32700 con 0x556715c02520 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.788+0000 7fe05d400700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c02520 0x556715c04910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.789+0000 7fe05d400700 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c02520 0x556715c04910 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b811a0f23a7e542 server_cookie=4aacc1067cb77293 in_seq=0 out_seq=0 2024-03-30T04:12:28.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.789+0000 7fe03fe00700 10 client.28397 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.789+0000 7fe03fe00700 1 -- 172.21.15.23:0/2608349688 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556715b32700 con 0x556715c02520 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.789+0000 7fe03fe00700 10 client.28397 handle_command_reply: tid=0 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.789+0000 7fe07892eb80 1 client.28397 shutdown 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 msgr2=0x556715bf27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe05000cec0 tx=0x7fe050096aa0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 shutdown_connections 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf27f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c02520 0x556715c04910 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 --2- 172.21.15.23:0/2608349688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715befae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 >> 172.21.15.23:0/2608349688 conn(0x556715b476a0 msgr2=0x556715be96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe03fe00700 0 client.28397 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:28.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 shutdown_connections 2024-03-30T04:12:28.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 1 -- 172.21.15.23:0/2608349688 wait complete. 2024-03-30T04:12:28.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.790+0000 7fe07892eb80 20 client.28397 trim_cache size 0 max 16384 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.805+0000 7fe07892eb80 1 Processor -- start 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.805+0000 7fe07892eb80 1 -- start start 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 0x556715bf9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715bf2df0 0x556715beae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715bea570 con 0x556715be75d0 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715bf0a70 con 0x556715be7ef0 2024-03-30T04:12:28.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe07892eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715bf9fb0 con 0x556715bf2df0 2024-03-30T04:12:28.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05d400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 0x556715bf9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39072/0 (socket says 172.21.15.23:39072) 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 -- 172.21.15.23:0/891461358 learned_addr learned my addr 172.21.15.23:0/891461358 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05e800700 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715bf2df0 0x556715beae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 -- 172.21.15.23:0/891461358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715bf2df0 msgr2=0x556715beae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715bf2df0 0x556715beae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 -- 172.21.15.23:0/891461358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 msgr2=0x556715bf9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 0x556715bf9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05de00700 1 -- 172.21.15.23:0/891461358 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556715affea0 con 0x556715be75d0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.806+0000 7fe05d400700 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 0x556715bf9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe05de00700 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe050096b50 tx=0x7fe050096550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7e20b1a7319cde server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe04be00700 1 -- 172.21.15.23:0/891461358 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500a21a0 con 0x556715be75d0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe04be00700 1 -- 172.21.15.23:0/891461358 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe05009a0e0 con 0x556715be75d0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe04be00700 1 -- 172.21.15.23:0/891461358 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050099870 con 0x556715be75d0 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe07892eb80 1 -- 172.21.15.23:0/891461358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 msgr2=0x556715c04910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.807+0000 7fe07892eb80 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe050096b50 tx=0x7fe050096550 comp rx=0 tx=0).stop 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 -- 172.21.15.23:0/891461358 shutdown_connections 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715bf2df0 0x556715beae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715be75d0 0x556715c04910 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 --2- 172.21.15.23:0/891461358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be7ef0 0x556715bf9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 -- 172.21.15.23:0/891461358 >> 172.21.15.23:0/891461358 conn(0x556715b476a0 msgr2=0x556715bdc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 -- 172.21.15.23:0/891461358 shutdown_connections 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 -- 172.21.15.23:0/891461358 wait complete. 2024-03-30T04:12:28.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.808+0000 7fe07892eb80 1 -- start start 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 Processor -- start 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715c55100 con 0x556715bf2df0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715c55270 con 0x556715be75d0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556715c55430 con 0x556715be7ef0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe07892eb80 10 client.0 _ll_register_callbacks cb 0x556715adf470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.809+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05e800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42898/0 (socket says 172.21.15.23:42898) 2024-03-30T04:12:28.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 -- 172.21.15.23:0/4266423943 learned_addr learned my addr 172.21.15.23:0/4266423943 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 -- 172.21.15.23:0/4266423943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 msgr2=0x556715bf7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 -- 172.21.15.23:0/4266423943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 msgr2=0x556715bf84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 -- 172.21.15.23:0/4266423943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06010e3a0 con 0x556715be75d0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05d400700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe05de00700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0500965e0 tx=0x7fe050099c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a157170633f76c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe03fe00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05009a340 con 0x556715be75d0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe05009d070 con 0x556715be75d0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.810+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe05009df10 con 0x556715be75d0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe05009a4a0 con 0x556715be75d0 2024-03-30T04:12:28.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 10 client.28403 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fe0601a63a0 con 0x556715be75d0 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe05009df10 con 0x556715be75d0 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe0500992b0 con 0x556715be75d0 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 10 client.28403 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 10 client.28403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 20 client.28403 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c56720 0x556715c58b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 4 client.28403 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.811+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556715988cf0 con 0x556715c56720 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe05d400700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c56720 0x556715c58b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe03fe00700 10 client.28403 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe05d400700 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c56720 0x556715c58b10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a65ec6f4973c376 server_cookie=83a18ec9b8c6b54f in_seq=0 out_seq=0 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe03fe00700 1 -- 172.21.15.23:0/4266423943 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556715988cf0 con 0x556715c56720 2024-03-30T04:12:28.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe03fe00700 10 client.28403 handle_command_reply: tid=0 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.812+0000 7fe07892eb80 1 client.28403 shutdown 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 msgr2=0x556715bf7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe0500965e0 tx=0x7fe050099c20 comp rx=0 tx=0).stop 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 shutdown_connections 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556715bf2df0 0x556715bf84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556715be75d0 0x556715bf7a50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556715c56720 0x556715c58b10 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 --2- 172.21.15.23:0/4266423943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556715be7ef0 0x556715bf7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe03fe00700 0 client.28403 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 >> 172.21.15.23:0/4266423943 conn(0x556715b476a0 msgr2=0x556715beae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 shutdown_connections 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 1 -- 172.21.15.23:0/4266423943 wait complete. 2024-03-30T04:12:28.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.813+0000 7fe07892eb80 20 client.28403 trim_cache size 0 max 16384 2024-03-30T04:12:28.816 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:12:28.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 msgr2=0x7fe0440a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 0x7fe0440a3740 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0580094f0 tx=0x7fe058009240 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 msgr2=0x7fe060199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe05000e9c0 tx=0x7fe05000ecd0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 shutdown_connections 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe06019dc70 0x7fe06019e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0440a1280 0x7fe0440a3740 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe060100e90 0x7fe060199390 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 --2- 172.21.15.23:0/1943048830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe060105fb0 0x7fe0601998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 >> 172.21.15.23:0/1943048830 conn(0x7fe0600fc710 msgr2=0x7fe0600fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 shutdown_connections 2024-03-30T04:12:28.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:28.814+0000 7fe064e00700 1 -- 172.21.15.23:0/1943048830 wait complete. 2024-03-30T04:12:29.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1042966 10.02 MB/sec execute 426 sec latency 25.607 ms 2024-03-30T04:12:29.831 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:12:29.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 -- 172.21.15.23:0/3296317535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8100f60 msgr2=0x7fd4a8101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3296317535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8100f60 0x7fd4a8101340 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd49c009990 tx=0x7fd49c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 -- 172.21.15.23:0/3296317535 shutdown_connections 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3296317535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8106080 0x7fd4a810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3296317535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8100f60 0x7fd4a8101340 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3296317535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a8101910 0x7fd4a8105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 -- 172.21.15.23:0/3296317535 >> 172.21.15.23:0/3296317535 conn(0x7fd4a80fc7c0 msgr2=0x7fd4a80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.046+0000 7fd4b0600700 1 -- 172.21.15.23:0/3296317535 shutdown_connections 2024-03-30T04:12:30.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.047+0000 7fd4b0600700 1 -- 172.21.15.23:0/3296317535 wait complete. 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.047+0000 7fd4b0600700 1 Processor -- start 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 -- start start 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 0x7fd4a8197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a819bb20 0x7fd4a819bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819c5a0 con 0x7fd4a8106080 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819c710 con 0x7fd4a819bb20 2024-03-30T04:12:30.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4b0600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819c8d0 con 0x7fd4a8100f60 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 0x7fd4a8197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39100/0 (socket says 172.21.15.23:39100) 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4ad200700 1 -- 172.21.15.23:0/3820704506 learned_addr learned my addr 172.21.15.23:0/3820704506 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.048+0000 7fd4ae600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a819bb20 0x7fd4a819bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 -- 172.21.15.23:0/3820704506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a819bb20 msgr2=0x7fd4a819bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a819bb20 0x7fd4a819bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 -- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 msgr2=0x7fd4a81977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 -- 172.21.15.23:0/3820704506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494009520 con 0x7fd4a8100f60 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 0x7fd4a8197290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0098e0 tx=0x7fd49c005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4a6000700 1 -- 172.21.15.23:0/3820704506 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c01d070 con 0x7fd4a8100f60 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd49c0095f0 con 0x7fd4a8100f60 2024-03-30T04:12:30.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4a6000700 1 -- 172.21.15.23:0/3820704506 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd49c022470 con 0x7fd4a8100f60 2024-03-30T04:12:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4a6000700 1 -- 172.21.15.23:0/3820704506 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c017650 con 0x7fd4a8100f60 2024-03-30T04:12:30.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.049+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a81a4060 con 0x7fd4a8100f60 2024-03-30T04:12:30.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.052+0000 7fd4a6000700 1 -- 172.21.15.23:0/3820704506 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd49c0225e0 con 0x7fd4a8100f60 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4a6000700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 0x7fd4900a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 Processor -- start 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4a6000700 1 -- 172.21.15.23:0/3820704506 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd49c0ce380 con 0x7fd4a8100f60 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 -- start start 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 0x7fd4900a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85da010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85ed510 0x55f4b85ef900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b8532a10 con 0x55f4b85d9b90 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b85303d0 con 0x55f4b85ed510 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.054+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b852bcc0 con 0x55f4b85db240 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4ae600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85ed510 0x55f4b85ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60580/0 (socket says 172.21.15.23:60580) 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 -- 172.21.15.23:0/664151862 learned_addr learned my addr 172.21.15.23:0/664151862 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4ad200700 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85da010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 -- 172.21.15.23:0/664151862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85ed510 msgr2=0x55f4b85ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85ed510 0x55f4b85ef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 -- 172.21.15.23:0/664151862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 msgr2=0x55f4b85da010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85da010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 -- 172.21.15.23:0/664151862 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4b83ad430 con 0x55f4b85db240 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 0x7fd4900a3750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8198790 tx=0x7fd494011ab0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd4adc00700 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd49c097c70 tx=0x7fd49c09afd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db012991e69f2b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.055+0000 7fd49aa00700 1 -- 172.21.15.23:0/664151862 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c09b900 con 0x55f4b85db240 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd49aa00700 1 -- 172.21.15.23:0/664151862 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd49c09b200 con 0x55f4b85db240 2024-03-30T04:12:30.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd49aa00700 1 -- 172.21.15.23:0/664151862 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0a10c0 con 0x55f4b85db240 2024-03-30T04:12:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 -- 172.21.15.23:0/664151862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 msgr2=0x55f4b85d9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd49c097c70 tx=0x7fd49c09afd0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 -- 172.21.15.23:0/664151862 shutdown_connections 2024-03-30T04:12:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85ed510 0x55f4b85ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85db240 0x55f4b85d9650 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/664151862 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85da010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 -- 172.21.15.23:0/664151862 >> 172.21.15.23:0/664151862 conn(0x55f4b8539720 msgr2=0x55f4b85d5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 -- 172.21.15.23:0/664151862 shutdown_connections 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.056+0000 7fd4c420cb80 1 -- 172.21.15.23:0/664151862 wait complete. 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.057+0000 7fd4c420cb80 1 -- start start 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.057+0000 7fd4c420cb80 1 Processor -- start 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85e03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85dde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b84c7e00 con 0x55f4b85d9b90 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b84c7aa0 con 0x55f4b85db240 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b83af720 con 0x55f4b85ed510 2024-03-30T04:12:30.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4c420cb80 10 client.0 _ll_register_callbacks cb 0x55f4b84d14f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85e03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42942/0 (socket says 172.21.15.23:42942) 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 -- 172.21.15.23:0/3972999502 learned_addr learned my addr 172.21.15.23:0/3972999502 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ae600700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85dde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 -- 172.21.15.23:0/3972999502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 msgr2=0x55f4b85dde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85dde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 -- 172.21.15.23:0/3972999502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 msgr2=0x55f4b85e03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85e03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4ad200700 1 -- 172.21.15.23:0/3972999502 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a81a4060 con 0x55f4b85db240 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.058+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85e03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd4ad200700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd49400fb60 tx=0x7fd4940047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96fb1daba4c7fe22 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd48f400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:30.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd494022070 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd49401da60 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4b84ec400 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd49401dc20 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd4c420cb80 10 client.28415 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.059+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f4b8359be0 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f4b84ec400 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd4940263c0 con 0x55f4b85db240 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 10 client.28415 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 10 client.28415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 20 client.28415 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b85fcee0 0x55f4b85ff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 4 client.28415 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:30.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4b8524780 con 0x55f4b85fcee0 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.060+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b85fcee0 0x55f4b85ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.061+0000 7fd4adc00700 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b85fcee0 0x55f4b85ff300 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d10f71155695d3 server_cookie=37e4d59841738437 in_seq=0 out_seq=0 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.061+0000 7fd48f400700 10 client.28415 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.061+0000 7fd48f400700 1 -- 172.21.15.23:0/3972999502 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f4b8524780 con 0x55f4b85fcee0 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.061+0000 7fd48f400700 10 client.28415 handle_command_reply: tid=0 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.061+0000 7fd4c420cb80 1 client.28415 shutdown 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 msgr2=0x55f4b85dd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd49400fb60 tx=0x7fd4940047c0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 shutdown_connections 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b85fcee0 0x55f4b85ff300 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85dde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85e03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/3972999502 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dd700 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 >> 172.21.15.23:0/3972999502 conn(0x55f4b8539720 msgr2=0x55f4b85db6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd48f400700 0 client.28415 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 shutdown_connections 2024-03-30T04:12:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 1 -- 172.21.15.23:0/3972999502 wait complete. 2024-03-30T04:12:30.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.062+0000 7fd4c420cb80 20 client.28415 trim_cache size 0 max 16384 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 Processor -- start 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 -- start start 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 0x55f4b85f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85ed510 0x55f4b85d55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b85d6790 con 0x55f4b85ed510 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b85decb0 con 0x55f4b85db240 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.081+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b85de650 con 0x55f4b85d9b90 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 0x55f4b85f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4ad200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42952/0 (socket says 172.21.15.23:42952) 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4ad200700 1 -- 172.21.15.23:0/1994916801 learned_addr learned my addr 172.21.15.23:0/1994916801 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4ae600700 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85ed510 0x55f4b85d55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 -- 172.21.15.23:0/1994916801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 msgr2=0x55f4b85dc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 -- 172.21.15.23:0/1994916801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85ed510 msgr2=0x55f4b85d55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85ed510 0x55f4b85d55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 -- 172.21.15.23:0/1994916801 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4b84f1f20 con 0x55f4b85d9b90 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4ad200700 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 0x55f4b85f3f60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd49c099550 tx=0x7fd49c0a1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2075cefdaafaf76 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.082+0000 7fd49b400700 1 -- 172.21.15.23:0/1994916801 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0b9040 con 0x55f4b85d9b90 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd49b400700 1 -- 172.21.15.23:0/1994916801 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd49c0a35f0 con 0x55f4b85d9b90 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd49b400700 1 -- 172.21.15.23:0/1994916801 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0ce020 con 0x55f4b85d9b90 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1994916801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 msgr2=0x55f4b85f3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 0x55f4b85f3f60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd49c099550 tx=0x7fd49c0a1040 comp rx=0 tx=0).stop 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1994916801 shutdown_connections 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85ed510 0x55f4b85d55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85d9b90 0x55f4b85f3f60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1994916801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85dc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1994916801 >> 172.21.15.23:0/1994916801 conn(0x55f4b8539720 msgr2=0x55f4b85e97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1994916801 shutdown_connections 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.083+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1994916801 wait complete. 2024-03-30T04:12:30.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.084+0000 7fd4c420cb80 1 -- start start 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.084+0000 7fd4c420cb80 1 Processor -- start 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85f3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b8650ed0 con 0x55f4b85d9b90 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b8651040 con 0x55f4b85db240 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4b8651200 con 0x55f4b85ed510 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4c420cb80 10 client.0 _ll_register_callbacks cb 0x55f4b84d14f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:30.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39126/0 (socket says 172.21.15.23:39126) 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 -- 172.21.15.23:0/1847332951 learned_addr learned my addr 172.21.15.23:0/1847332951 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4ad200700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4ae600700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 -- 172.21.15.23:0/1847332951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 msgr2=0x55f4b85f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 -- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 msgr2=0x55f4b85f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.085+0000 7fd4adc00700 1 -- 172.21.15.23:0/1847332951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4b8359be0 con 0x55f4b85d9b90 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd4ad200700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd4adc00700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0a30c0 tx=0x7fd49c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f6b3a2553d2444 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd48f400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c0b9040 con 0x55f4b85d9b90 2024-03-30T04:12:30.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd49c0995a0 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd49c09b200 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd49c066470 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd4c420cb80 10 client.18441 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.086+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fd49c0b3040 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd49c09b200 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd49c0cb910 con 0x55f4b85d9b90 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 10 client.18441 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 10 client.18441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 20 client.18441 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b8652580 0x55f4b8654970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 4 client.18441 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f4b837ad70 con 0x55f4b8652580 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.087+0000 7fd4ad200700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b8652580 0x55f4b8654970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.088+0000 7fd48f400700 10 client.18441 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.088+0000 7fd4ad200700 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b8652580 0x55f4b8654970 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27520dd9dacef900 server_cookie=8b7285efe9a6e1f5 in_seq=0 out_seq=0 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd48f400700 1 -- 172.21.15.23:0/1847332951 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f4b837ad70 con 0x55f4b8652580 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd48f400700 10 client.18441 handle_command_reply: tid=0 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 client.18441 shutdown 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 msgr2=0x55f4b85f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0a30c0 tx=0x7fd49c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 shutdown_connections 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f4b8652580 0x55f4b8654970 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4b85ed510 0x55f4b85f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4b85d9b90 0x55f4b85f29d0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 --2- 172.21.15.23:0/1847332951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f4b85db240 0x55f4b85f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 >> 172.21.15.23:0/1847332951 conn(0x55f4b8539720 msgr2=0x55f4b85d5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.089+0000 7fd48f400700 0 client.18441 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 shutdown_connections 2024-03-30T04:12:30.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4c420cb80 1 -- 172.21.15.23:0/1847332951 wait complete. 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4c420cb80 20 client.18441 trim_cache size 0 max 16384 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "5bf793f4-0601-455d-a2dd-fc27a7097456", 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:12:30.092 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 msgr2=0x7fd4900a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 0x7fd4900a3750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8198790 tx=0x7fd494011ab0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 msgr2=0x7fd4a8197290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.090+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 0x7fd4a8197290 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd49c0098e0 tx=0x7fd49c005740 comp rx=0 tx=0).stop 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 shutdown_connections 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4900a1290 0x7fd4900a3750 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a819bb20 0x7fd4a819bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd4a8100f60 0x7fd4a8197290 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 --2- 172.21.15.23:0/3820704506 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd4a8106080 0x7fd4a81977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 >> 172.21.15.23:0/3820704506 conn(0x7fd4a80fc7c0 msgr2=0x7fd4a80fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 shutdown_connections 2024-03-30T04:12:30.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.091+0000 7fd4b0600700 1 -- 172.21.15.23:0/3820704506 wait complete. 2024-03-30T04:12:30.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:12:30.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1044627 10.02 MB/sec execute 427 sec latency 28.444 ms 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 -- 172.21.15.23:0/2002022614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 msgr2=0x7f66701060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 --2- 172.21.15.23:0/2002022614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 0x7f66701060c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f66600099c0 tx=0x7f6660009cd0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 -- 172.21.15.23:0/2002022614 shutdown_connections 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 --2- 172.21.15.23:0/2002022614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66700687f0 0x7f667010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 --2- 172.21.15.23:0/2002022614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 0x7f66701060c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 --2- 172.21.15.23:0/2002022614 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6670067e30 0x7f66700682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.319 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 -- 172.21.15.23:0/2002022614 >> 172.21.15.23:0/2002022614 conn(0x7f6670076d40 msgr2=0x7f6670077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 -- 172.21.15.23:0/2002022614 shutdown_connections 2024-03-30T04:12:30.320 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.320+0000 7f6677000700 1 -- 172.21.15.23:0/2002022614 wait complete. 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.321+0000 7f6677000700 1 Processor -- start 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.321+0000 7f6677000700 1 -- start start 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 0x7f6670199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 0x7f667019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667019e340 con 0x7f66700687f0 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667019e4b0 con 0x7f6670067e30 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f6677000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667019e670 con 0x7f6670105ce0 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f666fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f666fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42970/0 (socket says 172.21.15.23:42970) 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f666fe00700 1 -- 172.21.15.23:0/56686437 learned_addr learned my addr 172.21.15.23:0/56686437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.322+0000 7f666f400700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 0x7f6670199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 -- 172.21.15.23:0/56686437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 msgr2=0x7f667019dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 0x7f667019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 -- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 msgr2=0x7f6670199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 0x7f6670199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 -- 172.21.15.23:0/56686437 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66600095f0 con 0x7f6670067e30 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666f400700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 0x7f6670199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666fe00700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f666000b9c0 tx=0x7f6660009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666cc00700 1 -- 172.21.15.23:0/56686437 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666001d070 con 0x7f6670067e30 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667019e8f0 con 0x7f6670067e30 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666cc00700 1 -- 172.21.15.23:0/56686437 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f666000bde0 con 0x7f6670067e30 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f666cc00700 1 -- 172.21.15.23:0/56686437 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666000f9d0 con 0x7f6670067e30 2024-03-30T04:12:30.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.323+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66701a6190 con 0x7f6670067e30 2024-03-30T04:12:30.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.326+0000 7f666cc00700 1 -- 172.21.15.23:0/56686437 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f6660003720 con 0x7f6670067e30 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f666cc00700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 0x7f66540a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f668abbdb80 1 Processor -- start 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f666cc00700 1 -- 172.21.15.23:0/56686437 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f66600cdec0 con 0x7f6670067e30 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f668abbdb80 1 -- start start 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f666f400700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 0x7f66540a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.328+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be3ef0 0x556965bee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 0x556965bf11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f668abbdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965b3c990 con 0x556965beedf0 2024-03-30T04:12:30.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f668abbdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965b3a350 con 0x556965be35d0 2024-03-30T04:12:30.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f668abbdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965b35c40 con 0x556965be3ef0 2024-03-30T04:12:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42986/0 (socket says 172.21.15.23:42986) 2024-03-30T04:12:30.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 -- 172.21.15.23:0/3102811262 learned_addr learned my addr 172.21.15.23:0/3102811262 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f6675000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 0x556965bf11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666f400700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 0x7f66540a3750 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f667019a970 tx=0x7f665c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666f400700 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be3ef0 0x556965bee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 -- 172.21.15.23:0/3102811262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be3ef0 msgr2=0x556965bee8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be3ef0 0x556965bee8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 -- 172.21.15.23:0/3102811262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 msgr2=0x556965bf11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 0x556965bf11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f666fe00700 1 -- 172.21.15.23:0/3102811262 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569659b73b0 con 0x556965be35d0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.329+0000 7f6675000700 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 0x556965bf11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.330+0000 7f666fe00700 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6660097120 tx=0x7f666009a600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba9a16d74a3dad64 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.330+0000 7f6665600700 1 -- 172.21.15.23:0/3102811262 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666009ea50 con 0x556965be35d0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.330+0000 7f6665600700 1 -- 172.21.15.23:0/3102811262 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66600a3010 con 0x556965be35d0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.330+0000 7f6665600700 1 -- 172.21.15.23:0/3102811262 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66600a2b80 con 0x556965be35d0 2024-03-30T04:12:30.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- 172.21.15.23:0/3102811262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 msgr2=0x556965be39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6660097120 tx=0x7f666009a600 comp rx=0 tx=0).stop 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- 172.21.15.23:0/3102811262 shutdown_connections 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965beedf0 0x556965bf11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be35d0 0x556965be39b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 --2- 172.21.15.23:0/3102811262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be3ef0 0x556965bee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- 172.21.15.23:0/3102811262 >> 172.21.15.23:0/3102811262 conn(0x556965b436a0 msgr2=0x556965bdfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- 172.21.15.23:0/3102811262 shutdown_connections 2024-03-30T04:12:30.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- 172.21.15.23:0/3102811262 wait complete. 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.331+0000 7f668abbdb80 1 -- start start 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 Processor -- start 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bee750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bec090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965ad1d80 con 0x556965be35d0 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965ad1a20 con 0x556965be3ef0 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569659b96a0 con 0x556965beedf0 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f668abbdb80 10 client.0 _ll_register_callbacks cb 0x556965adb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:30.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f666f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f666f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42994/0 (socket says 172.21.15.23:42994) 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f666f400700 1 -- 172.21.15.23:0/292138187 learned_addr learned my addr 172.21.15.23:0/292138187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.332+0000 7f666fe00700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f6675000700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 -- 172.21.15.23:0/292138187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 msgr2=0x556965bec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 -- 172.21.15.23:0/292138187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 msgr2=0x556965bee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bee750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 -- 172.21.15.23:0/292138187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666800e140 con 0x556965be3ef0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666fe00700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f666f400700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f665c0039b0 tx=0x7f665c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ac3f9b405a5d03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f665a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:30.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c01fa90 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f665c01fbf0 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.333+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556965af6380 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f665c01fdb0 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 10 client.28433 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f66701a6190 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556965af6380 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f665c0274b0 con 0x556965be3ef0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 10 client.28433 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 10 client.28433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 20 client.28433 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965bfe690 0x556965c00ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 4 client.28433 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.334+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556965b2e700 con 0x556965bfe690 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.335+0000 7f666fe00700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965bfe690 0x556965c00ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.335+0000 7f666fe00700 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965bfe690 0x556965c00ab0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3db752e1a6685def server_cookie=89cf970f1050b03d in_seq=0 out_seq=0 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.335+0000 7f665a000700 10 client.28433 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f665a000700 1 -- 172.21.15.23:0/292138187 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556965b2e700 con 0x556965bfe690 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f665a000700 10 client.28433 handle_command_reply: tid=0 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 client.28433 shutdown 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 msgr2=0x556965bebab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f665c0039b0 tx=0x7f665c0047c0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 shutdown_connections 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965bfe690 0x556965c00ab0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f665a000700 0 client.28433 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 --2- 172.21.15.23:0/292138187 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bebab0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 >> 172.21.15.23:0/292138187 conn(0x556965b436a0 msgr2=0x556965be5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 shutdown_connections 2024-03-30T04:12:30.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 1 -- 172.21.15.23:0/292138187 wait complete. 2024-03-30T04:12:30.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.336+0000 7f668abbdb80 20 client.28433 trim_cache size 0 max 16384 2024-03-30T04:12:30.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 Processor -- start 2024-03-30T04:12:30.353 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 -- start start 2024-03-30T04:12:30.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be35d0 0x556965bdef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965beedf0 0x556965be6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965c00eb0 con 0x556965be3ef0 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965bf3770 con 0x556965beedf0 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.355+0000 7f668abbdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965bec9a0 con 0x556965be35d0 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39170/0 (socket says 172.21.15.23:39170) 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 -- 172.21.15.23:0/2486791543 learned_addr learned my addr 172.21.15.23:0/2486791543 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 -- 172.21.15.23:0/2486791543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be35d0 msgr2=0x556965bdef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be35d0 0x556965bdef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 -- 172.21.15.23:0/2486791543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965beedf0 msgr2=0x556965be6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965beedf0 0x556965be6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 -- 172.21.15.23:0/2486791543 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556965963b60 con 0x556965be3ef0 2024-03-30T04:12:30.356 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.356+0000 7f666f400700 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f665c0094f0 tx=0x7f665c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a41a7426f9183dcb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f6666000700 1 -- 172.21.15.23:0/2486791543 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c027a80 con 0x556965be3ef0 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f6666000700 1 -- 172.21.15.23:0/2486791543 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f665c027be0 con 0x556965be3ef0 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f6666000700 1 -- 172.21.15.23:0/2486791543 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c036730 con 0x556965be3ef0 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 -- 172.21.15.23:0/2486791543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 msgr2=0x556965bf3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f665c0094f0 tx=0x7f665c011bf0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 -- 172.21.15.23:0/2486791543 shutdown_connections 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965beedf0 0x556965be6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965be35d0 0x556965bdef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 --2- 172.21.15.23:0/2486791543 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be3ef0 0x556965bf3200 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.357+0000 7f668abbdb80 1 -- 172.21.15.23:0/2486791543 >> 172.21.15.23:0/2486791543 conn(0x556965b436a0 msgr2=0x556965bd85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.358+0000 7f668abbdb80 1 -- 172.21.15.23:0/2486791543 shutdown_connections 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.358+0000 7f668abbdb80 1 -- 172.21.15.23:0/2486791543 wait complete. 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.358+0000 7f668abbdb80 1 -- start start 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.358+0000 7f668abbdb80 1 Processor -- start 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bf2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bf0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965bf14d0 con 0x556965be35d0 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965bf1640 con 0x556965be3ef0 2024-03-30T04:12:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556965bf1800 con 0x556965beedf0 2024-03-30T04:12:30.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f668abbdb80 10 client.0 _ll_register_callbacks cb 0x556965adb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:12:30.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bf2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:42998/0 (socket says 172.21.15.23:42998) 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 -- 172.21.15.23:0/1077874261 learned_addr learned my addr 172.21.15.23:0/1077874261 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f6675000700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bf0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 -- 172.21.15.23:0/1077874261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 msgr2=0x556965bf0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bf0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 -- 172.21.15.23:0/1077874261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 msgr2=0x556965bf2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bf2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666f400700 1 -- 172.21.15.23:0/1077874261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66701a6190 con 0x556965be3ef0 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.359+0000 7f666fe00700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bf2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f666f400700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f665c0278b0 tx=0x7f665c027f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc332fd4b848f2df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:12:30.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f665a000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c01f8f0 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66600a3010 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f665c01fa50 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f665c01fc10 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f668abbdb80 10 client.28436 fetch_fsmap learned FSMap version 30 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x556965afbea0 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.360+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66600a3010 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f665c004c40 con 0x556965be3ef0 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 10 client.28436 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 10 client.28436 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 20 client.28436 populate_metadata read hostname 'smithi023' 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965c532e0 0x556965c55700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 4 client.28436 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:12:30.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556965984cf0 con 0x556965c532e0 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f666fe00700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965c532e0 0x556965c55700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f666fe00700 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965c532e0 0x556965c55700 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6a37787516e60bc server_cookie=20d26edeab908cc8 in_seq=0 out_seq=0 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.361+0000 7f665a000700 10 client.28436 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f665a000700 1 -- 172.21.15.23:0/1077874261 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556965984cf0 con 0x556965c532e0 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f665a000700 10 client.28436 handle_command_reply: tid=0 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 client.28436 shutdown 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 msgr2=0x556965bf2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f665c0278b0 tx=0x7f665c027f80 comp rx=0 tx=0).stop 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 shutdown_connections 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x556965c532e0 0x556965c55700 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556965beedf0 0x556965bf0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556965be35d0 0x556965bf2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f665a000700 0 client.28436 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 --2- 172.21.15.23:0/1077874261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556965be3ef0 0x556965bf2ec0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 >> 172.21.15.23:0/1077874261 conn(0x556965b436a0 msgr2=0x556965be6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 shutdown_connections 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 1 -- 172.21.15.23:0/1077874261 wait complete. 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.362+0000 7f668abbdb80 20 client.28436 trim_cache size 0 max 16384 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "5bf793f4-0601-455d-a2dd-fc27a7097456": { 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:12:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "5bf793f4-0601-455d-a2dd-fc27a7097456", 2024-03-30T04:12:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:12:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:12:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:12:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:12:30.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 msgr2=0x7f66540a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 0x7f66540a3750 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f667019a970 tx=0x7f665c009260 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 msgr2=0x7f6670199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f666000b9c0 tx=0x7f6660009580 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 shutdown_connections 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6670105ce0 0x7f667019dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66540a1290 0x7f66540a3750 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6670067e30 0x7f6670199420 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 --2- 172.21.15.23:0/56686437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66700687f0 0x7f6670199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 >> 172.21.15.23:0/56686437 conn(0x7f6670076d40 msgr2=0x7f6670101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:12:30.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 shutdown_connections 2024-03-30T04:12:30.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:12:30.363+0000 7f6677000700 1 -- 172.21.15.23:0/56686437 wait complete. 2024-03-30T04:12:30.381 INFO:tasks.cephfs.filesystem:scrub status for tag:5bf793f4-0601-455d-a2dd-fc27a7097456 - {'path': '/', 'tag': '5bf793f4-0601-455d-a2dd-fc27a7097456', 'options': 'recursive,force'} 2024-03-30T04:12:30.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:30 smithi154 ceph-mon[79161]: pgmap v329: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-03-30T04:12:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:30 smithi193 ceph-mon[79581]: pgmap v329: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-03-30T04:12:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:30 smithi023 ceph-mon[38254]: pgmap v329: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 206 op/s 2024-03-30T04:12:31.226 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1046034 10.01 MB/sec execute 428 sec latency 22.518 ms 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:31 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:12:31.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:12:31.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:12:31.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:31 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:31.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:31 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:12:32.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1048024 10.02 MB/sec execute 429 sec latency 39.189 ms 2024-03-30T04:12:32.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:32 smithi154 ceph-mon[79161]: pgmap v330: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 146 op/s 2024-03-30T04:12:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:32 smithi193 ceph-mon[79581]: pgmap v330: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 146 op/s 2024-03-30T04:12:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:32 smithi023 ceph-mon[38254]: pgmap v330: 129 pgs: 129 active+clean; 564 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.1 MiB/s wr, 146 op/s 2024-03-30T04:12:33.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1048697 10.00 MB/sec execute 430 sec latency 34.448 ms 2024-03-30T04:12:34.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1050547 9.99 MB/sec execute 431 sec latency 26.339 ms 2024-03-30T04:12:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:34 smithi154 ceph-mon[79161]: pgmap v331: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-03-30T04:12:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:34 smithi193 ceph-mon[79581]: pgmap v331: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-03-30T04:12:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:34 smithi023 ceph-mon[38254]: pgmap v331: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 214 op/s 2024-03-30T04:12:35.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1051828 10.00 MB/sec execute 432 sec latency 24.063 ms 2024-03-30T04:12:36.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1053167 9.99 MB/sec execute 433 sec latency 31.187 ms 2024-03-30T04:12:36.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:36 smithi154 ceph-mon[79161]: pgmap v332: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-30T04:12:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:36 smithi193 ceph-mon[79581]: pgmap v332: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-30T04:12:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:36 smithi023 ceph-mon[38254]: pgmap v332: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 156 op/s 2024-03-30T04:12:37.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1055062 9.99 MB/sec execute 434 sec latency 48.147 ms 2024-03-30T04:12:38.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1055772 9.97 MB/sec execute 435 sec latency 37.995 ms 2024-03-30T04:12:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:38 smithi193 ceph-mon[79581]: pgmap v333: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-03-30T04:12:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:38 smithi023 ceph-mon[38254]: pgmap v333: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-03-30T04:12:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:38 smithi154 ceph-mon[79161]: pgmap v333: 129 pgs: 129 active+clean; 568 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 146 op/s 2024-03-30T04:12:39.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1057604 9.97 MB/sec execute 436 sec latency 31.838 ms 2024-03-30T04:12:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:39 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:12:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:39 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:12:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:39 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:12:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:39 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:12:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:39 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:12:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:39 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:12:40.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1058920 9.97 MB/sec execute 437 sec latency 19.208 ms 2024-03-30T04:12:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:40 smithi193 ceph-mon[79581]: pgmap v334: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 216 op/s 2024-03-30T04:12:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:40 smithi023 ceph-mon[38254]: pgmap v334: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 216 op/s 2024-03-30T04:12:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:40 smithi154 ceph-mon[79161]: pgmap v334: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.2 MiB/s wr, 216 op/s 2024-03-30T04:12:41.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1060258 9.96 MB/sec execute 438 sec latency 21.589 ms 2024-03-30T04:12:42.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1062230 9.96 MB/sec execute 439 sec latency 95.513 ms 2024-03-30T04:12:42.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:42 smithi154 ceph-mon[79161]: pgmap v335: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-03-30T04:12:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:42 smithi193 ceph-mon[79581]: pgmap v335: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-03-30T04:12:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:42 smithi023 ceph-mon[38254]: pgmap v335: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-03-30T04:12:43.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1063206 9.95 MB/sec execute 440 sec latency 64.846 ms 2024-03-30T04:12:44.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1065160 9.95 MB/sec execute 441 sec latency 69.324 ms 2024-03-30T04:12:44.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:44 smithi154 ceph-mon[79161]: pgmap v336: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-03-30T04:12:44.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:44 smithi193 ceph-mon[79581]: pgmap v336: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-03-30T04:12:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:44 smithi023 ceph-mon[38254]: pgmap v336: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.2 MiB/s wr, 211 op/s 2024-03-30T04:12:45.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1066256 9.94 MB/sec execute 442 sec latency 23.359 ms 2024-03-30T04:12:46.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1068035 9.94 MB/sec execute 443 sec latency 23.085 ms 2024-03-30T04:12:46.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:46 smithi154 ceph-mon[79161]: pgmap v337: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 148 op/s 2024-03-30T04:12:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:46 smithi193 ceph-mon[79581]: pgmap v337: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 148 op/s 2024-03-30T04:12:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:46 smithi023 ceph-mon[38254]: pgmap v337: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 148 op/s 2024-03-30T04:12:47.227 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1069452 9.94 MB/sec execute 444 sec latency 36.274 ms 2024-03-30T04:12:48.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1070731 9.93 MB/sec execute 445 sec latency 33.053 ms 2024-03-30T04:12:48.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:48 smithi154 ceph-mon[79161]: pgmap v338: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-03-30T04:12:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:48 smithi193 ceph-mon[79581]: pgmap v338: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-03-30T04:12:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:48 smithi023 ceph-mon[38254]: pgmap v338: 129 pgs: 129 active+clean; 571 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.9 MiB/s wr, 144 op/s 2024-03-30T04:12:49.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1072788 9.94 MB/sec execute 446 sec latency 20.292 ms 2024-03-30T04:12:50.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1073502 9.92 MB/sec execute 447 sec latency 29.830 ms 2024-03-30T04:12:50.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:50 smithi154 ceph-mon[79161]: pgmap v339: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 206 op/s 2024-03-30T04:12:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:50 smithi193 ceph-mon[79581]: pgmap v339: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 206 op/s 2024-03-30T04:12:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:50 smithi023 ceph-mon[38254]: pgmap v339: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 206 op/s 2024-03-30T04:12:51.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1075223 9.91 MB/sec execute 448 sec latency 28.652 ms 2024-03-30T04:12:52.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1076621 9.92 MB/sec execute 449 sec latency 19.061 ms 2024-03-30T04:12:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:52 smithi154 ceph-mon[79161]: pgmap v340: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-30T04:12:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:52 smithi193 ceph-mon[79581]: pgmap v340: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-30T04:12:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:52 smithi023 ceph-mon[38254]: pgmap v340: 129 pgs: 129 active+clean; 568 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-30T04:12:53.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1077820 9.91 MB/sec execute 450 sec latency 38.473 ms 2024-03-30T04:12:54.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1079704 9.90 MB/sec execute 451 sec latency 165.820 ms 2024-03-30T04:12:54.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:54 smithi154 ceph-mon[79161]: pgmap v341: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 199 op/s 2024-03-30T04:12:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:54 smithi193 ceph-mon[79581]: pgmap v341: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 199 op/s 2024-03-30T04:12:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:54 smithi023 ceph-mon[38254]: pgmap v341: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 199 op/s 2024-03-30T04:12:55.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1080465 9.89 MB/sec execute 452 sec latency 29.246 ms 2024-03-30T04:12:56.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1082278 9.89 MB/sec execute 453 sec latency 27.152 ms 2024-03-30T04:12:56.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:56 smithi154 ceph-mon[79161]: pgmap v342: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-03-30T04:12:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:56 smithi193 ceph-mon[79581]: pgmap v342: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-03-30T04:12:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:56 smithi023 ceph-mon[38254]: pgmap v342: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.1 MiB/s wr, 133 op/s 2024-03-30T04:12:57.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1083702 9.89 MB/sec execute 454 sec latency 24.731 ms 2024-03-30T04:12:58.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1084873 9.88 MB/sec execute 455 sec latency 48.774 ms 2024-03-30T04:12:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:12:58 smithi154 ceph-mon[79161]: pgmap v343: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-30T04:12:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:12:58 smithi193 ceph-mon[79581]: pgmap v343: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-30T04:12:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:12:58 smithi023 ceph-mon[38254]: pgmap v343: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 128 op/s 2024-03-30T04:12:59.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1086971 9.89 MB/sec execute 456 sec latency 19.826 ms 2024-03-30T04:13:00.228 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1087750 9.87 MB/sec execute 457 sec latency 30.084 ms 2024-03-30T04:13:00.382 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:13:00.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:00 smithi154 ceph-mon[79161]: pgmap v344: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 197 op/s 2024-03-30T04:13:00.463 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:00 smithi023 ceph-mon[38254]: pgmap v344: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 197 op/s 2024-03-30T04:13:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:00 smithi193 ceph-mon[79581]: pgmap v344: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 197 op/s 2024-03-30T04:13:00.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 -- 172.21.15.23:0/1194551738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184106080 msgr2=0x7f418410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 --2- 172.21.15.23:0/1194551738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184106080 0x7f418410dc00 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f417c00b0a0 tx=0x7f417c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 -- 172.21.15.23:0/1194551738 shutdown_connections 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 --2- 172.21.15.23:0/1194551738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184106080 0x7f418410dc00 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 --2- 172.21.15.23:0/1194551738 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184100f60 0x7f4184101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 --2- 172.21.15.23:0/1194551738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184101910 0x7f4184105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 -- 172.21.15.23:0/1194551738 >> 172.21.15.23:0/1194551738 conn(0x7f41840fc7c0 msgr2=0x7f41840febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 -- 172.21.15.23:0/1194551738 shutdown_connections 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.620+0000 7f4188a00700 1 -- 172.21.15.23:0/1194551738 wait complete. 2024-03-30T04:13:00.620 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.621+0000 7f4188a00700 1 Processor -- start 2024-03-30T04:13:00.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 -- start start 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184100f60 0x7f41840730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 0x7f41841a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841a5b40 con 0x7f4184073b30 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841a5c80 con 0x7f4184106080 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4188a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41841a5e40 con 0x7f4184100f60 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.622+0000 7f4181400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4181400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51122/0 (socket says 172.21.15.23:51122) 2024-03-30T04:13:00.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4181400700 1 -- 172.21.15.23:0/33313418 learned_addr learned my addr 172.21.15.23:0/33313418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 0x7f41841a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4181e00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184100f60 0x7f41840730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 -- 172.21.15.23:0/33313418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184100f60 msgr2=0x7f41840730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184100f60 0x7f41840730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 -- 172.21.15.23:0/33313418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 msgr2=0x7f41840735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 -- 172.21.15.23:0/33313418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c009d00 con 0x7f4184073b30 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4181400700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f4182800700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 0x7f41841a5570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f417c00b070 tx=0x7f417c00bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f417aa00700 1 -- 172.21.15.23:0/33313418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c00e040 con 0x7f4184073b30 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f417aa00700 1 -- 172.21.15.23:0/33313418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f417c01d480 con 0x7f4184073b30 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.623+0000 7f417aa00700 1 -- 172.21.15.23:0/33313418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0052a0 con 0x7f4184073b30 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.624+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41841a60c0 con 0x7f4184073b30 2024-03-30T04:13:00.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.624+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41841a6490 con 0x7f4184073b30 2024-03-30T04:13:00.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.627+0000 7f417aa00700 1 -- 172.21.15.23:0/33313418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f417c01d5f0 con 0x7f4184073b30 2024-03-30T04:13:00.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.628+0000 7f417aa00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 0x7f41680a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.628+0000 7f417aa00700 1 -- 172.21.15.23:0/33313418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f417c0ce0b0 con 0x7f4184073b30 2024-03-30T04:13:00.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.628+0000 7f419c599b80 1 Processor -- start 2024-03-30T04:13:00.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.628+0000 7f4181e00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 0x7f41680a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 -- start start 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef445d0 0x5648fef449b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef521e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fee9d990 con 0x5648fef445d0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fee9b350 con 0x5648fef4fdf0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f419c599b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fee96c40 con 0x5648fef44ef0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4181e00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 0x7f41680a3750 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4174009c60 tx=0x7f4174009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4181400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4181400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34092/0 (socket says 172.21.15.23:34092) 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4181400700 1 -- 172.21.15.23:0/2360034680 learned_addr learned my addr 172.21.15.23:0/2360034680 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4182800700 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.629+0000 7f4181e00700 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef445d0 0x5648fef449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 -- 172.21.15.23:0/2360034680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 msgr2=0x5648fef521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 -- 172.21.15.23:0/2360034680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef445d0 msgr2=0x5648fef449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef445d0 0x5648fef449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.630 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 -- 172.21.15.23:0/2360034680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648fed183b0 con 0x5648fef44ef0 2024-03-30T04:13:00.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f4181400700 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f41700115c0 tx=0x7f41700118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273c7b9c4abd3aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f416f000700 1 -- 172.21.15.23:0/2360034680 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170019630 con 0x5648fef44ef0 2024-03-30T04:13:00.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f416f000700 1 -- 172.21.15.23:0/2360034680 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4170019c50 con 0x5648fef44ef0 2024-03-30T04:13:00.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.630+0000 7f416f000700 1 -- 172.21.15.23:0/2360034680 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417000eda0 con 0x5648fef44ef0 2024-03-30T04:13:00.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 -- 172.21.15.23:0/2360034680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 msgr2=0x5648fef4f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f41700115c0 tx=0x7f41700118d0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 -- 172.21.15.23:0/2360034680 shutdown_connections 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef445d0 0x5648fef449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 --2- 172.21.15.23:0/2360034680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef44ef0 0x5648fef4f8b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 -- 172.21.15.23:0/2360034680 >> 172.21.15.23:0/2360034680 conn(0x5648feea46a0 msgr2=0x5648fef40a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 -- 172.21.15.23:0/2360034680 shutdown_connections 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.631+0000 7f419c599b80 1 -- 172.21.15.23:0/2360034680 wait complete. 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 -- start start 2024-03-30T04:13:00.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 Processor -- start 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef4d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fee32d80 con 0x5648fef44ef0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fee32a20 con 0x5648fef445d0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.632+0000 7f419c599b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fed1a6a0 con 0x5648fef4fdf0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f419c599b80 10 client.0 _ll_register_callbacks cb 0x5648fee3c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51144/0 (socket says 172.21.15.23:51144) 2024-03-30T04:13:00.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 -- 172.21.15.23:0/891562926 learned_addr learned my addr 172.21.15.23:0/891562926 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4182800700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef4d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 -- 172.21.15.23:0/891562926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 msgr2=0x5648fef4d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef4d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 -- 172.21.15.23:0/891562926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 msgr2=0x5648fef4cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.633+0000 7f4181e00700 1 -- 172.21.15.23:0/891562926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4170009d80 con 0x5648fef445d0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.634+0000 7f4181e00700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f417401eb80 tx=0x7f41740047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f39576c86500bb73 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.634+0000 7f4163400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.634+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400ec50 con 0x5648fef445d0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.634+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5648fee57380 con 0x5648fef445d0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.634+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f417400edb0 con 0x5648fef445d0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4174004830 con 0x5648fef445d0 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 10 client.28448 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:00.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5648fecc4b60 con 0x5648fef445d0 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5648fee57380 con 0x5648fef445d0 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f41740263f0 con 0x5648fef445d0 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 10 client.28448 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 10 client.28448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 20 client.28448 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fef5fa50 0x5648fef61e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 4 client.28448 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.635+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5648fee8f700 con 0x5648fef5fa50 2024-03-30T04:13:00.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.636+0000 7f4181400700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fef5fa50 0x5648fef61e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.636+0000 7f4181400700 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fef5fa50 0x5648fef61e70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82c99993f7df727b server_cookie=40621c8050f305a3 in_seq=0 out_seq=0 2024-03-30T04:13:00.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.636+0000 7f4163400700 10 client.28448 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:00.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.637+0000 7f4163400700 1 -- 172.21.15.23:0/891562926 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5648fee8f700 con 0x5648fef5fa50 2024-03-30T04:13:00.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.637+0000 7f4163400700 10 client.28448 handle_command_reply: tid=0 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.637+0000 7f419c599b80 1 client.28448 shutdown 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 msgr2=0x5648fef4f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f417401eb80 tx=0x7f41740047c0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 shutdown_connections 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fef5fa50 0x5648fef61e70 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef4d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef4f7f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f4163400700 0 client.28448 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 --2- 172.21.15.23:0/891562926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 >> 172.21.15.23:0/891562926 conn(0x5648feea46a0 msgr2=0x5648fef466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 shutdown_connections 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 1 -- 172.21.15.23:0/891562926 wait complete. 2024-03-30T04:13:00.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.638+0000 7f419c599b80 20 client.28448 trim_cache size 0 max 16384 2024-03-30T04:13:00.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 Processor -- start 2024-03-30T04:13:00.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 -- start start 2024-03-30T04:13:00.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 0x5648fef56aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef4c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fef416d0 con 0x5648fef44ef0 2024-03-30T04:13:00.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fef479d0 con 0x5648fef4fdf0 2024-03-30T04:13:00.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f419c599b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fef57060 con 0x5648fef445d0 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f4181400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.657+0000 7f4182800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef4c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4181e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 0x5648fef56aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4181400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36834/0 (socket says 172.21.15.23:36834) 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4181400700 1 -- 172.21.15.23:0/3009318982 learned_addr learned my addr 172.21.15.23:0/3009318982 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 -- 172.21.15.23:0/3009318982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 msgr2=0x5648fef56aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 0x5648fef56aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 -- 172.21.15.23:0/3009318982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 msgr2=0x5648fef4f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 -- 172.21.15.23:0/3009318982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5648fee5cea0 con 0x5648fef4fdf0 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4181400700 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4181e00700 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 0x5648fef56aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f4182800700 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef4c370 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f417c0cb3c0 tx=0x7f417c0a1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=655daa0b3b187866 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f416fa00700 1 -- 172.21.15.23:0/3009318982 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c09b510 con 0x5648fef4fdf0 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f416fa00700 1 -- 172.21.15.23:0/3009318982 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f417c09ae10 con 0x5648fef4fdf0 2024-03-30T04:13:00.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.658+0000 7f416fa00700 1 -- 172.21.15.23:0/3009318982 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c09b070 con 0x5648fef4fdf0 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 -- 172.21.15.23:0/3009318982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 msgr2=0x5648fef4c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef4c370 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f417c0cb3c0 tx=0x7f417c0a1040 comp rx=0 tx=0).stop 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 -- 172.21.15.23:0/3009318982 shutdown_connections 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef4fdf0 0x5648fef4c370 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef445d0 0x5648fef56aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 --2- 172.21.15.23:0/3009318982 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef4f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 -- 172.21.15.23:0/3009318982 >> 172.21.15.23:0/3009318982 conn(0x5648feea46a0 msgr2=0x5648fef4b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 -- 172.21.15.23:0/3009318982 shutdown_connections 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.659+0000 7f419c599b80 1 -- 172.21.15.23:0/3009318982 wait complete. 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 -- start start 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 Processor -- start 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef550a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef557e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fef55eb0 con 0x5648fef44ef0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fefb3890 con 0x5648fef445d0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.660+0000 7f419c599b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5648fefb39d0 con 0x5648fef4fdf0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f419c599b80 10 client.0 _ll_register_callbacks cb 0x5648fee3c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51168/0 (socket says 172.21.15.23:51168) 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 -- 172.21.15.23:0/3538321981 learned_addr learned my addr 172.21.15.23:0/3538321981 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181400700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4182800700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 -- 172.21.15.23:0/3538321981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 msgr2=0x5648fef557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 -- 172.21.15.23:0/3538321981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 msgr2=0x5648fef550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 -- 172.21.15.23:0/3538321981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5648fecc4b60 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4181e00700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4174005950 tx=0x7f417400e690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a06711d99bf02718 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.661+0000 7f4163400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417400ea50 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f417400ebb0 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4170019c50 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f417400ed70 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 10 client.28454 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5648fed183b0 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4170019c50 con 0x5648fef445d0 2024-03-30T04:13:00.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4174005020 con 0x5648fef445d0 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 10 client.28454 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 10 client.28454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 20 client.28454 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.662+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fefb5250 0x5648fefb7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f419c599b80 4 client.28454 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5648fece5cf0 con 0x5648fefb5250 2024-03-30T04:13:00.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f4181400700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fefb5250 0x5648fefb7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f4181400700 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fefb5250 0x5648fefb7670 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f92694fcc3725a2 server_cookie=4fe57ac7062aab13 in_seq=0 out_seq=0 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f4163400700 10 client.28454 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f4163400700 1 -- 172.21.15.23:0/3538321981 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5648fece5cf0 con 0x5648fefb5250 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f4163400700 10 client.28454 handle_command_reply: tid=0 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.663+0000 7f419c599b80 1 client.28454 shutdown 2024-03-30T04:13:00.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 msgr2=0x5648fef56ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f4174005950 tx=0x7f417400e690 comp rx=0 tx=0).stop 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 shutdown_connections 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5648fefb5250 0x5648fefb7670 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5648fef4fdf0 0x5648fef557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5648fef445d0 0x5648fef56ae0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 --2- 172.21.15.23:0/3538321981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5648fef44ef0 0x5648fef550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f4163400700 0 client.28454 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 >> 172.21.15.23:0/3538321981 conn(0x5648feea46a0 msgr2=0x5648fef4c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 shutdown_connections 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 1 -- 172.21.15.23:0/3538321981 wait complete. 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.664+0000 7f419c599b80 20 client.28454 trim_cache size 0 max 16384 2024-03-30T04:13:00.667 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 msgr2=0x7f41680a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 0x7f41680a3750 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4174009c60 tx=0x7f4174009260 comp rx=0 tx=0).stop 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 msgr2=0x7f41841a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 0x7f41841a5570 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f417c00b070 tx=0x7f417c00bcf0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 shutdown_connections 2024-03-30T04:13:00.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4184073b30 0x7f41841a5570 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4184100f60 0x7f41840730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41680a1290 0x7f41680a3750 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 --2- 172.21.15.23:0/33313418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4184106080 0x7f41840735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 >> 172.21.15.23:0/33313418 conn(0x7f41840fc7c0 msgr2=0x7f41840fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 shutdown_connections 2024-03-30T04:13:00.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.665+0000 7f4188a00700 1 -- 172.21.15.23:0/33313418 wait complete. 2024-03-30T04:13:00.683 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:13:00.683 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 -- 172.21.15.23:0/3936911457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d000731e0 msgr2=0x7f2d000735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3936911457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d000731e0 0x7f2d000735c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009990 tx=0x7f2cf0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 -- 172.21.15.23:0/3936911457 shutdown_connections 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3936911457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d0010d430 0x7f2d0010f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3936911457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d000731e0 0x7f2d000735c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3936911457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d0010cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 -- 172.21.15.23:0/3936911457 >> 172.21.15.23:0/3936911457 conn(0x7f2d000fc760 msgr2=0x7f2d000feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 -- 172.21.15.23:0/3936911457 shutdown_connections 2024-03-30T04:13:00.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.941+0000 7f2d06a00700 1 -- 172.21.15.23:0/3936911457 wait complete. 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.942+0000 7f2d06a00700 1 Processor -- start 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.942+0000 7f2d06a00700 1 -- start start 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0019e280 con 0x7f2d000731e0 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0019e3f0 con 0x7f2d0010d430 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d06a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0019e5b0 con 0x7f2d00073b00 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cffe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d04a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d04a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51180/0 (socket says 172.21.15.23:51180) 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cffe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:36848/0 (socket says 172.21.15.23:36848) 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2d04a00700 1 -- 172.21.15.23:0/3405313934 learned_addr learned my addr 172.21.15.23:0/3405313934 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34154/0 (socket says 172.21.15.23:34154) 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 -- 172.21.15.23:0/3405313934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 msgr2=0x7f2d0019dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 -- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 msgr2=0x7f2d00199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.943+0000 7f2cff400700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cff400700 1 -- 172.21.15.23:0/3405313934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf00095f0 con 0x7f2d00073b00 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cffe00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2d04a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:00.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cff400700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9f0 tx=0x7f2cec00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00c9c0 con 0x7f2d00073b00 2024-03-30T04:13:00.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2cec010430 con 0x7f2d00073b00 2024-03-30T04:13:00.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec018680 con 0x7f2d00073b00 2024-03-30T04:13:00.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0019e890 con 0x7f2d00073b00 2024-03-30T04:13:00.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.944+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d001a60d0 con 0x7f2d00073b00 2024-03-30T04:13:00.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.945+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d00108640 con 0x7f2d00073b00 2024-03-30T04:13:00.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.948+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2cec010860 con 0x7f2d00073b00 2024-03-30T04:13:00.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.949+0000 7f2cfcc00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 0x7f2ce40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:00.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.949+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2cec014070 con 0x7f2d00073b00 2024-03-30T04:13:00.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.949+0000 7f2cffe00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 0x7f2ce40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:00.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.950+0000 7f2cffe00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 0x7f2ce40a3750 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0006010 tx=0x7f2cf000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:00.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:00.951+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2cec096230 con 0x7f2d00073b00 2024-03-30T04:13:01.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1089232 9.86 MB/sec execute 458 sec latency 63.317 ms 2024-03-30T04:13:01.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.276+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d00074890 con 0x7f2d00073b00 2024-03-30T04:13:01.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.276+0000 7f2cfcc00700 1 -- 172.21.15.23:0/3405313934 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f2cec099850 con 0x7f2d00073b00 2024-03-30T04:13:01.276 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:13:01.276 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 msgr2=0x7f2ce40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 0x7f2ce40a3750 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0006010 tx=0x7f2cf000b260 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 msgr2=0x7f2d001998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9f0 tx=0x7f2cec00ed00 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 shutdown_connections 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ce40a1290 0x7f2ce40a3750 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d0010d430 0x7f2d0019dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d000731e0 0x7f2d00199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 --2- 172.21.15.23:0/3405313934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d00073b00 0x7f2d001998a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.279+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 >> 172.21.15.23:0/3405313934 conn(0x7f2d000fc760 msgr2=0x7f2d000fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.280+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 shutdown_connections 2024-03-30T04:13:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.280+0000 7f2d06a00700 1 -- 172.21.15.23:0/3405313934 wait complete. 2024-03-30T04:13:01.280 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:13:01.301 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:13:01.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 -- 172.21.15.23:0/2209919768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a80731e0 msgr2=0x7f41a80735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 --2- 172.21.15.23:0/2209919768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a80731e0 0x7f41a80735c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f419c009990 tx=0x7f419c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 -- 172.21.15.23:0/2209919768 shutdown_connections 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 --2- 172.21.15.23:0/2209919768 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a810d540 0x7f41a810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 --2- 172.21.15.23:0/2209919768 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a80731e0 0x7f41a80735c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 --2- 172.21.15.23:0/2209919768 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a8073b00 0x7f41a810d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 -- 172.21.15.23:0/2209919768 >> 172.21.15.23:0/2209919768 conn(0x7f41a80fc7f0 msgr2=0x7f41a80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 -- 172.21.15.23:0/2209919768 shutdown_connections 2024-03-30T04:13:01.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.553+0000 7f41af400700 1 -- 172.21.15.23:0/2209919768 wait complete. 2024-03-30T04:13:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.554+0000 7f41af400700 1 Processor -- start 2024-03-30T04:13:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 -- start start 2024-03-30T04:13:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a80731e0 0x7f41a8199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 0x7f41a8199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 0x7f41a819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a819e320 con 0x7f41a80731e0 2024-03-30T04:13:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a819e490 con 0x7f41a810d540 2024-03-30T04:13:01.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41af400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a819e650 con 0x7f41a8073b00 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 0x7f41a8199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 0x7f41a8199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34172/0 (socket says 172.21.15.23:34172) 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.555+0000 7f41a7e00700 1 -- 172.21.15.23:0/1602749888 learned_addr learned my addr 172.21.15.23:0/1602749888 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 0x7f41a819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41aca00700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a80731e0 0x7f41a8199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 -- 172.21.15.23:0/1602749888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 msgr2=0x7f41a8199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 0x7f41a8199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 -- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a80731e0 msgr2=0x7f41a8199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a80731e0 0x7f41a8199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 -- 172.21.15.23:0/1602749888 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a0009c40 con 0x7f41a810d540 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.556+0000 7f41ad400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 0x7f41a819dce0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f41a000efb0 tx=0x7f41a000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.557+0000 7f41a5600700 1 -- 172.21.15.23:0/1602749888 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0003a00 con 0x7f41a810d540 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.557+0000 7f41a5600700 1 -- 172.21.15.23:0/1602749888 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41a000f040 con 0x7f41a810d540 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.557+0000 7f41a5600700 1 -- 172.21.15.23:0/1602749888 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0004540 con 0x7f41a810d540 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.557+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c0095f0 con 0x7f41a810d540 2024-03-30T04:13:01.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.557+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a81a5f90 con 0x7f41a810d540 2024-03-30T04:13:01.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.561+0000 7f41a5600700 1 -- 172.21.15.23:0/1602749888 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f41a0003bf0 con 0x7f41a810d540 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.562+0000 7f41c3008b80 1 Processor -- start 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.562+0000 7f41c3008b80 1 -- start start 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.562+0000 7f41a5600700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 0x7f41900a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41a5600700 1 -- 172.21.15.23:0/1602749888 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f41a00ccbe0 con 0x7f41a810d540 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce9b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41a7e00700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 0x7f41900a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce9d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cde0990 con 0x555d4ce9b600 2024-03-30T04:13:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cdde350 con 0x555d4ce875d0 2024-03-30T04:13:01.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41c3008b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cdd9c40 con 0x555d4ce87ef0 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce9b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce9b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34188/0 (socket says 172.21.15.23:34188) 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41a7e00700 1 -- 172.21.15.23:0/563498408 learned_addr learned my addr 172.21.15.23:0/563498408 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.563+0000 7f41aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51206/0 (socket says 172.21.15.23:51206) 2024-03-30T04:13:01.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41ad400700 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce9d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 -- 172.21.15.23:0/563498408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 msgr2=0x555d4ce9b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce9b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 -- 172.21.15.23:0/563498408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 msgr2=0x555d4ce9d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce9d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 -- 172.21.15.23:0/563498408 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d4cc5b3b0 con 0x555d4ce875d0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41a7e00700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 0x7f41900a37d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f419400bfd0 tx=0x7f419400bd40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.564+0000 7f41aca00700 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098e0 tx=0x7f419c005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ad2acc02f724f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.565+0000 7f4199c00700 1 -- 172.21.15.23:0/563498408 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c01d070 con 0x555d4ce875d0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.565+0000 7f4199c00700 1 -- 172.21.15.23:0/563498408 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f419c01e040 con 0x555d4ce875d0 2024-03-30T04:13:01.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.565+0000 7f4199c00700 1 -- 172.21.15.23:0/563498408 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c004870 con 0x555d4ce875d0 2024-03-30T04:13:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 -- 172.21.15.23:0/563498408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 msgr2=0x555d4ce879b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098e0 tx=0x7f419c005c00 comp rx=0 tx=0).stop 2024-03-30T04:13:01.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 -- 172.21.15.23:0/563498408 shutdown_connections 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce9d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ce879b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 --2- 172.21.15.23:0/563498408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce9b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 -- 172.21.15.23:0/563498408 >> 172.21.15.23:0/563498408 conn(0x555d4cde76a0 msgr2=0x555d4ce83a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 -- 172.21.15.23:0/563498408 shutdown_connections 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.566+0000 7f41c3008b80 1 -- 172.21.15.23:0/563498408 wait complete. 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.567+0000 7f41c3008b80 1 -- start start 2024-03-30T04:13:01.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.567+0000 7f41c3008b80 1 Processor -- start 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 0x555d4ce922f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce9b600 0x555d4ce92d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cd75d80 con 0x555d4ce875d0 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cd75a20 con 0x555d4ce9b600 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4cc5d6a0 con 0x555d4ce87ef0 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41c3008b80 10 client.0 _ll_register_callbacks cb 0x555d4cd7f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34190/0 (socket says 172.21.15.23:34190) 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 -- 172.21.15.23:0/714640453 learned_addr learned my addr 172.21.15.23:0/714640453 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41ad400700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce9b600 0x555d4ce92d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41aca00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 0x555d4ce922f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 -- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce9b600 msgr2=0x555d4ce92d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce9b600 0x555d4ce92d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.568+0000 7f41a7e00700 1 -- 172.21.15.23:0/714640453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 msgr2=0x555d4ce922f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f41a7e00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 0x555d4ce922f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f41a7e00700 1 -- 172.21.15.23:0/714640453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a8110670 con 0x555d4ce87ef0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f41aca00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 0x555d4ce922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f41a7e00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f419400b9f0 tx=0x7f41940047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f33896fe43c7938 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f418e000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419400ee90 con 0x555d4ce87ef0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4194004020 con 0x555d4ce87ef0 2024-03-30T04:13:01.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.569+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d4cd9a380 con 0x555d4ce87ef0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4194012280 con 0x555d4ce87ef0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 10 client.28456 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555d4cc07b60 con 0x555d4ce87ef0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d4cd9a380 con 0x555d4ce87ef0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4194003710 con 0x555d4ce87ef0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 10 client.28456 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 10 client.28456 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 20 client.28456 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cea28b0 0x555d4cea4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 4 client.28456 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.570+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d4cdd2700 con 0x555d4cea28b0 2024-03-30T04:13:01.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.571+0000 7f41aca00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cea28b0 0x555d4cea4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.571+0000 7f41aca00700 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cea28b0 0x555d4cea4cd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86b58da057f625f8 server_cookie=fe1ad33eadde23ff in_seq=0 out_seq=0 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.571+0000 7f418e000700 10 client.28456 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.572+0000 7f418e000700 1 -- 172.21.15.23:0/714640453 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555d4cdd2700 con 0x555d4cea28b0 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.572+0000 7f418e000700 10 client.28456 handle_command_reply: tid=0 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.572+0000 7f41c3008b80 1 client.28456 shutdown 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.572+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 msgr2=0x555d4ce92830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.572+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f419400b9f0 tx=0x7f41940047c0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 shutdown_connections 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce9b600 0x555d4ce92d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce875d0 0x555d4ce922f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cea28b0 0x555d4cea4cd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 --2- 172.21.15.23:0/714640453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce92830 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 >> 172.21.15.23:0/714640453 conn(0x555d4cde76a0 msgr2=0x555d4ce896c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f418e000700 0 client.28456 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:01.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 shutdown_connections 2024-03-30T04:13:01.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 1 -- 172.21.15.23:0/714640453 wait complete. 2024-03-30T04:13:01.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.573+0000 7f41c3008b80 20 client.28456 trim_cache size 0 max 16384 2024-03-30T04:13:01.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 Processor -- start 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 -- start start 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce8d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce8dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.590+0000 7f41c3008b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ce97720 con 0x555d4ce9b600 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41c3008b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ce88640 con 0x555d4ce875d0 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41c3008b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ce93990 con 0x555d4ce87ef0 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41ad400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce8dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51224/0 (socket says 172.21.15.23:51224) 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 -- 172.21.15.23:0/2023405488 learned_addr learned my addr 172.21.15.23:0/2023405488 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41a7e00700 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce8d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 -- 172.21.15.23:0/2023405488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 msgr2=0x555d4ce8d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce8d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 -- 172.21.15.23:0/2023405488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 msgr2=0x555d4ce8dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce8dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 -- 172.21.15.23:0/2023405488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d4cd9fea0 con 0x555d4ce875d0 2024-03-30T04:13:01.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.591+0000 7f41aca00700 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f419c009990 tx=0x7f419c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29c7d0795370dc21 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.592+0000 7f419a600700 1 -- 172.21.15.23:0/2023405488 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c042070 con 0x555d4ce875d0 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.592+0000 7f419a600700 1 -- 172.21.15.23:0/2023405488 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f419c048070 con 0x555d4ce875d0 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.592+0000 7f419a600700 1 -- 172.21.15.23:0/2023405488 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c01d040 con 0x555d4ce875d0 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- 172.21.15.23:0/2023405488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 msgr2=0x555d4ceb24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f419c009990 tx=0x7f419c009310 comp rx=0 tx=0).stop 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- 172.21.15.23:0/2023405488 shutdown_connections 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ce8dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce875d0 0x555d4ceb24b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2023405488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce87ef0 0x555d4ce8d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- 172.21.15.23:0/2023405488 >> 172.21.15.23:0/2023405488 conn(0x555d4cde76a0 msgr2=0x555d4ce8cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- 172.21.15.23:0/2023405488 shutdown_connections 2024-03-30T04:13:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- 172.21.15.23:0/2023405488 wait complete. 2024-03-30T04:13:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.593+0000 7f41c3008b80 1 -- start start 2024-03-30T04:13:01.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 Processor -- start 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce87ef0 0x555d4ceaf790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ceafcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ceb0210 con 0x555d4ce9b600 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ceb0380 con 0x555d4ce87ef0 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d4ceb0540 con 0x555d4ce875d0 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.594+0000 7f41c3008b80 10 client.0 _ll_register_callbacks cb 0x555d4cd7f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34204/0 (socket says 172.21.15.23:34204) 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 -- 172.21.15.23:0/2438221196 learned_addr learned my addr 172.21.15.23:0/2438221196 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41a7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce87ef0 0x555d4ceaf790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 -- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce87ef0 msgr2=0x555d4ceaf790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce87ef0 0x555d4ceaf790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 -- 172.21.15.23:0/2438221196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 msgr2=0x555d4ceafcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ceafcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.595+0000 7f41aca00700 1 -- 172.21.15.23:0/2438221196 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d4cc5b3b0 con 0x555d4ce875d0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.596+0000 7f41aca00700 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098e0 tx=0x7f419c0442d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa4b5ab4e96ab8e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.596+0000 7f418e000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.596+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c042070 con 0x555d4ce875d0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.596+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f419c00b950 con 0x555d4ce875d0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.596+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f419c01e040 con 0x555d4ce875d0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.597+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f419c01d040 con 0x555d4ce875d0 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.597+0000 7f41c3008b80 10 client.28462 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:01.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.597+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555d4cc5fcd0 con 0x555d4ce875d0 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.597+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f419c01e040 con 0x555d4ce875d0 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f419c004c70 con 0x555d4ce875d0 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 10 client.28462 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 10 client.28462 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 20 client.28462 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cef6e50 0x555d4cef9240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 4 client.28462 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:01.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555d4cc28cf0 con 0x555d4cef6e50 2024-03-30T04:13:01.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.598+0000 7f41a7e00700 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cef6e50 0x555d4cef9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f418e000700 10 client.28462 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41a7e00700 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cef6e50 0x555d4cef9240 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d280c4ad3de10b11 server_cookie=60a75aff13f27619 in_seq=0 out_seq=0 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f418e000700 1 -- 172.21.15.23:0/2438221196 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555d4cc28cf0 con 0x555d4cef6e50 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f418e000700 10 client.28462 handle_command_reply: tid=0 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 client.28462 shutdown 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 msgr2=0x555d4ceb2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f419c0098e0 tx=0x7f419c0442d0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 shutdown_connections 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555d4ce9b600 0x555d4ceafcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555d4cef6e50 0x555d4cef9240 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555d4ce875d0 0x555d4ceb2300 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 --2- 172.21.15.23:0/2438221196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555d4ce87ef0 0x555d4ceaf790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 >> 172.21.15.23:0/2438221196 conn(0x555d4cde76a0 msgr2=0x555d4ce8d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.599+0000 7f418e000700 0 client.28462 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:01.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.600+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 shutdown_connections 2024-03-30T04:13:01.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.600+0000 7f41c3008b80 1 -- 172.21.15.23:0/2438221196 wait complete. 2024-03-30T04:13:01.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.600+0000 7f41c3008b80 20 client.28462 trim_cache size 0 max 16384 2024-03-30T04:13:01.601 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 msgr2=0x7f41900a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 0x7f41900a37d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f419400bfd0 tx=0x7f419400bd40 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 msgr2=0x7f41a819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 0x7f41a819dce0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f41a000efb0 tx=0x7f41a000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 shutdown_connections 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f41900a1310 0x7f41900a37d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41a810d540 0x7f41a819dce0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41a80731e0 0x7f41a8199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 --2- 172.21.15.23:0/1602749888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41a8073b00 0x7f41a8199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 >> 172.21.15.23:0/1602749888 conn(0x7f41a80fc7f0 msgr2=0x7f41a80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 shutdown_connections 2024-03-30T04:13:01.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.601+0000 7f41af400700 1 -- 172.21.15.23:0/1602749888 wait complete. 2024-03-30T04:13:01.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:13:01.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 -- 172.21.15.23:0/1018451428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073160 msgr2=0x7f3f78073540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.878 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 --2- 172.21.15.23:0/1018451428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073160 0x7f3f78073540 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009990 tx=0x7f3f68009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 -- 172.21.15.23:0/1018451428 shutdown_connections 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 --2- 172.21.15.23:0/1018451428 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f7810d5a0 0x7f3f7810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 --2- 172.21.15.23:0/1018451428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073160 0x7f3f78073540 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 --2- 172.21.15.23:0/1018451428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f78073a80 0x7f3f7810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 -- 172.21.15.23:0/1018451428 >> 172.21.15.23:0/1018451428 conn(0x7f3f780fc7f0 msgr2=0x7f3f780fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 -- 172.21.15.23:0/1018451428 shutdown_connections 2024-03-30T04:13:01.879 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.878+0000 7f3f7da00700 1 -- 172.21.15.23:0/1018451428 wait complete. 2024-03-30T04:13:01.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.880+0000 7f3f7da00700 1 Processor -- start 2024-03-30T04:13:01.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.880+0000 7f3f7da00700 1 -- start start 2024-03-30T04:13:01.880 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073a80 0x7f3f78199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 0x7f3f7819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819e320 con 0x7f3f78073a80 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819e490 con 0x7f3f78073160 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f7da00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819e650 con 0x7f3f7810d5a0 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f76000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073a80 0x7f3f78199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51244/0 (socket says 172.21.15.23:51244) 2024-03-30T04:13:01.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f76a00700 1 -- 172.21.15.23:0/2420462584 learned_addr learned my addr 172.21.15.23:0/2420462584 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.881+0000 7f3f77400700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 0x7f3f7819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 -- 172.21.15.23:0/2420462584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 msgr2=0x7f3f78199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 -- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073a80 msgr2=0x7f3f78199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073a80 0x7f3f78199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 -- 172.21.15.23:0/2420462584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64009520 con 0x7f3f7810d5a0 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f76a00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f77400700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 0x7f3f7819dce0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000efb0 tx=0x7f3f7000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.882+0000 7f3f6f400700 1 -- 172.21.15.23:0/2420462584 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70003a00 con 0x7f3f7810d5a0 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.883+0000 7f3f6f400700 1 -- 172.21.15.23:0/2420462584 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f7000f040 con 0x7f3f7810d5a0 2024-03-30T04:13:01.883 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.883+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f70009c40 con 0x7f3f7810d5a0 2024-03-30T04:13:01.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.883+0000 7f3f6f400700 1 -- 172.21.15.23:0/2420462584 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70004540 con 0x7f3f7810d5a0 2024-03-30T04:13:01.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.883+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f680095f0 con 0x7f3f7810d5a0 2024-03-30T04:13:01.886 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.887+0000 7f3f6f400700 1 -- 172.21.15.23:0/2420462584 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3f700046a0 con 0x7f3f7810d5a0 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.888+0000 7f3f9150ab80 1 Processor -- start 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.888+0000 7f3f6f400700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 0x7f3f5c0a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.888+0000 7f3f9150ab80 1 -- start start 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f6f400700 1 -- 172.21.15.23:0/2420462584 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f700ccb30 con 0x7f3f7810d5a0 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f76000700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 0x7f3f5c0a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b4b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b3fb990 con 0x55ac1b4addf0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b3f9350 con 0x55ac1b4a25d0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b3f4c40 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f77400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b4b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51250/0 (socket says 172.21.15.23:51250) 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.889+0000 7f3f76a00700 1 -- 172.21.15.23:0/3851141707 learned_addr learned my addr 172.21.15.23:0/3851141707 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 -- 172.21.15.23:0/3851141707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 msgr2=0x55ac1b4ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 -- 172.21.15.23:0/3851141707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 msgr2=0x55ac1b4a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 -- 172.21.15.23:0/3851141707 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac1b2763b0 con 0x55ac1b4addf0 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f76000700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 0x7f3f5c0a38a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f7819a950 tx=0x7f3f64011a80 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f76a00700 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f77400700 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b4b01e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3f700c9ea0 tx=0x7f3f700998e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b1a4132a50da20 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f63e00700 1 -- 172.21.15.23:0/3851141707 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700b2210 con 0x55ac1b4addf0 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f63e00700 1 -- 172.21.15.23:0/3851141707 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f700a1ce0 con 0x55ac1b4addf0 2024-03-30T04:13:01.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.890+0000 7f3f63e00700 1 -- 172.21.15.23:0/3851141707 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f700b90c0 con 0x55ac1b4addf0 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 -- 172.21.15.23:0/3851141707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 msgr2=0x55ac1b4b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b4b01e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3f700c9ea0 tx=0x7f3f700998e0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 -- 172.21.15.23:0/3851141707 shutdown_connections 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b4b01e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/3851141707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.891+0000 7f3f9150ab80 1 -- 172.21.15.23:0/3851141707 >> 172.21.15.23:0/3851141707 conn(0x55ac1b4026a0 msgr2=0x55ac1b49ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.892+0000 7f3f9150ab80 1 -- 172.21.15.23:0/3851141707 shutdown_connections 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.892+0000 7f3f9150ab80 1 -- 172.21.15.23:0/3851141707 wait complete. 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.892+0000 7f3f9150ab80 1 -- start start 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 Processor -- start 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4addf0 0x55ac1b4ab300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b390d80 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b390a20 con 0x55ac1b4a25d0 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b2786a0 con 0x55ac1b4addf0 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.893+0000 7f3f9150ab80 10 client.0 _ll_register_callbacks cb 0x55ac1b39a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:01.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51252/0 (socket says 172.21.15.23:51252) 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 -- 172.21.15.23:0/1281832652 learned_addr learned my addr 172.21.15.23:0/1281832652 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f77400700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4addf0 0x55ac1b4ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76000700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 -- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4addf0 msgr2=0x55ac1b4ab300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4addf0 0x55ac1b4ab300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 -- 172.21.15.23:0/1281832652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 msgr2=0x55ac1b4aabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4aabc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76a00700 1 -- 172.21.15.23:0/1281832652 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f78110650 con 0x55ac1b4a25d0 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.894+0000 7f3f76000700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f76a00700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7de019a41014ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f60c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6801d030 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f68004910 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac1b3b5380 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f68004ad0 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f9150ab80 10 client.28490 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.895+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3f680095f0 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ac1b3b5380 con 0x55ac1b4a25d0 2024-03-30T04:13:01.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f68024450 con 0x55ac1b4a25d0 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 10 client.28490 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 10 client.28490 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 20 client.28490 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b4bda90 0x55ac1b4bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 4 client.28490 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.896+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac1b3ed700 con 0x55ac1b4bda90 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.897+0000 7f3f76000700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b4bda90 0x55ac1b4bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.897+0000 7f3f76000700 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b4bda90 0x55ac1b4bfeb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ad1cbc48beaa4537 server_cookie=dec02b4a39b71803 in_seq=0 out_seq=0 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.897+0000 7f3f60c00700 10 client.28490 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:01.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f60c00700 1 -- 172.21.15.23:0/1281832652 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ac1b3ed700 con 0x55ac1b4bda90 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f60c00700 10 client.28490 handle_command_reply: tid=0 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 client.28490 shutdown 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 msgr2=0x55ac1b4ad880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006010 tx=0x7f3f68005270 comp rx=0 tx=0).stop 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 shutdown_connections 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b4bda90 0x55ac1b4bfeb0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4addf0 0x55ac1b4ab300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4ad880 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.898+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1281832652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4aabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.899+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 >> 172.21.15.23:0/1281832652 conn(0x55ac1b4026a0 msgr2=0x55ac1b4a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.899+0000 7f3f60c00700 0 client.28490 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.899+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 shutdown_connections 2024-03-30T04:13:01.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.899+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1281832652 wait complete. 2024-03-30T04:13:01.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.899+0000 7f3f9150ab80 20 client.28490 trim_cache size 0 max 16384 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 Processor -- start 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 -- start start 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b4a3570 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b4ab660 con 0x55ac1b4addf0 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.917+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b4b4fb0 con 0x55ac1b4a25d0 2024-03-30T04:13:01.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f76a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51260/0 (socket says 172.21.15.23:51260) 2024-03-30T04:13:01.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 -- 172.21.15.23:0/1461349901 learned_addr learned my addr 172.21.15.23:0/1461349901 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f76000700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 -- 172.21.15.23:0/1461349901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 msgr2=0x55ac1b4a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 -- 172.21.15.23:0/1461349901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 msgr2=0x55ac1b4b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 -- 172.21.15.23:0/1461349901 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac1b222b60 con 0x55ac1b4addf0 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f76000700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f76a00700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.918+0000 7f3f77400700 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3f70097d90 tx=0x7f3f700094e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265acebb8e0c2af server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.919+0000 7f3f6cc00700 1 -- 172.21.15.23:0/1461349901 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70096150 con 0x55ac1b4addf0 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.919+0000 7f3f6cc00700 1 -- 172.21.15.23:0/1461349901 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f700962b0 con 0x55ac1b4addf0 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.919+0000 7f3f6cc00700 1 -- 172.21.15.23:0/1461349901 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70098f30 con 0x55ac1b4addf0 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1461349901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 msgr2=0x55ac1b4975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f3f70097d90 tx=0x7f3f700094e0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1461349901 shutdown_connections 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4addf0 0x55ac1b4975a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/1461349901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1461349901 >> 172.21.15.23:0/1461349901 conn(0x55ac1b4026a0 msgr2=0x55ac1b4aa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1461349901 shutdown_connections 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- 172.21.15.23:0/1461349901 wait complete. 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.920+0000 7f3f9150ab80 1 -- start start 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 Processor -- start 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b510880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b510dc0 con 0x55ac1b4addf0 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b510f00 con 0x55ac1b4a25d0 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac1b5110c0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f9150ab80 10 client.0 _ll_register_callbacks cb 0x55ac1b39a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:01.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.921+0000 7f3f76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34266/0 (socket says 172.21.15.23:34266) 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 -- 172.21.15.23:0/2590531865 learned_addr learned my addr 172.21.15.23:0/2590531865 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f77400700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b510880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76a00700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 -- 172.21.15.23:0/2590531865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 msgr2=0x55ac1b4b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4b2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 -- 172.21.15.23:0/2590531865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 msgr2=0x55ac1b510880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b510880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 -- 172.21.15.23:0/2590531865 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac1b2763b0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f77400700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b510880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f76000700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55ac1b4addc0 tx=0x7f3f640047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e47ecf29f0ae1ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f60c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64025040 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3f64005ce0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.922+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3f700a1ce0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f6402a460 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 10 client.28480 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ac1b3baea0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f700a1ce0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3f6400f5b0 con 0x55ac1b4a2ef0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 10 client.28480 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 10 client.28480 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 20 client.28480 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b512840 0x55ac1b514c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 4 client.28480 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.923+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ac1b243cf0 con 0x55ac1b512840 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f76a00700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b512840 0x55ac1b514c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:01.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f76a00700 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b512840 0x55ac1b514c60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19bcfe4426bee95b server_cookie=1c2264693b536182 in_seq=0 out_seq=0 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f60c00700 10 client.28480 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f60c00700 1 -- 172.21.15.23:0/2590531865 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ac1b243cf0 con 0x55ac1b512840 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f60c00700 10 client.28480 handle_command_reply: tid=0 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.924+0000 7f3f9150ab80 1 client.28480 shutdown 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 msgr2=0x55ac1b4b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55ac1b4addc0 tx=0x7f3f640047c0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 shutdown_connections 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55ac1b512840 0x55ac1b514c60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ac1b4addf0 0x55ac1b510880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ac1b4a25d0 0x55ac1b4b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 --2- 172.21.15.23:0/2590531865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac1b4a2ef0 0x55ac1b4b31a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f60c00700 0 client.28480 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 >> 172.21.15.23:0/2590531865 conn(0x55ac1b4026a0 msgr2=0x55ac1b4a5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 shutdown_connections 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 1 -- 172.21.15.23:0/2590531865 wait complete. 2024-03-30T04:13:01.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.925+0000 7f3f9150ab80 20 client.28480 trim_cache size 0 max 16384 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 msgr2=0x7f3f5c0a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 0x7f3f5c0a38a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f7819a950 tx=0x7f3f64011a80 comp rx=0 tx=0).stop 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 msgr2=0x7f3f7819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 0x7f3f7819dce0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3f7000efb0 tx=0x7f3f7000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 shutdown_connections 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f7810d5a0 0x7f3f7819dce0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3f5c0a13e0 0x7f3f5c0a38a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3f78073160 0x7f3f78199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 --2- 172.21.15.23:0/2420462584 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3f78073a80 0x7f3f78199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 >> 172.21.15.23:0/2420462584 conn(0x7f3f780fc7f0 msgr2=0x7f3f780fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 shutdown_connections 2024-03-30T04:13:01.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:01.926+0000 7f3f7da00700 1 -- 172.21.15.23:0/2420462584 wait complete. 2024-03-30T04:13:01.947 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:13:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 -- 172.21.15.23:0/4279295011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3010103170 msgr2=0x7f3010103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 --2- 172.21.15.23:0/4279295011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3010103170 0x7f3010103550 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3000009990 tx=0x7f3000009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 -- 172.21.15.23:0/4279295011 shutdown_connections 2024-03-30T04:13:02.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 --2- 172.21.15.23:0/4279295011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f301010fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 --2- 172.21.15.23:0/4279295011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3010103170 0x7f3010103550 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 --2- 172.21.15.23:0/4279295011 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103b20 0x7f3010107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 -- 172.21.15.23:0/4279295011 >> 172.21.15.23:0/4279295011 conn(0x7f30100fe990 msgr2=0x7f3010100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 -- 172.21.15.23:0/4279295011 shutdown_connections 2024-03-30T04:13:02.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.191+0000 7f3016a00700 1 -- 172.21.15.23:0/4279295011 wait complete. 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.192+0000 7f3016a00700 1 Processor -- start 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 -- start start 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f3010199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f301019ddc0 0x7f301019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019e840 con 0x7f3010103170 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019e9b0 con 0x7f3010108290 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3016a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301019eb70 con 0x7f301019ddc0 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37004/0 (socket says 172.21.15.23:37004) 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f300fe00700 1 -- 172.21.15.23:0/3241434761 learned_addr learned my addr 172.21.15.23:0/3241434761 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f300f400700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f3010199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.193+0000 7f3014a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f301019ddc0 0x7f301019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 -- 172.21.15.23:0/3241434761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f301019ddc0 msgr2=0x7f301019e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f301019ddc0 0x7f301019e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 -- 172.21.15.23:0/3241434761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 msgr2=0x7f3010199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f3010199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 -- 172.21.15.23:0/3241434761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30000095f0 con 0x7f3010103170 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300fe00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300f400700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f3010199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300cc00700 1 -- 172.21.15.23:0/3241434761 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300001d070 con 0x7f3010103170 2024-03-30T04:13:02.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300cc00700 1 -- 172.21.15.23:0/3241434761 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3000022470 con 0x7f3010103170 2024-03-30T04:13:02.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30101a6070 con 0x7f3010103170 2024-03-30T04:13:02.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.194+0000 7f300cc00700 1 -- 172.21.15.23:0/3241434761 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000017650 con 0x7f3010103170 2024-03-30T04:13:02.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.195+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30101a64f0 con 0x7f3010103170 2024-03-30T04:13:02.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.198+0000 7f300cc00700 1 -- 172.21.15.23:0/3241434761 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f30000225e0 con 0x7f3010103170 2024-03-30T04:13:02.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.199+0000 7f300cc00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 0x7f2ff40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.199+0000 7f300cc00700 1 -- 172.21.15.23:0/3241434761 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f30000ce2f0 con 0x7f3010103170 2024-03-30T04:13:02.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.199+0000 7f302a564b80 1 Processor -- start 2024-03-30T04:13:02.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.199+0000 7f300f400700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 0x7f2ff40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 -- start start 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe98928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 0x559fe98951e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe97e0990 con 0x559fe9887ef0 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe97de350 con 0x559fe9892df0 2024-03-30T04:13:02.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f302a564b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe97d9c40 con 0x559fe98875d0 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f300fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f3014a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 0x559fe98951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f300f400700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 0x7f2ff40a37d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f301019aa30 tx=0x7f2ffc009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f300fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34300/0 (socket says 172.21.15.23:34300) 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f3014a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 0x559fe98951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51300/0 (socket says 172.21.15.23:51300) 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f300fe00700 1 -- 172.21.15.23:0/1610017583 learned_addr learned my addr 172.21.15.23:0/1610017583 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.200+0000 7f300f400700 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe98928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 -- 172.21.15.23:0/1610017583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 msgr2=0x559fe98951e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 0x559fe98951e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 -- 172.21.15.23:0/1610017583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 msgr2=0x559fe98928b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe98928b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 -- 172.21.15.23:0/1610017583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fe965b3b0 con 0x559fe98875d0 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f300fe00700 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3000097be0 tx=0x7f3000009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49aa919893d071be server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f3005200700 1 -- 172.21.15.23:0/1610017583 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300009aad0 con 0x559fe98875d0 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f3005200700 1 -- 172.21.15.23:0/1610017583 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f300009b870 con 0x559fe98875d0 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.201+0000 7f3005200700 1 -- 172.21.15.23:0/1610017583 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000a10c0 con 0x559fe98875d0 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 -- 172.21.15.23:0/1610017583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 msgr2=0x559fe98879b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3000097be0 tx=0x7f3000009310 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 -- 172.21.15.23:0/1610017583 shutdown_connections 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9892df0 0x559fe98951e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe98875d0 0x559fe98879b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 --2- 172.21.15.23:0/1610017583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe98928b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 -- 172.21.15.23:0/1610017583 >> 172.21.15.23:0/1610017583 conn(0x559fe97e76a0 msgr2=0x559fe9883a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 -- 172.21.15.23:0/1610017583 shutdown_connections 2024-03-30T04:13:02.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.202+0000 7f302a564b80 1 -- 172.21.15.23:0/1610017583 wait complete. 2024-03-30T04:13:02.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.203+0000 7f302a564b80 1 -- start start 2024-03-30T04:13:02.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.203+0000 7f302a564b80 1 Processor -- start 2024-03-30T04:13:02.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe988fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe9890250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe9775d80 con 0x559fe98875d0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe9775a20 con 0x559fe9887ef0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe965d6a0 con 0x559fe9892df0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f302a564b80 10 client.0 _ll_register_callbacks cb 0x559fe977f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe988fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37024/0 (socket says 172.21.15.23:37024) 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300fe00700 1 -- 172.21.15.23:0/18794184 learned_addr learned my addr 172.21.15.23:0/18794184 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f3014a00700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe9890250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 -- 172.21.15.23:0/18794184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 msgr2=0x559fe9890250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe9890250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 -- 172.21.15.23:0/18794184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 msgr2=0x559fe9892870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.204+0000 7f300f400700 1 -- 172.21.15.23:0/18794184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30101a64f0 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f300fe00700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f300f400700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe988fb60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f301006a3d0 tx=0x7f2ffc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf521393edc1c9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f2ff9600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc028670 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2ffc0287d0 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fe979a380 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ffc028990 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.205+0000 7f302a564b80 10 client.28508 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559fe9607b60 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559fe979a380 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2ffc01fc70 con 0x559fe9887ef0 2024-03-30T04:13:02.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 10 client.28508 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 10 client.28508 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 20 client.28508 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98a2730 0x559fe98a4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 4 client.28508 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fe97d2700 con 0x559fe98a2730 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.206+0000 7f300fe00700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98a2730 0x559fe98a4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.207+0000 7f300fe00700 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98a2730 0x559fe98a4b50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ecff936f4a563052 server_cookie=f462a1651d1ff53 in_seq=0 out_seq=0 2024-03-30T04:13:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.207+0000 7f2ff9600700 10 client.28508 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:02.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f2ff9600700 1 -- 172.21.15.23:0/18794184 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559fe97d2700 con 0x559fe98a2730 2024-03-30T04:13:02.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f2ff9600700 10 client.28508 handle_command_reply: tid=0 2024-03-30T04:13:02.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 client.28508 shutdown 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 msgr2=0x559fe988fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe988fb60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f301006a3d0 tx=0x7f2ffc0047c0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 shutdown_connections 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98a2730 0x559fe98a4b50 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe9890250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f2ff9600700 0 client.28508 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe9892870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 --2- 172.21.15.23:0/18794184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe988fb60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.209+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 >> 172.21.15.23:0/18794184 conn(0x559fe97e76a0 msgr2=0x559fe98896c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.210+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 shutdown_connections 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.210+0000 7f302a564b80 1 -- 172.21.15.23:0/18794184 wait complete. 2024-03-30T04:13:02.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.210+0000 7f302a564b80 20 client.28508 trim_cache size 0 max 16384 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 Processor -- start 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 -- start start 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe98997e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe987c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe98a4ee0 con 0x559fe98875d0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe9890830 con 0x559fe9887ef0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.225+0000 7f302a564b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe9899d20 con 0x559fe9892df0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37030/0 (socket says 172.21.15.23:37030) 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f3014a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe987c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe98997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 -- 172.21.15.23:0/3508652312 learned_addr learned my addr 172.21.15.23:0/3508652312 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 -- 172.21.15.23:0/3508652312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 msgr2=0x559fe987c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe987c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 -- 172.21.15.23:0/3508652312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 msgr2=0x559fe98997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe98997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 -- 172.21.15.23:0/3508652312 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fe979fea0 con 0x559fe98875d0 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300f400700 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe98997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f3014a00700 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe987c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.226+0000 7f300fe00700 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3000005820 tx=0x7f3000005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cddb0e4299c29f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f3005c00700 1 -- 172.21.15.23:0/3508652312 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300009b1f0 con 0x559fe98875d0 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f3005c00700 1 -- 172.21.15.23:0/3508652312 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30000974b0 con 0x559fe98875d0 2024-03-30T04:13:02.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f3005c00700 1 -- 172.21.15.23:0/3508652312 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000228c0 con 0x559fe98875d0 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f302a564b80 1 -- 172.21.15.23:0/3508652312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 msgr2=0x559fe988a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f302a564b80 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3000005820 tx=0x7f3000005fd0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.227+0000 7f302a564b80 1 -- 172.21.15.23:0/3508652312 shutdown_connections 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe987c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe98875d0 0x559fe988a570 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 --2- 172.21.15.23:0/3508652312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe9887ef0 0x559fe98997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 -- 172.21.15.23:0/3508652312 >> 172.21.15.23:0/3508652312 conn(0x559fe97e76a0 msgr2=0x559fe987cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 -- 172.21.15.23:0/3508652312 shutdown_connections 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 -- 172.21.15.23:0/3508652312 wait complete. 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 -- start start 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.228+0000 7f302a564b80 1 Processor -- start 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe9897ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe98f5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe98985b0 con 0x559fe9887ef0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe98f5aa0 con 0x559fe98875d0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe98f5be0 con 0x559fe9892df0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f302a564b80 10 client.0 _ll_register_callbacks cb 0x559fe977f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51334/0 (socket says 172.21.15.23:51334) 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 -- 172.21.15.23:0/837241582 learned_addr learned my addr 172.21.15.23:0/837241582 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300f400700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe9897ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f3014a00700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe98f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 -- 172.21.15.23:0/837241582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 msgr2=0x559fe98f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe98f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 -- 172.21.15.23:0/837241582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 msgr2=0x559fe9897ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe9897ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.229+0000 7f300fe00700 1 -- 172.21.15.23:0/837241582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc02d040 con 0x559fe98875d0 2024-03-30T04:13:02.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1089433 9.84 MB/sec execute 459 sec latency 88.321 ms 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f300fe00700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3000024000 tx=0x7f3000009ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec1c61b1c635d622 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30000a3560 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f300009b870 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f300009efa0 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3000099510 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 10 client.28514 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f30080115b0 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f300009b870 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f300009a660 con 0x559fe98875d0 2024-03-30T04:13:02.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 10 client.28514 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 10 client.28514 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 20 client.28514 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98f6ea0 0x559fe98f9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 4 client.28514 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.230+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559fe9628cf0 con 0x559fe98f6ea0 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f300f400700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98f6ea0 0x559fe98f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f300f400700 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98f6ea0 0x559fe98f9290 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61243f91a1038072 server_cookie=d8bcabf9b28e1313 in_seq=0 out_seq=0 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f2ff9600700 10 client.28514 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f2ff9600700 1 -- 172.21.15.23:0/837241582 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559fe9628cf0 con 0x559fe98f6ea0 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f2ff9600700 10 client.28514 handle_command_reply: tid=0 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f302a564b80 1 client.28514 shutdown 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 msgr2=0x559fe98979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.231+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3000024000 tx=0x7f3000009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 shutdown_connections 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x559fe98f6ea0 0x559fe98f9290 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559fe9892df0 0x559fe98f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559fe98875d0 0x559fe98979b0 secure :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3000024000 tx=0x7f3000009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 --2- 172.21.15.23:0/837241582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559fe9887ef0 0x559fe9897ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 >> 172.21.15.23:0/837241582 conn(0x559fe97e76a0 msgr2=0x559fe98834c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f2ff9600700 0 client.28514 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 shutdown_connections 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 1 -- 172.21.15.23:0/837241582 wait complete. 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.232+0000 7f302a564b80 20 client.28514 trim_cache size 0 max 16384 2024-03-30T04:13:02.236 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 msgr2=0x7f2ff40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 0x7f2ff40a37d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f301019aa30 tx=0x7f2ffc009260 comp rx=0 tx=0).stop 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 msgr2=0x7f30101994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3000000c00 tx=0x7f3000005740 comp rx=0 tx=0).stop 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 shutdown_connections 2024-03-30T04:13:02.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f301019ddc0 0x7f301019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3010103170 0x7f30101994e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ff40a1310 0x7f2ff40a37d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 --2- 172.21.15.23:0/3241434761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3010108290 0x7f3010199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 >> 172.21.15.23:0/3241434761 conn(0x7f30100fe990 msgr2=0x7f30100ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 shutdown_connections 2024-03-30T04:13:02.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.233+0000 7f3016a00700 1 -- 172.21.15.23:0/3241434761 wait complete. 2024-03-30T04:13:02.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:02 smithi023 ceph-mon[38254]: pgmap v345: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-03-30T04:13:02.249 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3405313934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:02.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:13:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:02 smithi154 ceph-mon[79161]: pgmap v345: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-03-30T04:13:02.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3405313934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 -- 172.21.15.23:0/3830936323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec100800 msgr2=0x7f2fec104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 --2- 172.21.15.23:0/3830936323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec100800 0x7f2fec104830 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009990 tx=0x7f2fd8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 -- 172.21.15.23:0/3830936323 shutdown_connections 2024-03-30T04:13:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 --2- 172.21.15.23:0/3830936323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec104f60 0x7f2fec10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 --2- 172.21.15.23:0/3830936323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 0x7f2fec100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 --2- 172.21.15.23:0/3830936323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec100800 0x7f2fec104830 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.470+0000 7f2ff3200700 1 -- 172.21.15.23:0/3830936323 >> 172.21.15.23:0/3830936323 conn(0x7f2fec076d40 msgr2=0x7f2fec077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.471+0000 7f2ff3200700 1 -- 172.21.15.23:0/3830936323 shutdown_connections 2024-03-30T04:13:02.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.471+0000 7f2ff3200700 1 -- 172.21.15.23:0/3830936323 wait complete. 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.471+0000 7f2ff3200700 1 Processor -- start 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 -- start start 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 0x7f2fec1018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec102350 0x7f2fec1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1a5a50 con 0x7f2fec102350 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1a5bc0 con 0x7f2fec0ffe50 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff3200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fec1a5d80 con 0x7f2fec104f60 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2feb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2febe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 0x7f2fec1018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2feb400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34328/0 (socket says 172.21.15.23:34328) 2024-03-30T04:13:02.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2feb400700 1 -- 172.21.15.23:0/1345719764 learned_addr learned my addr 172.21.15.23:0/1345719764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.472+0000 7f2ff1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec102350 0x7f2fec1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 -- 172.21.15.23:0/1345719764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 msgr2=0x7f2fec1018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 0x7f2fec1018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 -- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec102350 msgr2=0x7f2fec1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec102350 0x7f2fec1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 -- 172.21.15.23:0/1345719764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd80095f0 con 0x7f2fec104f60 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2feb400700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8000c00 tx=0x7f2fd8005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2fe8c00700 1 -- 172.21.15.23:0/1345719764 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd801d070 con 0x7f2fec104f60 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2fe8c00700 1 -- 172.21.15.23:0/1345719764 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd8022470 con 0x7f2fec104f60 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2fe8c00700 1 -- 172.21.15.23:0/1345719764 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8017650 con 0x7f2fec104f60 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fec1a6000 con 0x7f2fec104f60 2024-03-30T04:13:02.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.473+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec1a6420 con 0x7f2fec104f60 2024-03-30T04:13:02.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.476+0000 7f2fe8c00700 1 -- 172.21.15.23:0/1345719764 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2fd800fad0 con 0x7f2fec104f60 2024-03-30T04:13:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f2fe8c00700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 0x7f2fd00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f3006d7cb80 1 Processor -- start 2024-03-30T04:13:02.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f2fe8c00700 1 -- 172.21.15.23:0/1345719764 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2fd80ce350 con 0x7f2fec104f60 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f3006d7cb80 1 -- start start 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f2febe00700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 0x7f2fd00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f39d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.478+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f2f6990 con 0x55f01f3a8df0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f2f4350 con 0x55f01f39def0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f2efc40 con 0x55f01f39d5d0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2ff1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51366/0 (socket says 172.21.15.23:51366) 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2febe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f39d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 -- 172.21.15.23:0/512884767 learned_addr learned my addr 172.21.15.23:0/512884767 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2febe00700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 0x7f2fd00a37d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009cb0 tx=0x7f2fdc009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 -- 172.21.15.23:0/512884767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 msgr2=0x55f01f39d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f39d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 -- 172.21.15.23:0/512884767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 msgr2=0x55f01f3ab1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3ab1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 -- 172.21.15.23:0/512884767 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f01f1713b0 con 0x55f01f39def0 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.479+0000 7f2feb400700 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8097b20 tx=0x7f2fd800b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f19926b535179946 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.480+0000 7f2fe1600700 1 -- 172.21.15.23:0/512884767 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd809b7b0 con 0x55f01f39def0 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.480+0000 7f2fe1600700 1 -- 172.21.15.23:0/512884767 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd809b0b0 con 0x55f01f39def0 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.480+0000 7f2fe1600700 1 -- 172.21.15.23:0/512884767 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800fd80 con 0x55f01f39def0 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.480+0000 7f3006d7cb80 1 -- 172.21.15.23:0/512884767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 msgr2=0x55f01f3a88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.480+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8097b20 tx=0x7f2fd800b630 comp rx=0 tx=0).stop 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 -- 172.21.15.23:0/512884767 shutdown_connections 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3ab1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f39d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/512884767 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a88b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 -- 172.21.15.23:0/512884767 >> 172.21.15.23:0/512884767 conn(0x55f01f2fd6a0 msgr2=0x55f01f399a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 -- 172.21.15.23:0/512884767 shutdown_connections 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 -- 172.21.15.23:0/512884767 wait complete. 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 -- start start 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.481+0000 7f3006d7cb80 1 Processor -- start 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f28bd80 con 0x55f01f3a8df0 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f28ba20 con 0x55f01f39def0 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f1736a0 con 0x55f01f39d5d0 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f3006d7cb80 10 client.0 _ll_register_callbacks cb 0x55f01f295470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2febe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2febe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34342/0 (socket says 172.21.15.23:34342) 2024-03-30T04:13:02.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51374/0 (socket says 172.21.15.23:51374) 2024-03-30T04:13:02.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2febe00700 1 -- 172.21.15.23:0/3999375251 learned_addr learned my addr 172.21.15.23:0/3999375251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.482+0000 7f2ff1200700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 -- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 msgr2=0x55f01f3a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 -- 172.21.15.23:0/3999375251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 msgr2=0x55f01f3a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 -- 172.21.15.23:0/3999375251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fec1a6420 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2febe00700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009fd0 tx=0x7f2fdc0047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66ed9f77594baf38 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2fd6000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc003cf0 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2fdc025410 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.483+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f01f2b0380 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fdc003e50 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 10 client.28510 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f01f11db60 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f01f2b0380 con 0x55f01f39d5d0 2024-03-30T04:13:02.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2fdc01c660 con 0x55f01f39d5d0 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 10 client.28510 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 10 client.28510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 20 client.28510 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f3b8650 0x55f01f3baa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 4 client.28510 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.484+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f01f2e8700 con 0x55f01f3b8650 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.485+0000 7f2feb400700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f3b8650 0x55f01f3baa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.485+0000 7f2fd6000700 10 client.28510 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.485+0000 7f2feb400700 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f3b8650 0x55f01f3baa70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ade6834839d28a61 server_cookie=d95d4bca92c3f77c in_seq=0 out_seq=0 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f2fd6000700 1 -- 172.21.15.23:0/3999375251 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f01f2e8700 con 0x55f01f3b8650 2024-03-30T04:13:02.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f2fd6000700 10 client.28510 handle_command_reply: tid=0 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 client.28510 shutdown 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 msgr2=0x55f01f3a8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009fd0 tx=0x7f2fdc0047c0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 shutdown_connections 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f3a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f3b8650 0x55f01f3baa70 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3a8760 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/3999375251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 >> 172.21.15.23:0/3999375251 conn(0x55f01f2fd6a0 msgr2=0x55f01f39f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f2fd6000700 0 client.28510 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 shutdown_connections 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 1 -- 172.21.15.23:0/3999375251 wait complete. 2024-03-30T04:13:02.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.486+0000 7f3006d7cb80 20 client.28510 trim_cache size 0 max 16384 2024-03-30T04:13:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:02 smithi193 ceph-mon[79581]: pgmap v345: 129 pgs: 129 active+clean; 569 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 136 op/s 2024-03-30T04:13:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3405313934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.500+0000 7f3006d7cb80 1 Processor -- start 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.500+0000 7f3006d7cb80 1 -- start start 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f3a8df0 0x55f01f3af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f3a7050 con 0x55f01f39d5d0 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f39a660 con 0x55f01f39def0 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f3afd20 con 0x55f01f3a8df0 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2febe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2feb400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51376/0 (socket says 172.21.15.23:51376) 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2febe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37102/0 (socket says 172.21.15.23:37102) 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2feb400700 1 -- 172.21.15.23:0/93271704 learned_addr learned my addr 172.21.15.23:0/93271704 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2ff1200700 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f3a8df0 0x55f01f3af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2feb400700 1 -- 172.21.15.23:0/93271704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f3a8df0 msgr2=0x55f01f3af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.501+0000 7f2feb400700 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f3a8df0 0x55f01f3af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2feb400700 1 -- 172.21.15.23:0/93271704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 msgr2=0x55f01f3baa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2feb400700 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2feb400700 1 -- 172.21.15.23:0/93271704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f01f2b5ea0 con 0x55f01f39def0 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2febe00700 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2feb400700 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2fd809adb0 tx=0x7f2fd800b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31c1082228f67629 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2fe2000700 1 -- 172.21.15.23:0/93271704 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8083070 con 0x55f01f39def0 2024-03-30T04:13:02.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2fe2000700 1 -- 172.21.15.23:0/93271704 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd80a34a0 con 0x55f01f39def0 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.502+0000 7f2fe2000700 1 -- 172.21.15.23:0/93271704 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd80b9040 con 0x55f01f39def0 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- 172.21.15.23:0/93271704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 msgr2=0x55f01f398f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2fd809adb0 tx=0x7f2fd800b630 comp rx=0 tx=0).stop 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- 172.21.15.23:0/93271704 shutdown_connections 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f3a8df0 0x55f01f3af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f39d5d0 0x55f01f3baa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/93271704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f398f10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- 172.21.15.23:0/93271704 >> 172.21.15.23:0/93271704 conn(0x55f01f2fd6a0 msgr2=0x55f01f3925a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- 172.21.15.23:0/93271704 shutdown_connections 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- 172.21.15.23:0/93271704 wait complete. 2024-03-30T04:13:02.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.503+0000 7f3006d7cb80 1 -- start start 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 Processor -- start 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f4054c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f405a90 con 0x55f01f3a8df0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f405c00 con 0x55f01f39def0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f01f405dc0 con 0x55f01f39d5d0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f3006d7cb80 10 client.0 _ll_register_callbacks cb 0x55f01f295470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f2febe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f2febe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34350/0 (socket says 172.21.15.23:34350) 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f2febe00700 1 -- 172.21.15.23:0/2474227302 learned_addr learned my addr 172.21.15.23:0/2474227302 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f2feb400700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.504+0000 7f2ff1200700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f4054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 -- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 msgr2=0x55f01f3adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 -- 172.21.15.23:0/2474227302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 msgr2=0x55f01f4054c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f4054c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 -- 172.21.15.23:0/2474227302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f01f11db60 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2febe00700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc01ca40 tx=0x7f2fdc01ce10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d130f6b60669d9ad server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2fd6000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc005c20 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2fdc0045c0 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.505+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fd809b0b0 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fdc00f3c0 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 10 client.28516 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f2fd80b2040 con 0x55f01f39d5d0 2024-03-30T04:13:02.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fd809b0b0 con 0x55f01f39d5d0 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2fdc004a90 con 0x55f01f39d5d0 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 10 client.28516 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 10 client.28516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 20 client.28516 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f40d620 0x55f01f40fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 4 client.28516 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:02.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f01f13ecf0 con 0x55f01f40d620 2024-03-30T04:13:02.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.506+0000 7f2feb400700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f40d620 0x55f01f40fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f2feb400700 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f40d620 0x55f01f40fa40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=11e5066bde7d733a server_cookie=2717517c3b70ad3d in_seq=0 out_seq=0 2024-03-30T04:13:02.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f2fd6000700 10 client.28516 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:02.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f2fd6000700 1 -- 172.21.15.23:0/2474227302 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f01f13ecf0 con 0x55f01f40d620 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f2fd6000700 10 client.28516 handle_command_reply: tid=0 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 client.28516 shutdown 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 msgr2=0x55f01f3ad710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc01ca40 tx=0x7f2fdc01ce10 comp rx=0 tx=0).stop 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 shutdown_connections 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f01f3a8df0 0x55f01f4054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55f01f40d620 0x55f01f40fa40 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f01f39d5d0 0x55f01f3ad710 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 --2- 172.21.15.23:0/2474227302 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f01f39def0 0x55f01f3adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 >> 172.21.15.23:0/2474227302 conn(0x55f01f2fd6a0 msgr2=0x55f01f3994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.507+0000 7f2fd6000700 0 client.28516 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 shutdown_connections 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f3006d7cb80 1 -- 172.21.15.23:0/2474227302 wait complete. 2024-03-30T04:13:02.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f3006d7cb80 20 client.28516 trim_cache size 0 max 16384 2024-03-30T04:13:02.512 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:02.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 msgr2=0x7f2fd00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 0x7f2fd00a37d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009cb0 tx=0x7f2fdc009260 comp rx=0 tx=0).stop 2024-03-30T04:13:02.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 msgr2=0x7f2fec101e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8000c00 tx=0x7f2fd8005740 comp rx=0 tx=0).stop 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 shutdown_connections 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2fec102350 0x7f2fec1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2fd00a1310 0x7f2fd00a37d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fec0ffe50 0x7f2fec1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 --2- 172.21.15.23:0/1345719764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2fec104f60 0x7f2fec101e10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.508+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 >> 172.21.15.23:0/1345719764 conn(0x7f2fec076d40 msgr2=0x7f2fec068210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.509+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 shutdown_connections 2024-03-30T04:13:02.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.509+0000 7f2ff3200700 1 -- 172.21.15.23:0/1345719764 wait complete. 2024-03-30T04:13:02.537 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:13:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 -- 172.21.15.23:0/2186817013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 msgr2=0x7f522c1139c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 --2- 172.21.15.23:0/2186817013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c1139c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f52180099c0 tx=0x7f5218009cd0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 -- 172.21.15.23:0/2186817013 shutdown_connections 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 --2- 172.21.15.23:0/2186817013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c113f00 0x7f522c1162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 --2- 172.21.15.23:0/2186817013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c107db0 0x7f522c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 --2- 172.21.15.23:0/2186817013 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c1139c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.850+0000 7f5231a00700 1 -- 172.21.15.23:0/2186817013 >> 172.21.15.23:0/2186817013 conn(0x7f522c06bdf0 msgr2=0x7f522c06c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.851+0000 7f5231a00700 1 -- 172.21.15.23:0/2186817013 shutdown_connections 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.851+0000 7f5231a00700 1 -- 172.21.15.23:0/2186817013 wait complete. 2024-03-30T04:13:02.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.852+0000 7f5231a00700 1 Processor -- start 2024-03-30T04:13:02.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.853+0000 7f5231a00700 1 -- start start 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.853+0000 7f5231a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.853+0000 7f5231a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c109d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.853+0000 7f5231a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 0x7f522c10a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 0x7f522c10a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 0x7f522c10a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34368/0 (socket says 172.21.15.23:34368) 2024-03-30T04:13:02.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51402/0 (socket says 172.21.15.23:51402) 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522b400700 1 -- 172.21.15.23:0/238307675 learned_addr learned my addr 172.21.15.23:0/238307675 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1ade10 con 0x7f522c1086d0 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1adf80 con 0x7f522c107db0 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c1ae140 con 0x7f522c113f00 2024-03-30T04:13:02.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522a000700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c109d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522aa00700 1 -- 172.21.15.23:0/238307675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 msgr2=0x7f522c10a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.854+0000 7f522aa00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 0x7f522c10a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f522aa00700 1 -- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 msgr2=0x7f522c109d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f522aa00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c109d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f522aa00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180095f0 con 0x7f522c107db0 2024-03-30T04:13:02.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f522aa00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f522400bd70 tx=0x7f522400bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f5223400700 1 -- 172.21.15.23:0/238307675 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522400cc90 con 0x7f522c107db0 2024-03-30T04:13:02.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.855+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1ae420 con 0x7f522c107db0 2024-03-30T04:13:02.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.856+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c1ae950 con 0x7f522c107db0 2024-03-30T04:13:02.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.856+0000 7f5223400700 1 -- 172.21.15.23:0/238307675 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f522400cdf0 con 0x7f522c107db0 2024-03-30T04:13:02.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.856+0000 7f5223400700 1 -- 172.21.15.23:0/238307675 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224012760 con 0x7f522c107db0 2024-03-30T04:13:02.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.862+0000 7f5223400700 1 -- 172.21.15.23:0/238307675 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f52240128c0 con 0x7f522c107db0 2024-03-30T04:13:02.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.863+0000 7f5223400700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 0x7f52100a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.863+0000 7f524545db80 1 Processor -- start 2024-03-30T04:13:02.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.863+0000 7f5223400700 1 -- 172.21.15.23:0/238307675 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5224006e40 con 0x7f522c107db0 2024-03-30T04:13:02.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522a000700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 0x7f52100a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 -- start start 2024-03-30T04:13:02.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be5478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be495990 con 0x5604be547df0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be493350 con 0x5604be53c5d0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f524545db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be48ec40 con 0x5604be53cef0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51410/0 (socket says 172.21.15.23:51410) 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522aa00700 1 -- 172.21.15.23:0/3920725954 learned_addr learned my addr 172.21.15.23:0/3920725954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522a000700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 0x7f52100a3850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5218009990 tx=0x7f52180058e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.864+0000 7f522a000700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be5478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522b400700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 -- 172.21.15.23:0/3920725954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 msgr2=0x5604be5478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be5478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 -- 172.21.15.23:0/3920725954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 msgr2=0x5604be54a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 -- 172.21.15.23:0/3920725954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604be3103b0 con 0x5604be53c5d0 2024-03-30T04:13:02.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522a000700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be5478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:02.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f522aa00700 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f52240a1a10 tx=0x7f5224096000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=715859817958e071 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f5217e00700 1 -- 172.21.15.23:0/3920725954 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522409d980 con 0x5604be53c5d0 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f5217e00700 1 -- 172.21.15.23:0/3920725954 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f52240a1f40 con 0x5604be53c5d0 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.865+0000 7f5217e00700 1 -- 172.21.15.23:0/3920725954 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52240992a0 con 0x5604be53c5d0 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 -- 172.21.15.23:0/3920725954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 msgr2=0x5604be53c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f52240a1a10 tx=0x7f5224096000 comp rx=0 tx=0).stop 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 -- 172.21.15.23:0/3920725954 shutdown_connections 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be53c9b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 --2- 172.21.15.23:0/3920725954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be5478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 -- 172.21.15.23:0/3920725954 >> 172.21.15.23:0/3920725954 conn(0x5604be49c6a0 msgr2=0x5604be538a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 -- 172.21.15.23:0/3920725954 shutdown_connections 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.866+0000 7f524545db80 1 -- 172.21.15.23:0/3920725954 wait complete. 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.867+0000 7f524545db80 1 -- start start 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.867+0000 7f524545db80 1 Processor -- start 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be5477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 0x5604be544aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be42ad80 con 0x5604be53cef0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be42aa20 con 0x5604be53c5d0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be3126a0 con 0x5604be547df0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f524545db80 10 client.0 _ll_register_callbacks cb 0x5604be434470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522a000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 0x5604be544aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34382/0 (socket says 172.21.15.23:34382) 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522b400700 1 -- 172.21.15.23:0/4222314146 learned_addr learned my addr 172.21.15.23:0/4222314146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522aa00700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be5477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.868+0000 7f522b400700 1 -- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 msgr2=0x5604be5477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522b400700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be5477d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522b400700 1 -- 172.21.15.23:0/4222314146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 msgr2=0x5604be544aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522b400700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 0x5604be544aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522b400700 1 -- 172.21.15.23:0/4222314146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c1ae950 con 0x5604be547df0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522a000700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 0x5604be544aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522aa00700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be5477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f522b400700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f521c00bec0 tx=0x7f521c0116c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4fd596bd6280fe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f5214c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c011ee0 con 0x5604be547df0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604be44f380 con 0x5604be547df0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f521c012450 con 0x5604be547df0 2024-03-30T04:13:02.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f521c012610 con 0x5604be547df0 2024-03-30T04:13:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.869+0000 7f524545db80 10 client.28525 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.870+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5604be2bcb60 con 0x5604be547df0 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.870+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604be44f380 con 0x5604be547df0 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.870+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f521c025a00 con 0x5604be547df0 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 10 client.28525 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 10 client.28525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 20 client.28525 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be557a20 0x5604be559e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 4 client.28525 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604be487700 con 0x5604be557a20 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f522aa00700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be557a20 0x5604be559e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f5214c00700 10 client.28525 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.871+0000 7f522aa00700 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be557a20 0x5604be559e40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab29bc8f5590acbe server_cookie=ac57faabd6b737b1 in_seq=0 out_seq=0 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.872+0000 7f5214c00700 1 -- 172.21.15.23:0/4222314146 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5604be487700 con 0x5604be557a20 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.872+0000 7f5214c00700 10 client.28525 handle_command_reply: tid=0 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.872+0000 7f524545db80 1 client.28525 shutdown 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 msgr2=0x5604be545090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f521c00bec0 tx=0x7f521c0116c0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 shutdown_connections 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be557a20 0x5604be559e40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be547df0 0x5604be545090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be5477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 --2- 172.21.15.23:0/4222314146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be53cef0 0x5604be544aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 >> 172.21.15.23:0/4222314146 conn(0x5604be49c6a0 msgr2=0x5604be53e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f5214c00700 0 client.28525 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 shutdown_connections 2024-03-30T04:13:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 1 -- 172.21.15.23:0/4222314146 wait complete. 2024-03-30T04:13:02.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.873+0000 7f524545db80 20 client.28525 trim_cache size 0 max 16384 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 Processor -- start 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 -- start start 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be53fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be5402f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be545da0 con 0x5604be547df0 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be555150 con 0x5604be53c5d0 2024-03-30T04:13:02.887 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.887+0000 7f524545db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be5453f0 con 0x5604be53cef0 2024-03-30T04:13:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51432/0 (socket says 172.21.15.23:51432) 2024-03-30T04:13:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 -- 172.21.15.23:0/3451632037 learned_addr learned my addr 172.21.15.23:0/3451632037 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522a000700 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be53fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.888 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522b400700 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be5402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 -- 172.21.15.23:0/3451632037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 msgr2=0x5604be53fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be53fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 -- 172.21.15.23:0/3451632037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 msgr2=0x5604be5402f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be5402f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 -- 172.21.15.23:0/3451632037 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604be454ea0 con 0x5604be53c5d0 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f522aa00700 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f522400bd40 tx=0x7f52240999d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62039faa2feb9ea server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.888+0000 7f5220c00700 1 -- 172.21.15.23:0/3451632037 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224099240 con 0x5604be53c5d0 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f5220c00700 1 -- 172.21.15.23:0/3451632037 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5224097ff0 con 0x5604be53c5d0 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f5220c00700 1 -- 172.21.15.23:0/3451632037 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52240a1ab0 con 0x5604be53c5d0 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 -- 172.21.15.23:0/3451632037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 msgr2=0x5604be54eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f522400bd40 tx=0x7f52240999d0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 -- 172.21.15.23:0/3451632037 shutdown_connections 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be5402f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53c5d0 0x5604be54eb00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 --2- 172.21.15.23:0/3451632037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53cef0 0x5604be53fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 -- 172.21.15.23:0/3451632037 >> 172.21.15.23:0/3451632037 conn(0x5604be49c6a0 msgr2=0x5604be544000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 -- 172.21.15.23:0/3451632037 shutdown_connections 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.889+0000 7f524545db80 1 -- 172.21.15.23:0/3451632037 wait complete. 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 -- start start 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 Processor -- start 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53c5d0 0x5604be54cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53cef0 0x5604be54d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be5aa890 con 0x5604be547df0 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be5aaa00 con 0x5604be53cef0 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604be5aabc0 con 0x5604be53c5d0 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.890+0000 7f524545db80 10 client.0 _ll_register_callbacks cb 0x5604be434470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:02.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37170/0 (socket says 172.21.15.23:37170) 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 -- 172.21.15.23:0/2848419520 learned_addr learned my addr 172.21.15.23:0/2848419520 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522a000700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53cef0 0x5604be54d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522aa00700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53c5d0 0x5604be54cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 -- 172.21.15.23:0/2848419520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53c5d0 msgr2=0x5604be54cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53c5d0 0x5604be54cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 -- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53cef0 msgr2=0x5604be54d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53cef0 0x5604be54d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 -- 172.21.15.23:0/2848419520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604be3103b0 con 0x5604be547df0 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f522b400700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f521c025de0 tx=0x7f521c005ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453cd9aaa6fa7867 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:02.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.891+0000 7f5214c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:02.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c005450 con 0x5604be547df0 2024-03-30T04:13:02.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604be2ddcf0 con 0x5604be547df0 2024-03-30T04:13:02.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f521c0055b0 con 0x5604be547df0 2024-03-30T04:13:02.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f521c025470 con 0x5604be547df0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f524545db80 10 client.18567 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5604be314cd0 con 0x5604be547df0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604be2ddcf0 con 0x5604be547df0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f521c037070 con 0x5604be547df0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f524545db80 10 client.18567 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.892+0000 7f524545db80 10 client.18567 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f524545db80 20 client.18567 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be5ac390 0x5604be5ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f524545db80 4 client.18567 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5604be312ce0 con 0x5604be5ac390 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f522aa00700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be5ac390 0x5604be5ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f522aa00700 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be5ac390 0x5604be5ae7b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a90bcf93e8cb6b6 server_cookie=690b09b48329dfab in_seq=0 out_seq=0 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f5214c00700 10 client.18567 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:02.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f5214c00700 1 -- 172.21.15.23:0/2848419520 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5604be312ce0 con 0x5604be5ac390 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f5214c00700 10 client.18567 handle_command_reply: tid=0 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.893+0000 7f524545db80 1 client.18567 shutdown 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 msgr2=0x5604be54d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f521c025de0 tx=0x7f521c005ba0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 shutdown_connections 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5604be547df0 0x5604be54d560 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5604be5ac390 0x5604be5ae7b0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604be53c5d0 0x5604be54cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 --2- 172.21.15.23:0/2848419520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5604be53cef0 0x5604be54d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f5214c00700 0 client.18567 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 >> 172.21.15.23:0/2848419520 conn(0x5604be49c6a0 msgr2=0x5604be53fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 shutdown_connections 2024-03-30T04:13:02.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 1 -- 172.21.15.23:0/2848419520 wait complete. 2024-03-30T04:13:02.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.894+0000 7f524545db80 20 client.18567 trim_cache size 0 max 16384 2024-03-30T04:13:02.901 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 msgr2=0x7f52100a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 0x7f52100a3850 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5218009990 tx=0x7f52180058e0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 msgr2=0x7f522c109820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f522400bd70 tx=0x7f522400bda0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 shutdown_connections 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f522c113f00 0x7f522c10a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f52100a1390 0x7f52100a3850 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f522c107db0 0x7f522c109820 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 --2- 172.21.15.23:0/238307675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f522c1086d0 0x7f522c109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 >> 172.21.15.23:0/238307675 conn(0x7f522c06bdf0 msgr2=0x7f522c06fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 shutdown_connections 2024-03-30T04:13:02.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:02.895+0000 7f5231a00700 1 -- 172.21.15.23:0/238307675 wait complete. 2024-03-30T04:13:03.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1090646 9.84 MB/sec execute 460 sec latency 82.981 ms 2024-03-30T04:13:03.913 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:13:03.913 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 -- 172.21.15.23:0/1746873432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 msgr2=0x7f68ac101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 --2- 172.21.15.23:0/1746873432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac101340 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f68a0009990 tx=0x7f68a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 -- 172.21.15.23:0/1746873432 shutdown_connections 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 --2- 172.21.15.23:0/1746873432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac106080 0x7f68ac10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 --2- 172.21.15.23:0/1746873432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac101340 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 --2- 172.21.15.23:0/1746873432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac101910 0x7f68ac105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 -- 172.21.15.23:0/1746873432 >> 172.21.15.23:0/1746873432 conn(0x7f68ac0fc7c0 msgr2=0x7f68ac0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 -- 172.21.15.23:0/1746873432 shutdown_connections 2024-03-30T04:13:04.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.133+0000 7f68b4200700 1 -- 172.21.15.23:0/1746873432 wait complete. 2024-03-30T04:13:04.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 Processor -- start 2024-03-30T04:13:04.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 -- start start 2024-03-30T04:13:04.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 0x7f68ac1977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac19bb50 0x7f68ac19bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac19c5d0 con 0x7f68ac19bb50 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac19c740 con 0x7f68ac106080 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.135+0000 7f68b4200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac19c900 con 0x7f68ac100f60 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b1800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b1800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34398/0 (socket says 172.21.15.23:34398) 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b1800700 1 -- 172.21.15.23:0/1601988848 learned_addr learned my addr 172.21.15.23:0/1601988848 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 0x7f68ac1977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b2200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac19bb50 0x7f68ac19bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 -- 172.21.15.23:0/1601988848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 msgr2=0x7f68ac197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 -- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac19bb50 msgr2=0x7f68ac19bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac19bb50 0x7f68ac19bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 -- 172.21.15.23:0/1601988848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00095f0 con 0x7f68ac106080 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b1800700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.136+0000 7f68b0e00700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 0x7f68ac1977b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9c0 tx=0x7f689800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.137+0000 7f68aa000700 1 -- 172.21.15.23:0/1601988848 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689800c9c0 con 0x7f68ac106080 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.137+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac1a3e60 con 0x7f68ac106080 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.137+0000 7f68aa000700 1 -- 172.21.15.23:0/1601988848 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6898010430 con 0x7f68ac106080 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.137+0000 7f68aa000700 1 -- 172.21.15.23:0/1601988848 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898018680 con 0x7f68ac106080 2024-03-30T04:13:04.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.137+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1a42e0 con 0x7f68ac106080 2024-03-30T04:13:04.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.140+0000 7f68aa000700 1 -- 172.21.15.23:0/1601988848 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f689800cb20 con 0x7f68ac106080 2024-03-30T04:13:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.141+0000 7f68aa000700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 0x7f68940a7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.141+0000 7f68c7db5b80 1 Processor -- start 2024-03-30T04:13:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.141+0000 7f68aa000700 1 -- 172.21.15.23:0/1601988848 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f6898014070 con 0x7f68ac106080 2024-03-30T04:13:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.141+0000 7f68c7db5b80 1 -- start start 2024-03-30T04:13:04.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b1800700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 0x7f68940a7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd81a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd8c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563ccdaa10 con 0x55563cd81f70 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563ccd83d0 con 0x55563cd81650 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563ccd3cc0 con 0x55563cd8ce70 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd81a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd8c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd81a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51474/0 (socket says 172.21.15.23:51474) 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b1800700 1 -- 172.21.15.23:0/2624733235 learned_addr learned my addr 172.21.15.23:0/2624733235 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.142+0000 7f68b2200700 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b1800700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 0x7f68940a7b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f68a0006010 tx=0x7f68a0004580 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 -- 172.21.15.23:0/2624733235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 msgr2=0x55563cd8f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 -- 172.21.15.23:0/2624733235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 msgr2=0x55563cd81a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd81a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 -- 172.21.15.23:0/2624733235 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55563cb55430 con 0x55563cd81f70 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f68b0e00700 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd8c930 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f68980ca510 tx=0x7f6898096340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25948be1e0af4a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f689ea00700 1 -- 172.21.15.23:0/2624733235 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6898099ed0 con 0x55563cd81f70 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f689ea00700 1 -- 172.21.15.23:0/2624733235 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f689809dd00 con 0x55563cd81f70 2024-03-30T04:13:04.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.143+0000 7f689ea00700 1 -- 172.21.15.23:0/2624733235 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689809a130 con 0x55563cd81f70 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 -- 172.21.15.23:0/2624733235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 msgr2=0x55563cd8c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd8c930 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f68980ca510 tx=0x7f6898096340 comp rx=0 tx=0).stop 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 -- 172.21.15.23:0/2624733235 shutdown_connections 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd81a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/2624733235 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd8c930 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 -- 172.21.15.23:0/2624733235 >> 172.21.15.23:0/2624733235 conn(0x55563cce1720 msgr2=0x55563cd7db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 -- 172.21.15.23:0/2624733235 shutdown_connections 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.144+0000 7f68c7db5b80 1 -- 172.21.15.23:0/2624733235 wait complete. 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 -- start start 2024-03-30T04:13:04.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 Processor -- start 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd8c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cc6fe00 con 0x55563cd81650 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cc6faa0 con 0x55563cd81f70 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.145+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cb57720 con 0x55563cd8ce70 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68c7db5b80 10 client.0 _ll_register_callbacks cb 0x55563cc794f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b1800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd8c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51482/0 (socket says 172.21.15.23:51482) 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 -- 172.21.15.23:0/3095320432 learned_addr learned my addr 172.21.15.23:0/3095320432 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b2200700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 -- 172.21.15.23:0/3095320432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 msgr2=0x55563cd8a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 -- 172.21.15.23:0/3095320432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 msgr2=0x55563cd8c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd8c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.146+0000 7f68b0e00700 1 -- 172.21.15.23:0/3095320432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1a42e0 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f68980caa90 tx=0x7f68980a23e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dce45244b0362ee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f6893400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68980105a0 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f689809a130 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55563cc94400 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f689809a310 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f68c7db5b80 10 client.28562 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.147+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55563cb01be0 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6898063070 con 0x55563cd81f70 2024-03-30T04:13:04.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6898095cc0 con 0x55563cd81f70 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 10 client.28562 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 10 client.28562 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 20 client.28562 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cd9c6a0 0x55563cd9eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 4 client.28562 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55563cccc780 con 0x55563cd9c6a0 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.148+0000 7f68b1800700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cd9c6a0 0x55563cd9eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.149+0000 7f6893400700 10 client.28562 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.149+0000 7f68b1800700 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cd9c6a0 0x55563cd9eac0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=258992713fb4d177 server_cookie=16123308b160b3e5 in_seq=0 out_seq=0 2024-03-30T04:13:04.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.149+0000 7f6893400700 1 -- 172.21.15.23:0/3095320432 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55563cccc780 con 0x55563cd9c6a0 2024-03-30T04:13:04.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.149+0000 7f6893400700 10 client.28562 handle_command_reply: tid=0 2024-03-30T04:13:04.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.149+0000 7f68c7db5b80 1 client.28562 shutdown 2024-03-30T04:13:04.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 msgr2=0x55563cd89b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f68980caa90 tx=0x7f68980a23e0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 shutdown_connections 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cd9c6a0 0x55563cd9eac0 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd8a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd8c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3095320432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81f70 0x55563cd89b70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f6893400700 0 client.28562 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 >> 172.21.15.23:0/3095320432 conn(0x55563cce1720 msgr2=0x55563cd83710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 shutdown_connections 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3095320432 wait complete. 2024-03-30T04:13:04.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.150+0000 7f68c7db5b80 20 client.28562 trim_cache size 0 max 16384 2024-03-30T04:13:04.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 Processor -- start 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 -- start start 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cd9ee50 con 0x55563cd81f70 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cd82390 con 0x55563cd81650 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.170+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cd93fd0 con 0x55563cd8ce70 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b0e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b2200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51486/0 (socket says 172.21.15.23:51486) 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 -- 172.21.15.23:0/1733059661 learned_addr learned my addr 172.21.15.23:0/1733059661 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b2200700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34430/0 (socket says 172.21.15.23:34430) 2024-03-30T04:13:04.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 -- 172.21.15.23:0/1733059661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 msgr2=0x55563cd89320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 -- 172.21.15.23:0/1733059661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 msgr2=0x55563cd988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 -- 172.21.15.23:0/1733059661 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55563cda36b0 con 0x55563cd81650 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b2200700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b0e00700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.171+0000 7f68b1800700 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f68a000b690 tx=0x7f68a0022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=291397b3664b61d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f689f400700 1 -- 172.21.15.23:0/1733059661 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0036e10 con 0x55563cd81650 2024-03-30T04:13:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f689f400700 1 -- 172.21.15.23:0/1733059661 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68a001e040 con 0x55563cd81650 2024-03-30T04:13:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f689f400700 1 -- 172.21.15.23:0/1733059661 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a00336b0 con 0x55563cd81650 2024-03-30T04:13:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 -- 172.21.15.23:0/1733059661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 msgr2=0x55563cd93a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f68a000b690 tx=0x7f68a0022000 comp rx=0 tx=0).stop 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 -- 172.21.15.23:0/1733059661 shutdown_connections 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd8ce70 0x55563cd89320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd81650 0x55563cd93a90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/1733059661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81f70 0x55563cd988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.172+0000 7f68c7db5b80 1 -- 172.21.15.23:0/1733059661 >> 172.21.15.23:0/1733059661 conn(0x55563cce1720 msgr2=0x55563cd981e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.173+0000 7f68c7db5b80 1 -- 172.21.15.23:0/1733059661 shutdown_connections 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.173+0000 7f68c7db5b80 1 -- 172.21.15.23:0/1733059661 wait complete. 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.173+0000 7f68c7db5b80 1 -- start start 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.173+0000 7f68c7db5b80 1 Processor -- start 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd81f70 0x55563cd92970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 0x55563cdf04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cd93280 con 0x55563cd81650 2024-03-30T04:13:04.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cdf09e0 con 0x55563cd8ce70 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563cdf0b20 con 0x55563cd81f70 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68c7db5b80 10 client.0 _ll_register_callbacks cb 0x55563cc794f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68b1800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68b1800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37214/0 (socket says 172.21.15.23:37214) 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68b1800700 1 -- 172.21.15.23:0/3817183840 learned_addr learned my addr 172.21.15.23:0/3817183840 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68b2200700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 0x55563cdf04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.174+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd81f70 0x55563cd92970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 -- 172.21.15.23:0/3817183840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd81f70 msgr2=0x55563cd92970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd81f70 0x55563cd92970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 -- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 msgr2=0x55563cdf04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 0x55563cdf04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 -- 172.21.15.23:0/3817183840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a4009a20 con 0x55563cd81650 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b2200700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 0x55563cdf04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68b1800700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f68a000f4f0 tx=0x7f68a000fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ca4ccd3bc8cbf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f6893400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a0036450 con 0x55563cd81650 2024-03-30T04:13:04.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f68a002c040 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.175+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f689809dd00 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68a000fb60 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 10 client.18585 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f68ac10e470 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f689809dd00 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f68a003f430 con 0x55563cd81650 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 10 client.18585 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 10 client.18585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 20 client.18585 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cdf1e80 0x55563cdf42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 4 client.18585 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.176+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55563cb22d70 con 0x55563cdf1e80 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.177+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cdf1e80 0x55563cdf42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.177+0000 7f68b0e00700 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cdf1e80 0x55563cdf42a0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dfe8ff8af43d276 server_cookie=cc2f7720cfb1b479 in_seq=0 out_seq=0 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.177+0000 7f6893400700 10 client.18585 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.178+0000 7f6893400700 1 -- 172.21.15.23:0/3817183840 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55563cb22d70 con 0x55563cdf1e80 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.178+0000 7f6893400700 10 client.18585 handle_command_reply: tid=0 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.178+0000 7f68c7db5b80 1 client.18585 shutdown 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 msgr2=0x55563cd92430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f68a000f4f0 tx=0x7f68a000fd10 comp rx=0 tx=0).stop 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 shutdown_connections 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55563cd8ce70 0x55563cdf04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55563cd81650 0x55563cd92430 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55563cdf1e80 0x55563cdf42a0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 --2- 172.21.15.23:0/3817183840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55563cd81f70 0x55563cd92970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f6893400700 0 client.18585 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 >> 172.21.15.23:0/3817183840 conn(0x55563cce1720 msgr2=0x55563cd98a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 shutdown_connections 2024-03-30T04:13:04.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 1 -- 172.21.15.23:0/3817183840 wait complete. 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.179+0000 7f68c7db5b80 20 client.18585 trim_cache size 0 max 16384 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "fd560aef-80f4-42dc-a0a7-14b553ff5918", 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:13:04.181 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 msgr2=0x7f68940a7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 0x7f68940a7b60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f68a0006010 tx=0x7f68a0004580 comp rx=0 tx=0).stop 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 msgr2=0x7f68ac1977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 0x7f68ac1977b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f689800e9c0 tx=0x7f689800ecd0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 shutdown_connections 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f68ac19bb50 0x7f68ac19bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68ac100f60 0x7f68ac197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68940a56a0 0x7f68940a7b60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.182 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 --2- 172.21.15.23:0/1601988848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68ac106080 0x7f68ac1977b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 >> 172.21.15.23:0/1601988848 conn(0x7f68ac0fc7c0 msgr2=0x7f68ac0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 shutdown_connections 2024-03-30T04:13:04.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.180+0000 7f68b4200700 1 -- 172.21.15.23:0/1601988848 wait complete. 2024-03-30T04:13:04.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:13:04.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1091936 9.83 MB/sec execute 461 sec latency 31.623 ms 2024-03-30T04:13:04.258 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:04 smithi023 ceph-mon[38254]: pgmap v346: 129 pgs: 129 active+clean; 572 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-30T04:13:04.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 -- 172.21.15.23:0/4066412594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c0731e0 msgr2=0x7f681c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 --2- 172.21.15.23:0/4066412594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c0731e0 0x7f681c0735c0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6810009990 tx=0x7f6810009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 -- 172.21.15.23:0/4066412594 shutdown_connections 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 --2- 172.21.15.23:0/4066412594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 0x7f681c10f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 --2- 172.21.15.23:0/4066412594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c0731e0 0x7f681c0735c0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 --2- 172.21.15.23:0/4066412594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c073b00 0x7f681c10d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 -- 172.21.15.23:0/4066412594 >> 172.21.15.23:0/4066412594 conn(0x7f681c0fc7f0 msgr2=0x7f681c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.409+0000 7f6823400700 1 -- 172.21.15.23:0/4066412594 shutdown_connections 2024-03-30T04:13:04.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.410+0000 7f6823400700 1 -- 172.21.15.23:0/4066412594 wait complete. 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 Processor -- start 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 -- start start 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 0x7f681c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 0x7f681c19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c19e320 con 0x7f681c0731e0 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c19e490 con 0x7f681c073b00 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6823400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c19e650 con 0x7f681c10d540 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f681be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f681be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51516/0 (socket says 172.21.15.23:51516) 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6820a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 0x7f681c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f681be00700 1 -- 172.21.15.23:0/429597608 learned_addr learned my addr 172.21.15.23:0/429597608 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.411+0000 7f6821400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 0x7f681c19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 -- 172.21.15.23:0/429597608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 msgr2=0x7f681c19dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 0x7f681c19dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 -- 172.21.15.23:0/429597608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 msgr2=0x7f681c199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 -- 172.21.15.23:0/429597608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68100095f0 con 0x7f681c0731e0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f681be00700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6820a00700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 0x7f681c199400 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f68100098e0 tx=0x7f6810005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6819600700 1 -- 172.21.15.23:0/429597608 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681001d070 con 0x7f681c0731e0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6819600700 1 -- 172.21.15.23:0/429597608 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6810022470 con 0x7f681c0731e0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c19e8d0 con 0x7f681c0731e0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.412+0000 7f6819600700 1 -- 172.21.15.23:0/429597608 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6810017650 con 0x7f681c0731e0 2024-03-30T04:13:04.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.413+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c1a6170 con 0x7f681c0731e0 2024-03-30T04:13:04.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.417+0000 7f6819600700 1 -- 172.21.15.23:0/429597608 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f68100225e0 con 0x7f681c0731e0 2024-03-30T04:13:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.418+0000 7f6836f42b80 1 Processor -- start 2024-03-30T04:13:04.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.418+0000 7f6836f42b80 1 -- start start 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.418+0000 7f6819600700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 0x7f68040a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.418+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6836f42b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b414990 con 0x55e71b4c6df0 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6836f42b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b412350 con 0x55e71b4bb5d0 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6836f42b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b40dc40 con 0x55e71b4bbef0 2024-03-30T04:13:04.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6819600700 1 -- 172.21.15.23:0/429597608 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f68100ce490 con 0x7f681c0731e0 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6821400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34476/0 (socket says 172.21.15.23:34476) 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 -- 172.21.15.23:0/219541775 learned_addr learned my addr 172.21.15.23:0/219541775 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6821400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 0x7f68040a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f6820a00700 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 -- 172.21.15.23:0/219541775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 msgr2=0x55e71b4bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4bb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 -- 172.21.15.23:0/219541775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 msgr2=0x55e71b4c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.419+0000 7f681be00700 1 -- 172.21.15.23:0/219541775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e71b28f3b0 con 0x55e71b4bbef0 2024-03-30T04:13:04.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.420+0000 7f681be00700 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6808011680 tx=0x7f6808011990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=365fd9cdc42a1a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.420+0000 7f6821400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 0x7f68040a3750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6814009ec0 tx=0x7f681400b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.420+0000 7f680dc00700 1 -- 172.21.15.23:0/219541775 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808004030 con 0x55e71b4bbef0 2024-03-30T04:13:04.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.420+0000 7f680dc00700 1 -- 172.21.15.23:0/219541775 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6808003680 con 0x55e71b4bbef0 2024-03-30T04:13:04.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.420+0000 7f680dc00700 1 -- 172.21.15.23:0/219541775 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68080196b0 con 0x55e71b4bbef0 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 -- 172.21.15.23:0/219541775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 msgr2=0x55e71b4c68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6808011680 tx=0x7f6808011990 comp rx=0 tx=0).stop 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 -- 172.21.15.23:0/219541775 shutdown_connections 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 --2- 172.21.15.23:0/219541775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c68b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 -- 172.21.15.23:0/219541775 >> 172.21.15.23:0/219541775 conn(0x55e71b41b6a0 msgr2=0x55e71b4b7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 -- 172.21.15.23:0/219541775 shutdown_connections 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.421+0000 7f6836f42b80 1 -- 172.21.15.23:0/219541775 wait complete. 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.422+0000 7f6836f42b80 1 -- start start 2024-03-30T04:13:04.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.422+0000 7f6836f42b80 1 Processor -- start 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b3a9d80 con 0x55e71b4c6df0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b3a9a20 con 0x55e71b4bb5d0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b2916a0 con 0x55e71b4bbef0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6836f42b80 10 client.0 _ll_register_callbacks cb 0x55e71b3b3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6820a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6820a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51546/0 (socket says 172.21.15.23:51546) 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34480/0 (socket says 172.21.15.23:34480) 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6820a00700 1 -- 172.21.15.23:0/1886648505 learned_addr learned my addr 172.21.15.23:0/1886648505 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f6821400700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 -- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 msgr2=0x55e71b4c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 -- 172.21.15.23:0/1886648505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 msgr2=0x55e71b4c40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.423+0000 7f681be00700 1 -- 172.21.15.23:0/1886648505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6808009e10 con 0x55e71b4bbef0 2024-03-30T04:13:04.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f681be00700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6808000c00 tx=0x7f68080041a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e51e50c4be1ba374 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6802000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:04.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808005ce0 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f6808005030 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e71b3ce380 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68080051f0 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6836f42b80 10 client.28561 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.424+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e71b23bb60 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e71b3ce380 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6808019450 con 0x55e71b4bbef0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 10 client.28561 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 10 client.28561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 20 client.28561 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b4df070 0x55e71b4e1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 4 client.28561 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e71b406700 con 0x55e71b4df070 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.425+0000 7f6820a00700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b4df070 0x55e71b4e1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.426+0000 7f6820a00700 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b4df070 0x55e71b4e1490 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=865938d37de0fcfd server_cookie=7ebc202666d6c8df in_seq=0 out_seq=0 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.426+0000 7f6802000700 10 client.28561 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.426+0000 7f6802000700 1 -- 172.21.15.23:0/1886648505 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e71b406700 con 0x55e71b4df070 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.426+0000 7f6802000700 10 client.28561 handle_command_reply: tid=0 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 client.28561 shutdown 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 msgr2=0x55e71b4c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f6808000c00 tx=0x7f68080041a0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 shutdown_connections 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4c40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b4df070 0x55e71b4e1490 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 --2- 172.21.15.23:0/1886648505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4c3ab0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6802000700 0 client.28561 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 >> 172.21.15.23:0/1886648505 conn(0x55e71b41b6a0 msgr2=0x55e71b4bd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 shutdown_connections 2024-03-30T04:13:04.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 1 -- 172.21.15.23:0/1886648505 wait complete. 2024-03-30T04:13:04.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.427+0000 7f6836f42b80 20 client.28561 trim_cache size 0 max 16384 2024-03-30T04:13:04.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:04 smithi154 ceph-mon[79161]: pgmap v346: 129 pgs: 129 active+clean; 572 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-30T04:13:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 Processor -- start 2024-03-30T04:13:04.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 -- start start 2024-03-30T04:13:04.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4d6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4c50b0 con 0x55e71b4c6df0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4bc850 con 0x55e71b4bb5d0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.446+0000 7f6836f42b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4b8660 con 0x55e71b4bbef0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f6820a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f6821400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4d6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f6820a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51554/0 (socket says 172.21.15.23:51554) 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f6820a00700 1 -- 172.21.15.23:0/2577121050 learned_addr learned my addr 172.21.15.23:0/2577121050 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 -- 172.21.15.23:0/2577121050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 msgr2=0x55e71b4c4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 -- 172.21.15.23:0/2577121050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 msgr2=0x55e71b4b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 -- 172.21.15.23:0/2577121050 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e71b3d3ea0 con 0x55e71b4bbef0 2024-03-30T04:13:04.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.447+0000 7f681be00700 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4d6300 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6808011ce0 tx=0x7f68080199b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf6208807cb30825 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f680e600700 1 -- 172.21.15.23:0/2577121050 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808004210 con 0x55e71b4bbef0 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f680e600700 1 -- 172.21.15.23:0/2577121050 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6808019a80 con 0x55e71b4bbef0 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f680e600700 1 -- 172.21.15.23:0/2577121050 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808005070 con 0x55e71b4bbef0 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 -- 172.21.15.23:0/2577121050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 msgr2=0x55e71b4d6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4d6300 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f6808011ce0 tx=0x7f68080199b0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 -- 172.21.15.23:0/2577121050 shutdown_connections 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bb5d0 0x55e71b4c4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2577121050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bbef0 0x55e71b4d6300 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 -- 172.21.15.23:0/2577121050 >> 172.21.15.23:0/2577121050 conn(0x55e71b41b6a0 msgr2=0x55e71b4c3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 -- 172.21.15.23:0/2577121050 shutdown_connections 2024-03-30T04:13:04.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.448+0000 7f6836f42b80 1 -- 172.21.15.23:0/2577121050 wait complete. 2024-03-30T04:13:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.449+0000 7f6836f42b80 1 -- start start 2024-03-30T04:13:04.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.449+0000 7f6836f42b80 1 Processor -- start 2024-03-30T04:13:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bb5d0 0x55e71b4d4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4d5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4d6000 con 0x55e71b4c6df0 2024-03-30T04:13:04.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4d6170 con 0x55e71b4bbef0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e71b4d6330 con 0x55e71b4bb5d0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6836f42b80 10 client.0 _ll_register_callbacks cb 0x55e71b3b3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f681be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6821400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4d5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f681be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51556/0 (socket says 172.21.15.23:51556) 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f681be00700 1 -- 172.21.15.23:0/2167227178 learned_addr learned my addr 172.21.15.23:0/2167227178 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.450+0000 7f6820a00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bb5d0 0x55e71b4d4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 -- 172.21.15.23:0/2167227178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bb5d0 msgr2=0x55e71b4d4600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bb5d0 0x55e71b4d4600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 -- 172.21.15.23:0/2167227178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 msgr2=0x55e71b4d5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4d5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 -- 172.21.15.23:0/2167227178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e71b23bb60 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f6821400700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f681be00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f68080115a0 tx=0x7f68080183b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f903edfd7def5557 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f6802000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.451+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6808019650 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f6808000e40 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6808003680 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68080185f0 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 10 client.28586 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e71b293cd0 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6808003680 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6808011a00 con 0x55e71b4bbef0 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 10 client.28586 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 10 client.28586 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:04.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 20 client.28586 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b533cd0 0x55e71b5360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 4 client.28586 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.452+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e71b25ccf0 con 0x55e71b533cd0 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6820a00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b533cd0 0x55e71b5360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6820a00700 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b533cd0 0x55e71b5360f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbb946e412d8394d server_cookie=d024c2f37a665ae1 in_seq=0 out_seq=0 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6802000700 10 client.28586 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6802000700 1 -- 172.21.15.23:0/2167227178 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e71b25ccf0 con 0x55e71b533cd0 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6802000700 10 client.28586 handle_command_reply: tid=0 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6836f42b80 1 client.28586 shutdown 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 msgr2=0x55e71b4d4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f68080115a0 tx=0x7f68080183b0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.453+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 shutdown_connections 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e71b4c6df0 0x55e71b4d5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55e71b533cd0 0x55e71b5360f0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e71b4bb5d0 0x55e71b4d4600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 --2- 172.21.15.23:0/2167227178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e71b4bbef0 0x55e71b4d4b40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6802000700 0 client.28586 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 >> 172.21.15.23:0/2167227178 conn(0x55e71b41b6a0 msgr2=0x55e71b4c3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 shutdown_connections 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 1 -- 172.21.15.23:0/2167227178 wait complete. 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6836f42b80 20 client.28586 trim_cache size 0 max 16384 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "fd560aef-80f4-42dc-a0a7-14b553ff5918": { 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:13:04.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "fd560aef-80f4-42dc-a0a7-14b553ff5918", 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 msgr2=0x7f68040a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 0x7f68040a3750 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6814009ec0 tx=0x7f681400b040 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 msgr2=0x7f681c199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.454+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 0x7f681c199400 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f68100098e0 tx=0x7f6810005740 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 shutdown_connections 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f681c10d540 0x7f681c19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f681c0731e0 0x7f681c199400 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f68040a1290 0x7f68040a3750 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 --2- 172.21.15.23:0/429597608 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f681c073b00 0x7f681c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 >> 172.21.15.23:0/429597608 conn(0x7f681c0fc7f0 msgr2=0x7f681c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 shutdown_connections 2024-03-30T04:13:04.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:04.455+0000 7f6823400700 1 -- 172.21.15.23:0/429597608 wait complete. 2024-03-30T04:13:04.472 INFO:tasks.cephfs.filesystem:scrub status for tag:fd560aef-80f4-42dc-a0a7-14b553ff5918 - {'path': '/', 'tag': 'fd560aef-80f4-42dc-a0a7-14b553ff5918', 'options': 'recursive,force'} 2024-03-30T04:13:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:04 smithi193 ceph-mon[79581]: pgmap v346: 129 pgs: 129 active+clean; 572 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 206 op/s 2024-03-30T04:13:05.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1094061 9.84 MB/sec execute 462 sec latency 19.413 ms 2024-03-30T04:13:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:13:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:13:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:13:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:05 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:13:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:13:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:13:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:05 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:05 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:06.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1094861 9.82 MB/sec execute 463 sec latency 28.916 ms 2024-03-30T04:13:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:06 smithi193 ceph-mon[79581]: pgmap v347: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:13:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:06 smithi023 ceph-mon[38254]: pgmap v347: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:13:06.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:06 smithi154 ceph-mon[79161]: pgmap v347: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 170 op/s 2024-03-30T04:13:07.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1096551 9.82 MB/sec execute 464 sec latency 38.722 ms 2024-03-30T04:13:08.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1097974 9.82 MB/sec execute 465 sec latency 109.465 ms 2024-03-30T04:13:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:08 smithi193 ceph-mon[79581]: pgmap v348: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:13:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:08 smithi023 ceph-mon[38254]: pgmap v348: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:13:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:08 smithi154 ceph-mon[79161]: pgmap v348: 129 pgs: 129 active+clean; 572 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:13:09.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1099268 9.81 MB/sec execute 466 sec latency 38.133 ms 2024-03-30T04:13:10.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1101277 9.82 MB/sec execute 467 sec latency 51.267 ms 2024-03-30T04:13:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:10 smithi193 ceph-mon[79581]: pgmap v349: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-30T04:13:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:10 smithi193 ceph-mon[79581]: osdmap e100: 12 total, 12 up, 12 in 2024-03-30T04:13:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:10 smithi023 ceph-mon[38254]: pgmap v349: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-30T04:13:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:10 smithi023 ceph-mon[38254]: osdmap e100: 12 total, 12 up, 12 in 2024-03-30T04:13:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:10 smithi154 ceph-mon[79161]: pgmap v349: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 238 op/s 2024-03-30T04:13:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:10 smithi154 ceph-mon[79161]: osdmap e100: 12 total, 12 up, 12 in 2024-03-30T04:13:11.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1102597 9.81 MB/sec execute 468 sec latency 30.809 ms 2024-03-30T04:13:12.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1104702 9.82 MB/sec execute 469 sec latency 19.884 ms 2024-03-30T04:13:12.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:12 smithi154 ceph-mon[79161]: pgmap v351: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 203 op/s 2024-03-30T04:13:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:12 smithi193 ceph-mon[79581]: pgmap v351: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 203 op/s 2024-03-30T04:13:12.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:12 smithi023 ceph-mon[38254]: pgmap v351: 129 pgs: 129 active+clean; 574 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.5 MiB/s wr, 203 op/s 2024-03-30T04:13:13.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1105377 9.80 MB/sec execute 470 sec latency 79.357 ms 2024-03-30T04:13:14.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1106988 9.79 MB/sec execute 471 sec latency 45.421 ms 2024-03-30T04:13:14.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:14 smithi154 ceph-mon[79161]: pgmap v352: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-03-30T04:13:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:14 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:13:14.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:14 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:13:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:14 smithi193 ceph-mon[79581]: pgmap v352: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-03-30T04:13:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:14 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:13:14.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:14 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:13:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:14 smithi023 ceph-mon[38254]: pgmap v352: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 201 op/s 2024-03-30T04:13:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:14 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:13:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:14 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:13:15.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1108357 9.79 MB/sec execute 472 sec latency 34.770 ms 2024-03-30T04:13:16.229 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1109346 9.78 MB/sec execute 473 sec latency 30.433 ms 2024-03-30T04:13:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:16 smithi154 ceph-mon[79161]: pgmap v353: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 172 op/s 2024-03-30T04:13:16.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:16 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:13:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:16 smithi193 ceph-mon[79581]: pgmap v353: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 172 op/s 2024-03-30T04:13:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:16 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:13:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:16 smithi023 ceph-mon[38254]: pgmap v353: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 172 op/s 2024-03-30T04:13:16.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:16 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:13:17.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1111539 9.79 MB/sec execute 474 sec latency 25.940 ms 2024-03-30T04:13:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:17 smithi154 ceph-mon[79161]: osdmap e101: 12 total, 12 up, 12 in 2024-03-30T04:13:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:17 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:13:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:17 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:13:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:17 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:13:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:17 smithi193 ceph-mon[79581]: osdmap e101: 12 total, 12 up, 12 in 2024-03-30T04:13:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:17 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:13:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:17 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:13:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:17 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:13:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:17 smithi023 ceph-mon[38254]: osdmap e101: 12 total, 12 up, 12 in 2024-03-30T04:13:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:17 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:13:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:17 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:13:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:17 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:13:18.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1112574 9.78 MB/sec execute 475 sec latency 52.212 ms 2024-03-30T04:13:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:18 smithi193 ceph-mon[79581]: pgmap v355: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 108 op/s 2024-03-30T04:13:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:18 smithi023 ceph-mon[38254]: pgmap v355: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 108 op/s 2024-03-30T04:13:18.558 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:18 smithi154 ceph-mon[79161]: pgmap v355: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.2 MiB/s wr, 108 op/s 2024-03-30T04:13:19.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1114269 9.77 MB/sec execute 476 sec latency 41.348 ms 2024-03-30T04:13:20.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1115647 9.77 MB/sec execute 477 sec latency 19.480 ms 2024-03-30T04:13:20.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:20 smithi154 ceph-mon[79161]: pgmap v356: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-03-30T04:13:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:20 smithi193 ceph-mon[79581]: pgmap v356: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-03-30T04:13:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:20 smithi023 ceph-mon[38254]: pgmap v356: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.9 MiB/s wr, 189 op/s 2024-03-30T04:13:21.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1116995 9.77 MB/sec execute 478 sec latency 22.956 ms 2024-03-30T04:13:22.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1118979 9.77 MB/sec execute 479 sec latency 20.220 ms 2024-03-30T04:13:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:22 smithi193 ceph-mon[79581]: pgmap v357: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-03-30T04:13:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:22 smithi023 ceph-mon[38254]: pgmap v357: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-03-30T04:13:22.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:22 smithi154 ceph-mon[79161]: pgmap v357: 129 pgs: 129 active+clean; 573 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.5 MiB/s wr, 171 op/s 2024-03-30T04:13:23.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1119700 9.75 MB/sec execute 480 sec latency 84.645 ms 2024-03-30T04:13:24.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1121477 9.75 MB/sec execute 481 sec latency 44.237 ms 2024-03-30T04:13:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:24 smithi193 ceph-mon[79581]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:24 smithi023 ceph-mon[38254]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:24 smithi154 ceph-mon[79161]: pgmap v358: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:25.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1122800 9.75 MB/sec execute 482 sec latency 19.808 ms 2024-03-30T04:13:26.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1124123 9.74 MB/sec execute 483 sec latency 23.783 ms 2024-03-30T04:13:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:26 smithi193 ceph-mon[79581]: pgmap v359: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:26.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:26 smithi023 ceph-mon[38254]: pgmap v359: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:26.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:26 smithi154 ceph-mon[79161]: pgmap v359: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.4 MiB/s wr, 168 op/s 2024-03-30T04:13:27.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1126113 9.75 MB/sec execute 484 sec latency 22.340 ms 2024-03-30T04:13:28.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1127376 9.74 MB/sec execute 485 sec latency 46.237 ms 2024-03-30T04:13:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:28 smithi193 ceph-mon[79581]: pgmap v360: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-30T04:13:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:28 smithi023 ceph-mon[38254]: pgmap v360: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-30T04:13:28.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:28 smithi154 ceph-mon[79161]: pgmap v360: 129 pgs: 129 active+clean; 583 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-03-30T04:13:29.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1129150 9.74 MB/sec execute 486 sec latency 44.386 ms 2024-03-30T04:13:30.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1130075 9.73 MB/sec execute 487 sec latency 28.606 ms 2024-03-30T04:13:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:30 smithi193 ceph-mon[79581]: pgmap v361: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 214 op/s 2024-03-30T04:13:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:30 smithi023 ceph-mon[38254]: pgmap v361: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 214 op/s 2024-03-30T04:13:30.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:30 smithi154 ceph-mon[79161]: pgmap v361: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.3 MiB/s wr, 214 op/s 2024-03-30T04:13:31.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1131859 9.73 MB/sec execute 488 sec latency 34.785 ms 2024-03-30T04:13:32.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1133324 9.73 MB/sec execute 489 sec latency 22.976 ms 2024-03-30T04:13:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:32 smithi154 ceph-mon[79161]: pgmap v362: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 143 op/s 2024-03-30T04:13:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:32 smithi193 ceph-mon[79581]: pgmap v362: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 143 op/s 2024-03-30T04:13:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:32 smithi023 ceph-mon[38254]: pgmap v362: 129 pgs: 129 active+clean; 576 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.8 MiB/s wr, 143 op/s 2024-03-30T04:13:33.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1134629 9.72 MB/sec execute 490 sec latency 31.035 ms 2024-03-30T04:13:34.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1136395 9.72 MB/sec execute 491 sec latency 73.431 ms 2024-03-30T04:13:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:34 smithi154 ceph-mon[79161]: pgmap v363: 129 pgs: 129 active+clean; 569 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-30T04:13:34.473 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:13:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:34 smithi193 ceph-mon[79581]: pgmap v363: 129 pgs: 129 active+clean; 569 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-30T04:13:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:34 smithi023 ceph-mon[38254]: pgmap v363: 129 pgs: 129 active+clean; 569 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 209 op/s 2024-03-30T04:13:34.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 -- 172.21.15.23:0/2489541420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 msgr2=0x7f662c073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 --2- 172.21.15.23:0/2489541420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c073560 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6620009990 tx=0x7f6620009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 -- 172.21.15.23:0/2489541420 shutdown_connections 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 --2- 172.21.15.23:0/2489541420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c10d5a0 0x7f662c10f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 --2- 172.21.15.23:0/2489541420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c073560 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 --2- 172.21.15.23:0/2489541420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c073aa0 0x7f662c10d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 -- 172.21.15.23:0/2489541420 >> 172.21.15.23:0/2489541420 conn(0x7f662c0fc7f0 msgr2=0x7f662c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 -- 172.21.15.23:0/2489541420 shutdown_connections 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.687+0000 7f6634600700 1 -- 172.21.15.23:0/2489541420 wait complete. 2024-03-30T04:13:34.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.688+0000 7f6634600700 1 Processor -- start 2024-03-30T04:13:34.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 -- start start 2024-03-30T04:13:34.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c073aa0 0x7f662c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c10d5a0 0x7f662c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c19e3a0 con 0x7f662c10d5a0 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c19e510 con 0x7f662c073aa0 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6634600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662c19e6d0 con 0x7f662c073180 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6631c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6631200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c073aa0 0x7f662c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6632600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c10d5a0 0x7f662c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6631c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57564/0 (socket says 172.21.15.23:57564) 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.689+0000 7f6631c00700 1 -- 172.21.15.23:0/426288459 learned_addr learned my addr 172.21.15.23:0/426288459 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 -- 172.21.15.23:0/426288459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c073aa0 msgr2=0x7f662c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c073aa0 0x7f662c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 -- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c10d5a0 msgr2=0x7f662c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c10d5a0 0x7f662c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 -- 172.21.15.23:0/426288459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200095f0 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6631c00700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6620005910 tx=0x7f6620004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f662a000700 1 -- 172.21.15.23:0/426288459 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662001d070 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f662a000700 1 -- 172.21.15.23:0/426288459 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6620022470 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662c19e950 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f662a000700 1 -- 172.21.15.23:0/426288459 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620017650 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.690+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f662c1a61f0 con 0x7f662c073180 2024-03-30T04:13:34.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.693+0000 7f662a000700 1 -- 172.21.15.23:0/426288459 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f66200225e0 con 0x7f662c073180 2024-03-30T04:13:34.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.694+0000 7f664803ab80 1 Processor -- start 2024-03-30T04:13:34.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 -- start start 2024-03-30T04:13:34.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f662a000700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 0x7f66140a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f662a000700 1 -- 172.21.15.23:0/426288459 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f66200ce2a0 con 0x7f662c073180 2024-03-30T04:13:34.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d35d0 0x561f987d39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987e11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f9872c990 con 0x561f987d3ef0 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f9872a350 con 0x561f987dedf0 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f664803ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f98725c40 con 0x561f987d35d0 2024-03-30T04:13:34.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6632600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 0x7f66140a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6631200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6631200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58202/0 (socket says 172.21.15.23:58202) 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6631200700 1 -- 172.21.15.23:0/2274154599 learned_addr learned my addr 172.21.15.23:0/2274154599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6631c00700 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d35d0 0x561f987d39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.695+0000 7f6632600700 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987e11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 -- 172.21.15.23:0/2274154599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d35d0 msgr2=0x561f987d39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d35d0 0x561f987d39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 -- 172.21.15.23:0/2274154599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 msgr2=0x561f987e11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987e11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 -- 172.21.15.23:0/2274154599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f985a73b0 con 0x561f987d3ef0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6632600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 0x7f66140a37d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f662c06a3d0 tx=0x7f662400fc00 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f6631200700 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6618009d20 tx=0x7f6618011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6286d6d5faeb492b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.696+0000 7f661ea00700 1 -- 172.21.15.23:0/2274154599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618019630 con 0x561f987d3ef0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f661ea00700 1 -- 172.21.15.23:0/2274154599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6618019c50 con 0x561f987d3ef0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f661ea00700 1 -- 172.21.15.23:0/2274154599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661800eac0 con 0x561f987d3ef0 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 -- 172.21.15.23:0/2274154599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 msgr2=0x561f987de8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6618009d20 tx=0x7f6618011720 comp rx=0 tx=0).stop 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 -- 172.21.15.23:0/2274154599 shutdown_connections 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987e11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d35d0 0x561f987d39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 --2- 172.21.15.23:0/2274154599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987de8b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 -- 172.21.15.23:0/2274154599 >> 172.21.15.23:0/2274154599 conn(0x561f987336a0 msgr2=0x561f987cfa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 -- 172.21.15.23:0/2274154599 shutdown_connections 2024-03-30T04:13:34.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.697+0000 7f664803ab80 1 -- 172.21.15.23:0/2274154599 wait complete. 2024-03-30T04:13:34.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.698+0000 7f664803ab80 1 -- start start 2024-03-30T04:13:34.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.698+0000 7f664803ab80 1 Processor -- start 2024-03-30T04:13:34.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987dbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987dc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f986c1d80 con 0x561f987d35d0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f986c1a20 con 0x561f987dedf0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f985a96a0 con 0x561f987d3ef0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f664803ab80 10 client.0 _ll_register_callbacks cb 0x561f986cb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6632600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987dc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987dbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58216/0 (socket says 172.21.15.23:58216) 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6632600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987dc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37022/0 (socket says 172.21.15.23:37022) 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 -- 172.21.15.23:0/919206397 learned_addr learned my addr 172.21.15.23:0/919206397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 -- 172.21.15.23:0/919206397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 msgr2=0x561f987dbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987dbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 -- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 msgr2=0x561f987dc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987dc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.699+0000 7f6631c00700 1 -- 172.21.15.23:0/919206397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6618009980 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6631c00700 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f662009b0b0 tx=0x7f6620022c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f2f3f5c3142ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662009d070 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f662009b820 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561f986e6380 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f662009aa80 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f664803ab80 10 client.18615 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561f98553b60 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561f986e6380 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.700+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f662009b380 con 0x561f987d35d0 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 10 client.18615 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:34.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 10 client.18615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 20 client.18615 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f987ee560 0x561f987f0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 4 client.18615 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561f9871e700 con 0x561f987ee560 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f6631200700 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f987ee560 0x561f987f0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f6631200700 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f987ee560 0x561f987f0980 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62b3bb628101c3f server_cookie=8f11fa7c8e627bc9 in_seq=0 out_seq=0 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.701+0000 7f6613400700 10 client.18615 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f6613400700 1 -- 172.21.15.23:0/919206397 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x561f9871e700 con 0x561f987ee560 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f6613400700 10 client.18615 handle_command_reply: tid=0 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 client.18615 shutdown 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 msgr2=0x561f987de780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f662009b0b0 tx=0x7f6620022c90 comp rx=0 tx=0).stop 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 shutdown_connections 2024-03-30T04:13:34.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987dc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987de780 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f987ee560 0x561f987f0980 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 --2- 172.21.15.23:0/919206397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987dbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f6613400700 0 client.18615 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.702+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 >> 172.21.15.23:0/919206397 conn(0x561f987336a0 msgr2=0x561f987d56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.703+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 shutdown_connections 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.703+0000 7f664803ab80 1 -- 172.21.15.23:0/919206397 wait complete. 2024-03-30T04:13:34.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.703+0000 7f664803ab80 20 client.18615 trim_cache size 0 max 16384 2024-03-30T04:13:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.720+0000 7f664803ab80 1 Processor -- start 2024-03-30T04:13:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.720+0000 7f664803ab80 1 -- start start 2024-03-30T04:13:34.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987e5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987de2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f987d5140 con 0x561f987d35d0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f987d0660 con 0x561f987dedf0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f664803ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f987dc6b0 con 0x561f987d3ef0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6632600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987de2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6631200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6631c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987e5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6631200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57606/0 (socket says 172.21.15.23:57606) 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6632600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987de2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37024/0 (socket says 172.21.15.23:37024) 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.721+0000 7f6631200700 1 -- 172.21.15.23:0/457129271 learned_addr learned my addr 172.21.15.23:0/457129271 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 -- 172.21.15.23:0/457129271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 msgr2=0x561f987de2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987de2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 -- 172.21.15.23:0/457129271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 msgr2=0x561f987e5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987e5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 -- 172.21.15.23:0/457129271 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561f986ebea0 con 0x561f987d3ef0 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631c00700 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987e5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f6631200700 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6618009920 tx=0x7f66180058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c68fd72300b9bdd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f661f400700 1 -- 172.21.15.23:0/457129271 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618031070 con 0x561f987d3ef0 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f661f400700 1 -- 172.21.15.23:0/457129271 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6618019dc0 con 0x561f987d3ef0 2024-03-30T04:13:34.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.722+0000 7f661f400700 1 -- 172.21.15.23:0/457129271 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661802a040 con 0x561f987d3ef0 2024-03-30T04:13:34.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- 172.21.15.23:0/457129271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 msgr2=0x561f987c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6618009920 tx=0x7f66180058e0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- 172.21.15.23:0/457129271 shutdown_connections 2024-03-30T04:13:34.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987dedf0 0x561f987de2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d35d0 0x561f987e5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 --2- 172.21.15.23:0/457129271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987d3ef0 0x561f987c8f70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- 172.21.15.23:0/457129271 >> 172.21.15.23:0/457129271 conn(0x561f987336a0 msgr2=0x561f987c85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- 172.21.15.23:0/457129271 shutdown_connections 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- 172.21.15.23:0/457129271 wait complete. 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 -- start start 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.723+0000 7f664803ab80 1 Processor -- start 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987d35d0 0x561f987d60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987dedf0 0x561f987e5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f987e58f0 con 0x561f987d3ef0 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f98842360 con 0x561f987d35d0 2024-03-30T04:13:34.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561f988424a0 con 0x561f987dedf0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f664803ab80 10 client.0 _ll_register_callbacks cb 0x561f986cb470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f6631200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f6632600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987dedf0 0x561f987e5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f6631c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987d35d0 0x561f987d60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f6631200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58234/0 (socket says 172.21.15.23:58234) 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.724+0000 7f6631200700 1 -- 172.21.15.23:0/3552593838 learned_addr learned my addr 172.21.15.23:0/3552593838 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 -- 172.21.15.23:0/3552593838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987dedf0 msgr2=0x561f987e5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987dedf0 0x561f987e5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 -- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987d35d0 msgr2=0x561f987d60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987d35d0 0x561f987d60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 -- 172.21.15.23:0/3552593838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561f98553b60 con 0x561f987d3ef0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6631200700 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6618019f50 tx=0x7f6618011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c990d8d963a84d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6613400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618031070 con 0x561f987d3ef0 2024-03-30T04:13:34.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f661802d560 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.725+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6618019c50 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f661802a050 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 10 client.18621 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x561f985a73b0 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6618019c50 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f661802d820 con 0x561f987d3ef0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 10 client.18621 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 10 client.18621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 20 client.18621 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f98843870 0x561f98845c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 4 client.18621 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561f98574cf0 con 0x561f98843870 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.726+0000 7f6631c00700 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f98843870 0x561f98845c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f6631c00700 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f98843870 0x561f98845c60 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fb5971116cc01bd server_cookie=ffebd40ba7b1481e in_seq=0 out_seq=0 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f6613400700 10 client.18621 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f6613400700 1 -- 172.21.15.23:0/3552593838 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561f98574cf0 con 0x561f98843870 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f6613400700 10 client.18621 handle_command_reply: tid=0 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 client.18621 shutdown 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 msgr2=0x561f987e4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6618019f50 tx=0x7f6618011b20 comp rx=0 tx=0).stop 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 shutdown_connections 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x561f98843870 0x561f98845c60 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561f987dedf0 0x561f987e5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x561f987d35d0 0x561f987d60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f6613400700 0 client.18621 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 --2- 172.21.15.23:0/3552593838 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x561f987d3ef0 0x561f987e4ad0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.727+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 >> 172.21.15.23:0/3552593838 conn(0x561f987336a0 msgr2=0x561f987dba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 shutdown_connections 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f664803ab80 1 -- 172.21.15.23:0/3552593838 wait complete. 2024-03-30T04:13:34.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f664803ab80 20 client.18621 trim_cache size 0 max 16384 2024-03-30T04:13:34.731 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:34.731 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:13:34.731 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:13:34.731 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 msgr2=0x7f66140a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 0x7f66140a37d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f662c06a3d0 tx=0x7f662400fc00 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 msgr2=0x7f662c199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6620005910 tx=0x7f6620004730 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 shutdown_connections 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f662c10d5a0 0x7f662c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f662c073180 0x7f662c199480 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66140a1310 0x7f66140a37d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 --2- 172.21.15.23:0/426288459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f662c073aa0 0x7f662c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 >> 172.21.15.23:0/426288459 conn(0x7f662c0fc7f0 msgr2=0x7f662c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.728+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 shutdown_connections 2024-03-30T04:13:34.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.729+0000 7f6634600700 1 -- 172.21.15.23:0/426288459 wait complete. 2024-03-30T04:13:34.745 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:13:34.746 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:13:34.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 -- 172.21.15.23:0/4106068287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 msgr2=0x7f136810c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 --2- 172.21.15.23:0/4106068287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f136810c9b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f136000b0a0 tx=0x7f136000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 -- 172.21.15.23:0/4106068287 shutdown_connections 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 --2- 172.21.15.23:0/4106068287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f136810c9b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 --2- 172.21.15.23:0/4106068287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1368105b80 0x7f1368105f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 --2- 172.21.15.23:0/4106068287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f13680ffae0 0x7f13680fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 -- 172.21.15.23:0/4106068287 >> 172.21.15.23:0/4106068287 conn(0x7f13680fb5f0 msgr2=0x7f13680fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.954+0000 7f136f400700 1 -- 172.21.15.23:0/4106068287 shutdown_connections 2024-03-30T04:13:34.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.955+0000 7f136f400700 1 -- 172.21.15.23:0/4106068287 wait complete. 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.955+0000 7f136f400700 1 Processor -- start 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 -- start start 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 0x7f13680fbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1368105b80 0x7f13680fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13681a53a0 con 0x7f1368105b80 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13681a5510 con 0x7f13681004a0 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136f400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13681a56d0 con 0x7f13680ffae0 2024-03-30T04:13:34.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f1367e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 0x7f13680fbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f1367e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37052/0 (socket says 172.21.15.23:37052) 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f1367e00700 1 -- 172.21.15.23:0/3477556281 learned_addr learned my addr 172.21.15.23:0/3477556281 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.956+0000 7f136d400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1368105b80 0x7f13680fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 -- 172.21.15.23:0/3477556281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 msgr2=0x7f13680fbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 0x7f13680fbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 -- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1368105b80 msgr2=0x7f13680fc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1368105b80 0x7f13680fc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 -- 172.21.15.23:0/3477556281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1354009520 con 0x7f13681004a0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f136ca00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 0x7f13680fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1367e00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f135400eb30 tx=0x7f135400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354009880 con 0x7f13681004a0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13540044e0 con 0x7f13681004a0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360009d00 con 0x7f13681004a0 2024-03-30T04:13:34.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354003d80 con 0x7f13681004a0 2024-03-30T04:13:34.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.957+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13681a5d30 con 0x7f13681004a0 2024-03-30T04:13:34.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.958+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1368109070 con 0x7f13681004a0 2024-03-30T04:13:34.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.960+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1354003f20 con 0x7f13681004a0 2024-03-30T04:13:34.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.961+0000 7f1365600700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 0x7f13500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:34.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.962+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1354014070 con 0x7f13681004a0 2024-03-30T04:13:34.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.962+0000 7f136ca00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 0x7f13500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:34.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.962+0000 7f136ca00700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 0x7f13500a3750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f135c009fd0 tx=0x7f135c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:34.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:34.964+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f1354096a90 con 0x7f13681004a0 2024-03-30T04:13:35.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1137247 9.71 MB/sec execute 492 sec latency 26.312 ms 2024-03-30T04:13:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.250+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13680fd0c0 con 0x7f13681004a0 2024-03-30T04:13:35.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.251+0000 7f1365600700 1 -- 172.21.15.23:0/3477556281 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7f135409a0b0 con 0x7f13681004a0 2024-03-30T04:13:35.250 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:13:35.250 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 msgr2=0x7f13500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 0x7f13500a3750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f135c009fd0 tx=0x7f135c009260 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 msgr2=0x7f13680fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f135400eb30 tx=0x7f135400ee40 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 shutdown_connections 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1368105b80 0x7f13680fc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f13680ffae0 0x7f13680fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f13500a1290 0x7f13500a3750 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 --2- 172.21.15.23:0/3477556281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13681004a0 0x7f13680fc120 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 >> 172.21.15.23:0/3477556281 conn(0x7f13680fb5f0 msgr2=0x7f1368068150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 shutdown_connections 2024-03-30T04:13:35.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.254+0000 7f136f400700 1 -- 172.21.15.23:0/3477556281 wait complete. 2024-03-30T04:13:35.254 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:13:35.274 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:13:35.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 -- 172.21.15.23:0/2789627804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b40ffaf0 msgr2=0x7f66b410d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2789627804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b40ffaf0 0x7f66b410d1d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009990 tx=0x7f66a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 -- 172.21.15.23:0/2789627804 shutdown_connections 2024-03-30T04:13:35.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2789627804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b410d710 0x7f66b410fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2789627804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ff1d0 0x7f66b40ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2789627804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b40ffaf0 0x7f66b410d1d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 -- 172.21.15.23:0/2789627804 >> 172.21.15.23:0/2789627804 conn(0x7f66b4074b20 msgr2=0x7f66b4074f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.514+0000 7f66b9e00700 1 -- 172.21.15.23:0/2789627804 shutdown_connections 2024-03-30T04:13:35.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.515+0000 7f66b9e00700 1 -- 172.21.15.23:0/2789627804 wait complete. 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 Processor -- start 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 -- start start 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ffaf0 0x7f66b4199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b410d710 0x7f66b419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b419e350 con 0x7f66b40ffaf0 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b419e4c0 con 0x7f66b410d710 2024-03-30T04:13:35.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b9e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b419e680 con 0x7f66b40ff1d0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ffaf0 0x7f66b4199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57642/0 (socket says 172.21.15.23:57642) 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.516+0000 7f66b2a00700 1 -- 172.21.15.23:0/2816560458 learned_addr learned my addr 172.21.15.23:0/2816560458 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b3400700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b410d710 0x7f66b419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 -- 172.21.15.23:0/2816560458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b410d710 msgr2=0x7f66b419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b410d710 0x7f66b419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 -- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ffaf0 msgr2=0x7f66b4199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ffaf0 0x7f66b4199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 -- 172.21.15.23:0/2816560458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a00095f0 con 0x7f66b40ff1d0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e9a0 tx=0x7f66a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b8a00700 1 -- 172.21.15.23:0/2816560458 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4009790 con 0x7f66b40ff1d0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b419e960 con 0x7f66b40ff1d0 2024-03-30T04:13:35.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b8a00700 1 -- 172.21.15.23:0/2816560458 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66a4010430 con 0x7f66b40ff1d0 2024-03-30T04:13:35.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.517+0000 7f66b8a00700 1 -- 172.21.15.23:0/2816560458 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4018680 con 0x7f66b40ff1d0 2024-03-30T04:13:35.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.518+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b41a61a0 con 0x7f66b40ff1d0 2024-03-30T04:13:35.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.521+0000 7f66b8a00700 1 -- 172.21.15.23:0/2816560458 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f66a40105a0 con 0x7f66b40ff1d0 2024-03-30T04:13:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.522+0000 7f66b8a00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 0x7f66980a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.522+0000 7f66cd830b80 1 Processor -- start 2024-03-30T04:13:35.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.522+0000 7f66b8a00700 1 -- 172.21.15.23:0/2816560458 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f66a4014070 con 0x7f66b40ff1d0 2024-03-30T04:13:35.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.522+0000 7f66b2000700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 0x7f66980a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.522+0000 7f66cd830b80 1 -- start start 2024-03-30T04:13:35.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b57ef0 0x555809b628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b62df0 0x555809b651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809ab0990 con 0x555809b575d0 2024-03-30T04:13:35.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809aae350 con 0x555809b57ef0 2024-03-30T04:13:35.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66cd830b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809aa9c40 con 0x555809b62df0 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58276/0 (socket says 172.21.15.23:58276) 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66b2a00700 1 -- 172.21.15.23:0/3933979264 learned_addr learned my addr 172.21.15.23:0/3933979264 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66b3400700 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b62df0 0x555809b651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.523+0000 7f66b2000700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 0x7f66980a3750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f66b419a980 tx=0x7f66a000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2000700 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b57ef0 0x555809b628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 -- 172.21.15.23:0/3933979264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b62df0 msgr2=0x555809b651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b62df0 0x555809b651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 -- 172.21.15.23:0/3933979264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b57ef0 msgr2=0x555809b628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b57ef0 0x555809b628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 -- 172.21.15.23:0/3933979264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55580992b3b0 con 0x555809b575d0 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f66a4096a30 tx=0x7f66a4099c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e66396c391d773 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.524+0000 7f669fe00700 1 -- 172.21.15.23:0/3933979264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a409ddf0 con 0x555809b575d0 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f669fe00700 1 -- 172.21.15.23:0/3933979264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66a40ade90 con 0x555809b575d0 2024-03-30T04:13:35.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f669fe00700 1 -- 172.21.15.23:0/3933979264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a40994b0 con 0x555809b575d0 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 -- 172.21.15.23:0/3933979264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 msgr2=0x555809b579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f66a4096a30 tx=0x7f66a4099c30 comp rx=0 tx=0).stop 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 -- 172.21.15.23:0/3933979264 shutdown_connections 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b62df0 0x555809b651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b579b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3933979264 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b57ef0 0x555809b628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.525+0000 7f66cd830b80 1 -- 172.21.15.23:0/3933979264 >> 172.21.15.23:0/3933979264 conn(0x555809ab76a0 msgr2=0x555809b53a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.526+0000 7f66cd830b80 1 -- 172.21.15.23:0/3933979264 shutdown_connections 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.526+0000 7f66cd830b80 1 -- 172.21.15.23:0/3933979264 wait complete. 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.526+0000 7f66cd830b80 1 -- start start 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 Processor -- start 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b5fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b62df0 0x555809b60200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809a45d80 con 0x555809b62df0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809a45a20 con 0x555809b575d0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55580992d6a0 con 0x555809b57ef0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.527+0000 7f66cd830b80 10 client.0 _ll_register_callbacks cb 0x555809a4f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37088/0 (socket says 172.21.15.23:37088) 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b62df0 0x555809b60200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 -- 172.21.15.23:0/2182882157 learned_addr learned my addr 172.21.15.23:0/2182882157 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2000700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b5fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 -- 172.21.15.23:0/2182882157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 msgr2=0x555809b5fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b5fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 -- 172.21.15.23:0/2182882157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b62df0 msgr2=0x555809b60200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b62df0 0x555809b60200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 -- 172.21.15.23:0/2182882157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b41a61a0 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f66a40a1f10 tx=0x7f66a40cd220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c67e0046da9383f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f669cc00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a40b8070 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f66a409ddf0 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.528+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555809a6a380 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66a409a430 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 10 client.28622 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5558098d7b60 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555809a6a380 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f66a40a2040 con 0x555809b575d0 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 10 client.28622 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:35.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 10 client.28622 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:13:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 20 client.28622 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809b7acc0 0x555809b7d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 4 client.28622 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.529+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555809aa2700 con 0x555809b7acc0 2024-03-30T04:13:35.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.530+0000 7f66b2000700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809b7acc0 0x555809b7d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.530+0000 7f66b2000700 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809b7acc0 0x555809b7d0b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c6911339a2e0ad5 server_cookie=861bdc4b85caa56e in_seq=0 out_seq=0 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.530+0000 7f669cc00700 10 client.28622 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f669cc00700 1 -- 172.21.15.23:0/2182882157 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555809aa2700 con 0x555809b7acc0 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f669cc00700 10 client.28622 handle_command_reply: tid=0 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 client.28622 shutdown 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 msgr2=0x555809b62840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f66a40a1f10 tx=0x7f66a40cd220 comp rx=0 tx=0).stop 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 shutdown_connections 2024-03-30T04:13:35.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b62df0 0x555809b60200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b575d0 0x555809b62840 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809b7acc0 0x555809b7d0b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2182882157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b5fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 >> 172.21.15.23:0/2182882157 conn(0x555809ab76a0 msgr2=0x555809b59690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f669cc00700 0 client.28622 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 shutdown_connections 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.531+0000 7f66cd830b80 1 -- 172.21.15.23:0/2182882157 wait complete. 2024-03-30T04:13:35.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.532+0000 7f66cd830b80 20 client.28622 trim_cache size 0 max 16384 2024-03-30T04:13:35.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.547+0000 7f66cd830b80 1 Processor -- start 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.547+0000 7f66cd830b80 1 -- start start 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b57ef0 0x555809b72220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809b62370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809b607e0 con 0x555809b57ef0 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809b72760 con 0x555809b62df0 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66cd830b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809b582d0 con 0x555809b575d0 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b57ef0 0x555809b72220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57668/0 (socket says 172.21.15.23:57668) 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2a00700 1 -- 172.21.15.23:0/2190738644 learned_addr learned my addr 172.21.15.23:0/2190738644 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b3400700 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809b62370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2a00700 1 -- 172.21.15.23:0/2190738644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 msgr2=0x555809b62370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.548+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809b62370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66b2a00700 1 -- 172.21.15.23:0/2190738644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b57ef0 msgr2=0x555809b72220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b57ef0 0x555809b72220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66b2a00700 1 -- 172.21.15.23:0/2190738644 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555809a6fea0 con 0x555809b575d0 2024-03-30T04:13:35.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66b2a00700 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f66a4096a30 tx=0x7f66a4099c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c650478167cf4bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66a9200700 1 -- 172.21.15.23:0/2190738644 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a40b8040 con 0x555809b575d0 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66a9200700 1 -- 172.21.15.23:0/2190738644 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f66a409a6c0 con 0x555809b575d0 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.549+0000 7f66a9200700 1 -- 172.21.15.23:0/2190738644 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4004750 con 0x555809b575d0 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 -- 172.21.15.23:0/2190738644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 msgr2=0x555809b5a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f66a4096a30 tx=0x7f66a4099c30 comp rx=0 tx=0).stop 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 -- 172.21.15.23:0/2190738644 shutdown_connections 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809b62370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b575d0 0x555809b5a540 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 --2- 172.21.15.23:0/2190738644 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b57ef0 0x555809b72220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 -- 172.21.15.23:0/2190738644 >> 172.21.15.23:0/2190738644 conn(0x555809ab76a0 msgr2=0x555809b4c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 -- 172.21.15.23:0/2190738644 shutdown_connections 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.550+0000 7f66cd830b80 1 -- 172.21.15.23:0/2190738644 wait complete. 2024-03-30T04:13:35.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.551+0000 7f66cd830b80 1 -- start start 2024-03-30T04:13:35.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.551+0000 7f66cd830b80 1 Processor -- start 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b703c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809bc79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809bc7f00 con 0x555809b575d0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809bc8070 con 0x555809b62df0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555809bc8230 con 0x555809b57ef0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66cd830b80 10 client.0 _ll_register_callbacks cb 0x555809a4f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57672/0 (socket says 172.21.15.23:57672) 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 -- 172.21.15.23:0/3506994673 learned_addr learned my addr 172.21.15.23:0/3506994673 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b3400700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809bc79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 -- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 msgr2=0x555809bc79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809bc79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 -- 172.21.15.23:0/3506994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 msgr2=0x555809b703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b703c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2000700 1 -- 172.21.15.23:0/3506994673 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558098d7b60 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.552+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f66b2000700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f66a0000c00 tx=0x7f66a0009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d8efa2d555b5247 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f669cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0034430 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f66a000bcd0 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558098f8cf0 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66a0034590 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f66cd830b80 10 client.28609 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55580992fcd0 con 0x555809b57ef0 2024-03-30T04:13:35.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.553+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558098f8cf0 con 0x555809b57ef0 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f66a0004500 con 0x555809b57ef0 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 10 client.28609 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 10 client.28609 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 20 client.28609 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809bcfa40 0x555809bd1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 4 client.28609 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55580992dce0 con 0x555809bcfa40 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809bcfa40 0x555809bd1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f66b2a00700 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809bcfa40 0x555809bd1e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f1194733a0c1b8d4 server_cookie=c98fd5b4ddf2c490 in_seq=0 out_seq=0 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.554+0000 7f669cc00700 10 client.28609 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f669cc00700 1 -- 172.21.15.23:0/3506994673 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55580992dce0 con 0x555809bcfa40 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f669cc00700 10 client.28609 handle_command_reply: tid=0 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 client.28609 shutdown 2024-03-30T04:13:35.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 msgr2=0x555809b70900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f66a0000c00 tx=0x7f66a0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 shutdown_connections 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555809b62df0 0x555809bc79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555809b575d0 0x555809b703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x555809bcfa40 0x555809bd1e60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 --2- 172.21.15.23:0/3506994673 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555809b57ef0 0x555809b70900 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 >> 172.21.15.23:0/3506994673 conn(0x555809ab76a0 msgr2=0x555809b5adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f669cc00700 0 client.28609 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 shutdown_connections 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 1 -- 172.21.15.23:0/3506994673 wait complete. 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.555+0000 7f66cd830b80 20 client.28609 trim_cache size 0 max 16384 2024-03-30T04:13:35.557 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:35.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 msgr2=0x7f66980a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 0x7f66980a3750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f66b419a980 tx=0x7f66a000b260 comp rx=0 tx=0).stop 2024-03-30T04:13:35.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 msgr2=0x7f66b4199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e9a0 tx=0x7f66a400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 shutdown_connections 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f66980a1290 0x7f66980a3750 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66b410d710 0x7f66b419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66b40ff1d0 0x7f66b4199430 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 --2- 172.21.15.23:0/2816560458 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f66b40ffaf0 0x7f66b4199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 >> 172.21.15.23:0/2816560458 conn(0x7f66b4074b20 msgr2=0x7f66b410dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 shutdown_connections 2024-03-30T04:13:35.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.556+0000 7f66b9e00700 1 -- 172.21.15.23:0/2816560458 wait complete. 2024-03-30T04:13:35.574 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 -- 172.21.15.23:0/1589833464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 msgr2=0x7f4e30103bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 --2- 172.21.15.23:0/1589833464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30103bb0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0098f0 tx=0x7f4e1c009c00 comp rx=0 tx=0).stop 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 -- 172.21.15.23:0/1589833464 shutdown_connections 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 --2- 172.21.15.23:0/1589833464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e301045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 --2- 172.21.15.23:0/1589833464 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e300ff5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 --2- 172.21.15.23:0/1589833464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30103bb0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 -- 172.21.15.23:0/1589833464 >> 172.21.15.23:0/1589833464 conn(0x7f4e30074b20 msgr2=0x7f4e30074f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 -- 172.21.15.23:0/1589833464 shutdown_connections 2024-03-30T04:13:35.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.813+0000 7f4e35a00700 1 -- 172.21.15.23:0/1589833464 wait complete. 2024-03-30T04:13:35.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.815+0000 7f4e35a00700 1 Processor -- start 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.815+0000 7f4e35a00700 1 -- start start 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e30106ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30107530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e30107a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e30108fc0 con 0x7f4e301040f0 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3019b3f0 con 0x7f4e300ff1d0 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.816+0000 7f4e35a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3019b5b0 con 0x7f4e300ffaf0 2024-03-30T04:13:35.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30107530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e30106ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e30107a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e30107a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58332/0 (socket says 172.21.15.23:58332) 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2f400700 1 -- 172.21.15.23:0/2772305005 learned_addr learned my addr 172.21.15.23:0/2772305005 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 -- 172.21.15.23:0/2772305005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 msgr2=0x7f4e30107530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30107530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 -- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 msgr2=0x7f4e30107a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e30107a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.817+0000 7f4e2ea00700 1 -- 172.21.15.23:0/2772305005 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1c009520 con 0x7f4e300ff1d0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e30106ff0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000d950 tx=0x7f4e2000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e27400700 1 -- 172.21.15.23:0/2772305005 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20009b40 con 0x7f4e300ff1d0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e27400700 1 -- 172.21.15.23:0/2772305005 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e2000be10 con 0x7f4e300ff1d0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e27400700 1 -- 172.21.15.23:0/2772305005 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2000f620 con 0x7f4e300ff1d0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3019b890 con 0x7f4e300ff1d0 2024-03-30T04:13:35.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.818+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e3019bda0 con 0x7f4e300ff1d0 2024-03-30T04:13:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.825+0000 7f4e27400700 1 -- 172.21.15.23:0/2772305005 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f4e20010460 con 0x7f4e300ff1d0 2024-03-30T04:13:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.825+0000 7f4e495e9b80 1 Processor -- start 2024-03-30T04:13:35.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.825+0000 7f4e495e9b80 1 -- start start 2024-03-30T04:13:35.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c7679b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c7728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c6c0990 con 0x55e70c7675d0 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c6be350 con 0x55e70c772df0 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.826+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c6b9c40 con 0x55e70c767ef0 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37126/0 (socket says 172.21.15.23:37126) 2024-03-30T04:13:35.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 -- 172.21.15.23:0/3065240089 learned_addr learned my addr 172.21.15.23:0/3065240089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2f400700 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c7679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 -- 172.21.15.23:0/3065240089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 msgr2=0x55e70c7728b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c7728b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 -- 172.21.15.23:0/3065240089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 msgr2=0x55e70c7679b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c7679b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e2e000700 1 -- 172.21.15.23:0/3065240089 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e70c53b3b0 con 0x55e70c772df0 2024-03-30T04:13:35.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.827+0000 7f4e27400700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 0x7f4e140a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.828+0000 7f4e27400700 1 -- 172.21.15.23:0/2772305005 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4e200ccd60 con 0x7f4e300ff1d0 2024-03-30T04:13:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.828+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c000c00 tx=0x7f4e1c009480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=211bdefdd52ad309 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.828+0000 7f4e2f400700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 0x7f4e140a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.829+0000 7f4e1be00700 1 -- 172.21.15.23:0/3065240089 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c01d070 con 0x55e70c772df0 2024-03-30T04:13:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.829+0000 7f4e1be00700 1 -- 172.21.15.23:0/3065240089 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1c00f920 con 0x55e70c772df0 2024-03-30T04:13:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.829+0000 7f4e1be00700 1 -- 172.21.15.23:0/3065240089 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c005040 con 0x55e70c772df0 2024-03-30T04:13:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.830+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3065240089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 msgr2=0x55e70c7751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.830+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c000c00 tx=0x7f4e1c009480 comp rx=0 tx=0).stop 2024-03-30T04:13:35.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.830+0000 7f4e2f400700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 0x7f4e140a3850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009dc0 tx=0x7f4e28009040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3065240089 shutdown_connections 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c7751e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c7679b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3065240089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c7728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3065240089 >> 172.21.15.23:0/3065240089 conn(0x55e70c6c76a0 msgr2=0x55e70c763a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3065240089 shutdown_connections 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.831+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3065240089 wait complete. 2024-03-30T04:13:35.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.832+0000 7f4e495e9b80 1 -- start start 2024-03-30T04:13:35.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.833+0000 7f4e495e9b80 1 Processor -- start 2024-03-30T04:13:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c772870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c770250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c655d80 con 0x55e70c7675d0 2024-03-30T04:13:35.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c655a20 con 0x55e70c772df0 2024-03-30T04:13:35.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c53d6a0 con 0x55e70c767ef0 2024-03-30T04:13:35.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e495e9b80 10 client.0 _ll_register_callbacks cb 0x55e70c65f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:35.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57702/0 (socket says 172.21.15.23:57702) 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e2e000700 1 -- 172.21.15.23:0/3789750447 learned_addr learned my addr 172.21.15.23:0/3789750447 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.834+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c772870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2e000700 1 -- 172.21.15.23:0/3789750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 msgr2=0x55e70c770250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c770250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2e000700 1 -- 172.21.15.23:0/3789750447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 msgr2=0x55e70c772870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c772870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2e000700 1 -- 172.21.15.23:0/3789750447 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e30106300 con 0x55e70c767ef0 2024-03-30T04:13:35.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.835+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c772870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.836+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0098c0 tx=0x7f4e1c00bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70e1ce7e2df1e674 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.836+0000 7f4e18c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.836+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c01d040 con 0x55e70c767ef0 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.836+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4e1c02b450 con 0x55e70c767ef0 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.836+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e70c67a380 con 0x55e70c767ef0 2024-03-30T04:13:35.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.837+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e1c02b610 con 0x55e70c767ef0 2024-03-30T04:13:35.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.837+0000 7f4e495e9b80 10 client.28615 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:35.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.837+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e70c4e7b60 con 0x55e70c767ef0 2024-03-30T04:13:35.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.838+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e70c67a380 con 0x55e70c767ef0 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.838+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e1c015020 con 0x55e70c767ef0 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.838+0000 7f4e495e9b80 10 client.28615 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.838+0000 7f4e495e9b80 10 client.28615 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.839+0000 7f4e495e9b80 20 client.28615 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.839+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7824c0 0x55e70c7848e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.839+0000 7f4e495e9b80 4 client.28615 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:35.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.839+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e70c6b2700 con 0x55e70c7824c0 2024-03-30T04:13:35.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.839+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7824c0 0x55e70c7848e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.844+0000 7f4e18c00700 10 client.28615 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:35.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.844+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7824c0 0x55e70c7848e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc3e847f07e2abb4 server_cookie=91a6804e4dcba323 in_seq=0 out_seq=0 2024-03-30T04:13:35.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.845+0000 7f4e18c00700 1 -- 172.21.15.23:0/3789750447 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55e70c6b2700 con 0x55e70c7824c0 2024-03-30T04:13:35.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.845+0000 7f4e18c00700 10 client.28615 handle_command_reply: tid=0 2024-03-30T04:13:35.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.845+0000 7f4e495e9b80 1 client.28615 shutdown 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 msgr2=0x55e70c76fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c0098c0 tx=0x7f4e1c00bcb0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 shutdown_connections 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7824c0 0x55e70c7848e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e18c00700 0 client.28615 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:35.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c772df0 0x55e70c770250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c7675d0 0x55e70c772870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3789750447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c767ef0 0x55e70c76fb60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 >> 172.21.15.23:0/3789750447 conn(0x55e70c6c76a0 msgr2=0x55e70c769530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 shutdown_connections 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.846+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3789750447 wait complete. 2024-03-30T04:13:35.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.847+0000 7f4e495e9b80 20 client.28615 trim_cache size 0 max 16384 2024-03-30T04:13:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 Processor -- start 2024-03-30T04:13:35.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 -- start start 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7848b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c77e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c768880 con 0x55e70c772df0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c770850 con 0x55e70c767ef0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.863+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c765160 con 0x55e70c7675d0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37140/0 (socket says 172.21.15.23:37140) 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 -- 172.21.15.23:0/2695025287 learned_addr learned my addr 172.21.15.23:0/2695025287 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2f400700 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c77e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7848b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 -- 172.21.15.23:0/2695025287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 msgr2=0x55e70c7848b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7848b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 -- 172.21.15.23:0/2695025287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 msgr2=0x55e70c77e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c77e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 -- 172.21.15.23:0/2695025287 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e70c67fea0 con 0x55e70c767ef0 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.864+0000 7f4e2e000700 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00b2e0 tx=0x7f4e1c00b9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7fd32113fd2377a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.865+0000 7f4e24c00700 1 -- 172.21.15.23:0/2695025287 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c01d070 con 0x55e70c767ef0 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.865+0000 7f4e24c00700 1 -- 172.21.15.23:0/2695025287 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e1c029eb0 con 0x55e70c767ef0 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.865+0000 7f4e24c00700 1 -- 172.21.15.23:0/2695025287 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c029680 con 0x55e70c767ef0 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.865+0000 7f4e495e9b80 1 -- 172.21.15.23:0/2695025287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 msgr2=0x55e70c76fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.865+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c00b2e0 tx=0x7f4e1c00b9b0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 -- 172.21.15.23:0/2695025287 shutdown_connections 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c77e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.866 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7848b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/2695025287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c76fb20 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 -- 172.21.15.23:0/2695025287 >> 172.21.15.23:0/2695025287 conn(0x55e70c6c76a0 msgr2=0x55e70c789200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 -- 172.21.15.23:0/2695025287 shutdown_connections 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 -- 172.21.15.23:0/2695025287 wait complete. 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 -- start start 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.866+0000 7f4e495e9b80 1 Processor -- start 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c777f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c778470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c7d52b0 con 0x55e70c772df0 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c7d53f0 con 0x55e70c767ef0 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e70c7d55b0 con 0x55e70c7675d0 2024-03-30T04:13:35.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e495e9b80 10 client.0 _ll_register_callbacks cb 0x55e70c65f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c778470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57732/0 (socket says 172.21.15.23:57732) 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2e000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c777f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2ea00700 1 -- 172.21.15.23:0/3118548197 learned_addr learned my addr 172.21.15.23:0/3118548197 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.867+0000 7f4e2f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c778470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58364/0 (socket says 172.21.15.23:58364) 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 -- 172.21.15.23:0/3118548197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 msgr2=0x55e70c777f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c777f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 -- 172.21.15.23:0/3118548197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 msgr2=0x55e70c778470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c778470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 -- 172.21.15.23:0/3118548197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e70c4e7b60 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e2ea00700 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4e20095f30 tx=0x7f4e2000df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1abaf5ae65c3ad03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e18c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e20099a30 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4e2009d860 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e70c508cf0 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e200a1eb0 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e495e9b80 10 client.28621 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55e70c53fcd0 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.868+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e70c508cf0 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e200b2170 con 0x55e70c7675d0 2024-03-30T04:13:35.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 10 client.28621 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 10 client.28621 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 20 client.28621 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7d6d30 0x55e70c7d9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 4 client.28621 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e70c53dce0 con 0x55e70c7d6d30 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7d6d30 0x55e70c7d9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e2e000700 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7d6d30 0x55e70c7d9150 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=893fad15c1fc610a server_cookie=7d05be769421bbe3 in_seq=0 out_seq=0 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.869+0000 7f4e18c00700 10 client.28621 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e18c00700 1 -- 172.21.15.23:0/3118548197 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e70c53dce0 con 0x55e70c7d6d30 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e18c00700 10 client.28621 handle_command_reply: tid=0 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 client.28621 shutdown 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 msgr2=0x55e70c7779f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4e20095f30 tx=0x7f4e2000df50 comp rx=0 tx=0).stop 2024-03-30T04:13:35.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 shutdown_connections 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55e70c7d6d30 0x55e70c7d9150 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55e70c772df0 0x55e70c778470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e70c7675d0 0x55e70c7779f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 --2- 172.21.15.23:0/3118548197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55e70c767ef0 0x55e70c777f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 >> 172.21.15.23:0/3118548197 conn(0x55e70c6c76a0 msgr2=0x55e70c75c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e18c00700 0 client.28621 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 shutdown_connections 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 1 -- 172.21.15.23:0/3118548197 wait complete. 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.870+0000 7f4e495e9b80 20 client.28621 trim_cache size 0 max 16384 2024-03-30T04:13:35.873 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 msgr2=0x7f4e140a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 0x7f4e140a3850 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009dc0 tx=0x7f4e28009040 comp rx=0 tx=0).stop 2024-03-30T04:13:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 msgr2=0x7f4e30106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e30106ff0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000d950 tx=0x7f4e2000dd10 comp rx=0 tx=0).stop 2024-03-30T04:13:35.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 shutdown_connections 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4e301040f0 0x7f4e30107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4e140a1390 0x7f4e140a3850 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e300ff1d0 0x7f4e30106ff0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 --2- 172.21.15.23:0/2772305005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e300ffaf0 0x7f4e30107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 >> 172.21.15.23:0/2772305005 conn(0x7f4e30074b20 msgr2=0x7f4e300fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 shutdown_connections 2024-03-30T04:13:35.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:35.871+0000 7f4e35a00700 1 -- 172.21.15.23:0/2772305005 wait complete. 2024-03-30T04:13:35.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 -- 172.21.15.23:0/667026367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4101910 msgr2=0x7f1dc4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/667026367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4101910 0x7f1dc4105950 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1db00098f0 tx=0x7f1db0009c00 comp rx=0 tx=0).stop 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 -- 172.21.15.23:0/667026367 shutdown_connections 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/667026367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4106080 0x7f1dc410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/667026367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4100f60 0x7f1dc4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/667026367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc4101910 0x7f1dc4105950 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 -- 172.21.15.23:0/667026367 >> 172.21.15.23:0/667026367 conn(0x7f1dc40fc7c0 msgr2=0x7f1dc40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 -- 172.21.15.23:0/667026367 shutdown_connections 2024-03-30T04:13:36.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.111+0000 7f1dcbe00700 1 -- 172.21.15.23:0/667026367 wait complete. 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.112+0000 7f1dcbe00700 1 Processor -- start 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.112+0000 7f1dcbe00700 1 -- start start 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 0x7f1dc41999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc419dd40 0x7f1dc419e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc419e7c0 con 0x7f1dc419dd40 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc419e930 con 0x7f1dc4106080 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dcbe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc419eaf0 con 0x7f1dc4100f60 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 0x7f1dc41999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc419dd40 0x7f1dc419e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57744/0 (socket says 172.21.15.23:57744) 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 -- 172.21.15.23:0/3040805807 learned_addr learned my addr 172.21.15.23:0/3040805807 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 -- 172.21.15.23:0/3040805807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 msgr2=0x7f1dc41999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 0x7f1dc41999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 -- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc419dd40 msgr2=0x7f1dc419e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc419dd40 0x7f1dc419e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc9400700 1 -- 172.21.15.23:0/3040805807 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0009520 con 0x7f1dc4100f60 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.113+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 0x7f1dc41999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1db800e9c0 tx=0x7f1db800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dc2000700 1 -- 172.21.15.23:0/3040805807 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8004a30 con 0x7f1dc4100f60 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dc2000700 1 -- 172.21.15.23:0/3040805807 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1db800cc80 con 0x7f1dc4100f60 2024-03-30T04:13:36.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc41a5ff0 con 0x7f1dc4100f60 2024-03-30T04:13:36.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dc2000700 1 -- 172.21.15.23:0/3040805807 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80054e0 con 0x7f1dc4100f60 2024-03-30T04:13:36.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.114+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc41a6420 con 0x7f1dc4100f60 2024-03-30T04:13:36.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.117+0000 7f1dc2000700 1 -- 172.21.15.23:0/3040805807 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1db8004b90 con 0x7f1dc4100f60 2024-03-30T04:13:36.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.119+0000 7f1dc2000700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 0x7f1dac0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.119+0000 7f1ddf8f9b80 1 Processor -- start 2024-03-30T04:13:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.119+0000 7f1dc2000700 1 -- 172.21.15.23:0/3040805807 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1db8014070 con 0x7f1dc4100f60 2024-03-30T04:13:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.119+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 0x7f1dac0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 -- start start 2024-03-30T04:13:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6f14990 con 0x5557b6fbbef0 2024-03-30T04:13:36.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6f12350 con 0x5557b6fbb5d0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6f0dc40 con 0x5557b6fc6df0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.120+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58388/0 (socket says 172.21.15.23:58388) 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 -- 172.21.15.23:0/3151838321 learned_addr learned my addr 172.21.15.23:0/3151838321 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 0x7f1dac0a3750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005e90 tx=0x7f1db0019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc9e00700 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 -- 172.21.15.23:0/3151838321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 msgr2=0x5557b6fc91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 -- 172.21.15.23:0/3151838321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 msgr2=0x5557b6fbb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 -- 172.21.15.23:0/3151838321 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557b6d8f3b0 con 0x5557b6fbbef0 2024-03-30T04:13:36.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.121+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1dc406a380 tx=0x7f1db001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e62acba66ea104a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.122+0000 7f1db6600700 1 -- 172.21.15.23:0/3151838321 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000b7c0 con 0x5557b6fbbef0 2024-03-30T04:13:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.122+0000 7f1db6600700 1 -- 172.21.15.23:0/3151838321 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1db001d040 con 0x5557b6fbbef0 2024-03-30T04:13:36.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.122+0000 7f1db6600700 1 -- 172.21.15.23:0/3151838321 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0004a50 con 0x5557b6fbbef0 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.122+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3151838321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 msgr2=0x5557b6fc68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.122+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f1dc406a380 tx=0x7f1db001f000 comp rx=0 tx=0).stop 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3151838321 shutdown_connections 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3151838321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc68b0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3151838321 >> 172.21.15.23:0/3151838321 conn(0x5557b6f1b6a0 msgr2=0x5557b6fb7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3151838321 shutdown_connections 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3151838321 wait complete. 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 -- start start 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.123+0000 7f1ddf8f9b80 1 Processor -- start 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6ea9d80 con 0x5557b6fbbef0 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6ea9a20 con 0x5557b6fbb5d0 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6d916a0 con 0x5557b6fc6df0 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1ddf8f9b80 10 client.0 _ll_register_callbacks cb 0x5557b6eb3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37196/0 (socket says 172.21.15.23:37196) 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9400700 1 -- 172.21.15.23:0/3676014685 learned_addr learned my addr 172.21.15.23:0/3676014685 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9400700 1 -- 172.21.15.23:0/3676014685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 msgr2=0x5557b6fc4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.124+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1dc9400700 1 -- 172.21.15.23:0/3676014685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 msgr2=0x5557b6fc3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1dc9400700 1 -- 172.21.15.23:0/3676014685 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc410e470 con 0x5557b6fbb5d0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1dc9e00700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1dc9400700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1db8005950 tx=0x7f1db8096180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e867db00532bc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1daaa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db809a2a0 con 0x5557b6fbb5d0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1db809dc70 con 0x5557b6fbb5d0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5557b6ece380 con 0x5557b6fbb5d0 2024-03-30T04:13:36.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1db80a22c0 con 0x5557b6fbb5d0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1ddf8f9b80 10 client.28655 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.125+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5557b6d3bb60 con 0x5557b6fbb5d0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5557b6ece380 con 0x5557b6fbb5d0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1db8064020 con 0x5557b6fbb5d0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 10 client.28655 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 10 client.28655 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 20 client.28655 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b6fd6aa0 0x5557b6fd8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 4 client.28655 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557b6f06700 con 0x5557b6fd6aa0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.126+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b6fd6aa0 0x5557b6fd8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.127+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b6fd6aa0 0x5557b6fd8ec0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5cbbaf2ad8e50926 server_cookie=b28aa28d0d43e816 in_seq=0 out_seq=0 2024-03-30T04:13:36.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.127+0000 7f1daaa00700 10 client.28655 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1daaa00700 1 -- 172.21.15.23:0/3676014685 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5557b6f06700 con 0x5557b6fd6aa0 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1daaa00700 10 client.28655 handle_command_reply: tid=0 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 client.28655 shutdown 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 msgr2=0x5557b6fc6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f1db8005950 tx=0x7f1db8096180 comp rx=0 tx=0).stop 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 shutdown_connections 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b6fd6aa0 0x5557b6fd8ec0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fc6df0 0x5557b6fc4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbb5d0 0x5557b6fc6870 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/3676014685 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbbef0 0x5557b6fc3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1daaa00700 0 client.28655 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:36.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 >> 172.21.15.23:0/3676014685 conn(0x5557b6f1b6a0 msgr2=0x5557b6fbd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 shutdown_connections 2024-03-30T04:13:36.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/3676014685 wait complete. 2024-03-30T04:13:36.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.128+0000 7f1ddf8f9b80 20 client.28655 trim_cache size 0 max 16384 2024-03-30T04:13:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.143+0000 7f1ddf8f9b80 1 Processor -- start 2024-03-30T04:13:36.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.143+0000 7f1ddf8f9b80 1 -- start start 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbe5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fc6df0 0x5557b6fb05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6fc4850 con 0x5557b6fc6df0 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6fcdf80 con 0x5557b6fbbef0 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b6fc20f0 con 0x5557b6fbb5d0 2024-03-30T04:13:36.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbe5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37208/0 (socket says 172.21.15.23:37208) 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1dc8a00700 1 -- 172.21.15.23:0/1974539589 learned_addr learned my addr 172.21.15.23:0/1974539589 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.144+0000 7f1dc9e00700 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fc6df0 0x5557b6fb05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 -- 172.21.15.23:0/1974539589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 msgr2=0x5557b6fbe5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbe5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 -- 172.21.15.23:0/1974539589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fc6df0 msgr2=0x5557b6fb05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fc6df0 0x5557b6fb05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 -- 172.21.15.23:0/1974539589 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557b6ed3ea0 con 0x5557b6fbbef0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc9400700 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1db00094b0 tx=0x7f1db0009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e12932bfaae462f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1db7000700 1 -- 172.21.15.23:0/1974539589 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db003f430 con 0x5557b6fbbef0 2024-03-30T04:13:36.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1db7000700 1 -- 172.21.15.23:0/1974539589 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1db003f590 con 0x5557b6fbbef0 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.145+0000 7f1db7000700 1 -- 172.21.15.23:0/1974539589 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0003c90 con 0x5557b6fbbef0 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/1974539589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 msgr2=0x5557b6fcda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1db00094b0 tx=0x7f1db0009c00 comp rx=0 tx=0).stop 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/1974539589 shutdown_connections 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fc6df0 0x5557b6fb05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbb5d0 0x5557b6fbe5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/1974539589 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fbbef0 0x5557b6fcda40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/1974539589 >> 172.21.15.23:0/1974539589 conn(0x5557b6f1b6a0 msgr2=0x5557b6fc33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/1974539589 shutdown_connections 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/1974539589 wait complete. 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.146+0000 7f1ddf8f9b80 1 -- start start 2024-03-30T04:13:36.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 Processor -- start 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbb5d0 0x5557b6fcba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 0x5557b6fcc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b70299c0 con 0x5557b6fbb5d0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b7029b30 con 0x5557b6fc6df0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.147+0000 7f1ddf8f9b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557b7029cf0 con 0x5557b6fbbef0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1ddf8f9b80 10 client.0 _ll_register_callbacks cb 0x5557b6eb3470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 0x5557b6fcc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57780/0 (socket says 172.21.15.23:57780) 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc9400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbb5d0 0x5557b6fcba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 0x5557b6fcc4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37224/0 (socket says 172.21.15.23:37224) 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 -- 172.21.15.23:0/2241878941 learned_addr learned my addr 172.21.15.23:0/2241878941 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 -- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 msgr2=0x5557b6fcc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 0x5557b6fcc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 -- 172.21.15.23:0/2241878941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbb5d0 msgr2=0x5557b6fcba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbb5d0 0x5557b6fcba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 -- 172.21.15.23:0/2241878941 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557b6d3bb60 con 0x5557b6fbbef0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1dc8a00700 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1db000f5d0 tx=0x7f1db00058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b567dc8c95fabcd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.148+0000 7f1daaa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000bbc0 con 0x5557b6fbbef0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1db0003680 con 0x5557b6fbbef0 2024-03-30T04:13:36.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1db001d040 con 0x5557b6fbbef0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1db0003950 con 0x5557b6fbbef0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 10 client.28639 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5557b6d8f3b0 con 0x5557b6fbbef0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1db001d040 con 0x5557b6fbbef0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1db003f730 con 0x5557b6fbbef0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 10 client.28639 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 10 client.28639 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 20 client.28639 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b702b060 0x5557b702d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 4 client.28639 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.149+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5557b6d5ccf0 con 0x5557b702b060 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1dc9400700 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b702b060 0x5557b702d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1dc9400700 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b702b060 0x5557b702d480 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f9420584d5ffb4f server_cookie=194c6623685a0c5d in_seq=0 out_seq=0 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1daaa00700 10 client.28639 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1daaa00700 1 -- 172.21.15.23:0/2241878941 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5557b6d5ccf0 con 0x5557b702b060 2024-03-30T04:13:36.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1daaa00700 10 client.28639 handle_command_reply: tid=0 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.150+0000 7f1ddf8f9b80 1 client.28639 shutdown 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 msgr2=0x5557b6fcbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1db000f5d0 tx=0x7f1db00058e0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 shutdown_connections 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5557b702b060 0x5557b702d480 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5557b6fc6df0 0x5557b6fcc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5557b6fbb5d0 0x5557b6fcba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1daaa00700 0 client.28639 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 --2- 172.21.15.23:0/2241878941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5557b6fbbef0 0x5557b6fcbf60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 >> 172.21.15.23:0/2241878941 conn(0x5557b6f1b6a0 msgr2=0x5557b6fb05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 shutdown_connections 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 1 -- 172.21.15.23:0/2241878941 wait complete. 2024-03-30T04:13:36.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.151+0000 7f1ddf8f9b80 20 client.28639 trim_cache size 0 max 16384 2024-03-30T04:13:36.155 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:36.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 msgr2=0x7f1dac0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 0x7f1dac0a3750 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1db0005e90 tx=0x7f1db0019040 comp rx=0 tx=0).stop 2024-03-30T04:13:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 msgr2=0x7f1dc4199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1db800e9c0 tx=0x7f1db800ecd0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 shutdown_connections 2024-03-30T04:13:36.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1dc419dd40 0x7f1dc419e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dc4100f60 0x7f1dc4199460 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f1dac0a1290 0x7f1dac0a3750 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 --2- 172.21.15.23:0/3040805807 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1dc4106080 0x7f1dc41999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 >> 172.21.15.23:0/3040805807 conn(0x7f1dc40fc7c0 msgr2=0x7f1dc40fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 shutdown_connections 2024-03-30T04:13:36.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.152+0000 7f1dcbe00700 1 -- 172.21.15.23:0/3040805807 wait complete. 2024-03-30T04:13:36.170 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:13:36.230 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1139043 9.70 MB/sec execute 493 sec latency 32.027 ms 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 -- 172.21.15.23:0/205612518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8106080 msgr2=0x7f8ab810dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 --2- 172.21.15.23:0/205612518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8106080 0x7f8ab810dc00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8ab0009f90 tx=0x7f8ab000b330 comp rx=0 tx=0).stop 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 -- 172.21.15.23:0/205612518 shutdown_connections 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 --2- 172.21.15.23:0/205612518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8106080 0x7f8ab810dc00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 --2- 172.21.15.23:0/205612518 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8100f60 0x7f8ab8101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 --2- 172.21.15.23:0/205612518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8101910 0x7f8ab8105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 -- 172.21.15.23:0/205612518 >> 172.21.15.23:0/205612518 conn(0x7f8ab80fc7c0 msgr2=0x7f8ab80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 -- 172.21.15.23:0/205612518 shutdown_connections 2024-03-30T04:13:36.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.397+0000 7f8abee00700 1 -- 172.21.15.23:0/205612518 wait complete. 2024-03-30T04:13:36.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.398+0000 7f8abee00700 1 Processor -- start 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.398+0000 7f8abee00700 1 -- start start 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.398+0000 7f8abee00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.398+0000 7f8abee00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8106080 0x7f8ab80735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8abee00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8074d70 0x7f8ab8073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8abee00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab81a9f70 con 0x7f8ab8106080 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8abee00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab81aa0e0 con 0x7f8ab8100f60 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8abee00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ab81aa2a0 con 0x7f8ab8074d70 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37238/0 (socket says 172.21.15.23:37238) 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 -- 172.21.15.23:0/236811985 learned_addr learned my addr 172.21.15.23:0/236811985 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8abce00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8074d70 0x7f8ab8073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8aafe00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8106080 0x7f8ab80735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 -- 172.21.15.23:0/236811985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8074d70 msgr2=0x7f8ab8073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8074d70 0x7f8ab8073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 -- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8106080 msgr2=0x7f8ab80735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8106080 0x7f8ab80735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.399+0000 7f8ab7e00700 1 -- 172.21.15.23:0/236811985 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8009520 con 0x7f8ab8100f60 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800eb30 tx=0x7f8aa800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8ab5600700 1 -- 172.21.15.23:0/236811985 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8009880 con 0x7f8ab8100f60 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa00095f0 con 0x7f8ab8100f60 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8ab5600700 1 -- 172.21.15.23:0/236811985 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8aa80044e0 con 0x7f8ab8100f60 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8ab5600700 1 -- 172.21.15.23:0/236811985 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8003d80 con 0x7f8ab8100f60 2024-03-30T04:13:36.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.400+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab0009c40 con 0x7f8ab8100f60 2024-03-30T04:13:36.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.403+0000 7f8ab5600700 1 -- 172.21.15.23:0/236811985 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8aa8003f20 con 0x7f8ab8100f60 2024-03-30T04:13:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 Processor -- start 2024-03-30T04:13:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 -- start start 2024-03-30T04:13:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ab5600700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 0x7f8a9c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ab5600700 1 -- 172.21.15.23:0/236811985 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8aa8014070 con 0x7f8ab8100f60 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c4ffd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c500b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4f56990 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4f54350 con 0x55c4c4ffdef0 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.405+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4f4fc40 con 0x55c4c5008df0 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8abce00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 0x7f8a9c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37240/0 (socket says 172.21.15.23:37240) 2024-03-30T04:13:36.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 -- 172.21.15.23:0/934534668 learned_addr learned my addr 172.21.15.23:0/934534668 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c4ffd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 -- 172.21.15.23:0/934534668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 msgr2=0x55c4c500b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c500b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 -- 172.21.15.23:0/934534668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 msgr2=0x55c4c4ffd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c4ffd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.406+0000 7f8aafe00700 1 -- 172.21.15.23:0/934534668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4c4dd13b0 con 0x55c4c4ffdef0 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.407+0000 7f8abce00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 0x7f8a9c0a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ab80751c0 tx=0x7f8ab00074e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.407+0000 7f8aafe00700 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009f50 tx=0x7f8aa0011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48b1467fba56460 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.407+0000 7f8aad600700 1 -- 172.21.15.23:0/934534668 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0004310 con 0x55c4c4ffdef0 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.407+0000 7f8aad600700 1 -- 172.21.15.23:0/934534668 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8aa0004470 con 0x55c4c4ffdef0 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.407+0000 7f8aad600700 1 -- 172.21.15.23:0/934534668 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0019940 con 0x55c4c4ffdef0 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- 172.21.15.23:0/934534668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 msgr2=0x55c4c50088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009f50 tx=0x7f8aa0011720 comp rx=0 tx=0).stop 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- 172.21.15.23:0/934534668 shutdown_connections 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c500b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c4ffd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/934534668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c50088b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- 172.21.15.23:0/934534668 >> 172.21.15.23:0/934534668 conn(0x55c4c4f5d6a0 msgr2=0x55c4c4ff9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- 172.21.15.23:0/934534668 shutdown_connections 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- 172.21.15.23:0/934534668 wait complete. 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.408+0000 7f8ad2924b80 1 -- start start 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 Processor -- start 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffdef0 0x55c4c5005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4eebd80 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4eeba20 con 0x55c4c5008df0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4dd36a0 con 0x55c4c4ffdef0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ad2924b80 10 client.0 _ll_register_callbacks cb 0x55c4c4ef5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8abce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58452/0 (socket says 172.21.15.23:58452) 2024-03-30T04:13:36.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8abce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37246/0 (socket says 172.21.15.23:37246) 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.409+0000 7f8ab7e00700 1 -- 172.21.15.23:0/1250884705 learned_addr learned my addr 172.21.15.23:0/1250884705 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8aafe00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffdef0 0x55c4c5005a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 -- 172.21.15.23:0/1250884705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffdef0 msgr2=0x55c4c5005a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffdef0 0x55c4c5005a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 -- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 msgr2=0x55c4c50060c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 -- 172.21.15.23:0/1250884705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa0009b80 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8abce00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809a070 tx=0x7f8aa800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c585151e33f81e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8aa6000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa809a370 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8aa809e1a0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.410+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4c4f10380 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aa80a27f0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 10 client.18693 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c4c4d7db60 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c4c4f10380 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8aa8099860 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 10 client.18693 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 10 client.18693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 20 client.18693 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c50189d0 0x55c4c501adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 4 client.18693 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:36.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.411+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c4c4f48700 con 0x55c4c50189d0 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.412+0000 7f8aafe00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c50189d0 0x55c4c501adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.412+0000 7f8aafe00700 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c50189d0 0x55c4c501adc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e57a995c91ee67c3 server_cookie=b68a43258459b74 in_seq=0 out_seq=0 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.412+0000 7f8aa6000700 10 client.18693 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8aa6000700 1 -- 172.21.15.23:0/1250884705 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c4c4f48700 con 0x55c4c50189d0 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8aa6000700 10 client.18693 handle_command_reply: tid=0 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 client.18693 shutdown 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 msgr2=0x55c4c50087a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809a070 tx=0x7f8aa800cc50 comp rx=0 tx=0).stop 2024-03-30T04:13:36.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 shutdown_connections 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c50060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c50087a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c50189d0 0x55c4c501adc0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/1250884705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffdef0 0x55c4c5005a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 >> 172.21.15.23:0/1250884705 conn(0x55c4c4f5d6a0 msgr2=0x55c4c4fff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8aa6000700 0 client.18693 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 shutdown_connections 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.413+0000 7f8ad2924b80 1 -- 172.21.15.23:0/1250884705 wait complete. 2024-03-30T04:13:36.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.414+0000 7f8ad2924b80 20 client.18693 trim_cache size 0 max 16384 2024-03-30T04:13:36.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.430+0000 7f8ad2924b80 1 Processor -- start 2024-03-30T04:13:36.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.430+0000 7f8ad2924b80 1 -- start start 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c5006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c500fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c5000df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c5015510 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4ffe850 con 0x55c4c4ffdef0 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c4ffd3f0 con 0x55c4c5008df0 2024-03-30T04:13:36.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c5006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c5006040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58462/0 (socket says 172.21.15.23:58462) 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8aafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c500fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.431+0000 7f8ab7e00700 1 -- 172.21.15.23:0/2611950557 learned_addr learned my addr 172.21.15.23:0/2611950557 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8abce00700 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c5000df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 -- 172.21.15.23:0/2611950557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 msgr2=0x55c4c5000df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c5000df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 -- 172.21.15.23:0/2611950557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 msgr2=0x55c4c5006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c5006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 -- 172.21.15.23:0/2611950557 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4c4f15ea0 con 0x55c4c4ffdef0 2024-03-30T04:13:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:36 smithi154 ceph-mon[79161]: pgmap v364: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-30T04:13:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:36 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3477556281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:36.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aafe00700 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c500fa70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00194c0 tx=0x7f8aa0019520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30f0641728b56925 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aae000700 1 -- 172.21.15.23:0/2611950557 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0009290 con 0x55c4c4ffdef0 2024-03-30T04:13:36.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aae000700 1 -- 172.21.15.23:0/2611950557 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8aa0005510 con 0x55c4c4ffdef0 2024-03-30T04:13:36.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.432+0000 7f8aae000700 1 -- 172.21.15.23:0/2611950557 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0003c30 con 0x55c4c4ffdef0 2024-03-30T04:13:36.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.433+0000 7f8ad2924b80 1 -- 172.21.15.23:0/2611950557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 msgr2=0x55c4c500fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.433+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c500fa70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f8aa00194c0 tx=0x7f8aa0019520 comp rx=0 tx=0).stop 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 -- 172.21.15.23:0/2611950557 shutdown_connections 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c5008df0 0x55c4c5000df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffd5d0 0x55c4c5006040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/2611950557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c4ffdef0 0x55c4c500fa70 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 -- 172.21.15.23:0/2611950557 >> 172.21.15.23:0/2611950557 conn(0x55c4c4f5d6a0 msgr2=0x55c4c4ff25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 -- 172.21.15.23:0/2611950557 shutdown_connections 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 -- 172.21.15.23:0/2611950557 wait complete. 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 -- start start 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.434+0000 7f8ad2924b80 1 Processor -- start 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffdef0 0x55c4c500dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c506b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c500e350 con 0x55c4c4ffdef0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c506bd10 con 0x55c4c5008df0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4c506bed0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ad2924b80 10 client.0 _ll_register_callbacks cb 0x55c4c4ef5470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8abce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c506b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57818/0 (socket says 172.21.15.23:57818) 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 -- 172.21.15.23:0/4107417963 learned_addr learned my addr 172.21.15.23:0/4107417963 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8abce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c506b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37260/0 (socket says 172.21.15.23:37260) 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8aafe00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffdef0 0x55c4c500dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 -- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 msgr2=0x55c4c506b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c506b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 -- 172.21.15.23:0/4107417963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffdef0 msgr2=0x55c4c500dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffdef0 0x55c4c500dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.435+0000 7f8ab7e00700 1 -- 172.21.15.23:0/4107417963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa001e040 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8ab7e00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809a300 tx=0x7f8aa8098920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90fc613c0eb82f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8aa6000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa809a7d0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8aa809e1a0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8aa0004470 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aa80a2400 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8ad2924b80 10 client.28654 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55c4c4dd5cd0 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.436+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8aa0004470 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8aa807f060 con 0x55c4c4ffd5d0 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 10 client.28654 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 10 client.28654 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:13:36.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 20 client.28654 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c506d290 0x55c4c506f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 4 client.28654 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:36.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c4c4d9ecf0 con 0x55c4c506d290 2024-03-30T04:13:36.439 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.437+0000 7f8aafe00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c506d290 0x55c4c506f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8aa6000700 10 client.28654 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8aafe00700 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c506d290 0x55c4c506f6b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=544ab38a1778e0f3 server_cookie=c3194d5dd7ea8e23 in_seq=0 out_seq=0 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8aa6000700 1 -- 172.21.15.23:0/4107417963 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c4c4d9ecf0 con 0x55c4c506d290 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8aa6000700 10 client.28654 handle_command_reply: tid=0 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 client.28654 shutdown 2024-03-30T04:13:36.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 msgr2=0x55c4c500d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8aa809a300 tx=0x7f8aa8098920 comp rx=0 tx=0).stop 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 shutdown_connections 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c4c5008df0 0x55c4c506b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c4c506d290 0x55c4c506f6b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4c4ffd5d0 0x55c4c500d670 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 --2- 172.21.15.23:0/4107417963 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c4c4ffdef0 0x55c4c500dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8aa6000700 0 client.28654 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 >> 172.21.15.23:0/4107417963 conn(0x55c4c4f5d6a0 msgr2=0x55c4c5000df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 shutdown_connections 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 1 -- 172.21.15.23:0/4107417963 wait complete. 2024-03-30T04:13:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.438+0000 7f8ad2924b80 20 client.28654 trim_cache size 0 max 16384 2024-03-30T04:13:36.442 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 msgr2=0x7f8a9c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 0x7f8a9c0a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8ab80751c0 tx=0x7f8ab00074e0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 msgr2=0x7f8ab80730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800eb30 tx=0x7f8aa800ee40 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 shutdown_connections 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8ab8074d70 0x7f8ab8073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8a9c0a1290 0x7f8a9c0a3750 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ab8100f60 0x7f8ab80730b0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 --2- 172.21.15.23:0/236811985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8ab8106080 0x7f8ab80735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 >> 172.21.15.23:0/236811985 conn(0x7f8ab80fc7c0 msgr2=0x7f8ab80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 shutdown_connections 2024-03-30T04:13:36.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.439+0000 7f8abee00700 1 -- 172.21.15.23:0/236811985 wait complete. 2024-03-30T04:13:36.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:36 smithi023 ceph-mon[38254]: pgmap v364: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-30T04:13:36.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:36 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3477556281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:36.457 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:13:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:36 smithi193 ceph-mon[79581]: pgmap v364: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 146 op/s 2024-03-30T04:13:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:36 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3477556281' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 -- 172.21.15.23:0/4212304887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8067e30 msgr2=0x7f3aa80682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 --2- 172.21.15.23:0/4212304887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8067e30 0x7f3aa80682b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3a940099c0 tx=0x7f3a94009cd0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 -- 172.21.15.23:0/4212304887 shutdown_connections 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 --2- 172.21.15.23:0/4212304887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 --2- 172.21.15.23:0/4212304887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8105ce0 0x7f3aa81060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 --2- 172.21.15.23:0/4212304887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8067e30 0x7f3aa80682b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 -- 172.21.15.23:0/4212304887 >> 172.21.15.23:0/4212304887 conn(0x7f3aa8076d40 msgr2=0x7f3aa8077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 -- 172.21.15.23:0/4212304887 shutdown_connections 2024-03-30T04:13:36.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.690+0000 7f3aad200700 1 -- 172.21.15.23:0/4212304887 wait complete. 2024-03-30T04:13:36.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.692+0000 7f3aad200700 1 Processor -- start 2024-03-30T04:13:36.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.692+0000 7f3aad200700 1 -- start start 2024-03-30T04:13:36.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa81999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8105ce0 0x7f3aa819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819e3a0 con 0x7f3aa8067e30 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819e510 con 0x7f3aa80687f0 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aad200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa819e6d0 con 0x7f3aa8105ce0 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58486/0 (socket says 172.21.15.23:58486) 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5e00700 1 -- 172.21.15.23:0/2701715178 learned_addr learned my addr 172.21.15.23:0/2701715178 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa81999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa6800700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8105ce0 0x7f3aa819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 -- 172.21.15.23:0/2701715178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8105ce0 msgr2=0x7f3aa819dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8105ce0 0x7f3aa819dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 -- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 msgr2=0x7f3aa8199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5400700 1 -- 172.21.15.23:0/2701715178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a940095f0 con 0x7f3aa80687f0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.693+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3aa5400700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa81999c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3a940058e0 tx=0x7f3a9400b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3a9ea00700 1 -- 172.21.15.23:0/2701715178 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9401d070 con 0x7f3aa80687f0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa819e950 con 0x7f3aa80687f0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3a9ea00700 1 -- 172.21.15.23:0/2701715178 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a9400bdc0 con 0x7f3aa80687f0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3a9ea00700 1 -- 172.21.15.23:0/2701715178 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9400fa60 con 0x7f3aa80687f0 2024-03-30T04:13:36.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.694+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa81a61f0 con 0x7f3aa80687f0 2024-03-30T04:13:36.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.697+0000 7f3a9ea00700 1 -- 172.21.15.23:0/2701715178 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3a9402b440 con 0x7f3aa80687f0 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.698+0000 7f3ac0da1b80 1 Processor -- start 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3a9ea00700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 0x7f3a8c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 -- start start 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3a9ea00700 1 -- 172.21.15.23:0/2701715178 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a940cdbc0 con 0x7f3aa80687f0 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3aa6800700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 0x7f3a8c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e60d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e558990 con 0x563b4e5ffef0 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e556350 con 0x563b4e60adf0 2024-03-30T04:13:36.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e551c40 con 0x563b4e5ff5d0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57844/0 (socket says 172.21.15.23:57844) 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.699+0000 7f3aa5e00700 1 -- 172.21.15.23:0/297800540 learned_addr learned my addr 172.21.15.23:0/297800540 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5400700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa6800700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e60d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 -- 172.21.15.23:0/297800540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 msgr2=0x563b4e60d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e60d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 -- 172.21.15.23:0/297800540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 msgr2=0x563b4e60a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 -- 172.21.15.23:0/297800540 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b4e3d33b0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa6800700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e60d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa6800700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 0x7f3a8c0a3750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000f590 tx=0x7f3aa000b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a980117e0 tx=0x7f3a98011af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ccbf8ac23362c496 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3a93400700 1 -- 172.21.15.23:0/297800540 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9800f480 con 0x563b4e5ff5d0 2024-03-30T04:13:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3a93400700 1 -- 172.21.15.23:0/297800540 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a9800faa0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.700+0000 7f3a93400700 1 -- 172.21.15.23:0/297800540 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98019b40 con 0x563b4e5ff5d0 2024-03-30T04:13:36.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/297800540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 msgr2=0x563b4e5ff9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3a980117e0 tx=0x7f3a98011af0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/297800540 shutdown_connections 2024-03-30T04:13:36.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e60d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e5ff9b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/297800540 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/297800540 >> 172.21.15.23:0/297800540 conn(0x563b4e55f6a0 msgr2=0x563b4e5fba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/297800540 shutdown_connections 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.701+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/297800540 wait complete. 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.702+0000 7f3ac0da1b80 1 -- start start 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.702+0000 7f3ac0da1b80 1 Processor -- start 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e607aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e4edd80 con 0x563b4e5ffef0 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e4eda20 con 0x563b4e60adf0 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e3d56a0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3ac0da1b80 10 client.0 _ll_register_callbacks cb 0x563b4e4f7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57860/0 (socket says 172.21.15.23:57860) 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37284/0 (socket says 172.21.15.23:37284) 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 -- 172.21.15.23:0/474485577 learned_addr learned my addr 172.21.15.23:0/474485577 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5400700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e607aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 -- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 msgr2=0x563b4e6080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 -- 172.21.15.23:0/474485577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 msgr2=0x563b4e607aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e607aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa5e00700 1 -- 172.21.15.23:0/474485577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a98011440 con 0x563b4e5ff5d0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.703+0000 7f3aa6800700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a98014040 tx=0x7f3a9800f6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaeec053957215c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3a87e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:13:36.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a98005ce0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b4e512380 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a9800f8a0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a98004250 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3ac0da1b80 10 client.28666 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.704+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x563b4e37fb60 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563b4e512380 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3a98003710 con 0x563b4e5ff5d0 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 10 client.28666 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 10 client.28666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 20 client.28666 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e61a790 0x563b4e61cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 4 client.28666 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:36.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b4e54a700 con 0x563b4e61a790 2024-03-30T04:13:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.705+0000 7f3aa5400700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e61a790 0x563b4e61cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.706+0000 7f3aa5400700 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e61a790 0x563b4e61cbb0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e22c0fcdedfc56 server_cookie=8b45209fa1af6701 in_seq=0 out_seq=0 2024-03-30T04:13:36.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.706+0000 7f3a87e00700 10 client.28666 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.706+0000 7f3a87e00700 1 -- 172.21.15.23:0/474485577 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x563b4e54a700 con 0x563b4e61a790 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.706+0000 7f3a87e00700 10 client.28666 handle_command_reply: tid=0 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 client.28666 shutdown 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 msgr2=0x563b4e60a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3a98014040 tx=0x7f3a9800f6b0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 shutdown_connections 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e6080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e61a790 0x563b4e61cbb0 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60a7b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/474485577 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e607aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 >> 172.21.15.23:0/474485577 conn(0x563b4e55f6a0 msgr2=0x563b4e6016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3a87e00700 0 client.28666 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 shutdown_connections 2024-03-30T04:13:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/474485577 wait complete. 2024-03-30T04:13:36.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.707+0000 7f3ac0da1b80 20 client.28666 trim_cache size 0 max 16384 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 Processor -- start 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 -- start start 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e608070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e611aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e611fe0 con 0x563b4e5ffef0 2024-03-30T04:13:36.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e6086b0 con 0x563b4e60adf0 2024-03-30T04:13:36.722 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e605ff0 con 0x563b4e5ff5d0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3aa6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3aa6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37298/0 (socket says 172.21.15.23:37298) 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3aa6800700 1 -- 172.21.15.23:0/3666842798 learned_addr learned my addr 172.21.15.23:0/3666842798 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e608070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.723+0000 7f3aa5400700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e611aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 -- 172.21.15.23:0/3666842798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 msgr2=0x563b4e608070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e608070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 -- 172.21.15.23:0/3666842798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 msgr2=0x563b4e611aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e611aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 -- 172.21.15.23:0/3666842798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b4e517ea0 con 0x563b4e60adf0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e608070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3aa6800700 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0017c60 tx=0x7f3aa0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de4ca298f95ebfbd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3a93e00700 1 -- 172.21.15.23:0/3666842798 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000e040 con 0x563b4e60adf0 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3a93e00700 1 -- 172.21.15.23:0/3666842798 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3aa0026070 con 0x563b4e60adf0 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.724+0000 7f3a93e00700 1 -- 172.21.15.23:0/3666842798 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0004870 con 0x563b4e60adf0 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3666842798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 msgr2=0x563b4e602d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0017c60 tx=0x7f3aa0018000 comp rx=0 tx=0).stop 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3666842798 shutdown_connections 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e602d50 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e608070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3666842798 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e611aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3666842798 >> 172.21.15.23:0/3666842798 conn(0x563b4e55f6a0 msgr2=0x563b4e61c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3666842798 shutdown_connections 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.725+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3666842798 wait complete. 2024-03-30T04:13:36.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 -- start start 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 Processor -- start 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e66d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e66db00 con 0x563b4e5ffef0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e66dc70 con 0x563b4e60adf0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b4e66de30 con 0x563b4e5ff5d0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3ac0da1b80 10 client.0 _ll_register_callbacks cb 0x563b4e4f7470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.726+0000 7f3aa5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa6800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e66d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58542/0 (socket says 172.21.15.23:58542) 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 -- 172.21.15.23:0/3863120906 learned_addr learned my addr 172.21.15.23:0/3863120906 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 -- 172.21.15.23:0/3863120906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 msgr2=0x563b4e60f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 -- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 msgr2=0x563b4e66d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e66d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 -- 172.21.15.23:0/3863120906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b4e37fb60 con 0x563b4e5ffef0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa6800700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e66d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3aa5400700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3a94097390 tx=0x7f3a9400f560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67acf92b7bad0b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.727+0000 7f3a87e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9409e750 con 0x563b4e5ffef0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3a940a2d10 con 0x563b4e5ffef0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a9800faa0 con 0x563b4e5ffef0 2024-03-30T04:13:36.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a9409ab80 con 0x563b4e5ffef0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 10 client.18717 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3aa0017910 con 0x563b4e5ffef0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a9800faa0 con 0x563b4e5ffef0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3a940ba0b0 con 0x563b4e5ffef0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 10 client.18717 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 10 client.18717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 20 client.18717 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.728+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e66f2b0 0x563b4e6716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.729+0000 7f3ac0da1b80 4 client.18717 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.729+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563b4e3a0cf0 con 0x563b4e66f2b0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.729+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e66f2b0 0x563b4e6716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.729+0000 7f3a87e00700 10 client.18717 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:36.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.729+0000 7f3aa5e00700 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e66f2b0 0x563b4e6716a0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ab5eb92f0240732 server_cookie=5d46df88bae11ee1 in_seq=0 out_seq=0 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3a87e00700 1 -- 172.21.15.23:0/3863120906 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563b4e3a0cf0 con 0x563b4e66f2b0 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3a87e00700 10 client.18717 handle_command_reply: tid=0 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 client.18717 shutdown 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 msgr2=0x563b4e60fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3a94097390 tx=0x7f3a9400f560 comp rx=0 tx=0).stop 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 shutdown_connections 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x563b4e60adf0 0x563b4e66d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x563b4e66f2b0 0x563b4e6716a0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b4e5ff5d0 0x563b4e60f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 --2- 172.21.15.23:0/3863120906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x563b4e5ffef0 0x563b4e60fe70 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3a87e00700 0 client.18717 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:36.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 >> 172.21.15.23:0/3863120906 conn(0x563b4e55f6a0 msgr2=0x563b4e602d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 shutdown_connections 2024-03-30T04:13:36.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 1 -- 172.21.15.23:0/3863120906 wait complete. 2024-03-30T04:13:36.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.730+0000 7f3ac0da1b80 20 client.18717 trim_cache size 0 max 16384 2024-03-30T04:13:36.733 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 msgr2=0x7f3a8c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 0x7f3a8c0a3750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000f590 tx=0x7f3aa000b040 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 msgr2=0x7f3aa81999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa81999c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3a940058e0 tx=0x7f3a9400b9c0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 shutdown_connections 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3aa8105ce0 0x7f3aa819dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3aa8067e30 0x7f3aa8199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3a8c0a1290 0x7f3a8c0a3750 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 --2- 172.21.15.23:0/2701715178 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3aa80687f0 0x7f3aa81999c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 >> 172.21.15.23:0/2701715178 conn(0x7f3aa8076d40 msgr2=0x7f3aa810c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:36.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 shutdown_connections 2024-03-30T04:13:36.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:36.731+0000 7f3aad200700 1 -- 172.21.15.23:0/2701715178 wait complete. 2024-03-30T04:13:37.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1140601 9.71 MB/sec execute 494 sec latency 19.789 ms 2024-03-30T04:13:37.749 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:13:37.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:13:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 -- 172.21.15.23:0/2353092741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00069260 msgr2=0x7f9b00069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 --2- 172.21.15.23:0/2353092741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00069260 0x7f9b00069640 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9af4009990 tx=0x7f9af4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 -- 172.21.15.23:0/2353092741 shutdown_connections 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 --2- 172.21.15.23:0/2353092741 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b001060b0 0x7f9b0010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 --2- 172.21.15.23:0/2353092741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00069260 0x7f9b00069640 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 --2- 172.21.15.23:0/2353092741 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b00069c10 0x7f9b00105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.965+0000 7f9b08000700 1 -- 172.21.15.23:0/2353092741 >> 172.21.15.23:0/2353092741 conn(0x7f9b00076d80 msgr2=0x7f9b00077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.966+0000 7f9b08000700 1 -- 172.21.15.23:0/2353092741 shutdown_connections 2024-03-30T04:13:37.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.966+0000 7f9b08000700 1 -- 172.21.15.23:0/2353092741 wait complete. 2024-03-30T04:13:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 Processor -- start 2024-03-30T04:13:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 -- start start 2024-03-30T04:13:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b001060b0 0x7f9b001001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00101930 0x7f9b00100730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b001a9e10 con 0x7f9b001060b0 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b001a9f80 con 0x7f9b00101930 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b08000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b001aa140 con 0x7f9b00069260 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b05600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b05600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57880/0 (socket says 172.21.15.23:57880) 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b05600700 1 -- 172.21.15.23:0/575157016 learned_addr learned my addr 172.21.15.23:0/575157016 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.967+0000 7f9b06000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00101930 0x7f9b00100730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b04c00700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b001060b0 0x7f9b001001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 -- 172.21.15.23:0/575157016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00101930 msgr2=0x7f9b00100730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00101930 0x7f9b00100730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 -- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b001060b0 msgr2=0x7f9b001001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:37.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b001060b0 0x7f9b001001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 -- 172.21.15.23:0/575157016 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec009520 con 0x7f9b00069260 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b05600700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9af4005910 tx=0x7f9af4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9afe000700 1 -- 172.21.15.23:0/575157016 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af401d070 con 0x7f9b00069260 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9afe000700 1 -- 172.21.15.23:0/575157016 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9af4022470 con 0x7f9b00069260 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af40095f0 con 0x7f9b00069260 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.968+0000 7f9afe000700 1 -- 172.21.15.23:0/575157016 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af4017650 con 0x7f9b00069260 2024-03-30T04:13:37.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.969+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b001aa650 con 0x7f9b00069260 2024-03-30T04:13:37.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.972+0000 7f9afe000700 1 -- 172.21.15.23:0/575157016 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9af40225e0 con 0x7f9b00069260 2024-03-30T04:13:37.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 Processor -- start 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 -- start start 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9afe000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 0x7f9ae80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34650 0x55f7b8c34a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9afe000700 1 -- 172.21.15.23:0/575157016 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9af40ce410 con 0x7f9b00069260 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c42260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b8da10 con 0x55f7b8c34650 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b8b3d0 con 0x55f7b8c34f70 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.974+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b86cc0 con 0x55f7b8c3fe70 2024-03-30T04:13:37.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b06000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 0x7f9ae80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b06000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c42260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37336/0 (socket says 172.21.15.23:37336) 2024-03-30T04:13:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 -- 172.21.15.23:0/469162903 learned_addr learned my addr 172.21.15.23:0/469162903 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:37.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b05600700 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34650 0x55f7b8c34a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 -- 172.21.15.23:0/469162903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c3fe70 msgr2=0x55f7b8c42260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c42260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 -- 172.21.15.23:0/469162903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34650 msgr2=0x55f7b8c34a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34650 0x55f7b8c34a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 -- 172.21.15.23:0/469162903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b8a08430 con 0x55f7b8c34f70 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b06000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 0x7f9ae80a3750 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9b00101d80 tx=0x7f9af8003e20 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.975+0000 7f9b04c00700 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009e40 tx=0x7f9aec011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af6ef80dff059f30 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.976+0000 7f9af2600700 1 -- 172.21.15.23:0/469162903 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec004250 con 0x55f7b8c34f70 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.976+0000 7f9af2600700 1 -- 172.21.15.23:0/469162903 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9aec0043b0 con 0x55f7b8c34f70 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.976+0000 7f9af2600700 1 -- 172.21.15.23:0/469162903 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec019790 con 0x55f7b8c34f70 2024-03-30T04:13:37.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.976+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/469162903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 msgr2=0x55f7b8c3f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.976+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009e40 tx=0x7f9aec011720 comp rx=0 tx=0).stop 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/469162903 shutdown_connections 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c42260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34650 0x55f7b8c34a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/469162903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3f930 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/469162903 >> 172.21.15.23:0/469162903 conn(0x55f7b8b94720 msgr2=0x55f7b8c30b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/469162903 shutdown_connections 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/469162903 wait complete. 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 -- start start 2024-03-30T04:13:37.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.977+0000 7f9b1ba7ab80 1 Processor -- start 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c3f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c3d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b22e00 con 0x55f7b8c3fe70 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b22aa0 con 0x55f7b8c34f70 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8a0a720 con 0x55f7b8c34650 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b1ba7ab80 10 client.0 _ll_register_callbacks cb 0x55f7b8b2c4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:37.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37338/0 (socket says 172.21.15.23:37338) 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b06000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c3d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b04c00700 1 -- 172.21.15.23:0/3613837512 learned_addr learned my addr 172.21.15.23:0/3613837512 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b05600700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c3f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.978+0000 7f9b04c00700 1 -- 172.21.15.23:0/3613837512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 msgr2=0x55f7b8c3f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c3f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b04c00700 1 -- 172.21.15.23:0/3613837512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 msgr2=0x55f7b8c3d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c3d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b04c00700 1 -- 172.21.15.23:0/3613837512 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aec009a70 con 0x55f7b8c34f70 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009840 tx=0x7f9aec003930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc6cd614c910453 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9ae6a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec004520 con 0x55f7b8c34f70 2024-03-30T04:13:37.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9aec005040 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7b8b47400 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec005200 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b1ba7ab80 10 client.28706 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.979+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f7b89b4be0 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f7b8b47400 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9aec021070 con 0x55f7b8c34f70 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 10 client.28706 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 10 client.28706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 20 client.28706 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8c4f9d0 0x55f7b8c51df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 4 client.28706 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:37.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7b8b7f780 con 0x55f7b8c4f9d0 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.980+0000 7f9b05600700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8c4f9d0 0x55f7b8c51df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.981+0000 7f9b05600700 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8c4f9d0 0x55f7b8c51df0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=554801827f97954c server_cookie=6f16c330c277f7b6 in_seq=0 out_seq=0 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.981+0000 7f9ae6a00700 10 client.28706 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.981+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3613837512 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f7b8b7f780 con 0x55f7b8c4f9d0 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.981+0000 7f9ae6a00700 10 client.28706 handle_command_reply: tid=0 2024-03-30T04:13:37.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.981+0000 7f9b1ba7ab80 1 client.28706 shutdown 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 msgr2=0x55f7b8c3cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009840 tx=0x7f9aec003930 comp rx=0 tx=0).stop 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 shutdown_connections 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c3d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8c4f9d0 0x55f7b8c51df0 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c3f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9ae6a00700 0 client.28706 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3613837512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34f70 0x55f7b8c3cba0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 >> 172.21.15.23:0/3613837512 conn(0x55f7b8b94720 msgr2=0x55f7b8c36590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 shutdown_connections 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3613837512 wait complete. 2024-03-30T04:13:37.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:37.982+0000 7f9b1ba7ab80 20 client.28706 trim_cache size 0 max 16384 2024-03-30T04:13:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.000+0000 7f9b1ba7ab80 1 Processor -- start 2024-03-30T04:13:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.000+0000 7f9b1ba7ab80 1 -- start start 2024-03-30T04:13:38.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 0x55f7b8c4b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c4b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8c3dd10 con 0x55f7b8c3fe70 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8b7f780 con 0x55f7b8c34650 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8c37540 con 0x55f7b8c34f70 2024-03-30T04:13:38.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b05600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b05600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37342/0 (socket says 172.21.15.23:37342) 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b05600700 1 -- 172.21.15.23:0/2577906249 learned_addr learned my addr 172.21.15.23:0/2577906249 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 0x55f7b8c4b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b06000700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c4b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 -- 172.21.15.23:0/2577906249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 msgr2=0x55f7b8c46af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 -- 172.21.15.23:0/2577906249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 msgr2=0x55f7b8c4b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c4b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 -- 172.21.15.23:0/2577906249 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7b8c56a20 con 0x55f7b8c34f70 2024-03-30T04:13:38.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b05600700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.001+0000 7f9b04c00700 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 0x55f7b8c4b240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d60 tx=0x7f9aec009950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d414119d94765b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9af3000700 1 -- 172.21.15.23:0/2577906249 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec015070 con 0x55f7b8c34f70 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9af3000700 1 -- 172.21.15.23:0/2577906249 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9aec01a630 con 0x55f7b8c34f70 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9af3000700 1 -- 172.21.15.23:0/2577906249 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec005510 con 0x55f7b8c34f70 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/2577906249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 msgr2=0x55f7b8c4b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 0x55f7b8c4b240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d60 tx=0x7f9aec009950 comp rx=0 tx=0).stop 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/2577906249 shutdown_connections 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c4b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c34650 0x55f7b8c46af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/2577906249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34f70 0x55f7b8c4b240 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.002+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/2577906249 >> 172.21.15.23:0/2577906249 conn(0x55f7b8b94720 msgr2=0x55f7b8c3b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.003+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/2577906249 shutdown_connections 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.003+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/2577906249 wait complete. 2024-03-30T04:13:38.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.003+0000 7f9b1ba7ab80 1 -- start start 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.003+0000 7f9b1ba7ab80 1 Processor -- start 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c452b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c45d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8ca3810 con 0x55f7b8c34f70 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8ca3950 con 0x55f7b8c3fe70 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7b8ca3a90 con 0x55f7b8c34650 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b1ba7ab80 10 client.0 _ll_register_callbacks cb 0x55f7b8b2c4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58602/0 (socket says 172.21.15.23:58602) 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 -- 172.21.15.23:0/3285982247 learned_addr learned my addr 172.21.15.23:0/3285982247 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b06000700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c45d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 -- 172.21.15.23:0/3285982247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 msgr2=0x55f7b8c452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b05600700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c452b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 -- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 msgr2=0x55f7b8c45d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c45d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b04c00700 1 -- 172.21.15.23:0/3285982247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af40cbe70 con 0x55f7b8c34f70 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b06000700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c45d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.004+0000 7f9b05600700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9b04c00700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9aec01a4b0 tx=0x7f9aec003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adc7b01b6af08ca server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9ae6a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec015070 con 0x55f7b8c34f70 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9aec0043b0 con 0x55f7b8c34f70 2024-03-30T04:13:38.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9aec000e40 con 0x55f7b8c34f70 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec011a70 con 0x55f7b8c34f70 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9b1ba7ab80 10 client.18735 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f7b89b4be0 con 0x55f7b8c34f70 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.005+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9aec0043b0 con 0x55f7b8c34f70 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9aec021030 con 0x55f7b8c34f70 2024-03-30T04:13:38.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 10 client.18735 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 10 client.18735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 20 client.18735 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8ca4e50 0x55f7b8ca7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 4 client.18735 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f7b89d5d70 con 0x55f7b8ca4e50 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b05600700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8ca4e50 0x55f7b8ca7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9b05600700 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8ca4e50 0x55f7b8ca7270 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5355f66b175dce server_cookie=f2d73528526ce270 in_seq=0 out_seq=0 2024-03-30T04:13:38.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.006+0000 7f9ae6a00700 10 client.18735 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.007+0000 7f9ae6a00700 1 -- 172.21.15.23:0/3285982247 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f7b89d5d70 con 0x55f7b8ca4e50 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.007+0000 7f9ae6a00700 10 client.18735 handle_command_reply: tid=0 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.007+0000 7f9b1ba7ab80 1 client.18735 shutdown 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 msgr2=0x55f7b8c457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9aec01a4b0 tx=0x7f9aec003fb0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 shutdown_connections 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f7b8c3fe70 0x55f7b8c45d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f7b8ca4e50 0x55f7b8ca7270 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f7b8c34650 0x55f7b8c452b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 --2- 172.21.15.23:0/3285982247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f7b8c34f70 0x55f7b8c457f0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 >> 172.21.15.23:0/3285982247 conn(0x55f7b8b94720 msgr2=0x55f7b8c4b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9ae6a00700 0 client.18735 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 shutdown_connections 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 1 -- 172.21.15.23:0/3285982247 wait complete. 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.008+0000 7f9b1ba7ab80 20 client.18735 trim_cache size 0 max 16384 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "52fdeff0-1705-452b-be6f-ba6de15252d1", 2024-03-30T04:13:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:13:38.017 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 msgr2=0x7f9ae80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 0x7f9ae80a3750 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9b00101d80 tx=0x7f9af8003e20 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 msgr2=0x7f9b000ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9af4005910 tx=0x7f9af4004730 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 shutdown_connections 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ae80a1290 0x7f9ae80a3750 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b00101930 0x7f9b00100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b00069260 0x7f9b000ffc70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 --2- 172.21.15.23:0/575157016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b001060b0 0x7f9b001001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 >> 172.21.15.23:0/575157016 conn(0x7f9b00076d80 msgr2=0x7f9b001071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 shutdown_connections 2024-03-30T04:13:38.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.009+0000 7f9b08000700 1 -- 172.21.15.23:0/575157016 wait complete. 2024-03-30T04:13:38.027 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:13:38.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1142116 9.70 MB/sec execute 495 sec latency 33.739 ms 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 -- 172.21.15.23:0/2969437135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea0101920 msgr2=0x7f9ea0105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2969437135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea0101920 0x7f9ea0105960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0098f0 tx=0x7f9e8c009c00 comp rx=0 tx=0).stop 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 -- 172.21.15.23:0/2969437135 shutdown_connections 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2969437135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0106090 0x7f9ea010dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2969437135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0100f70 0x7f9ea0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2969437135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea0101920 0x7f9ea0105960 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 -- 172.21.15.23:0/2969437135 >> 172.21.15.23:0/2969437135 conn(0x7f9ea00fc7f0 msgr2=0x7f9ea00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 -- 172.21.15.23:0/2969437135 shutdown_connections 2024-03-30T04:13:38.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.243+0000 7f9ea8400700 1 -- 172.21.15.23:0/2969437135 wait complete. 2024-03-30T04:13:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.244+0000 7f9ea8400700 1 Processor -- start 2024-03-30T04:13:38.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 -- start start 2024-03-30T04:13:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0100f70 0x7f9ea0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea019de30 0x7f9ea019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019e8b0 con 0x7f9ea019de30 2024-03-30T04:13:38.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019ea20 con 0x7f9ea0106090 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea8400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea019ebe0 con 0x7f9ea0100f70 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0100f70 0x7f9ea0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.245+0000 7f9ea6400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea019de30 0x7f9ea019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37288/0 (socket says 172.21.15.23:37288) 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 -- 172.21.15.23:0/2329437370 learned_addr learned my addr 172.21.15.23:0/2329437370 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 -- 172.21.15.23:0/2329437370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0100f70 msgr2=0x7f9ea0199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0100f70 0x7f9ea0199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 -- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea019de30 msgr2=0x7f9ea019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea019de30 0x7f9ea019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 -- 172.21.15.23:0/2329437370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940095f0 con 0x7f9ea0106090 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009230 tx=0x7f9e8c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9e9e000700 1 -- 172.21.15.23:0/2329437370 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c01c070 con 0x7f9ea0106090 2024-03-30T04:13:38.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8c009520 con 0x7f9ea0106090 2024-03-30T04:13:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.246+0000 7f9e9e000700 1 -- 172.21.15.23:0/2329437370 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e8c00b9c0 con 0x7f9ea0106090 2024-03-30T04:13:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.247+0000 7f9e9e000700 1 -- 172.21.15.23:0/2329437370 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0038e0 con 0x7f9ea0106090 2024-03-30T04:13:38.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.247+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea01a6310 con 0x7f9ea0106090 2024-03-30T04:13:38.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.250+0000 7f9e9e000700 1 -- 172.21.15.23:0/2329437370 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9e8c020070 con 0x7f9ea0106090 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.251+0000 7f9e9e000700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 0x7f9e880a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 Processor -- start 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9e9e000700 1 -- 172.21.15.23:0/2329437370 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9e8c0cdb50 con 0x7f9ea0106090 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 0x7f9e880a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 -- start start 2024-03-30T04:13:38.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f0689179b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f0689251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068870990 con 0x55f0689175d0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f06886e350 con 0x55f068917ef0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.252+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068869c40 con 0x55f068922df0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea6400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f0689251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37300/0 (socket says 172.21.15.23:37300) 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 -- 172.21.15.23:0/3388190845 learned_addr learned my addr 172.21.15.23:0/3388190845 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f0689179b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 0x7f9e880a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e94005fd0 tx=0x7f9e94009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 -- 172.21.15.23:0/3388190845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 msgr2=0x55f0689251e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f0689251e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 -- 172.21.15.23:0/3388190845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 msgr2=0x55f0689179b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f0689179b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 -- 172.21.15.23:0/3388190845 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0686eb3b0 con 0x55f068917ef0 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea6400700 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f0689251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c096e10 tx=0x7f9e8c009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3967b8c12b872a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.253+0000 7f9e92a00700 1 -- 172.21.15.23:0/3388190845 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c09e740 con 0x55f068917ef0 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9e92a00700 1 -- 172.21.15.23:0/3388190845 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e8c0a2d00 con 0x55f068917ef0 2024-03-30T04:13:38.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9e92a00700 1 -- 172.21.15.23:0/3388190845 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c0ba0b0 con 0x55f068917ef0 2024-03-30T04:13:38.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3388190845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 msgr2=0x55f0689228b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c096e10 tx=0x7f9e8c009c00 comp rx=0 tx=0).stop 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3388190845 shutdown_connections 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f0689251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f0689179b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3388190845 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f0689228b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3388190845 >> 172.21.15.23:0/3388190845 conn(0x55f0688776a0 msgr2=0x55f068913a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3388190845 shutdown_connections 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.254+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3388190845 wait complete. 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.255+0000 7f9ebbf1bb80 1 -- start start 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.255+0000 7f9ebbf1bb80 1 Processor -- start 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f06891fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068922df0 0x55f068920180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068805d80 con 0x55f068922df0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068805a20 con 0x55f068917ef0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0686ed6a0 con 0x55f0689175d0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ebbf1bb80 10 client.0 _ll_register_callbacks cb 0x55f06880f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f06891fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea6400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068922df0 0x55f068920180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35978/0 (socket says 172.21.15.23:35978) 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5a00700 1 -- 172.21.15.23:0/3876744065 learned_addr learned my addr 172.21.15.23:0/3876744065 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 -- 172.21.15.23:0/3876744065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 msgr2=0x55f0689227f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 -- 172.21.15.23:0/3876744065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068922df0 msgr2=0x55f068920180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068922df0 0x55f068920180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5000700 1 -- 172.21.15.23:0/3876744065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8c0b8e70 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.256+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ea5000700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f06891fb70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009fd0 tx=0x7f9e8c0972d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d37ebc52fd04526c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c09ad70 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f06882a380 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9e8c099e00 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e8c0052e0 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 10 client.28721 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f068697b60 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e8c064080 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e8c0ba040 con 0x55f068917ef0 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 10 client.28721 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:38.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 10 client.28721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 20 client.28721 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.257+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f068932160 0x55f068934580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.258+0000 7f9ebbf1bb80 4 client.28721 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.258+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f068862700 con 0x55f068932160 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.258+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f068932160 0x55f068934580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.258+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f068932160 0x55f068934580 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c837d81473c4d4e server_cookie=6b86db79974c5429 in_seq=0 out_seq=0 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.258+0000 7f9e87400700 10 client.28721 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9e87400700 1 -- 172.21.15.23:0/3876744065 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f068862700 con 0x55f068932160 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9e87400700 10 client.28721 handle_command_reply: tid=0 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 client.28721 shutdown 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 msgr2=0x55f06891fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f06891fb70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009fd0 tx=0x7f9e8c0972d0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 shutdown_connections 2024-03-30T04:13:38.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068922df0 0x55f068920180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f068932160 0x55f068934580 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689227f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/3876744065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f06891fb70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9e87400700 0 client.28721 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 >> 172.21.15.23:0/3876744065 conn(0x55f0688776a0 msgr2=0x55f068919530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 shutdown_connections 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.259+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/3876744065 wait complete. 2024-03-30T04:13:38.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.260+0000 7f9ebbf1bb80 20 client.28721 trim_cache size 0 max 16384 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 Processor -- start 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 -- start start 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068917ef0 0x55f06890c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 0x55f06891f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f06891ff30 con 0x55f068917ef0 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0689147d0 con 0x55f068922df0 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.282+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068929d20 con 0x55f0689175d0 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea6400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 0x55f06891f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068917ef0 0x55f06890c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35992/0 (socket says 172.21.15.23:35992) 2024-03-30T04:13:38.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea6400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 0x55f06891f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:37314/0 (socket says 172.21.15.23:37314) 2024-03-30T04:13:38.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2007257956 learned_addr learned my addr 172.21.15.23:0/2007257956 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2007257956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 msgr2=0x55f06891f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 0x55f06891f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2007257956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068917ef0 msgr2=0x55f06890c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068917ef0 0x55f06890c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2007257956 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f06882fea0 con 0x55f0689175d0 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.283+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e94011ab0 tx=0x7f9e94014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea300078426750 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.284+0000 7f9e93400700 1 -- 172.21.15.23:0/2007257956 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94023040 con 0x55f0689175d0 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.284+0000 7f9e93400700 1 -- 172.21.15.23:0/2007257956 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e94004500 con 0x55f0689175d0 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.284+0000 7f9e93400700 1 -- 172.21.15.23:0/2007257956 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9401ee40 con 0x55f0689175d0 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.284+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2007257956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 msgr2=0x55f0689297e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.284+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9e94011ab0 tx=0x7f9e94014000 comp rx=0 tx=0).stop 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2007257956 shutdown_connections 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068922df0 0x55f06891f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0689175d0 0x55f0689297e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2007257956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f068917ef0 0x55f06890c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2007257956 >> 172.21.15.23:0/2007257956 conn(0x55f0688776a0 msgr2=0x55f06890cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2007257956 shutdown_connections 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2007257956 wait complete. 2024-03-30T04:13:38.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 -- start start 2024-03-30T04:13:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.285+0000 7f9ebbf1bb80 1 Processor -- start 2024-03-30T04:13:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f068928300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f068928840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f068985f60 con 0x55f0689175d0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0689860a0 con 0x55f068917ef0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0689861e0 con 0x55f068922df0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ebbf1bb80 10 client.0 _ll_register_callbacks cb 0x55f06880f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50934/0 (socket says 172.21.15.23:50934) 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2802697800 learned_addr learned my addr 172.21.15.23:0/2802697800 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea6400700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f068928840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f068928300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2802697800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 msgr2=0x55f068928840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f068928840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 msgr2=0x55f068928300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:13:38.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f068928300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.286+0000 7f9ea5a00700 1 -- 172.21.15.23:0/2802697800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e94017070 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9ea5a00700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009860 tx=0x7f9e940295d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8b7caedf1fb3d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9e87400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94023070 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9e94004240 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e8c0a2d00 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e9401e8b0 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9ebbf1bb80 10 client.18753 fetch_fsmap learned FSMap version 30 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.287+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55f0686efcd0 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e8c0a2d00 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9e94004750 con 0x55f0689175d0 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 10 client.18753 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:13:38.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 10 client.18753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 20 client.18753 populate_metadata read hostname 'smithi023' 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f0689878b0 0x55f068989cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 4 client.18753 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f0686b8cf0 con 0x55f0689878b0 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.288+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f0689878b0 0x55f068989cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ea5000700 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f0689878b0 0x55f068989cd0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5f26a3ec4740d19 server_cookie=77aaccb77e22d689 in_seq=0 out_seq=0 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9e87400700 10 client.18753 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9e87400700 1 -- 172.21.15.23:0/2802697800 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f0686b8cf0 con 0x55f0689878b0 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9e87400700 10 client.18753 handle_command_reply: tid=0 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 client.18753 shutdown 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 msgr2=0x55f068927dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9e94009860 tx=0x7f9e940295d0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 shutdown_connections 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f0689878b0 0x55f068989cd0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f068922df0 0x55f068928840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f0689175d0 0x55f068927dc0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 --2- 172.21.15.23:0/2802697800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f068917ef0 0x55f068928300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9e87400700 0 client.18753 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 >> 172.21.15.23:0/2802697800 conn(0x55f0688776a0 msgr2=0x55f06892e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 shutdown_connections 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.289+0000 7f9ebbf1bb80 1 -- 172.21.15.23:0/2802697800 wait complete. 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ebbf1bb80 20 client.18753 trim_cache size 0 max 16384 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:13:38.292 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: "52fdeff0-1705-452b-be6f-ba6de15252d1": { 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "52fdeff0-1705-452b-be6f-ba6de15252d1", 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:13:38.293 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 msgr2=0x7f9e880a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 0x7f9e880a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9e94005fd0 tx=0x7f9e94009310 comp rx=0 tx=0).stop 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 msgr2=0x7f9ea0199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c009230 tx=0x7f9e8c00fa00 comp rx=0 tx=0).stop 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 shutdown_connections 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9ea019de30 0x7f9ea019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ea0100f70 0x7f9ea0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9e880a1290 0x7f9e880a3750 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 --2- 172.21.15.23:0/2329437370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ea0106090 0x7f9ea0199a90 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:13:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 >> 172.21.15.23:0/2329437370 conn(0x7f9ea00fc7f0 msgr2=0x7f9ea00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:13:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.290+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 shutdown_connections 2024-03-30T04:13:38.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:13:38.291+0000 7f9ea8400700 1 -- 172.21.15.23:0/2329437370 wait complete. 2024-03-30T04:13:38.307 INFO:tasks.cephfs.filesystem:scrub status for tag:52fdeff0-1705-452b-be6f-ba6de15252d1 - {'path': '/', 'tag': '52fdeff0-1705-452b-be6f-ba6de15252d1', 'options': 'recursive,force'} 2024-03-30T04:13:38.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:38 smithi154 ceph-mon[79161]: pgmap v365: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 142 op/s 2024-03-30T04:13:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:38 smithi193 ceph-mon[79581]: pgmap v365: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 142 op/s 2024-03-30T04:13:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:38 smithi023 ceph-mon[38254]: pgmap v365: 129 pgs: 129 active+clean; 570 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 142 op/s 2024-03-30T04:13:39.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1143818 9.71 MB/sec execute 496 sec latency 71.154 ms 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:39 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:13:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:13:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:13:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.494 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:39 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:39 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:13:40.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1144627 9.69 MB/sec execute 497 sec latency 29.398 ms 2024-03-30T04:13:40.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:40 smithi154 ceph-mon[79161]: pgmap v366: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 210 op/s 2024-03-30T04:13:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:40 smithi193 ceph-mon[79581]: pgmap v366: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 210 op/s 2024-03-30T04:13:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:40 smithi023 ceph-mon[38254]: pgmap v366: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 210 op/s 2024-03-30T04:13:41.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1146643 9.69 MB/sec execute 498 sec latency 32.826 ms 2024-03-30T04:13:42.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1147707 9.69 MB/sec execute 499 sec latency 21.989 ms 2024-03-30T04:13:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:42 smithi154 ceph-mon[79161]: pgmap v367: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:42 smithi193 ceph-mon[79581]: pgmap v367: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:42 smithi023 ceph-mon[38254]: pgmap v367: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:43.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1148788 9.68 MB/sec execute 500 sec latency 30.323 ms 2024-03-30T04:13:44.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1150481 9.68 MB/sec execute 501 sec latency 94.666 ms 2024-03-30T04:13:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:44 smithi193 ceph-mon[79581]: pgmap v368: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-03-30T04:13:44.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:44 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:13:44.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:44 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:13:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:44 smithi023 ceph-mon[38254]: pgmap v368: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-03-30T04:13:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:44 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:13:44.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:44 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:13:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:44 smithi154 ceph-mon[79161]: pgmap v368: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.1 MiB/s wr, 200 op/s 2024-03-30T04:13:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:44 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:13:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:44 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:13:45.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1151293 9.67 MB/sec execute 502 sec latency 37.401 ms 2024-03-30T04:13:46.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1152729 9.66 MB/sec execute 503 sec latency 25.832 ms 2024-03-30T04:13:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:46 smithi193 ceph-mon[79581]: pgmap v369: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:46 smithi023 ceph-mon[38254]: pgmap v369: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:46 smithi154 ceph-mon[79161]: pgmap v369: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:47.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1154469 9.67 MB/sec execute 504 sec latency 109.778 ms 2024-03-30T04:13:48.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1155318 9.65 MB/sec execute 505 sec latency 23.478 ms 2024-03-30T04:13:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:48 smithi193 ceph-mon[79581]: pgmap v370: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:48 smithi023 ceph-mon[38254]: pgmap v370: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:48 smithi154 ceph-mon[79161]: pgmap v370: 129 pgs: 129 active+clean; 569 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:13:49.231 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1156801 9.64 MB/sec execute 506 sec latency 122.601 ms 2024-03-30T04:13:50.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1158161 9.65 MB/sec execute 507 sec latency 19.675 ms 2024-03-30T04:13:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:50 smithi193 ceph-mon[79581]: pgmap v371: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:13:50.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:50 smithi023 ceph-mon[38254]: pgmap v371: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:13:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:50 smithi154 ceph-mon[79161]: pgmap v371: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:13:51.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1159379 9.64 MB/sec execute 508 sec latency 196.571 ms 2024-03-30T04:13:52.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1161471 9.65 MB/sec execute 509 sec latency 24.649 ms 2024-03-30T04:13:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:52 smithi193 ceph-mon[79581]: pgmap v372: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:13:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:52 smithi023 ceph-mon[38254]: pgmap v372: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:13:52.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:52 smithi154 ceph-mon[79161]: pgmap v372: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:13:53.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1162196 9.63 MB/sec execute 510 sec latency 28.023 ms 2024-03-30T04:13:54.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1163674 9.62 MB/sec execute 511 sec latency 122.279 ms 2024-03-30T04:13:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:54 smithi193 ceph-mon[79581]: pgmap v373: 129 pgs: 129 active+clean; 571 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-03-30T04:13:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:54 smithi023 ceph-mon[38254]: pgmap v373: 129 pgs: 129 active+clean; 571 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-03-30T04:13:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:54 smithi154 ceph-mon[79161]: pgmap v373: 129 pgs: 129 active+clean; 571 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-03-30T04:13:55.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1165138 9.63 MB/sec execute 512 sec latency 45.475 ms 2024-03-30T04:13:55.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:55 smithi023 ceph-mon[38254]: pgmap v374: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:55 smithi154 ceph-mon[79161]: pgmap v374: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:55.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:55 smithi193 ceph-mon[79581]: pgmap v374: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:13:56.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1166275 9.61 MB/sec execute 513 sec latency 28.386 ms 2024-03-30T04:13:57.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1168506 9.63 MB/sec execute 514 sec latency 20.876 ms 2024-03-30T04:13:58.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1169377 9.61 MB/sec execute 515 sec latency 28.956 ms 2024-03-30T04:13:58.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:13:58 smithi154 ceph-mon[79161]: pgmap v375: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 135 op/s 2024-03-30T04:13:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:13:58 smithi193 ceph-mon[79581]: pgmap v375: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 135 op/s 2024-03-30T04:13:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:13:58 smithi023 ceph-mon[38254]: pgmap v375: 129 pgs: 129 active+clean; 570 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 135 op/s 2024-03-30T04:13:59.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1171016 9.61 MB/sec execute 516 sec latency 85.219 ms 2024-03-30T04:14:00.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1172250 9.61 MB/sec execute 517 sec latency 60.766 ms 2024-03-30T04:14:00.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:00 smithi154 ceph-mon[79161]: pgmap v376: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 203 op/s 2024-03-30T04:14:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:00 smithi193 ceph-mon[79581]: pgmap v376: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 203 op/s 2024-03-30T04:14:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:00 smithi023 ceph-mon[38254]: pgmap v376: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.1 MiB/s wr, 203 op/s 2024-03-30T04:14:01.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1173123 9.59 MB/sec execute 518 sec latency 72.472 ms 2024-03-30T04:14:02.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1173360 9.58 MB/sec execute 519 sec latency 89.932 ms 2024-03-30T04:14:02.433 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:02 smithi154 ceph-mon[79161]: pgmap v377: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:14:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:02 smithi193 ceph-mon[79581]: pgmap v377: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:14:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:02 smithi023 ceph-mon[38254]: pgmap v377: 129 pgs: 129 active+clean; 572 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 136 op/s 2024-03-30T04:14:03.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1175486 9.58 MB/sec execute 520 sec latency 18.989 ms 2024-03-30T04:14:04.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1176336 9.57 MB/sec execute 521 sec latency 23.219 ms 2024-03-30T04:14:04.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:04 smithi154 ceph-mon[79161]: pgmap v378: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-03-30T04:14:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:04 smithi193 ceph-mon[79581]: pgmap v378: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-03-30T04:14:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:04 smithi023 ceph-mon[38254]: pgmap v378: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.0 MiB/s wr, 204 op/s 2024-03-30T04:14:05.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1177985 9.57 MB/sec execute 522 sec latency 38.636 ms 2024-03-30T04:14:06.232 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1179360 9.57 MB/sec execute 523 sec latency 61.644 ms 2024-03-30T04:14:06.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:06 smithi154 ceph-mon[79161]: pgmap v379: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:14:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:06 smithi193 ceph-mon[79581]: pgmap v379: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:14:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:06 smithi023 ceph-mon[38254]: pgmap v379: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.9 MiB/s wr, 167 op/s 2024-03-30T04:14:07.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1180693 9.56 MB/sec execute 524 sec latency 34.138 ms 2024-03-30T04:14:08.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1182755 9.57 MB/sec execute 525 sec latency 27.632 ms 2024-03-30T04:14:08.308 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:14:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:08 smithi193 ceph-mon[79581]: pgmap v380: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 -- 172.21.15.23:0/837005906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb074100f60 msgr2=0x7fb074101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 --2- 172.21.15.23:0/837005906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb074100f60 0x7fb074101340 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb068009990 tx=0x7fb068009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 -- 172.21.15.23:0/837005906 shutdown_connections 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 --2- 172.21.15.23:0/837005906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074106080 0x7fb07410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 --2- 172.21.15.23:0/837005906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb074100f60 0x7fb074101340 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 --2- 172.21.15.23:0/837005906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074101910 0x7fb074105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 -- 172.21.15.23:0/837005906 >> 172.21.15.23:0/837005906 conn(0x7fb0740fc7c0 msgr2=0x7fb0740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 -- 172.21.15.23:0/837005906 shutdown_connections 2024-03-30T04:14:08.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.522+0000 7fb07bc00700 1 -- 172.21.15.23:0/837005906 wait complete. 2024-03-30T04:14:08.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:08 smithi023 ceph-mon[38254]: pgmap v380: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.523+0000 7fb07bc00700 1 Processor -- start 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.523+0000 7fb07bc00700 1 -- start start 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074106080 0x7fb074199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 0x7fb07419e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07419e7b0 con 0x7fb07419dd30 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07419e920 con 0x7fb074106080 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb07bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07419eae0 con 0x7fb074100f60 2024-03-30T04:14:08.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34924/0 (socket says 172.21.15.23:34924) 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 -- 172.21.15.23:0/3755183890 learned_addr learned my addr 172.21.15.23:0/3755183890 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb073e00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074106080 0x7fb074199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079c00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 0x7fb07419e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 -- 172.21.15.23:0/3755183890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074106080 msgr2=0x7fb074199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074106080 0x7fb074199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 -- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 msgr2=0x7fb07419e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 0x7fb07419e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079200700 1 -- 172.21.15.23:0/3755183890 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680095f0 con 0x7fb074100f60 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.524+0000 7fb079c00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 0x7fb07419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:08.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb079200700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb068005910 tx=0x7fb068004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb071600700 1 -- 172.21.15.23:0/3755183890 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06801d070 con 0x7fb074100f60 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb071600700 1 -- 172.21.15.23:0/3755183890 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb068022470 con 0x7fb074100f60 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0741a5fe0 con 0x7fb074100f60 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb071600700 1 -- 172.21.15.23:0/3755183890 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb068017650 con 0x7fb074100f60 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.525+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0741a63a0 con 0x7fb074100f60 2024-03-30T04:14:08.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.528+0000 7fb071600700 1 -- 172.21.15.23:0/3755183890 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb0680225e0 con 0x7fb074100f60 2024-03-30T04:14:08.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.529+0000 7fb08f77cb80 1 Processor -- start 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 -- start start 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb071600700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 0x7fb05c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655c9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655ca11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655bec990 con 0x558655c93ef0 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655bea350 con 0x558655c9edf0 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655be5c40 con 0x558655c935d0 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb071600700 1 -- 172.21.15.23:0/3755183890 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb0680ce430 con 0x7fb074100f60 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb079c00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 0x7fb05c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb073e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655c9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb079c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655ca11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34934/0 (socket says 172.21.15.23:34934) 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.530+0000 7fb079200700 1 -- 172.21.15.23:0/535972420 learned_addr learned my addr 172.21.15.23:0/535972420 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 -- 172.21.15.23:0/535972420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 msgr2=0x558655ca11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655ca11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 -- 172.21.15.23:0/535972420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 msgr2=0x558655c9e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655c9e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 -- 172.21.15.23:0/535972420 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558655a673b0 con 0x558655c935d0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb073e00700 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655c9e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079200700 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb0680a3170 tx=0x7fb068009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dfff470861390a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb079c00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 0x7fb05c0a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0094f0 tx=0x7fb06c009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb066000700 1 -- 172.21.15.23:0/535972420 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06809ac10 con 0x558655c935d0 2024-03-30T04:14:08.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb066000700 1 -- 172.21.15.23:0/535972420 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0680cbe90 con 0x558655c935d0 2024-03-30T04:14:08.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.531+0000 7fb066000700 1 -- 172.21.15.23:0/535972420 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0680a36a0 con 0x558655c935d0 2024-03-30T04:14:08.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 -- 172.21.15.23:0/535972420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 msgr2=0x558655c939b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb0680a3170 tx=0x7fb068009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 -- 172.21.15.23:0/535972420 shutdown_connections 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655ca11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655c939b0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/535972420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655c9e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 -- 172.21.15.23:0/535972420 >> 172.21.15.23:0/535972420 conn(0x558655bf36a0 msgr2=0x558655c8fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 -- 172.21.15.23:0/535972420 shutdown_connections 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.532+0000 7fb08f77cb80 1 -- 172.21.15.23:0/535972420 wait complete. 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 -- start start 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 Processor -- start 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c9e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c93ef0 0x558655c9ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655b81d80 con 0x558655c935d0 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655b81a20 con 0x558655c9edf0 2024-03-30T04:14:08.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655a696a0 con 0x558655c93ef0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.533+0000 7fb08f77cb80 10 client.0 _ll_register_callbacks cb 0x558655b8b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb073e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c93ef0 0x558655c9ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56478/0 (socket says 172.21.15.23:56478) 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 -- 172.21.15.23:0/3415660859 learned_addr learned my addr 172.21.15.23:0/3415660859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079200700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c9e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 -- 172.21.15.23:0/3415660859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c93ef0 msgr2=0x558655c9ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c93ef0 0x558655c9ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 -- 172.21.15.23:0/3415660859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 msgr2=0x558655c9e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c9e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079c00700 1 -- 172.21.15.23:0/3415660859 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0600098e0 con 0x558655c9edf0 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.534+0000 7fb079200700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c9e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:08.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb079c00700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb06c01b6d0 tx=0x7fb06c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65dbcb526a95ae13 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb05aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c020070 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb06c017e60 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558655ba6380 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb06c01bea0 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb08f77cb80 10 client.28739 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.535+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fb0741a63a0 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558655ba6380 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb06c02e070 con 0x558655c9edf0 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 10 client.28739 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 10 client.28739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 20 client.28739 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655cae5e0 0x558655cb0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 4 client.28739 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558655bde700 con 0x558655cae5e0 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.536+0000 7fb079200700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655cae5e0 0x558655cb0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.537+0000 7fb079200700 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655cae5e0 0x558655cb0a00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=535de0882309db6c server_cookie=e4b652f8692ca06c in_seq=0 out_seq=0 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.537+0000 7fb05aa00700 10 client.28739 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.537+0000 7fb05aa00700 1 -- 172.21.15.23:0/3415660859 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558655bde700 con 0x558655cae5e0 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.537+0000 7fb05aa00700 10 client.28739 handle_command_reply: tid=0 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.537+0000 7fb08f77cb80 1 client.28739 shutdown 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 msgr2=0x558655c9bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb06c01b6d0 tx=0x7fb06c00b040 comp rx=0 tx=0).stop 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 shutdown_connections 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655c9bfc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c9e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655cae5e0 0x558655cb0a00 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/3415660859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c93ef0 0x558655c9ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb05aa00700 0 client.28739 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 >> 172.21.15.23:0/3415660859 conn(0x558655bf36a0 msgr2=0x558655c956c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 shutdown_connections 2024-03-30T04:14:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 1 -- 172.21.15.23:0/3415660859 wait complete. 2024-03-30T04:14:08.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.538+0000 7fb08f77cb80 20 client.28739 trim_cache size 0 max 16384 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.556+0000 7fb08f77cb80 1 Processor -- start 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.556+0000 7fb08f77cb80 1 -- start start 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.556+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c9edf0 0x558655c885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655c8f560 con 0x558655c935d0 2024-03-30T04:14:08.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655c90690 con 0x558655c93ef0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655c907f0 con 0x558655c9edf0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb079200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56494/0 (socket says 172.21.15.23:56494) 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb079200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54080/0 (socket says 172.21.15.23:54080) 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 -- 172.21.15.23:0/1706816344 learned_addr learned my addr 172.21.15.23:0/1706816344 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb079c00700 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c9edf0 0x558655c885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 -- 172.21.15.23:0/1706816344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c9edf0 msgr2=0x558655c885a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c9edf0 0x558655c885a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 -- 172.21.15.23:0/1706816344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 msgr2=0x558655c8ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 -- 172.21.15.23:0/1706816344 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558655a13b60 con 0x558655c93ef0 2024-03-30T04:14:08.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb079200700 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.557+0000 7fb073e00700 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb0600104b0 tx=0x7fb0600107c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24e2cdefb8f1bedd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb066a00700 1 -- 172.21.15.23:0/1706816344 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060016050 con 0x558655c93ef0 2024-03-30T04:14:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb066a00700 1 -- 172.21.15.23:0/1706816344 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb060017040 con 0x558655c93ef0 2024-03-30T04:14:08.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb066a00700 1 -- 172.21.15.23:0/1706816344 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb060005280 con 0x558655c93ef0 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1706816344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 msgr2=0x558655ca3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb0600104b0 tx=0x7fb0600107c0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1706816344 shutdown_connections 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c9edf0 0x558655c885a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c935d0 0x558655c8ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1706816344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c93ef0 0x558655ca3200 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.558+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1706816344 >> 172.21.15.23:0/1706816344 conn(0x558655bf36a0 msgr2=0x558655cb5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.559+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1706816344 shutdown_connections 2024-03-30T04:14:08.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.559+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1706816344 wait complete. 2024-03-30T04:14:08.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.559+0000 7fb08f77cb80 1 -- start start 2024-03-30T04:14:08.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.559+0000 7fb08f77cb80 1 Processor -- start 2024-03-30T04:14:08.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655ca19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655cfa8d0 con 0x558655c93ef0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655cfaa40 con 0x558655c9edf0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558655cfac00 con 0x558655c935d0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb08f77cb80 10 client.0 _ll_register_callbacks cb 0x558655b8b470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb079c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34978/0 (socket says 172.21.15.23:34978) 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb079200700 1 -- 172.21.15.23:0/1675602788 learned_addr learned my addr 172.21.15.23:0/1675602788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb073e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655ca19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.560+0000 7fb079c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56506/0 (socket says 172.21.15.23:56506) 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 -- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 msgr2=0x558655cfa390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 -- 172.21.15.23:0/1675602788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 msgr2=0x558655ca19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655ca19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 -- 172.21.15.23:0/1675602788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558655a673b0 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079c00700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb079200700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb0680977b0 tx=0x7fb06801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52e3e9679652ca0a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb05aa00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06809b9b0 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0680cbe90 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.561+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb0680b1020 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0680a36a0 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 10 client.28720 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fb060009d10 con 0x558655c935d0 2024-03-30T04:14:08.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb0680cbe90 con 0x558655c935d0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb068081020 con 0x558655c935d0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 10 client.28720 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 10 client.28720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 20 client.28720 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655d031a0 0x558655d05590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 4 client.28720 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558655a34cf0 con 0x558655d031a0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.562+0000 7fb073e00700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655d031a0 0x558655d05590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb073e00700 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655d031a0 0x558655d05590 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e72ac1f1b3b577f server_cookie=6c488cffed217951 in_seq=0 out_seq=0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb05aa00700 10 client.28720 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb05aa00700 1 -- 172.21.15.23:0/1675602788 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558655a34cf0 con 0x558655d031a0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb05aa00700 10 client.28720 handle_command_reply: tid=0 2024-03-30T04:14:08.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 client.28720 shutdown 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 msgr2=0x558655ca1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fb0680977b0 tx=0x7fb06801a040 comp rx=0 tx=0).stop 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 shutdown_connections 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558655c9edf0 0x558655cfa390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558655d031a0 0x558655d05590 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558655c935d0 0x558655ca1470 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 --2- 172.21.15.23:0/1675602788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558655c93ef0 0x558655ca19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 >> 172.21.15.23:0/1675602788 conn(0x558655bf36a0 msgr2=0x558655c885a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb05aa00700 0 client.28720 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 shutdown_connections 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.563+0000 7fb08f77cb80 1 -- 172.21.15.23:0/1675602788 wait complete. 2024-03-30T04:14:08.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb08f77cb80 20 client.28720 trim_cache size 0 max 16384 2024-03-30T04:14:08.570 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:14:08.570 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:14:08.570 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 msgr2=0x7fb05c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 0x7fb05c0a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb06c0094f0 tx=0x7fb06c009920 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 msgr2=0x7fb074199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fb068005910 tx=0x7fb068004730 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 shutdown_connections 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb07419dd30 0x7fb07419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb074100f60 0x7fb074199450 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb05c0a1290 0x7fb05c0a3750 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 --2- 172.21.15.23:0/3755183890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb074106080 0x7fb074199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.564+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 >> 172.21.15.23:0/3755183890 conn(0x7fb0740fc7c0 msgr2=0x7fb0740fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.565+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 shutdown_connections 2024-03-30T04:14:08.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.565+0000 7fb07bc00700 1 -- 172.21.15.23:0/3755183890 wait complete. 2024-03-30T04:14:08.581 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:14:08.582 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:14:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:08 smithi154 ceph-mon[79161]: pgmap v380: 129 pgs: 129 active+clean; 567 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-03-30T04:14:08.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.794+0000 7efde8600700 1 -- 172.21.15.23:0/776705355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde0100e60 msgr2=0x7efde0101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.794+0000 7efde8600700 1 --2- 172.21.15.23:0/776705355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde0100e60 0x7efde0101240 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009990 tx=0x7efdd4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 -- 172.21.15.23:0/776705355 shutdown_connections 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 --2- 172.21.15.23:0/776705355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 --2- 172.21.15.23:0/776705355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde0100e60 0x7efde0101240 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 --2- 172.21.15.23:0/776705355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0101810 0x7efde0105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 -- 172.21.15.23:0/776705355 >> 172.21.15.23:0/776705355 conn(0x7efde00fc700 msgr2=0x7efde00feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 -- 172.21.15.23:0/776705355 shutdown_connections 2024-03-30T04:14:08.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.795+0000 7efde8600700 1 -- 172.21.15.23:0/776705355 wait complete. 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 Processor -- start 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 -- start start 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde0199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 0x7efde019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde019e660 con 0x7efde0105f80 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde019e7d0 con 0x7efde0100e60 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde8600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde019e990 con 0x7efde019dbe0 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde5c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde5200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde0199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde5c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56526/0 (socket says 172.21.15.23:56526) 2024-03-30T04:14:08.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.796+0000 7efde5c00700 1 -- 172.21.15.23:0/2705351251 learned_addr learned my addr 172.21.15.23:0/2705351251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde6600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 0x7efde019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 -- 172.21.15.23:0/2705351251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 msgr2=0x7efde019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 0x7efde019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 -- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 msgr2=0x7efde0199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde0199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 -- 172.21.15.23:0/2705351251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd40095f0 con 0x7efde0100e60 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5200700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde0199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde6600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 0x7efde019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efde5c00700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efdd4004760 tx=0x7efdd4004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.797+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd401d070 con 0x7efde0100e60 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.798+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdd4022470 con 0x7efde0100e60 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.798+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde01a5e90 con 0x7efde0100e60 2024-03-30T04:14:08.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.798+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4017650 con 0x7efde0100e60 2024-03-30T04:14:08.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.798+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde01a6250 con 0x7efde0100e60 2024-03-30T04:14:08.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.799+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde0102330 con 0x7efde0100e60 2024-03-30T04:14:08.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.801+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7efdd40225e0 con 0x7efde0100e60 2024-03-30T04:14:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.802+0000 7efdde000700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 0x7efdc80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:08.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.802+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9512+0+0 (secure 0 0 0) 0x7efdd40ce2b0 con 0x7efde0100e60 2024-03-30T04:14:08.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.802+0000 7efde5200700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 0x7efdc80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:08.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.803+0000 7efde5200700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 0x7efdc80a3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efde019a850 tx=0x7efdcc009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:08.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:08.806+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7efdd405d9f0 con 0x7efde0100e60 2024-03-30T04:14:09.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.118+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efde0067a10 con 0x7efde0100e60 2024-03-30T04:14:09.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.120+0000 7efdde000700 1 -- 172.21.15.23:0/2705351251 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7efdd40973b0 con 0x7efde0100e60 2024-03-30T04:14:09.119 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:14:09.119 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:14:09.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.124+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 msgr2=0x7efdc80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.124+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 0x7efdc80a3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7efde019a850 tx=0x7efdcc009260 comp rx=0 tx=0).stop 2024-03-30T04:14:09.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.124+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 msgr2=0x7efde0199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.124+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7efdd4004760 tx=0x7efdd4004790 comp rx=0 tx=0).stop 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 shutdown_connections 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efde019dbe0 0x7efde019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7efdc80a1290 0x7efdc80a3750 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efde0100e60 0x7efde0199300 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 --2- 172.21.15.23:0/2705351251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efde0105f80 0x7efde0199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.125+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 >> 172.21.15.23:0/2705351251 conn(0x7efde00fc700 msgr2=0x7efde00fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.126+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 shutdown_connections 2024-03-30T04:14:09.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.126+0000 7efde8600700 1 -- 172.21.15.23:0/2705351251 wait complete. 2024-03-30T04:14:09.128 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:14:09.162 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:14:09.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1183470 9.56 MB/sec execute 526 sec latency 78.669 ms 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 -- 172.21.15.23:0/892498205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c101510 msgr2=0x7f3c1c10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 --2- 172.21.15.23:0/892498205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c101510 0x7f3c1c10d9d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400b0a0 tx=0x7f3c1400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 -- 172.21.15.23:0/892498205 shutdown_connections 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 --2- 172.21.15.23:0/892498205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c101510 0x7f3c1c10d9d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 --2- 172.21.15.23:0/892498205 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c106c10 0x7f3c1c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 --2- 172.21.15.23:0/892498205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c100b50 0x7f3c1c100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 -- 172.21.15.23:0/892498205 >> 172.21.15.23:0/892498205 conn(0x7f3c1c0fc620 msgr2=0x7f3c1c0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 -- 172.21.15.23:0/892498205 shutdown_connections 2024-03-30T04:14:09.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.448+0000 7f3c23000700 1 -- 172.21.15.23:0/892498205 wait complete. 2024-03-30T04:14:09.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.449+0000 7f3c23000700 1 Processor -- start 2024-03-30T04:14:09.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.449+0000 7f3c23000700 1 -- start start 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c100b50 0x7f3c1c199290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c106c10 0x7f3c1c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c19e1b0 con 0x7f3c1c100b50 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c19e320 con 0x7f3c1c101510 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c23000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c19e4e0 con 0x7f3c1c106c10 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56540/0 (socket says 172.21.15.23:56540) 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c21200700 1 -- 172.21.15.23:0/459307854 learned_addr learned my addr 172.21.15.23:0/459307854 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c21c00700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c100b50 0x7f3c1c199290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.450+0000 7f3c22600700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c106c10 0x7f3c1c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 -- 172.21.15.23:0/459307854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c106c10 msgr2=0x7f3c1c19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c106c10 0x7f3c1c19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 -- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c100b50 msgr2=0x7f3c1c199290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c100b50 0x7f3c1c199290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 -- 172.21.15.23:0/459307854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c14009d00 con 0x7f3c1c101510 2024-03-30T04:14:09.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.451+0000 7f3c21200700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9a0 tx=0x7f3c0800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.452+0000 7f3c1a000700 1 -- 172.21.15.23:0/459307854 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08009790 con 0x7f3c1c101510 2024-03-30T04:14:09.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.452+0000 7f3c1a000700 1 -- 172.21.15.23:0/459307854 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c08010430 con 0x7f3c1c101510 2024-03-30T04:14:09.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.452+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c19e7c0 con 0x7f3c1c101510 2024-03-30T04:14:09.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.452+0000 7f3c1a000700 1 -- 172.21.15.23:0/459307854 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018680 con 0x7f3c1c101510 2024-03-30T04:14:09.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.452+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c1a6000 con 0x7f3c1c101510 2024-03-30T04:14:09.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.455+0000 7f3c1a000700 1 -- 172.21.15.23:0/459307854 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3c080105a0 con 0x7f3c1c101510 2024-03-30T04:14:09.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.457+0000 7f3c1a000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 0x7f3c040a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.457+0000 7f3c1a000700 1 -- 172.21.15.23:0/459307854 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f3c08014070 con 0x7f3c1c101510 2024-03-30T04:14:09.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.457+0000 7f3c21c00700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 0x7f3c040a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 Processor -- start 2024-03-30T04:14:09.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 -- start start 2024-03-30T04:14:09.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c21c00700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 0x7f3c040a3750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009c60 tx=0x7f3c10009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610aba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60ff9990 con 0x564e610a10c0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60ff7350 con 0x564e610a07a0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.458+0000 7f3c38045b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60ff2c40 con 0x564e610abfc0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610aba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56548/0 (socket says 172.21.15.23:56548) 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 -- 172.21.15.23:0/4213020355 learned_addr learned my addr 172.21.15.23:0/4213020355 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c22600700 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 -- 172.21.15.23:0/4213020355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 msgr2=0x564e610ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 -- 172.21.15.23:0/4213020355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 msgr2=0x564e610aba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610aba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 -- 172.21.15.23:0/4213020355 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e60e743b0 con 0x564e610a07a0 2024-03-30T04:14:09.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21200700 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.459+0000 7f3c21c00700 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e4e0 tx=0x7f3c10014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8968d6ce5e3c1baf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.460+0000 7f3c0ea00700 1 -- 172.21.15.23:0/4213020355 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000ee00 con 0x564e610a07a0 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.460+0000 7f3c0ea00700 1 -- 172.21.15.23:0/4213020355 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c10021070 con 0x564e610a07a0 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.460+0000 7f3c0ea00700 1 -- 172.21.15.23:0/4213020355 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1001d720 con 0x564e610a07a0 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 -- 172.21.15.23:0/4213020355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 msgr2=0x564e610a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e4e0 tx=0x7f3c10014000 comp rx=0 tx=0).stop 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 -- 172.21.15.23:0/4213020355 shutdown_connections 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610a0b80 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 --2- 172.21.15.23:0/4213020355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610aba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 -- 172.21.15.23:0/4213020355 >> 172.21.15.23:0/4213020355 conn(0x564e610006a0 msgr2=0x564e6109cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 -- 172.21.15.23:0/4213020355 shutdown_connections 2024-03-30T04:14:09.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.461+0000 7f3c38045b80 1 -- 172.21.15.23:0/4213020355 wait complete. 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.462+0000 7f3c38045b80 1 -- start start 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.462+0000 7f3c38045b80 1 Processor -- start 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 0x564e610aba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610a8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610abfc0 0x564e610a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60f8ed80 con 0x564e610abfc0 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60f8ea20 con 0x564e610a10c0 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e60e766a0 con 0x564e610a07a0 2024-03-30T04:14:09.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c38045b80 10 client.0 _ll_register_callbacks cb 0x564e60f98470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610a8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610a8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56556/0 (socket says 172.21.15.23:56556) 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 0x564e610aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c21200700 1 -- 172.21.15.23:0/3622256023 learned_addr learned my addr 172.21.15.23:0/3622256023 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.463+0000 7f3c22600700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610abfc0 0x564e610a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 -- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 msgr2=0x564e610a8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610a8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 -- 172.21.15.23:0/3622256023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610abfc0 msgr2=0x564e610a94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610abfc0 0x564e610a94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 -- 172.21.15.23:0/3622256023 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c1a6000 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 0x564e610aba30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000ead0 tx=0x7f3c1000ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd8f2cdf8f9721e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c03400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10005250 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f3c100053b0 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.464+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e60fb3380 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c10005570 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 10 client.28738 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3c10017070 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e60fb3380 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3c1002f640 con 0x564e610a07a0 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 10 client.28738 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:09.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 10 client.28738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 20 client.28738 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e610bbaa0 0x564e610bdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 4 client.28738 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.465+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e60feb700 con 0x564e610bbaa0 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.466+0000 7f3c21200700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e610bbaa0 0x564e610bdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.466+0000 7f3c21200700 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e610bbaa0 0x564e610bdec0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=88c371b78bb21d5f server_cookie=e8cf3200d27e7695 in_seq=0 out_seq=0 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.466+0000 7f3c03400700 10 client.28738 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c03400700 1 -- 172.21.15.23:0/3622256023 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564e60feb700 con 0x564e610bbaa0 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c03400700 10 client.28738 handle_command_reply: tid=0 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 client.28738 shutdown 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 msgr2=0x564e610aba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 0x564e610aba30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000ead0 tx=0x7f3c1000ec00 comp rx=0 tx=0).stop 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 shutdown_connections 2024-03-30T04:14:09.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610abfc0 0x564e610a94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e610bbaa0 0x564e610bdec0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610a07a0 0x564e610aba30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3622256023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610a8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c03400700 0 client.28738 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 >> 172.21.15.23:0/3622256023 conn(0x564e610006a0 msgr2=0x564e610a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 shutdown_connections 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.467+0000 7f3c38045b80 1 -- 172.21.15.23:0/3622256023 wait complete. 2024-03-30T04:14:09.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.468+0000 7f3c38045b80 20 client.28738 trim_cache size 0 max 16384 2024-03-30T04:14:09.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.482+0000 7f3c38045b80 1 Processor -- start 2024-03-30T04:14:09.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.482+0000 7f3c38045b80 1 -- start start 2024-03-30T04:14:09.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a07a0 0x564e610b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e610a2130 con 0x564e610a07a0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e610a9810 con 0x564e610a10c0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c38045b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e610b31b0 con 0x564e610abfc0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c22600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a07a0 0x564e610b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56572/0 (socket says 172.21.15.23:56572) 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c22600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35034/0 (socket says 172.21.15.23:35034) 2024-03-30T04:14:09.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 -- 172.21.15.23:0/2763751564 learned_addr learned my addr 172.21.15.23:0/2763751564 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 -- 172.21.15.23:0/2763751564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 msgr2=0x564e61096010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 -- 172.21.15.23:0/2763751564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a07a0 msgr2=0x564e610b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a07a0 0x564e610b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c21200700 1 -- 172.21.15.23:0/2763751564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e60e20b60 con 0x564e610a10c0 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.483+0000 7f3c22600700 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.484+0000 7f3c21200700 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009230 tx=0x7f3c08010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83de2be0ceaee333 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.484+0000 7f3c0f400700 1 -- 172.21.15.23:0/2763751564 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0809dde0 con 0x564e610a10c0 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.484+0000 7f3c0f400700 1 -- 172.21.15.23:0/2763751564 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c080a2430 con 0x564e610a10c0 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.484+0000 7f3c0f400700 1 -- 172.21.15.23:0/2763751564 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0809a470 con 0x564e610a10c0 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- 172.21.15.23:0/2763751564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 msgr2=0x564e610b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c08009230 tx=0x7f3c08010c80 comp rx=0 tx=0).stop 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- 172.21.15.23:0/2763751564 shutdown_connections 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e61096010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a07a0 0x564e610b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 --2- 172.21.15.23:0/2763751564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a10c0 0x564e610b2c70 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- 172.21.15.23:0/2763751564 >> 172.21.15.23:0/2763751564 conn(0x564e610006a0 msgr2=0x564e610957f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- 172.21.15.23:0/2763751564 shutdown_connections 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- 172.21.15.23:0/2763751564 wait complete. 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.485+0000 7f3c38045b80 1 -- start start 2024-03-30T04:14:09.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 Processor -- start 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610b1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e610b1a50 con 0x564e610a10c0 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e610b1bc0 con 0x564e610a07a0 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e6110f8e0 con 0x564e610abfc0 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c38045b80 10 client.0 _ll_register_callbacks cb 0x564e60f98470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:09.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c22600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c21c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56584/0 (socket says 172.21.15.23:56584) 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c21c00700 1 -- 172.21.15.23:0/3686007572 learned_addr learned my addr 172.21.15.23:0/3686007572 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.486+0000 7f3c21200700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 -- 172.21.15.23:0/3686007572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 msgr2=0x564e610b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610b1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 -- 172.21.15.23:0/3686007572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 msgr2=0x564e610b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 -- 172.21.15.23:0/3686007572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c10e2d0 con 0x564e610a07a0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c21c00700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c10011ab0 tx=0x7f3c1002fba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3eab9503eb0da93 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c03400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1002fc70 con 0x564e610a07a0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f3c1002f690 con 0x564e610a07a0 2024-03-30T04:14:09.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.487+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c10021070 con 0x564e610a07a0 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c1002f850 con 0x564e610a07a0 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 10 client.28769 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f3c1c19e7c0 con 0x564e610a07a0 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3c10021070 con 0x564e610a07a0 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3c10003af0 con 0x564e610a07a0 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 10 client.28769 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 10 client.28769 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 20 client.28769 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e61110c00 0x564e61113020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 4 client.28769 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.488+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564e60e41cf0 con 0x564e61110c00 2024-03-30T04:14:09.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c21200700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e61110c00 0x564e61113020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c21200700 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e61110c00 0x564e61113020 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f7d44971c4f6614d server_cookie=ff70e72e9cf3b8be in_seq=0 out_seq=0 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c03400700 10 client.28769 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c03400700 1 -- 172.21.15.23:0/3686007572 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564e60e41cf0 con 0x564e61110c00 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c03400700 10 client.28769 handle_command_reply: tid=0 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c38045b80 1 client.28769 shutdown 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 msgr2=0x564e610b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.489+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c10011ab0 tx=0x7f3c1002fba0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 shutdown_connections 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x564e61110c00 0x564e61113020 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564e610abfc0 0x564e610b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564e610a07a0 0x564e610b2a40 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c03400700 0 client.28769 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 --2- 172.21.15.23:0/3686007572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564e610a10c0 0x564e610b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 >> 172.21.15.23:0/3686007572 conn(0x564e610006a0 msgr2=0x564e61096010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 shutdown_connections 2024-03-30T04:14:09.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 1 -- 172.21.15.23:0/3686007572 wait complete. 2024-03-30T04:14:09.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c38045b80 20 client.28769 trim_cache size 0 max 16384 2024-03-30T04:14:09.493 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 msgr2=0x7f3c040a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 0x7f3c040a3750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3c10009c60 tx=0x7f3c10009260 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 msgr2=0x7f3c1c1997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.490+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9a0 tx=0x7f3c0800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 shutdown_connections 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1c106c10 0x7f3c1c19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c1c100b50 0x7f3c1c199290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3c040a1290 0x7f3c040a3750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 --2- 172.21.15.23:0/459307854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c1c101510 0x7f3c1c1997d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 >> 172.21.15.23:0/459307854 conn(0x7f3c1c0fc620 msgr2=0x7f3c1c10c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 shutdown_connections 2024-03-30T04:14:09.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.491+0000 7f3c23000700 1 -- 172.21.15.23:0/459307854 wait complete. 2024-03-30T04:14:09.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:09 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2705351251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:09.508 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:09 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2705351251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:09.511 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:14:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:09 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2705351251' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 -- 172.21.15.23:0/686927173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc106090 msgr2=0x7f11dc10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 --2- 172.21.15.23:0/686927173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc106090 0x7f11dc10dc10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009f90 tx=0x7f11d400b330 comp rx=0 tx=0).stop 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 -- 172.21.15.23:0/686927173 shutdown_connections 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 --2- 172.21.15.23:0/686927173 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc106090 0x7f11dc10dc10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 --2- 172.21.15.23:0/686927173 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc100f70 0x7f11dc101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.723 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 --2- 172.21.15.23:0/686927173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc101920 0x7f11dc105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.724+0000 7f11e2a00700 1 -- 172.21.15.23:0/686927173 >> 172.21.15.23:0/686927173 conn(0x7f11dc0fc7f0 msgr2=0x7f11dc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.725+0000 7f11e2a00700 1 -- 172.21.15.23:0/686927173 shutdown_connections 2024-03-30T04:14:09.724 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.725+0000 7f11e2a00700 1 -- 172.21.15.23:0/686927173 wait complete. 2024-03-30T04:14:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 Processor -- start 2024-03-30T04:14:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 -- start start 2024-03-30T04:14:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc100f70 0x7f11dc1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc19ddd0 0x7f11dc19e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc19e850 con 0x7f11dc100f70 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc19e9c0 con 0x7f11dc19ddd0 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e2a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc19eb80 con 0x7f11dc106090 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11db400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc100f70 0x7f11dc1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11db400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35058/0 (socket says 172.21.15.23:35058) 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc19ddd0 0x7f11dc19e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.726+0000 7f11db400700 1 -- 172.21.15.23:0/3433235919 learned_addr learned my addr 172.21.15.23:0/3433235919 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 -- 172.21.15.23:0/3433235919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc19ddd0 msgr2=0x7f11dc19e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc19ddd0 0x7f11dc19e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 -- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc100f70 msgr2=0x7f11dc1994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc100f70 0x7f11dc1994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 -- 172.21.15.23:0/3433235919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc0095f0 con 0x7f11dc106090 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11db400700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f11c800e970 tx=0x7f11c800ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11d8c00700 1 -- 172.21.15.23:0/3433235919 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8009790 con 0x7f11dc106090 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11d8c00700 1 -- 172.21.15.23:0/3433235919 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c800c7e0 con 0x7f11dc106090 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4009c40 con 0x7f11dc106090 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.727+0000 7f11d8c00700 1 -- 172.21.15.23:0/3433235919 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80054e0 con 0x7f11dc106090 2024-03-30T04:14:09.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.728+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc1a6480 con 0x7f11dc106090 2024-03-30T04:14:09.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.731+0000 7f11d8c00700 1 -- 172.21.15.23:0/3433235919 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f11c800c950 con 0x7f11dc106090 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.732+0000 7f11f6517b80 1 Processor -- start 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.732+0000 7f11f6517b80 1 -- start start 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.732+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250cff79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.732+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 0x56250d0028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11f6517b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cf50990 con 0x56250cff7ef0 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11f6517b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cf4e350 con 0x56250d002df0 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11f6517b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cf49c40 con 0x56250cff75d0 2024-03-30T04:14:09.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11d8c00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 0x7f11c00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11d8c00700 1 -- 172.21.15.23:0/3433235919 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f11c8014070 con 0x7f11dc106090 2024-03-30T04:14:09.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 0x56250d0028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250cff79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11e0a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250cff79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35070/0 (socket says 172.21.15.23:35070) 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56618/0 (socket says 172.21.15.23:56618) 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11e0a00700 1 -- 172.21.15.23:0/3018095027 learned_addr learned my addr 172.21.15.23:0/3018095027 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11e0a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 0x7f11c00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 -- 172.21.15.23:0/3018095027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 msgr2=0x56250cff79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250cff79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 -- 172.21.15.23:0/3018095027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 msgr2=0x56250d0028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 0x56250d0028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11db400700 1 -- 172.21.15.23:0/3018095027 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56250cdcb3b0 con 0x56250d002df0 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.733+0000 7f11dbe00700 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 0x56250d0028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.734+0000 7f11db400700 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f11c80ca5a0 tx=0x7f11c8099960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=123e2b579626a228 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.734+0000 7f11e0a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 0x7f11c00a37d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f11d40060b0 tx=0x7f11d4009700 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.734+0000 7f11d1200700 1 -- 172.21.15.23:0/3018095027 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c809dd90 con 0x56250d002df0 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.734+0000 7f11d1200700 1 -- 172.21.15.23:0/3018095027 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c80a23e0 con 0x56250d002df0 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.734+0000 7f11d1200700 1 -- 172.21.15.23:0/3018095027 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c8074990 con 0x56250d002df0 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- 172.21.15.23:0/3018095027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 msgr2=0x56250d0051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f11c80ca5a0 tx=0x7f11c8099960 comp rx=0 tx=0).stop 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- 172.21.15.23:0/3018095027 shutdown_connections 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250d0051e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250cff79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 --2- 172.21.15.23:0/3018095027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff7ef0 0x56250d0028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- 172.21.15.23:0/3018095027 >> 172.21.15.23:0/3018095027 conn(0x56250cf576a0 msgr2=0x56250cff3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- 172.21.15.23:0/3018095027 shutdown_connections 2024-03-30T04:14:09.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- 172.21.15.23:0/3018095027 wait complete. 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.735+0000 7f11f6517b80 1 -- start start 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 Processor -- start 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250cfffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0000f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cee5d80 con 0x56250d002df0 2024-03-30T04:14:09.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cee5a20 con 0x56250cff75d0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cdcd6a0 con 0x56250cff7ef0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11f6517b80 10 client.0 _ll_register_callbacks cb 0x56250ceef470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11db400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250cfffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56622/0 (socket says 172.21.15.23:56622) 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11dbe00700 1 -- 172.21.15.23:0/2725460016 learned_addr learned my addr 172.21.15.23:0/2725460016 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.736+0000 7f11e0a00700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 -- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 msgr2=0x56250cfffae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250cfffae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 -- 172.21.15.23:0/2725460016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 msgr2=0x56250d0000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 -- 172.21.15.23:0/2725460016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc1a6480 con 0x56250cff75d0 2024-03-30T04:14:09.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11db400700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250cfffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11dbe00700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0127a0 tx=0x7f11cc012ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ebe2a65f2ac3dbc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11c5600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.737+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc017070 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56250cf0a380 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f11cc004030 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11cc0041f0 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 10 client.28781 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56250cd77b60 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56250cf0a380 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f11cc00f540 con 0x56250cff75d0 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 10 client.28781 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 10 client.28781 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 20 client.28781 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:09.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d0128c0 0x56250d014ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 4 client.28781 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.738+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56250cf42700 con 0x56250d0128c0 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.739+0000 7f11db400700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d0128c0 0x56250d014ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.739+0000 7f11c5600700 10 client.28781 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.739+0000 7f11db400700 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d0128c0 0x56250d014ce0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d989604cae00ace3 server_cookie=ad2a4cde339290fc in_seq=0 out_seq=0 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11c5600700 1 -- 172.21.15.23:0/2725460016 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56250cf42700 con 0x56250d0128c0 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11c5600700 10 client.28781 handle_command_reply: tid=0 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 client.28781 shutdown 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 msgr2=0x56250d0027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f11cc0127a0 tx=0x7f11cc012ab0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 shutdown_connections 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d0128c0 0x56250d014ce0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff75d0 0x56250d0027c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 --2- 172.21.15.23:0/2725460016 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250cfffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11c5600700 0 client.28781 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 >> 172.21.15.23:0/2725460016 conn(0x56250cf576a0 msgr2=0x56250cff9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 shutdown_connections 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.740+0000 7f11f6517b80 1 -- 172.21.15.23:0/2725460016 wait complete. 2024-03-30T04:14:09.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.741+0000 7f11f6517b80 20 client.28781 trim_cache size 0 max 16384 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 Processor -- start 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 -- start start 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff75d0 0x56250d00e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250cfff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cff9110 con 0x56250cff75d0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250d009fb0 con 0x56250d002df0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11f6517b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250cff73c0 con 0x56250cff7ef0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11db400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11db400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35080/0 (socket says 172.21.15.23:35080) 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11db400700 1 -- 172.21.15.23:0/1067168659 learned_addr learned my addr 172.21.15.23:0/1067168659 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11dbe00700 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff75d0 0x56250d00e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.755+0000 7f11e0a00700 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250cfff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 -- 172.21.15.23:0/1067168659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 msgr2=0x56250cfff3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250cfff3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 -- 172.21.15.23:0/1067168659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff75d0 msgr2=0x56250d00e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff75d0 0x56250d00e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 -- 172.21.15.23:0/1067168659 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56250cf0fea0 con 0x56250cff7ef0 2024-03-30T04:14:09.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11db400700 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f11c800cc00 tx=0x7f11c8099e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e6c761cabe2a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11d1c00700 1 -- 172.21.15.23:0/1067168659 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c809dd90 con 0x56250cff7ef0 2024-03-30T04:14:09.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11d1c00700 1 -- 172.21.15.23:0/1067168659 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c8005660 con 0x56250cff7ef0 2024-03-30T04:14:09.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.756+0000 7f11d1c00700 1 -- 172.21.15.23:0/1067168659 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c809a1c0 con 0x56250cff7ef0 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- 172.21.15.23:0/1067168659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 msgr2=0x56250d009a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f11c800cc00 tx=0x7f11c8099e60 comp rx=0 tx=0).stop 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- 172.21.15.23:0/1067168659 shutdown_connections 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250d002df0 0x56250cfff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250cff75d0 0x56250d00e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1067168659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff7ef0 0x56250d009a70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- 172.21.15.23:0/1067168659 >> 172.21.15.23:0/1067168659 conn(0x56250cf576a0 msgr2=0x56250d00e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- 172.21.15.23:0/1067168659 shutdown_connections 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- 172.21.15.23:0/1067168659 wait complete. 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.757+0000 7f11f6517b80 1 -- start start 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 Processor -- start 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff7ef0 0x56250d007bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250d065c30 con 0x56250d002df0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250d065d70 con 0x56250cff7ef0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250d065eb0 con 0x56250cff75d0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11f6517b80 10 client.0 _ll_register_callbacks cb 0x56250ceef470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11dbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35086/0 (socket says 172.21.15.23:35086) 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11dbe00700 1 -- 172.21.15.23:0/1282621889 learned_addr learned my addr 172.21.15.23:0/1282621889 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.758+0000 7f11db400700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff7ef0 0x56250d007bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11e0a00700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 -- 172.21.15.23:0/1282621889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff7ef0 msgr2=0x56250d007bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff7ef0 0x56250d007bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 -- 172.21.15.23:0/1282621889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 msgr2=0x56250d0656f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0656f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 -- 172.21.15.23:0/1282621889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56250cdcb3b0 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11dbe00700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11cc012420 tx=0x7f11cc005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47d859267a29c8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11c5600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc017040 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f11cc000e40 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c80a23e0 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11cc00f750 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11f6517b80 10 client.28762 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.759+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56250cdcfcd0 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11c80a23e0 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f11cc003b40 con 0x56250cff75d0 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 10 client.28762 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 10 client.28762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:14:09.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 20 client.28762 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:09.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d067530 0x56250d069950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:09.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 4 client.28762 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:09.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56250cd98cf0 con 0x56250d067530 2024-03-30T04:14:09.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.760+0000 7f11db400700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d067530 0x56250d069950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11c5600700 10 client.28762 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11db400700 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d067530 0x56250d069950 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2ca7b00eb791428 server_cookie=6ce9398953ddb635 in_seq=0 out_seq=0 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11c5600700 1 -- 172.21.15.23:0/1282621889 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56250cd98cf0 con 0x56250d067530 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11c5600700 10 client.28762 handle_command_reply: tid=0 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 client.28762 shutdown 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 msgr2=0x56250d007680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11cc012420 tx=0x7f11cc005e20 comp rx=0 tx=0).stop 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 shutdown_connections 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56250d067530 0x56250d069950 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56250d002df0 0x56250d0656f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11c5600700 0 client.28762 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56250cff75d0 0x56250d007680 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 --2- 172.21.15.23:0/1282621889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56250cff7ef0 0x56250d007bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 >> 172.21.15.23:0/1282621889 conn(0x56250cf576a0 msgr2=0x56250cff94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 shutdown_connections 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 1 -- 172.21.15.23:0/1282621889 wait complete. 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.761+0000 7f11f6517b80 20 client.28762 trim_cache size 0 max 16384 2024-03-30T04:14:09.764 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:09.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 msgr2=0x7f11c00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 0x7f11c00a37d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f11d40060b0 tx=0x7f11d4009700 comp rx=0 tx=0).stop 2024-03-30T04:14:09.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 msgr2=0x7f11dc199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:09.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f11c800e970 tx=0x7f11c800ec80 comp rx=0 tx=0).stop 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 shutdown_connections 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f11c00a1310 0x7f11c00a37d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11dc19ddd0 0x7f11dc19e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f11dc100f70 0x7f11dc1994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 --2- 172.21.15.23:0/3433235919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11dc106090 0x7f11dc199a30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 >> 172.21.15.23:0/3433235919 conn(0x7f11dc0fc7f0 msgr2=0x7f11dc10c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 shutdown_connections 2024-03-30T04:14:09.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:09.762+0000 7f11e2a00700 1 -- 172.21.15.23:0/3433235919 wait complete. 2024-03-30T04:14:09.779 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:14:10.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 -- 172.21.15.23:0/3292546618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd5010b250 msgr2=0x7fcd5010d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 --2- 172.21.15.23:0/3292546618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd5010b250 0x7fcd5010d6b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009990 tx=0x7fcd3c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 -- 172.21.15.23:0/3292546618 shutdown_connections 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 --2- 172.21.15.23:0/3292546618 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd5010b250 0x7fcd5010d6b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 --2- 172.21.15.23:0/3292546618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd50101de0 0x7fcd501021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 --2- 172.21.15.23:0/3292546618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd5010abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 -- 172.21.15.23:0/3292546618 >> 172.21.15.23:0/3292546618 conn(0x7fcd500fb5c0 msgr2=0x7fcd500fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 -- 172.21.15.23:0/3292546618 shutdown_connections 2024-03-30T04:14:10.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.067+0000 7fcd58600700 1 -- 172.21.15.23:0/3292546618 wait complete. 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.069+0000 7fcd58600700 1 Processor -- start 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.069+0000 7fcd58600700 1 -- start start 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd50101de0 0x7fcd501081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 0x7fcd50102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50103130 con 0x7fcd50101de0 2024-03-30T04:14:10.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd501032a0 con 0x7fcd5010b250 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd58600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50103460 con 0x7fcd50102700 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd55200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd55200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35110/0 (socket says 172.21.15.23:35110) 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd55200700 1 -- 172.21.15.23:0/4014164539 learned_addr learned my addr 172.21.15.23:0/4014164539 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.070+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 0x7fcd50102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd55c00700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd50101de0 0x7fcd501081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd56600700 1 -- 172.21.15.23:0/4014164539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 msgr2=0x7fcd50108700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd56600700 1 -- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd50101de0 msgr2=0x7fcd501081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd50101de0 0x7fcd501081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd56600700 1 -- 172.21.15.23:0/4014164539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3c0095f0 con 0x7fcd5010b250 2024-03-30T04:14:10.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.071+0000 7fcd55200700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.072+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 0x7fcd50102b60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0098e0 tx=0x7fcd3c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.073+0000 7fcd4e000700 1 -- 172.21.15.23:0/4014164539 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c01d070 con 0x7fcd5010b250 2024-03-30T04:14:10.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.073+0000 7fcd4e000700 1 -- 172.21.15.23:0/4014164539 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd3c005700 con 0x7fcd5010b250 2024-03-30T04:14:10.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.073+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd501036e0 con 0x7fcd5010b250 2024-03-30T04:14:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.073+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd501b2d30 con 0x7fcd5010b250 2024-03-30T04:14:10.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.073+0000 7fcd4e000700 1 -- 172.21.15.23:0/4014164539 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c004a40 con 0x7fcd5010b250 2024-03-30T04:14:10.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.079+0000 7fcd6c060b80 1 Processor -- start 2024-03-30T04:14:10.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.080+0000 7fcd4e000700 1 -- 172.21.15.23:0/4014164539 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fcd3c004ba0 con 0x7fcd5010b250 2024-03-30T04:14:10.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 -- start start 2024-03-30T04:14:10.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea2f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea2fdf0 0x55d05ea321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e97d990 con 0x55d05ea24ef0 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e97b350 con 0x55d05ea2fdf0 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.081+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e976c40 con 0x55d05ea245d0 2024-03-30T04:14:10.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.082+0000 7fcd55200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.082+0000 7fcd55200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35126/0 (socket says 172.21.15.23:35126) 2024-03-30T04:14:10.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.082+0000 7fcd55200700 1 -- 172.21.15.23:0/2249131086 learned_addr learned my addr 172.21.15.23:0/2249131086 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.083+0000 7fcd55c00700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea2f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea2fdf0 0x55d05ea321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd4e000700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 0x7fcd340a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd4e000700 1 -- 172.21.15.23:0/4014164539 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fcd3c0cddd0 con 0x7fcd5010b250 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 -- 172.21.15.23:0/2249131086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea2fdf0 msgr2=0x55d05ea321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea2fdf0 0x55d05ea321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 -- 172.21.15.23:0/2249131086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 msgr2=0x55d05ea2f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea2f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55200700 1 -- 172.21.15.23:0/2249131086 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d05e7f83b0 con 0x55d05ea245d0 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd55c00700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.084+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 0x7fcd340a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.085+0000 7fcd55200700 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007ef0 tx=0x7fcd40009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a84bb498e34f390c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.085+0000 7fcd56600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 0x7fcd340a3890 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c005b40 tx=0x7fcd3c01a040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.086+0000 7fcd46a00700 1 -- 172.21.15.23:0/2249131086 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40003a00 con 0x55d05ea245d0 2024-03-30T04:14:10.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.086+0000 7fcd46a00700 1 -- 172.21.15.23:0/2249131086 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd4000f070 con 0x55d05ea245d0 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.086+0000 7fcd46a00700 1 -- 172.21.15.23:0/2249131086 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40004270 con 0x55d05ea245d0 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 -- 172.21.15.23:0/2249131086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 msgr2=0x55d05ea249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcd40007ef0 tx=0x7fcd40009230 comp rx=0 tx=0).stop 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 -- 172.21.15.23:0/2249131086 shutdown_connections 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea2fdf0 0x55d05ea321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea249b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/2249131086 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 -- 172.21.15.23:0/2249131086 >> 172.21.15.23:0/2249131086 conn(0x55d05e9846a0 msgr2=0x55d05ea20a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.086 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.087+0000 7fcd6c060b80 1 -- 172.21.15.23:0/2249131086 shutdown_connections 2024-03-30T04:14:10.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.088+0000 7fcd6c060b80 1 -- 172.21.15.23:0/2249131086 wait complete. 2024-03-30T04:14:10.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.088+0000 7fcd6c060b80 1 -- start start 2024-03-30T04:14:10.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 Processor -- start 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea2f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 0x55d05ea2cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea2d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e912d80 con 0x55d05ea2fdf0 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e912a20 con 0x55d05ea245d0 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05e7fa6a0 con 0x55d05ea24ef0 2024-03-30T04:14:10.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.089+0000 7fcd6c060b80 10 client.0 _ll_register_callbacks cb 0x55d05e91c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.090+0000 7fcd55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea2f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.090+0000 7fcd55c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 0x55d05ea2cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.090+0000 7fcd55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea2f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56684/0 (socket says 172.21.15.23:56684) 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.090+0000 7fcd55200700 1 -- 172.21.15.23:0/1254638211 learned_addr learned my addr 172.21.15.23:0/1254638211 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.090+0000 7fcd55200700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea2d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.089 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 -- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 msgr2=0x55d05ea2f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea2f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 -- 172.21.15.23:0/1254638211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 msgr2=0x55d05ea2d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea2d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 -- 172.21.15.23:0/1254638211 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4800a890 con 0x55d05ea24ef0 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd55c00700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 0x55d05ea2cad0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcd48010a20 tx=0x7fcd48010d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfd13d9dc5071fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd3b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:10.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.091+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd48015070 con 0x55d05ea24ef0 2024-03-30T04:14:10.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.092+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fcd4800fb60 con 0x55d05ea24ef0 2024-03-30T04:14:10.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.092+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d05e937380 con 0x55d05ea24ef0 2024-03-30T04:14:10.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.093+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd4800fd20 con 0x55d05ea24ef0 2024-03-30T04:14:10.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.093+0000 7fcd6c060b80 10 client.28774 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.093+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fcd40011040 con 0x55d05ea24ef0 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.093+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d05e937380 con 0x55d05ea24ef0 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcd48005020 con 0x55d05ea24ef0 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 10 client.28774 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 10 client.28774 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:14:10.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 20 client.28774 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea3fb00 0x55d05ea41f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 4 client.28774 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:10.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d05e96f700 con 0x55d05ea3fb00 2024-03-30T04:14:10.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.094+0000 7fcd55200700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea3fb00 0x55d05ea41f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.095+0000 7fcd3b400700 10 client.28774 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:10.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.096+0000 7fcd55200700 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea3fb00 0x55d05ea41f20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3a3fc951fa75e6c0 server_cookie=1d3c5e81d7d03738 in_seq=0 out_seq=0 2024-03-30T04:14:10.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.097+0000 7fcd3b400700 1 -- 172.21.15.23:0/1254638211 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d05e96f700 con 0x55d05ea3fb00 2024-03-30T04:14:10.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.097+0000 7fcd3b400700 10 client.28774 handle_command_reply: tid=0 2024-03-30T04:14:10.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.097+0000 7fcd6c060b80 1 client.28774 shutdown 2024-03-30T04:14:10.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 msgr2=0x55d05ea2cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 0x55d05ea2cad0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fcd48010a20 tx=0x7fcd48010d30 comp rx=0 tx=0).stop 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 shutdown_connections 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea3fb00 0x55d05ea41f20 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea2d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea2f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd3b400700 0 client.28774 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1254638211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea24ef0 0x55d05ea2cad0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 >> 172.21.15.23:0/1254638211 conn(0x55d05e9846a0 msgr2=0x55d05ea266c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.098+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 shutdown_connections 2024-03-30T04:14:10.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.099+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1254638211 wait complete. 2024-03-30T04:14:10.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.099+0000 7fcd6c060b80 20 client.28774 trim_cache size 0 max 16384 2024-03-30T04:14:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.114+0000 7fcd6c060b80 1 Processor -- start 2024-03-30T04:14:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.114+0000 7fcd6c060b80 1 -- start start 2024-03-30T04:14:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea41ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 0x55d05ea1ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea36b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea3d180 con 0x55d05ea2fdf0 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea255a0 con 0x55d05ea24ef0 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea370c0 con 0x55d05ea245d0 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd55c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 0x55d05ea1ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd55c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 0x55d05ea1ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56694/0 (socket says 172.21.15.23:56694) 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd55c00700 1 -- 172.21.15.23:0/273209640 learned_addr learned my addr 172.21.15.23:0/273209640 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd55200700 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea41ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.115+0000 7fcd55200700 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea36b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 -- 172.21.15.23:0/273209640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 msgr2=0x55d05ea1ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 0x55d05ea1ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 -- 172.21.15.23:0/273209640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 msgr2=0x55d05ea36b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea36b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 -- 172.21.15.23:0/273209640 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d05e7a4b60 con 0x55d05ea245d0 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd55200700 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea41ef0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd40003ba0 tx=0x7fcd40007c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d802739031dbc755 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd47400700 1 -- 172.21.15.23:0/273209640 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400178b0 con 0x55d05ea245d0 2024-03-30T04:14:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd47400700 1 -- 172.21.15.23:0/273209640 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd40017ed0 con 0x55d05ea245d0 2024-03-30T04:14:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.116+0000 7fcd47400700 1 -- 172.21.15.23:0/273209640 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4001ba60 con 0x55d05ea245d0 2024-03-30T04:14:10.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 -- 172.21.15.23:0/273209640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 msgr2=0x55d05ea41ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea41ef0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcd40003ba0 tx=0x7fcd40007c50 comp rx=0 tx=0).stop 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 -- 172.21.15.23:0/273209640 shutdown_connections 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea2fdf0 0x55d05ea36b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea245d0 0x55d05ea41ef0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/273209640 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea24ef0 0x55d05ea1ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.117+0000 7fcd6c060b80 1 -- 172.21.15.23:0/273209640 >> 172.21.15.23:0/273209640 conn(0x55d05e9846a0 msgr2=0x55d05ea2c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.118+0000 7fcd6c060b80 1 -- 172.21.15.23:0/273209640 shutdown_connections 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.118+0000 7fcd6c060b80 1 -- 172.21.15.23:0/273209640 wait complete. 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.118+0000 7fcd6c060b80 1 -- start start 2024-03-30T04:14:10.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 Processor -- start 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea34c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea2fdf0 0x55d05ea35170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea35790 con 0x55d05ea24ef0 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea35900 con 0x55d05ea245d0 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05ea35ac0 con 0x55d05ea2fdf0 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd6c060b80 10 client.0 _ll_register_callbacks cb 0x55d05e91c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.119+0000 7fcd55200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56708/0 (socket says 172.21.15.23:56708) 2024-03-30T04:14:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 -- 172.21.15.23:0/1080046455 learned_addr learned my addr 172.21.15.23:0/1080046455 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea2fdf0 0x55d05ea35170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 -- 172.21.15.23:0/1080046455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea2fdf0 msgr2=0x55d05ea35170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea2fdf0 0x55d05ea35170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 -- 172.21.15.23:0/1080046455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 msgr2=0x55d05ea34c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea34c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 -- 172.21.15.23:0/1080046455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd40031040 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd55200700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcd4001d040 tx=0x7fcd400060c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=554576f031f2d116 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.120+0000 7fcd3b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40017cf0 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7fcd40037440 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd4000f070 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd40037600 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd6c060b80 10 client.28802 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d05e93cea0 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.121+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd4000f070 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcd40035070 con 0x55d05ea245d0 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 10 client.28802 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 10 client.28802 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:14:10.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 20 client.28802 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea94b90 0x55d05ea96f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 4 client.28802 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d05e7c5cf0 con 0x55d05ea94b90 2024-03-30T04:14:10.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.122+0000 7fcd55c00700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea94b90 0x55d05ea96f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.123+0000 7fcd55c00700 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea94b90 0x55d05ea96f80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c8b730efc307f9 server_cookie=ad0c8544edd393a4 in_seq=0 out_seq=0 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.123+0000 7fcd3b400700 10 client.28802 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd3b400700 1 -- 172.21.15.23:0/1080046455 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d05e7c5cf0 con 0x55d05ea94b90 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd3b400700 10 client.28802 handle_command_reply: tid=0 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 client.28802 shutdown 2024-03-30T04:14:10.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 msgr2=0x55d05ea367f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fcd4001d040 tx=0x7fcd400060c0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 shutdown_connections 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55d05ea94b90 0x55d05ea96f80 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d05ea2fdf0 0x55d05ea35170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d05ea245d0 0x55d05ea367f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd3b400700 0 client.28802 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 --2- 172.21.15.23:0/1080046455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d05ea24ef0 0x55d05ea34c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 >> 172.21.15.23:0/1080046455 conn(0x55d05e9846a0 msgr2=0x55d05ea2b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 shutdown_connections 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.124+0000 7fcd6c060b80 1 -- 172.21.15.23:0/1080046455 wait complete. 2024-03-30T04:14:10.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd6c060b80 20 client.28802 trim_cache size 0 max 16384 2024-03-30T04:14:10.125 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 msgr2=0x7fcd340a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 0x7fcd340a3890 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c005b40 tx=0x7fcd3c01a040 comp rx=0 tx=0).stop 2024-03-30T04:14:10.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 msgr2=0x7fcd50102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 0x7fcd50102b60 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0098e0 tx=0x7fcd3c00f690 comp rx=0 tx=0).stop 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 shutdown_connections 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcd340a13d0 0x7fcd340a3890 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcd5010b250 0x7fcd50102b60 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd50101de0 0x7fcd501081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 --2- 172.21.15.23:0/4014164539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcd50102700 0x7fcd50108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.125+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 >> 172.21.15.23:0/4014164539 conn(0x7fcd500fb5c0 msgr2=0x7fcd500ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.126+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 shutdown_connections 2024-03-30T04:14:10.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.126+0000 7fcd58600700 1 -- 172.21.15.23:0/4014164539 wait complete. 2024-03-30T04:14:10.143 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:14:10.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1184997 9.55 MB/sec execute 527 sec latency 51.249 ms 2024-03-30T04:14:10.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.366+0000 7f9fac800700 1 -- 172.21.15.23:0/68060752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa4100f60 msgr2=0x7f9fa4101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.366+0000 7f9fac800700 1 --2- 172.21.15.23:0/68060752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa4100f60 0x7f9fa4101340 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9f98009990 tx=0x7f9f98009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 -- 172.21.15.23:0/68060752 shutdown_connections 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 --2- 172.21.15.23:0/68060752 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4106080 0x7f9fa410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 --2- 172.21.15.23:0/68060752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa4100f60 0x7f9fa4101340 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 --2- 172.21.15.23:0/68060752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4101910 0x7f9fa4105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 -- 172.21.15.23:0/68060752 >> 172.21.15.23:0/68060752 conn(0x7f9fa40fc7c0 msgr2=0x7f9fa40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 -- 172.21.15.23:0/68060752 shutdown_connections 2024-03-30T04:14:10.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.367+0000 7f9fac800700 1 -- 172.21.15.23:0/68060752 wait complete. 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.368+0000 7f9fac800700 1 Processor -- start 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.368+0000 7f9fac800700 1 -- start start 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.368+0000 7f9fac800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 0x7f9fa4199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.368+0000 7f9fac800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.368+0000 7f9fac800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fac800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa419e7d0 con 0x7f9fa4100f60 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fac800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa419e940 con 0x7f9fa419dd50 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fac800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa419eb00 con 0x7f9fa4106080 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 0x7f9fa4199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9faa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35180/0 (socket says 172.21.15.23:35180) 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9faa800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56720/0 (socket says 172.21.15.23:56720) 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 -- 172.21.15.23:0/2762668002 learned_addr learned my addr 172.21.15.23:0/2762668002 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 0x7f9fa4199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54290/0 (socket says 172.21.15.23:54290) 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 -- 172.21.15.23:0/2762668002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 msgr2=0x7f9fa419e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 -- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 msgr2=0x7f9fa4199470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 0x7f9fa4199470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9fa9400700 1 -- 172.21.15.23:0/2762668002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f980095f0 con 0x7f9fa4106080 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.369+0000 7f9faa800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9fa9400700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400e970 tx=0x7f9f9400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9f9ea00700 1 -- 172.21.15.23:0/2762668002 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f94009790 con 0x7f9fa4106080 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9f9ea00700 1 -- 172.21.15.23:0/2762668002 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f9400c7e0 con 0x7f9fa4106080 2024-03-30T04:14:10.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa41a6060 con 0x7f9fa4106080 2024-03-30T04:14:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9f9ea00700 1 -- 172.21.15.23:0/2762668002 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f940054e0 con 0x7f9fa4106080 2024-03-30T04:14:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.370+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa41a6540 con 0x7f9fa4106080 2024-03-30T04:14:10.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.373+0000 7f9f9ea00700 1 -- 172.21.15.23:0/2762668002 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9f9400c950 con 0x7f9fa4106080 2024-03-30T04:14:10.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.376+0000 7f9f9ea00700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 0x7f9f8c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.376+0000 7f9f9ea00700 1 -- 172.21.15.23:0/2762668002 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9f94014070 con 0x7f9fa4106080 2024-03-30T04:14:10.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.376+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 0x7f9f8c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.376+0000 7f9fc0362b80 1 Processor -- start 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 -- start start 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 0x7f9f8c0a3750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9f98006010 tx=0x7f9f9800b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80d989b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80cf1990 con 0x564c80d985d0 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80cef350 con 0x564c80d98ef0 2024-03-30T04:14:10.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.377+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80ceac40 con 0x564c80da3df0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80d989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80d989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54300/0 (socket says 172.21.15.23:54300) 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9fa9e00700 1 -- 172.21.15.23:0/941808075 learned_addr learned my addr 172.21.15.23:0/941808075 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9fa9400700 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 -- 172.21.15.23:0/941808075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 msgr2=0x564c80da38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 -- 172.21.15.23:0/941808075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 msgr2=0x564c80d989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80d989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 -- 172.21.15.23:0/941808075 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c80b6c3b0 con 0x564c80da3df0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.378+0000 7f9faa800700 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da61e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0003db0 tx=0x7f9fa0009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e02c693306af39d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.379+0000 7f9f93000700 1 -- 172.21.15.23:0/941808075 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0004bb0 con 0x564c80da3df0 2024-03-30T04:14:10.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.379+0000 7f9f93000700 1 -- 172.21.15.23:0/941808075 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fa000f040 con 0x564c80da3df0 2024-03-30T04:14:10.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.379+0000 7f9f93000700 1 -- 172.21.15.23:0/941808075 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0007c20 con 0x564c80da3df0 2024-03-30T04:14:10.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.379+0000 7f9fc0362b80 1 -- 172.21.15.23:0/941808075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 msgr2=0x564c80da61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.379+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da61e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9fa0003db0 tx=0x7f9fa0009920 comp rx=0 tx=0).stop 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 -- 172.21.15.23:0/941808075 shutdown_connections 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da61e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80d989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/941808075 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 -- 172.21.15.23:0/941808075 >> 172.21.15.23:0/941808075 conn(0x564c80cf86a0 msgr2=0x564c80d94a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 -- 172.21.15.23:0/941808075 shutdown_connections 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 -- 172.21.15.23:0/941808075 wait complete. 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.380+0000 7f9fc0362b80 1 -- start start 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 Processor -- start 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80c86d80 con 0x564c80d985d0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80c86a20 con 0x564c80d98ef0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80b6e6a0 con 0x564c80da3df0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fc0362b80 10 client.0 _ll_register_callbacks cb 0x564c80c90470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fa9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54302/0 (socket says 172.21.15.23:54302) 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.381+0000 7f9fa9e00700 1 -- 172.21.15.23:0/382751182 learned_addr learned my addr 172.21.15.23:0/382751182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9faa800700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 -- 172.21.15.23:0/382751182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 msgr2=0x564c80da1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 -- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 msgr2=0x564c80da0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 -- 172.21.15.23:0/382751182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa0003a60 con 0x564c80d985d0 2024-03-30T04:14:10.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9faa800700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9400700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800be80 tx=0x7f9f98009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfc146c3f538760 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9f87400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98004d20 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9f98004e80 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.382+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564c80cab380 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f9800f460 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 10 client.18840 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564c80b18b60 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564c80cab380 con 0x564c80d985d0 2024-03-30T04:14:10.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9f9802f070 con 0x564c80d985d0 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 10 client.18840 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 10 client.18840 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 20 client.18840 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80db3960 0x564c80db5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 4 client.18840 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.383+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c80ce3700 con 0x564c80db3960 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.384+0000 7f9fa9400700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80db3960 0x564c80db5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.384+0000 7f9fa9400700 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80db3960 0x564c80db5d80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c9917a6335b168dd server_cookie=7545362750b99e74 in_seq=0 out_seq=0 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.384+0000 7f9f87400700 10 client.18840 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9f87400700 1 -- 172.21.15.23:0/382751182 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564c80ce3700 con 0x564c80db3960 2024-03-30T04:14:10.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9f87400700 10 client.18840 handle_command_reply: tid=0 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 client.18840 shutdown 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 msgr2=0x564c80da3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f9f9800be80 tx=0x7f9f98009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 shutdown_connections 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80db3960 0x564c80db5d80 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80da1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d985d0 0x564c80da3750 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/382751182 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80da0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9f87400700 0 client.18840 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 >> 172.21.15.23:0/382751182 conn(0x564c80cf86a0 msgr2=0x564c80d9a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 shutdown_connections 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 1 -- 172.21.15.23:0/382751182 wait complete. 2024-03-30T04:14:10.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.385+0000 7f9fc0362b80 20 client.18840 trim_cache size 0 max 16384 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 Processor -- start 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 -- start start 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 0x564c80db5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 0x564c80d8d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80d9b5b0 con 0x564c80da3df0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80da1420 con 0x564c80d98ef0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80da2040 con 0x564c80d985d0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fa9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9faa800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 0x564c80d8d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 0x564c80db5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fa9400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56732/0 (socket says 172.21.15.23:56732) 2024-03-30T04:14:10.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.400+0000 7f9fa9400700 1 -- 172.21.15.23:0/1685448071 learned_addr learned my addr 172.21.15.23:0/1685448071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 -- 172.21.15.23:0/1685448071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 msgr2=0x564c80db5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 0x564c80db5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 -- 172.21.15.23:0/1685448071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 msgr2=0x564c80d8d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 0x564c80d8d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 -- 172.21.15.23:0/1685448071 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c80cb0ea0 con 0x564c80d98ef0 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 0x564c80db5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9faa800700 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 0x564c80d8d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9fa9400700 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9f94096520 tx=0x7f9f940cd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4676cf5d2a75d6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9f93a00700 1 -- 172.21.15.23:0/1685448071 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9409de50 con 0x564c80d98ef0 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9f93a00700 1 -- 172.21.15.23:0/1685448071 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f940a24a0 con 0x564c80d98ef0 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.401+0000 7f9f93a00700 1 -- 172.21.15.23:0/1685448071 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9409a280 con 0x564c80d98ef0 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 -- 172.21.15.23:0/1685448071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 msgr2=0x564c80daaaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9f94096520 tx=0x7f9f940cd040 comp rx=0 tx=0).stop 2024-03-30T04:14:10.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 -- 172.21.15.23:0/1685448071 shutdown_connections 2024-03-30T04:14:10.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80da3df0 0x564c80d8d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80d985d0 0x564c80db5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/1685448071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d98ef0 0x564c80daaaa0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 -- 172.21.15.23:0/1685448071 >> 172.21.15.23:0/1685448071 conn(0x564c80cf86a0 msgr2=0x564c80dba6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 -- 172.21.15.23:0/1685448071 shutdown_connections 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.402+0000 7f9fc0362b80 1 -- 172.21.15.23:0/1685448071 wait complete. 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.403+0000 7f9fc0362b80 1 -- start start 2024-03-30T04:14:10.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.403+0000 7f9fc0362b80 1 Processor -- start 2024-03-30T04:14:10.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d98ef0 0x564c80da91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80e00760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80e00ca0 con 0x564c80d98ef0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80e00e10 con 0x564c80d985d0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c80e00fd0 con 0x564c80da3df0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fc0362b80 10 client.0 _ll_register_callbacks cb 0x564c80c90470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56742/0 (socket says 172.21.15.23:56742) 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 -- 172.21.15.23:0/3096059361 learned_addr learned my addr 172.21.15.23:0/3096059361 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9400700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d98ef0 0x564c80da91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9faa800700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80e00760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 -- 172.21.15.23:0/3096059361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 msgr2=0x564c80e00760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80e00760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 -- 172.21.15.23:0/3096059361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d98ef0 msgr2=0x564c80da91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d98ef0 0x564c80da91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.404+0000 7f9fa9e00700 1 -- 172.21.15.23:0/3096059361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c80b18b60 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9fa9e00700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9f98024620 tx=0x7f9f98024a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30f4b96be1a092bc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9f87400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f98024410 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fa000f040 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f9f98004b70 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f98004d70 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9fc0362b80 10 client.28820 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x564c80b6c3b0 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.405+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fa000f040 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9f9802f050 con 0x564c80d985d0 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 10 client.28820 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 10 client.28820 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:14:10.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 20 client.28820 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80e08370 0x564c80e0a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 4 client.28820 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564c80b39cf0 con 0x564c80e08370 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fa9400700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80e08370 0x564c80e0a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9f87400700 10 client.28820 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.406+0000 7f9fa9400700 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80e08370 0x564c80e0a760 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=910ab96a69fe7f4 server_cookie=38b151879ffae6c4 in_seq=0 out_seq=0 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9f87400700 1 -- 172.21.15.23:0/3096059361 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564c80b39cf0 con 0x564c80e08370 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9f87400700 10 client.28820 handle_command_reply: tid=0 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 client.28820 shutdown 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 msgr2=0x564c80da8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9f98024620 tx=0x7f9f98024a70 comp rx=0 tx=0).stop 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 shutdown_connections 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x564c80e08370 0x564c80e0a760 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564c80da3df0 0x564c80e00760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564c80d985d0 0x564c80da8c70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 --2- 172.21.15.23:0/3096059361 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564c80d98ef0 0x564c80da91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9f87400700 0 client.28820 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 >> 172.21.15.23:0/3096059361 conn(0x564c80cf86a0 msgr2=0x564c80d8d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 shutdown_connections 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 1 -- 172.21.15.23:0/3096059361 wait complete. 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.407+0000 7f9fc0362b80 20 client.28820 trim_cache size 0 max 16384 2024-03-30T04:14:10.415 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 msgr2=0x7f9f8c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 0x7f9f8c0a3750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9f98006010 tx=0x7f9f9800b260 comp rx=0 tx=0).stop 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 msgr2=0x7f9fa41999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400e970 tx=0x7f9f9400ec80 comp rx=0 tx=0).stop 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 shutdown_connections 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9f8c0a1290 0x7f9f8c0a3750 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fa419dd50 0x7f9fa419e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fa4100f60 0x7f9fa4199470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 --2- 172.21.15.23:0/2762668002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9fa4106080 0x7f9fa41999b0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 >> 172.21.15.23:0/2762668002 conn(0x7f9fa40fc7c0 msgr2=0x7f9fa40fdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 shutdown_connections 2024-03-30T04:14:10.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.408+0000 7f9fac800700 1 -- 172.21.15.23:0/2762668002 wait complete. 2024-03-30T04:14:10.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:10 smithi023 ceph-mon[38254]: pgmap v381: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 233 op/s 2024-03-30T04:14:10.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:10 smithi023 ceph-mon[38254]: osdmap e102: 12 total, 12 up, 12 in 2024-03-30T04:14:10.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:14:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:10 smithi193 ceph-mon[79581]: pgmap v381: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 233 op/s 2024-03-30T04:14:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:10 smithi193 ceph-mon[79581]: osdmap e102: 12 total, 12 up, 12 in 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 -- 172.21.15.23:0/4019759550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 msgr2=0x7f7e28114260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/4019759550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e28114260 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b0a0 tx=0x7f7e2000b3b0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 -- 172.21.15.23:0/4019759550 shutdown_connections 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/4019759550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e28114260 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/4019759550 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 0x7f7e281079f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/4019759550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e28107fc0 0x7f7e2810bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 -- 172.21.15.23:0/4019759550 >> 172.21.15.23:0/4019759550 conn(0x7f7e28076b90 msgr2=0x7f7e28105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 -- 172.21.15.23:0/4019759550 shutdown_connections 2024-03-30T04:14:10.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.655+0000 7f7e2fe00700 1 -- 172.21.15.23:0/4019759550 wait complete. 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.656+0000 7f7e2fe00700 1 Processor -- start 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.656+0000 7f7e2fe00700 1 -- start start 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 0x7f7e281a3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e2819d630 0x7f7e2819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2819e130 con 0x7f7e28107610 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2819e2a0 con 0x7f7e2810c6e0 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2fe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2819e460 con 0x7f7e2819d630 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56772/0 (socket says 172.21.15.23:56772) 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1888645739 learned_addr learned my addr 172.21.15.23:0/1888645739 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2d400700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 0x7f7e281a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.657+0000 7f7e2de00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e2819d630 0x7f7e2819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1888645739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e2819d630 msgr2=0x7f7e2819dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e2819d630 0x7f7e2819dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 msgr2=0x7f7e281a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 0x7f7e281a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1888645739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e20009d00 con 0x7f7e2810c6e0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e970 tx=0x7f7e1400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e26000700 1 -- 172.21.15.23:0/1888645739 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400cbd0 con 0x7f7e2810c6e0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e26000700 1 -- 172.21.15.23:0/1888645739 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e1400eec0 con 0x7f7e2810c6e0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e26000700 1 -- 172.21.15.23:0/1888645739 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14005610 con 0x7f7e2810c6e0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.658+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e281aa3f0 con 0x7f7e2810c6e0 2024-03-30T04:14:10.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.659+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281aa8b0 con 0x7f7e2810c6e0 2024-03-30T04:14:10.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.662+0000 7f7e26000700 1 -- 172.21.15.23:0/1888645739 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f7e14004750 con 0x7f7e2810c6e0 2024-03-30T04:14:10.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.663+0000 7f7e26000700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 0x7f7e100a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.663+0000 7f7e26000700 1 -- 172.21.15.23:0/1888645739 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7e14014070 con 0x7f7e2810c6e0 2024-03-30T04:14:10.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.663+0000 7f7e43990b80 1 Processor -- start 2024-03-30T04:14:10.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.663+0000 7f7e2d400700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 0x7f7e100a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.663+0000 7f7e43990b80 1 -- start start 2024-03-30T04:14:10.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c25d0 0x559de96c29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de961b990 con 0x559de96c2ef0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de9619350 con 0x559de96c25d0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e43990b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de9614c40 con 0x559de96cddf0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2d400700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 0x7f7e100a37d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009c60 tx=0x7f7e1c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54336/0 (socket says 172.21.15.23:54336) 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1738946557 learned_addr learned my addr 172.21.15.23:0/1738946557 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2de00700 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2d400700 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c25d0 0x559de96c29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1738946557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 msgr2=0x559de96d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1738946557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c25d0 msgr2=0x559de96c29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.664+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c25d0 0x559de96c29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.665+0000 7f7e2ca00700 1 -- 172.21.15.23:0/1738946557 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559de94963b0 con 0x559de96c2ef0 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.665+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7e140a1fc0 tx=0x7f7e14000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8fcaca0a58bca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.665+0000 7f7e1a600700 1 -- 172.21.15.23:0/1738946557 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1409a070 con 0x559de96c2ef0 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.665+0000 7f7e1a600700 1 -- 172.21.15.23:0/1738946557 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e1409dea0 con 0x559de96c2ef0 2024-03-30T04:14:10.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.665+0000 7f7e1a600700 1 -- 172.21.15.23:0/1738946557 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14096780 con 0x559de96c2ef0 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 -- 172.21.15.23:0/1738946557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 msgr2=0x559de96cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7e140a1fc0 tx=0x7f7e14000bc0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 -- 172.21.15.23:0/1738946557 shutdown_connections 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c25d0 0x559de96c29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 --2- 172.21.15.23:0/1738946557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96cd8b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 -- 172.21.15.23:0/1738946557 >> 172.21.15.23:0/1738946557 conn(0x559de96226a0 msgr2=0x559de96bea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 -- 172.21.15.23:0/1738946557 shutdown_connections 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.666+0000 7f7e43990b80 1 -- 172.21.15.23:0/1738946557 wait complete. 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.667+0000 7f7e43990b80 1 -- start start 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.667+0000 7f7e43990b80 1 Processor -- start 2024-03-30T04:14:10.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de96cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de95b0d80 con 0x559de96c25d0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de95b0a20 con 0x559de96c2ef0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de94986a0 con 0x559de96cddf0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e43990b80 10 client.0 _ll_register_callbacks cb 0x559de95ba470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de96cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54338/0 (socket says 172.21.15.23:54338) 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 -- 172.21.15.23:0/3562803210 learned_addr learned my addr 172.21.15.23:0/3562803210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2de00700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 -- 172.21.15.23:0/3562803210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 msgr2=0x559de96cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 -- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 msgr2=0x559de96cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de96cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2d400700 1 -- 172.21.15.23:0/3562803210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281aa8b0 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.668+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de96cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e2d400700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c01ebe0 tx=0x7f7e1c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390ef384a4f03146 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00ec50 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559de95d5380 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7e1c009e60 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e1c00edb0 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e43990b80 10 client.18858 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.669+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559de9442b60 con 0x559de96c25d0 2024-03-30T04:14:10.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559de95d5380 con 0x559de96c25d0 2024-03-30T04:14:10.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7e1c01b520 con 0x559de96c25d0 2024-03-30T04:14:10.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 10 client.18858 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 10 client.18858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:10.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 20 client.18858 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de96ddab0 0x559de96dfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 4 client.18858 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559de960d700 con 0x559de96ddab0 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.670+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de96ddab0 0x559de96dfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de96ddab0 0x559de96dfed0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4216aa6633c0bddf server_cookie=bacf7eacadcff1d2 in_seq=0 out_seq=0 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e0ea00700 10 client.18858 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e0ea00700 1 -- 172.21.15.23:0/3562803210 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559de960d700 con 0x559de96ddab0 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e0ea00700 10 client.18858 handle_command_reply: tid=0 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e43990b80 1 client.18858 shutdown 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 msgr2=0x559de96cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.671+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c01ebe0 tx=0x7f7e1c0047c0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 shutdown_connections 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de96ddab0 0x559de96dfed0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de96cacb0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 --2- 172.21.15.23:0/3562803210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de96cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e0ea00700 0 client.18858 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 >> 172.21.15.23:0/3562803210 conn(0x559de96226a0 msgr2=0x559de96c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 shutdown_connections 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 1 -- 172.21.15.23:0/3562803210 wait complete. 2024-03-30T04:14:10.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.672+0000 7f7e43990b80 20 client.18858 trim_cache size 0 max 16384 2024-03-30T04:14:10.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:10 smithi154 ceph-mon[79161]: pgmap v381: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.4 MiB/s wr, 233 op/s 2024-03-30T04:14:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:10 smithi154 ceph-mon[79161]: osdmap e102: 12 total, 12 up, 12 in 2024-03-30T04:14:10.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.687+0000 7f7e43990b80 1 Processor -- start 2024-03-30T04:14:10.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.687+0000 7f7e43990b80 1 -- start start 2024-03-30T04:14:10.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.687+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96c25d0 0x559de96bdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e43990b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96dd850 con 0x559de96c2ef0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e43990b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96bedc0 con 0x559de96cddf0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e43990b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96e0290 con 0x559de96c25d0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96d4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54350/0 (socket says 172.21.15.23:54350) 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2ca00700 1 -- 172.21.15.23:0/519050542 learned_addr learned my addr 172.21.15.23:0/519050542 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56786/0 (socket says 172.21.15.23:56786) 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2d400700 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96c25d0 0x559de96bdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 -- 172.21.15.23:0/519050542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96c25d0 msgr2=0x559de96bdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96c25d0 0x559de96bdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 -- 172.21.15.23:0/519050542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 msgr2=0x559de96d4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96d4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.688+0000 7f7e2de00700 1 -- 172.21.15.23:0/519050542 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559de95daea0 con 0x559de96cddf0 2024-03-30T04:14:10.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.689+0000 7f7e2de00700 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b780 tx=0x7f7e20007b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f89f380957b6e068 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.689+0000 7f7e1b000700 1 -- 172.21.15.23:0/519050542 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000e040 con 0x559de96cddf0 2024-03-30T04:14:10.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.689+0000 7f7e1b000700 1 -- 172.21.15.23:0/519050542 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e20021070 con 0x559de96cddf0 2024-03-30T04:14:10.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.689+0000 7f7e1b000700 1 -- 172.21.15.23:0/519050542 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e20007950 con 0x559de96cddf0 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- 172.21.15.23:0/519050542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 msgr2=0x559de96cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f7e2000b780 tx=0x7f7e20007b80 comp rx=0 tx=0).stop 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- 172.21.15.23:0/519050542 shutdown_connections 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96cddf0 0x559de96cb550 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96c25d0 0x559de96bdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 --2- 172.21.15.23:0/519050542 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c2ef0 0x559de96d4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- 172.21.15.23:0/519050542 >> 172.21.15.23:0/519050542 conn(0x559de96226a0 msgr2=0x559de96b75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- 172.21.15.23:0/519050542 shutdown_connections 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- 172.21.15.23:0/519050542 wait complete. 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.690+0000 7f7e43990b80 1 -- start start 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 Processor -- start 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de972b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96d3050 con 0x559de96c25d0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96d31c0 con 0x559de96c2ef0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559de96d3380 con 0x559de96cddf0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e43990b80 10 client.0 _ll_register_callbacks cb 0x559de95ba470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e2d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e2de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de972b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e2d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54354/0 (socket says 172.21.15.23:54354) 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.691+0000 7f7e2d400700 1 -- 172.21.15.23:0/575120397 learned_addr learned my addr 172.21.15.23:0/575120397 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 -- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 msgr2=0x559de972b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de972b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 -- 172.21.15.23:0/575120397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 msgr2=0x559de972ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 -- 172.21.15.23:0/575120397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e28114ad0 con 0x559de96cddf0 2024-03-30T04:14:10.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2d400700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2ca00700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de972b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:10.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e2de00700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d2a80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x559de960d840 tx=0x7f7e20006040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ceab662372ee85a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:10.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e0ea00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2000e030 con 0x559de96cddf0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e1409dea0 con 0x559de96cddf0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.692+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7e20023070 con 0x559de96cddf0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e200074a0 con 0x559de96cddf0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 10 client.28816 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x559de949acd0 con 0x559de96cddf0 2024-03-30T04:14:10.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e1409dea0 con 0x559de96cddf0 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7e2000b0d0 con 0x559de96cddf0 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 10 client.28816 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 10 client.28816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 20 client.28816 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de9732730 0x559de9734b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 4 client.28816 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.693+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559de9463cf0 con 0x559de9732730 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e2d400700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de9732730 0x559de9734b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e2d400700 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de9732730 0x559de9734b20 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaddf35977b2e76c server_cookie=c0affff342dcab61 in_seq=0 out_seq=0 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e0ea00700 10 client.28816 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e0ea00700 1 -- 172.21.15.23:0/575120397 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559de9463cf0 con 0x559de9732730 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e0ea00700 10 client.28816 handle_command_reply: tid=0 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e43990b80 1 client.28816 shutdown 2024-03-30T04:14:10.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 msgr2=0x559de96d2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.694+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d2a80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x559de960d840 tx=0x7f7e20006040 comp rx=0 tx=0).stop 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 shutdown_connections 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x559de9732730 0x559de9734b20 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559de96cddf0 0x559de96d2a80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559de96c25d0 0x559de972ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 --2- 172.21.15.23:0/575120397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x559de96c2ef0 0x559de972b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e0ea00700 0 client.28816 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 >> 172.21.15.23:0/575120397 conn(0x559de96226a0 msgr2=0x559de96be4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 shutdown_connections 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 1 -- 172.21.15.23:0/575120397 wait complete. 2024-03-30T04:14:10.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e43990b80 20 client.28816 trim_cache size 0 max 16384 2024-03-30T04:14:10.698 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:10.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 msgr2=0x7f7e100a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 0x7f7e100a37d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c009c60 tx=0x7f7e1c009260 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 msgr2=0x7f7e281a3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.695+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f7e1400e970 tx=0x7f7e1400ec80 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 shutdown_connections 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e2819d630 0x7f7e2819dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e28107610 0x7f7e281a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f7e100a1310 0x7f7e100a37d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 --2- 172.21.15.23:0/1888645739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e2810c6e0 0x7f7e281a3ad0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 >> 172.21.15.23:0/1888645739 conn(0x7f7e28076b90 msgr2=0x7f7e28077270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 shutdown_connections 2024-03-30T04:14:10.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:10.696+0000 7f7e2fe00700 1 -- 172.21.15.23:0/1888645739 wait complete. 2024-03-30T04:14:11.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1186502 9.55 MB/sec execute 528 sec latency 17.422 ms 2024-03-30T04:14:11.713 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:14:11.714 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 -- 172.21.15.23:0/2386320624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694101510 msgr2=0x7f769410d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 --2- 172.21.15.23:0/2386320624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694101510 0x7f769410d9d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f768c00b0d0 tx=0x7f768c00b3e0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 -- 172.21.15.23:0/2386320624 shutdown_connections 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 --2- 172.21.15.23:0/2386320624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694101510 0x7f769410d9d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 --2- 172.21.15.23:0/2386320624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694106c10 0x7f7694106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 --2- 172.21.15.23:0/2386320624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694100b50 0x7f7694100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 -- 172.21.15.23:0/2386320624 >> 172.21.15.23:0/2386320624 conn(0x7f76940fc620 msgr2=0x7f76940fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:11.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 -- 172.21.15.23:0/2386320624 shutdown_connections 2024-03-30T04:14:11.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.955+0000 7f7693200700 1 -- 172.21.15.23:0/2386320624 wait complete. 2024-03-30T04:14:11.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.956+0000 7f7693200700 1 Processor -- start 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.956+0000 7f7693200700 1 -- start start 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.956+0000 7f7693200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7693200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 0x7f76941a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7693200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 0x7f769410c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7693200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410d120 con 0x7f7694106c10 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7693200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410d290 con 0x7f7694100b50 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7693200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769410d450 con 0x7f7694101510 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56804/0 (socket says 172.21.15.23:56804) 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 -- 172.21.15.23:0/3535862322 learned_addr learned my addr 172.21.15.23:0/3535862322 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7692800700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 0x7f769410c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691400700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 0x7f76941a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 -- 172.21.15.23:0/3535862322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 msgr2=0x7f76941a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 0x7f76941a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 -- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 msgr2=0x7f769410c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 0x7f769410c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691e00700 1 -- 172.21.15.23:0/3535862322 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c009d00 con 0x7f7694100b50 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7691400700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 0x7f76941a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.957+0000 7f7692800700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 0x7f769410c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f7691e00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f768400e9a0 tx=0x7f768400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f768aa00700 1 -- 172.21.15.23:0/3535862322 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684009790 con 0x7f7694100b50 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f768aa00700 1 -- 172.21.15.23:0/3535862322 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7684010430 con 0x7f7694100b50 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f768aa00700 1 -- 172.21.15.23:0/3535862322 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684018680 con 0x7f7694100b50 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769410d730 con 0x7f7694100b50 2024-03-30T04:14:11.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.958+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941aa3e0 con 0x7f7694100b50 2024-03-30T04:14:11.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.962+0000 7f768aa00700 1 -- 172.21.15.23:0/3535862322 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f76840105a0 con 0x7f7694100b50 2024-03-30T04:14:11.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.963+0000 7f76accdfb80 1 Processor -- start 2024-03-30T04:14:11.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.963+0000 7f76accdfb80 1 -- start start 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 0x56216c4a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 0x56216c4a8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c3f3a10 con 0x56216c4a6040 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c3f13d0 con 0x56216c49b140 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f76accdfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c3eccc0 con 0x56216c49a820 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f7692800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f7692800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35294/0 (socket says 172.21.15.23:35294) 2024-03-30T04:14:11.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.964+0000 7f7692800700 1 -- 172.21.15.23:0/2138691146 learned_addr learned my addr 172.21.15.23:0/2138691146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7691400700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 0x56216c4a8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7691e00700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 0x56216c4a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 -- 172.21.15.23:0/2138691146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 msgr2=0x56216c4a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 0x56216c4a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 -- 172.21.15.23:0/2138691146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 msgr2=0x56216c4a8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 0x56216c4a8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 -- 172.21.15.23:0/2138691146 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56216c26e430 con 0x56216c49a820 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7691400700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 0x56216c4a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7691e00700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 0x56216c4a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f768aa00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 0x7f76780a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7692800700 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f768c005ba0 tx=0x7f768c009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e87bd18310045979 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f768aa00700 1 -- 172.21.15.23:0/3535862322 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7684014070 con 0x7f7694100b50 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f7691e00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 0x7f76780a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.965+0000 7f767f400700 1 -- 172.21.15.23:0/2138691146 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00e040 con 0x56216c49a820 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.966+0000 7f767f400700 1 -- 172.21.15.23:0/2138691146 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768c009460 con 0x56216c49a820 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.966+0000 7f767f400700 1 -- 172.21.15.23:0/2138691146 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c005330 con 0x56216c49a820 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.966+0000 7f76accdfb80 1 -- 172.21.15.23:0/2138691146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 msgr2=0x56216c49ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.966+0000 7f7691e00700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 0x7f76780a37d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f76840963c0 tx=0x7f7684010c80 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:11.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.966+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f768c005ba0 tx=0x7f768c009b80 comp rx=0 tx=0).stop 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 -- 172.21.15.23:0/2138691146 shutdown_connections 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c4a6040 0x56216c4a8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49a820 0x56216c49ac00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2138691146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49b140 0x56216c4a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 -- 172.21.15.23:0/2138691146 >> 172.21.15.23:0/2138691146 conn(0x56216c3fa720 msgr2=0x56216c496ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 -- 172.21.15.23:0/2138691146 shutdown_connections 2024-03-30T04:14:11.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 -- 172.21.15.23:0/2138691146 wait complete. 2024-03-30T04:14:11.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.967+0000 7f76accdfb80 1 -- start start 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 Processor -- start 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c4a3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c388e00 con 0x56216c49a820 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c388aa0 con 0x56216c4a6040 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c270720 con 0x56216c49b140 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.968+0000 7f76accdfb80 10 client.0 _ll_register_callbacks cb 0x56216c3924f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7692800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c4a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7692800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c4a3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56818/0 (socket says 172.21.15.23:56818) 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7692800700 1 -- 172.21.15.23:0/2490560472 learned_addr learned my addr 172.21.15.23:0/2490560472 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7692800700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 -- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 msgr2=0x56216c4a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c4a3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 -- 172.21.15.23:0/2490560472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 msgr2=0x56216c4a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.969+0000 7f7691400700 1 -- 172.21.15.23:0/2490560472 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769410e2d0 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f7691400700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4a2d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7680005950 tx=0x7f7680012e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e653d67c7b0173e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f7673e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680010b90 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56216c3ad400 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7680010cf0 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7680004020 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f76accdfb80 10 client.28828 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.970+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56216c21abe0 con 0x56216c49b140 2024-03-30T04:14:11.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56216c3ad400 con 0x56216c49b140 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7680003c30 con 0x56216c49b140 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 10 client.28828 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 10 client.28828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 20 client.28828 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c4b5be0 0x56216c4b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 4 client.28828 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:11.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.971+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56216c3e5780 con 0x56216c4b5be0 2024-03-30T04:14:11.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.972+0000 7f7692800700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c4b5be0 0x56216c4b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.972+0000 7f7692800700 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c4b5be0 0x56216c4b8000 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8950e5ad96ec850c server_cookie=610b19033b036fa7 in_seq=0 out_seq=0 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.972+0000 7f7673e00700 10 client.28828 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f7673e00700 1 -- 172.21.15.23:0/2490560472 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x7f768c032730 con 0x56216c4b5be0 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f7673e00700 10 client.28828 handle_command_reply: tid=0 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 client.28828 shutdown 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 msgr2=0x56216c4a2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4a2d50 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7680005950 tx=0x7f7680012e40 comp rx=0 tx=0).stop 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 shutdown_connections 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c4a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c4b5be0 0x56216c4b8000 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 --2- 172.21.15.23:0/2490560472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4a2d50 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f7673e00700 0 client.28828 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.973+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 >> 172.21.15.23:0/2490560472 conn(0x56216c3fa720 msgr2=0x56216c49c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:11.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.974+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 shutdown_connections 2024-03-30T04:14:11.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.974+0000 7f76accdfb80 1 -- 172.21.15.23:0/2490560472 wait complete. 2024-03-30T04:14:11.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.974+0000 7f76accdfb80 20 client.28828 trim_cache size 0 max 16384 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 Processor -- start 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 -- start start 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49b140 0x56216c4a1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c4a6040 0x56216c4a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c4a20a0 con 0x56216c49b140 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c496fd0 con 0x56216c49a820 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.996+0000 7f76accdfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c4ad1a0 con 0x56216c4a6040 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7691400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49b140 0x56216c4a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56826/0 (socket says 172.21.15.23:56826) 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 -- 172.21.15.23:0/1293937877 learned_addr learned my addr 172.21.15.23:0/1293937877 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c4a6040 0x56216c4a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 -- 172.21.15.23:0/1293937877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c4a6040 msgr2=0x56216c4a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c4a6040 0x56216c4a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 -- 172.21.15.23:0/1293937877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49b140 msgr2=0x56216c4a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49b140 0x56216c4a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 -- 172.21.15.23:0/1293937877 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56216c3b2f20 con 0x56216c49a820 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f7692800700 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f768c005040 tx=0x7f768c028040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96c58392276e660c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f767fe00700 1 -- 172.21.15.23:0/1293937877 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00e040 con 0x56216c49a820 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.997+0000 7f767fe00700 1 -- 172.21.15.23:0/1293937877 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768c02e070 con 0x56216c49a820 2024-03-30T04:14:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f767fe00700 1 -- 172.21.15.23:0/1293937877 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c0057b0 con 0x56216c49a820 2024-03-30T04:14:12.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 -- 172.21.15.23:0/1293937877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 msgr2=0x56216c4acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f768c005040 tx=0x7f768c028040 comp rx=0 tx=0).stop 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 -- 172.21.15.23:0/1293937877 shutdown_connections 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c4a6040 0x56216c4a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c49a820 0x56216c4acc60 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 --2- 172.21.15.23:0/1293937877 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49b140 0x56216c4a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 -- 172.21.15.23:0/1293937877 >> 172.21.15.23:0/1293937877 conn(0x56216c3fa720 msgr2=0x56216c48f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 -- 172.21.15.23:0/1293937877 shutdown_connections 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.998+0000 7f76accdfb80 1 -- 172.21.15.23:0/1293937877 wait complete. 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.999+0000 7f76accdfb80 1 -- start start 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:11.999+0000 7f76accdfb80 1 Processor -- start 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c5099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c509f20 con 0x56216c49a820 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c50a090 con 0x56216c4a6040 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56216c50a250 con 0x56216c49b140 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f76accdfb80 10 client.0 _ll_register_callbacks cb 0x56216c3924f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f7691400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f7692800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f7691400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35326/0 (socket says 172.21.15.23:35326) 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f7691400700 1 -- 172.21.15.23:0/4117392610 learned_addr learned my addr 172.21.15.23:0/4117392610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.000+0000 7f7692800700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c5099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 -- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 msgr2=0x56216c5099e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c5099e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 -- 172.21.15.23:0/4117392610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 msgr2=0x56216c4ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4ab9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 -- 172.21.15.23:0/4117392610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c030040 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7692800700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7691400700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56216c4a4320 tx=0x7f768001a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a54e4abad9b9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7673e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680003f90 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f768c009460 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f7680000e40 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76800040f0 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f76accdfb80 10 client.28834 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.001+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56216c272d50 con 0x56216c49b140 2024-03-30T04:14:12.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768c009460 con 0x56216c49b140 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7680005030 con 0x56216c49b140 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 10 client.28834 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 10 client.28834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 20 client.28834 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c50bab0 0x56216c50ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 4 client.28834 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56216c23bd70 con 0x56216c50bab0 2024-03-30T04:14:12.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.002+0000 7f7692800700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c50bab0 0x56216c50ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.003+0000 7f7692800700 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c50bab0 0x56216c50ded0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83fbcece019f8479 server_cookie=f379fb4fcce06102 in_seq=0 out_seq=0 2024-03-30T04:14:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.003+0000 7f7673e00700 10 client.28834 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f7673e00700 1 -- 172.21.15.23:0/4117392610 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56216c23bd70 con 0x56216c50bab0 2024-03-30T04:14:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f7673e00700 10 client.28834 handle_command_reply: tid=0 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 client.28834 shutdown 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 msgr2=0x56216c4abf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x56216c4a4320 tx=0x7f768001a610 comp rx=0 tx=0).stop 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 shutdown_connections 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56216c4a6040 0x56216c5099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56216c49a820 0x56216c4ab9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56216c50bab0 0x56216c50ded0 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 --2- 172.21.15.23:0/4117392610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56216c49b140 0x56216c4abf20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 >> 172.21.15.23:0/4117392610 conn(0x56216c3fa720 msgr2=0x56216c4a14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f7673e00700 0 client.28834 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 shutdown_connections 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 1 -- 172.21.15.23:0/4117392610 wait complete. 2024-03-30T04:14:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.004+0000 7f76accdfb80 20 client.28834 trim_cache size 0 max 16384 2024-03-30T04:14:12.007 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:12.007 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:14:12.007 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "1f2f7a13-af73-40cf-8a7d-827d969b79f0", 2024-03-30T04:14:12.007 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:14:12.007 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 msgr2=0x7f76780a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 0x7f76780a37d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f76840963c0 tx=0x7f7684010c80 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 msgr2=0x7f76941a33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f768400e9a0 tx=0x7f768400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 shutdown_connections 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7694106c10 0x7f769410c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f76780a1310 0x7f76780a37d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7694100b50 0x7f76941a33f0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 --2- 172.21.15.23:0/3535862322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7694101510 0x7f76941a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 >> 172.21.15.23:0/3535862322 conn(0x7f76940fc620 msgr2=0x7f76940fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 shutdown_connections 2024-03-30T04:14:12.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.005+0000 7f7693200700 1 -- 172.21.15.23:0/3535862322 wait complete. 2024-03-30T04:14:12.024 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:14:12.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1187809 9.55 MB/sec execute 529 sec latency 28.413 ms 2024-03-30T04:14:12.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 -- 172.21.15.23:0/1895579595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 msgr2=0x7f6bb01060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/1895579595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb01060c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4009990 tx=0x7f6ba4009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 -- 172.21.15.23:0/1895579595 shutdown_connections 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/1895579595 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 0x7f6bb010db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/1895579595 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb01060c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/1895579595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb00682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 -- 172.21.15.23:0/1895579595 >> 172.21.15.23:0/1895579595 conn(0x7f6bb0076d40 msgr2=0x7f6bb0077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 -- 172.21.15.23:0/1895579595 shutdown_connections 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.251+0000 7f6bb7a00700 1 -- 172.21.15.23:0/1895579595 wait complete. 2024-03-30T04:14:12.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.252+0000 7f6bb7a00700 1 Processor -- start 2024-03-30T04:14:12.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.252+0000 7f6bb7a00700 1 -- start start 2024-03-30T04:14:12.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 0x7f6bb01977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb019c180 con 0x7f6bb0105ce0 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb019c2f0 con 0x7f6bb00687f0 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb7a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb019c4b0 con 0x7f6bb0067e30 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb5a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35340/0 (socket says 172.21.15.23:35340) 2024-03-30T04:14:12.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bb5000700 1 -- 172.21.15.23:0/2298349482 learned_addr learned my addr 172.21.15.23:0/2298349482 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.253+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 0x7f6bb01977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 -- 172.21.15.23:0/2298349482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 msgr2=0x7f6bb01977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 0x7f6bb01977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 -- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 msgr2=0x7f6bb019bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb019bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 -- 172.21.15.23:0/2298349482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba40095f0 con 0x7f6bb0067e30 2024-03-30T04:14:12.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb019bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb5000700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4000c00 tx=0x7f6ba4005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bad600700 1 -- 172.21.15.23:0/2298349482 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba401d070 con 0x7f6bb0067e30 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bad600700 1 -- 172.21.15.23:0/2298349482 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ba4022470 con 0x7f6bb0067e30 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb019c730 con 0x7f6bb0067e30 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bad600700 1 -- 172.21.15.23:0/2298349482 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4017650 con 0x7f6bb0067e30 2024-03-30T04:14:12.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.254+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01a3fd0 con 0x7f6bb0067e30 2024-03-30T04:14:12.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.258+0000 7f6bad600700 1 -- 172.21.15.23:0/2298349482 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f6ba40225e0 con 0x7f6bb0067e30 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.259+0000 7f6bcb50eb80 1 Processor -- start 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.259+0000 7f6bad600700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 0x7f6b980a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.259+0000 7f6bcb50eb80 1 -- start start 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.259+0000 7f6bad600700 1 -- 172.21.15.23:0/2298349482 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f6ba40ce400 con 0x7f6bb0067e30 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 0x7f6b980a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 0x55aeb40561e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3fa1990 con 0x55aeb4053df0 2024-03-30T04:14:12.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3f9f350 con 0x55aeb4048ef0 2024-03-30T04:14:12.259 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3f9ac40 con 0x55aeb40485d0 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35354/0 (socket says 172.21.15.23:35354) 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bb5000700 1 -- 172.21.15.23:0/1221643188 learned_addr learned my addr 172.21.15.23:0/1221643188 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bafe00700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.260+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 0x55aeb40561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 -- 172.21.15.23:0/1221643188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 msgr2=0x55aeb40538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 0x7f6b980a3750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8007ae0 tx=0x7f6ba8007ea0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 -- 172.21.15.23:0/1221643188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 msgr2=0x55aeb40561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 0x55aeb40561e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 -- 172.21.15.23:0/1221643188 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeb3e1c3b0 con 0x55aeb40485d0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 0x55aeb40561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4097750 tx=0x7f6ba409b020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a0e6ff2cb77ccc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6ba2000700 1 -- 172.21.15.23:0/1221643188 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40b1020 con 0x55aeb40485d0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6ba2000700 1 -- 172.21.15.23:0/1221643188 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ba409b950 con 0x55aeb40485d0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.261+0000 7f6ba2000700 1 -- 172.21.15.23:0/1221643188 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4022990 con 0x55aeb40485d0 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1221643188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 msgr2=0x55aeb40489b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4097750 tx=0x7f6ba409b020 comp rx=0 tx=0).stop 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1221643188 shutdown_connections 2024-03-30T04:14:12.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb4053df0 0x55aeb40561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb40485d0 0x55aeb40489b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1221643188 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1221643188 >> 172.21.15.23:0/1221643188 conn(0x55aeb3fa86a0 msgr2=0x55aeb4044a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1221643188 shutdown_connections 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.262+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1221643188 wait complete. 2024-03-30T04:14:12.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.263+0000 7f6bcb50eb80 1 -- start start 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.263+0000 7f6bcb50eb80 1 Processor -- start 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb4057b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4053df0 0x55aeb405c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3f36d80 con 0x55aeb40485d0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3f36a20 con 0x55aeb4053df0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb3e1e6a0 con 0x55aeb4048ef0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bcb50eb80 10 client.0 _ll_register_callbacks cb 0x55aeb3f40470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bb5a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4053df0 0x55aeb405c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35356/0 (socket says 172.21.15.23:35356) 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb4057b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 -- 172.21.15.23:0/2453447705 learned_addr learned my addr 172.21.15.23:0/2453447705 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 -- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4053df0 msgr2=0x55aeb405c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4053df0 0x55aeb405c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 -- 172.21.15.23:0/2453447705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 msgr2=0x55aeb4057b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb4057b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.264+0000 7f6bafe00700 1 -- 172.21.15.23:0/2453447705 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba8017a00 con 0x55aeb4048ef0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0127f0 tx=0x7f6b9c012b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=612fa431f8cc76bd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c017070 con 0x55aeb4048ef0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f6b9c00f4e0 con 0x55aeb4048ef0 2024-03-30T04:14:12.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aeb3f5b380 con 0x55aeb4048ef0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b9c00f6a0 con 0x55aeb4048ef0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6bcb50eb80 10 client.28846 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f6bb01a3fd0 con 0x55aeb4048ef0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55aeb3f5b380 con 0x55aeb4048ef0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.265+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6b9c0107e0 con 0x55aeb4048ef0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 10 client.28846 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 10 client.28846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 20 client.28846 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb4067aa0 0x55aeb4069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 4 client.28846 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aeb3f93700 con 0x55aeb4067aa0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bb5000700 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb4067aa0 0x55aeb4069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.266+0000 7f6bb5000700 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb4067aa0 0x55aeb4069ec0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b9d836175d7f1d server_cookie=c95adb22b2695e80 in_seq=0 out_seq=0 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6b96a00700 10 client.28846 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6b96a00700 1 -- 172.21.15.23:0/2453447705 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55aeb3f93700 con 0x55aeb4067aa0 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6b96a00700 10 client.28846 handle_command_reply: tid=0 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 client.28846 shutdown 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 msgr2=0x55aeb4058050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0127f0 tx=0x7f6b9c012b00 comp rx=0 tx=0).stop 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 shutdown_connections 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4053df0 0x55aeb405c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb4057b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb4067aa0 0x55aeb4069ec0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2453447705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4048ef0 0x55aeb4058050 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6b96a00700 0 client.28846 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.267+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 >> 172.21.15.23:0/2453447705 conn(0x55aeb3fa86a0 msgr2=0x55aeb404a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.268+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 shutdown_connections 2024-03-30T04:14:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.268+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2453447705 wait complete. 2024-03-30T04:14:12.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.268+0000 7f6bcb50eb80 20 client.28846 trim_cache size 0 max 16384 2024-03-30T04:14:12.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 Processor -- start 2024-03-30T04:14:12.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 -- start start 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb40567b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb406a250 con 0x55aeb40485d0 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb4046160 con 0x55aeb4048ef0 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.287+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb4044f20 con 0x55aeb4053df0 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56876/0 (socket says 172.21.15.23:56876) 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 -- 172.21.15.23:0/2392252099 learned_addr learned my addr 172.21.15.23:0/2392252099 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb40567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 -- 172.21.15.23:0/2392252099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 msgr2=0x55aeb40567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb40567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 -- 172.21.15.23:0/2392252099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 msgr2=0x55aeb40537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.288+0000 7f6bafe00700 1 -- 172.21.15.23:0/2392252099 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeb3dc8b60 con 0x55aeb4048ef0 2024-03-30T04:14:12.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.289+0000 7f6bafe00700 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009340 tx=0x7f6b9c012e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8c30df478aec41 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.289+0000 7f6ba2a00700 1 -- 172.21.15.23:0/2392252099 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c017070 con 0x55aeb4048ef0 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.289+0000 7f6ba2a00700 1 -- 172.21.15.23:0/2392252099 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b9c018040 con 0x55aeb4048ef0 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.289+0000 7f6ba2a00700 1 -- 172.21.15.23:0/2392252099 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c003680 con 0x55aeb4048ef0 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2392252099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 msgr2=0x55aeb4057170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009340 tx=0x7f6b9c012e50 comp rx=0 tx=0).stop 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2392252099 shutdown_connections 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb40567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/2392252099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb4057170 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2392252099 >> 172.21.15.23:0/2392252099 conn(0x55aeb3fa86a0 msgr2=0x55aeb403d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2392252099 shutdown_connections 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/2392252099 wait complete. 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.290+0000 7f6bcb50eb80 1 -- start start 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 Processor -- start 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40640f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb40612b0 con 0x55aeb40485d0 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb40613f0 con 0x55aeb4048ef0 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb40bb8d0 con 0x55aeb4053df0 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bcb50eb80 10 client.0 _ll_register_callbacks cb 0x55aeb3f40470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bb5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bb5000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40640f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bb5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35382/0 (socket says 172.21.15.23:35382) 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bafe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:56878/0 (socket says 172.21.15.23:56878) 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.291+0000 7f6bb5a00700 1 -- 172.21.15.23:0/1605034166 learned_addr learned my addr 172.21.15.23:0/1605034166 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 -- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 msgr2=0x55aeb40605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 -- 172.21.15.23:0/1605034166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 msgr2=0x55aeb40640f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40640f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 -- 172.21.15.23:0/1605034166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb01a3fd0 con 0x55aeb4053df0 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bb5a00700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6ba801a7a0 tx=0x7f6ba8004b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=182f32192447e255 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6b96a00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:12.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8025860 con 0x55aeb4053df0 2024-03-30T04:14:12.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ba409b950 con 0x55aeb4053df0 2024-03-30T04:14:12.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.292+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9600+0+0 (secure 0 0 0) 0x7f6ba80259c0 con 0x55aeb4053df0 2024-03-30T04:14:12.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ba8025b80 con 0x55aeb4053df0 2024-03-30T04:14:12.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 10 client.28852 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:12.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55aeb3f60ea0 con 0x55aeb4053df0 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ba409b950 con 0x55aeb4053df0 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6ba802b020 con 0x55aeb4053df0 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 10 client.28852 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 10 client.28852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 20 client.28852 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb40bd0e0 0x55aeb40bf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 4 client.28852 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55aeb3de9cf0 con 0x55aeb40bd0e0 2024-03-30T04:14:12.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.293+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb40bd0e0 0x55aeb40bf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bb5000700 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb40bd0e0 0x55aeb40bf500 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=120bb406b481b18a server_cookie=caa3e2a0705b043e in_seq=0 out_seq=0 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6b96a00700 10 client.28852 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6b96a00700 1 -- 172.21.15.23:0/1605034166 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55aeb3de9cf0 con 0x55aeb40bd0e0 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6b96a00700 10 client.28852 handle_command_reply: tid=0 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 client.28852 shutdown 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 msgr2=0x55aeb4060bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6ba801a7a0 tx=0x7f6ba8004b40 comp rx=0 tx=0).stop 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 shutdown_connections 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55aeb40bd0e0 0x55aeb40bf500 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55aeb4053df0 0x55aeb4060bd0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55aeb40485d0 0x55aeb40640f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 --2- 172.21.15.23:0/1605034166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55aeb4048ef0 0x55aeb40605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6b96a00700 0 client.28852 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:12.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.294+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 >> 172.21.15.23:0/1605034166 conn(0x55aeb3fa86a0 msgr2=0x55aeb40576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 shutdown_connections 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bcb50eb80 1 -- 172.21.15.23:0/1605034166 wait complete. 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bcb50eb80 20 client.28852 trim_cache size 0 max 16384 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stdout: "1f2f7a13-af73-40cf-8a7d-827d969b79f0": { 2024-03-30T04:14:12.298 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:14:12.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "1f2f7a13-af73-40cf-8a7d-827d969b79f0", 2024-03-30T04:14:12.299 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:14:12.299 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:14:12.299 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:14:12.299 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 msgr2=0x7f6b980a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 0x7f6b980a3750 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8007ae0 tx=0x7f6ba8007ea0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 msgr2=0x7f6bb0197260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ba4000c00 tx=0x7f6ba4005740 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 shutdown_connections 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bb0105ce0 0x7f6bb019bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bb0067e30 0x7f6bb0197260 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6b980a1290 0x7f6b980a3750 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 --2- 172.21.15.23:0/2298349482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bb00687f0 0x7f6bb01977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.295+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 >> 172.21.15.23:0/2298349482 conn(0x7f6bb0076d40 msgr2=0x7f6bb00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.296+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 shutdown_connections 2024-03-30T04:14:12.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:12.296+0000 7f6bb7a00700 1 -- 172.21.15.23:0/2298349482 wait complete. 2024-03-30T04:14:12.315 INFO:tasks.cephfs.filesystem:scrub status for tag:1f2f7a13-af73-40cf-8a7d-827d969b79f0 - {'path': '/', 'tag': '1f2f7a13-af73-40cf-8a7d-827d969b79f0', 'options': 'recursive,force'} 2024-03-30T04:14:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:12 smithi193 ceph-mon[79581]: pgmap v383: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-03-30T04:14:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:12 smithi023 ceph-mon[38254]: pgmap v383: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-03-30T04:14:12.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:12 smithi154 ceph-mon[79161]: pgmap v383: 129 pgs: 129 active+clean; 569 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-03-30T04:14:13.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1189924 9.55 MB/sec execute 530 sec latency 21.558 ms 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:13 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:13 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:13.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:13 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:14.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1190819 9.54 MB/sec execute 531 sec latency 31.993 ms 2024-03-30T04:14:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:14 smithi193 ceph-mon[79581]: pgmap v384: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-03-30T04:14:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:14 smithi023 ceph-mon[38254]: pgmap v384: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-03-30T04:14:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:14 smithi154 ceph-mon[79161]: pgmap v384: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.0 MiB/s wr, 201 op/s 2024-03-30T04:14:15.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1192410 9.53 MB/sec execute 532 sec latency 46.298 ms 2024-03-30T04:14:15.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:15 smithi193 ceph-mon[79581]: pgmap v385: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 168 op/s 2024-03-30T04:14:15.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:15 smithi023 ceph-mon[38254]: pgmap v385: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 168 op/s 2024-03-30T04:14:15.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:15 smithi154 ceph-mon[79161]: pgmap v385: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.8 MiB/s wr, 168 op/s 2024-03-30T04:14:16.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1193626 9.54 MB/sec execute 533 sec latency 24.758 ms 2024-03-30T04:14:17.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1195019 9.53 MB/sec execute 534 sec latency 31.749 ms 2024-03-30T04:14:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:17 smithi154 ceph-mon[79161]: osdmap e103: 12 total, 12 up, 12 in 2024-03-30T04:14:17.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:17 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:14:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:17 smithi193 ceph-mon[79581]: osdmap e103: 12 total, 12 up, 12 in 2024-03-30T04:14:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:17 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:14:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:17 smithi023 ceph-mon[38254]: osdmap e103: 12 total, 12 up, 12 in 2024-03-30T04:14:17.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:17 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:14:18.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1197000 9.53 MB/sec execute 535 sec latency 25.427 ms 2024-03-30T04:14:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:18 smithi154 ceph-mon[79161]: pgmap v387: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-03-30T04:14:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:18 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:18 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:18 smithi193 ceph-mon[79581]: pgmap v387: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-03-30T04:14:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:18 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:18 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:18 smithi023 ceph-mon[38254]: pgmap v387: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 105 op/s 2024-03-30T04:14:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:18 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:18.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:18 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:19.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1197757 9.52 MB/sec execute 536 sec latency 36.172 ms 2024-03-30T04:14:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:19 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:14:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:19 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:14:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:14:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:14:19.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:19 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:14:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:19 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:14:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:14:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:14:19.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:19 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:14:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:19 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:14:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:14:19.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:14:19.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:14:20.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1199365 9.52 MB/sec execute 537 sec latency 54.685 ms 2024-03-30T04:14:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:20 smithi193 ceph-mon[79581]: pgmap v388: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-03-30T04:14:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:20 smithi023 ceph-mon[38254]: pgmap v388: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-03-30T04:14:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:20 smithi154 ceph-mon[79161]: pgmap v388: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-03-30T04:14:21.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1200921 9.52 MB/sec execute 538 sec latency 18.356 ms 2024-03-30T04:14:22.233 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1202368 9.51 MB/sec execute 539 sec latency 29.517 ms 2024-03-30T04:14:22.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:22 smithi193 ceph-mon[79581]: pgmap v389: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 168 op/s 2024-03-30T04:14:22.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:22 smithi023 ceph-mon[38254]: pgmap v389: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 168 op/s 2024-03-30T04:14:22.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:22 smithi154 ceph-mon[79161]: pgmap v389: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.7 MiB/s wr, 168 op/s 2024-03-30T04:14:23.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1204135 9.52 MB/sec execute 540 sec latency 31.151 ms 2024-03-30T04:14:24.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1205204 9.51 MB/sec execute 541 sec latency 29.229 ms 2024-03-30T04:14:24.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:24 smithi193 ceph-mon[79581]: pgmap v390: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:24.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:24 smithi023 ceph-mon[38254]: pgmap v390: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:24 smithi154 ceph-mon[79161]: pgmap v390: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:25.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1206474 9.50 MB/sec execute 542 sec latency 83.461 ms 2024-03-30T04:14:26.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1207787 9.50 MB/sec execute 543 sec latency 20.632 ms 2024-03-30T04:14:26.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:26 smithi193 ceph-mon[79581]: pgmap v391: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:26.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:26 smithi023 ceph-mon[38254]: pgmap v391: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:26.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:26 smithi154 ceph-mon[79161]: pgmap v391: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-03-30T04:14:27.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1209082 9.49 MB/sec execute 544 sec latency 22.603 ms 2024-03-30T04:14:27.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:27 smithi193 ceph-mon[79581]: pgmap v392: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 164 op/s 2024-03-30T04:14:27.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:27 smithi023 ceph-mon[38254]: pgmap v392: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 164 op/s 2024-03-30T04:14:27.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:27 smithi154 ceph-mon[79161]: pgmap v392: 129 pgs: 129 active+clean; 572 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 164 op/s 2024-03-30T04:14:28.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1211118 9.50 MB/sec execute 545 sec latency 41.962 ms 2024-03-30T04:14:29.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1211797 9.49 MB/sec execute 546 sec latency 28.403 ms 2024-03-30T04:14:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:29 smithi154 ceph-mon[79161]: pgmap v393: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-03-30T04:14:29.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:29 smithi193 ceph-mon[79581]: pgmap v393: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-03-30T04:14:29.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:29 smithi023 ceph-mon[38254]: pgmap v393: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 208 op/s 2024-03-30T04:14:30.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1212939 9.48 MB/sec execute 547 sec latency 72.541 ms 2024-03-30T04:14:31.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1214780 9.48 MB/sec execute 548 sec latency 29.128 ms 2024-03-30T04:14:32.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1215769 9.47 MB/sec execute 549 sec latency 30.979 ms 2024-03-30T04:14:32.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:32 smithi154 ceph-mon[79161]: pgmap v394: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-03-30T04:14:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:32 smithi193 ceph-mon[79581]: pgmap v394: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-03-30T04:14:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:32 smithi023 ceph-mon[38254]: pgmap v394: 129 pgs: 129 active+clean; 574 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 137 op/s 2024-03-30T04:14:33.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1217635 9.47 MB/sec execute 550 sec latency 60.221 ms 2024-03-30T04:14:34.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1218603 9.47 MB/sec execute 551 sec latency 18.581 ms 2024-03-30T04:14:34.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:34 smithi154 ceph-mon[79161]: pgmap v395: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 205 op/s 2024-03-30T04:14:34.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:34 smithi193 ceph-mon[79581]: pgmap v395: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 205 op/s 2024-03-30T04:14:34.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:34 smithi023 ceph-mon[38254]: pgmap v395: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 205 op/s 2024-03-30T04:14:35.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1219819 9.46 MB/sec execute 552 sec latency 39.219 ms 2024-03-30T04:14:36.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1221826 9.47 MB/sec execute 553 sec latency 22.508 ms 2024-03-30T04:14:36.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:36 smithi154 ceph-mon[79161]: pgmap v396: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:36.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:36 smithi193 ceph-mon[79581]: pgmap v396: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:36.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:36 smithi023 ceph-mon[38254]: pgmap v396: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:37.234 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1222662 9.45 MB/sec execute 554 sec latency 26.550 ms 2024-03-30T04:14:38.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1224667 9.46 MB/sec execute 555 sec latency 20.390 ms 2024-03-30T04:14:38.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:38 smithi154 ceph-mon[79161]: pgmap v397: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:38 smithi193 ceph-mon[79581]: pgmap v397: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:38 smithi023 ceph-mon[38254]: pgmap v397: 129 pgs: 129 active+clean; 568 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-03-30T04:14:39.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1226042 9.45 MB/sec execute 556 sec latency 21.536 ms 2024-03-30T04:14:40.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1227442 9.45 MB/sec execute 557 sec latency 79.489 ms 2024-03-30T04:14:40.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:40 smithi193 ceph-mon[79581]: pgmap v398: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 195 op/s 2024-03-30T04:14:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:40 smithi023 ceph-mon[38254]: pgmap v398: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 195 op/s 2024-03-30T04:14:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:40 smithi154 ceph-mon[79161]: pgmap v398: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 195 op/s 2024-03-30T04:14:41.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1228977 9.45 MB/sec execute 558 sec latency 23.753 ms 2024-03-30T04:14:42.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1230094 9.44 MB/sec execute 559 sec latency 25.566 ms 2024-03-30T04:14:42.316 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:14:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:42 smithi193 ceph-mon[79581]: pgmap v399: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-03-30T04:14:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:42 smithi023 ceph-mon[38254]: pgmap v399: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-03-30T04:14:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 -- 172.21.15.23:0/331695953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 msgr2=0x7fe47410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 --2- 172.21.15.23:0/331695953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe47410dc00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009f90 tx=0x7fe46c00b330 comp rx=0 tx=0).stop 2024-03-30T04:14:42.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 -- 172.21.15.23:0/331695953 shutdown_connections 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 --2- 172.21.15.23:0/331695953 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe47410dc00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 --2- 172.21.15.23:0/331695953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 --2- 172.21.15.23:0/331695953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474101910 0x7fe474105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 -- 172.21.15.23:0/331695953 >> 172.21.15.23:0/331695953 conn(0x7fe4740fc7c0 msgr2=0x7fe4740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 -- 172.21.15.23:0/331695953 shutdown_connections 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.533+0000 7fe479600700 1 -- 172.21.15.23:0/331695953 wait complete. 2024-03-30T04:14:42.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.534+0000 7fe479600700 1 Processor -- start 2024-03-30T04:14:42.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.534+0000 7fe479600700 1 -- start start 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe4741081c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 0x7fe4741a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47410be20 con 0x7fe474106080 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4741aa080 con 0x7fe474100f60 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe479600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4741aa210 con 0x7fe474108700 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe472a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe472a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38152/0 (socket says 172.21.15.23:38152) 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe472a00700 1 -- 172.21.15.23:0/4040061805 learned_addr learned my addr 172.21.15.23:0/4040061805 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe473400700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 0x7fe4741a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.535+0000 7fe472000700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe4741081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 -- 172.21.15.23:0/4040061805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 msgr2=0x7fe474107c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 -- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 msgr2=0x7fe4741081c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe4741081c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 -- 172.21.15.23:0/4040061805 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640095f0 con 0x7fe474108700 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe472a00700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe473400700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 0x7fe4741a9b40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00b6c0 tx=0x7fe46c009440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe46b400700 1 -- 172.21.15.23:0/4040061805 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c00e040 con 0x7fe474108700 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe46b400700 1 -- 172.21.15.23:0/4040061805 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe46c00be50 con 0x7fe474108700 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe46c009c40 con 0x7fe474108700 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe46b400700 1 -- 172.21.15.23:0/4040061805 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c005040 con 0x7fe474108700 2024-03-30T04:14:42.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.536+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4741aa730 con 0x7fe474108700 2024-03-30T04:14:42.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.539+0000 7fe46b400700 1 -- 172.21.15.23:0/4040061805 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe46c01d690 con 0x7fe474108700 2024-03-30T04:14:42.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.541+0000 7fe46b400700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 0x7fe4580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.541+0000 7fe48d0afb80 1 Processor -- start 2024-03-30T04:14:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.541+0000 7fe46b400700 1 -- 172.21.15.23:0/4040061805 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe46c0ce400 con 0x7fe474108700 2024-03-30T04:14:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.541+0000 7fe472a00700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 0x7fe4580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.541+0000 7fe48d0afb80 1 -- start start 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c657ef0 0x55fe5c6628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c662df0 0x55fe5c6651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c5b0990 con 0x55fe5c657ef0 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c5ae350 con 0x55fe5c662df0 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c5a9c40 con 0x55fe5c6575d0 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 0x7fe4580a3750 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe464005d10 tx=0x7fe464009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c657ef0 0x55fe5c6628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe473400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c662df0 0x55fe5c6651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43728/0 (socket says 172.21.15.23:43728) 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 -- 172.21.15.23:0/3876480844 learned_addr learned my addr 172.21.15.23:0/3876480844 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 -- 172.21.15.23:0/3876480844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c662df0 msgr2=0x55fe5c6651e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c662df0 0x55fe5c6651e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 -- 172.21.15.23:0/3876480844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c657ef0 msgr2=0x55fe5c6628b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c657ef0 0x55fe5c6628b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.542+0000 7fe472a00700 1 -- 172.21.15.23:0/3876480844 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe5c42b3b0 con 0x55fe5c6575d0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.543+0000 7fe472a00700 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe464011800 tx=0x7fe464014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e68b78a204ff36 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.543+0000 7fe45f400700 1 -- 172.21.15.23:0/3876480844 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464004220 con 0x55fe5c6575d0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.543+0000 7fe45f400700 1 -- 172.21.15.23:0/3876480844 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe464021070 con 0x55fe5c6575d0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.543+0000 7fe45f400700 1 -- 172.21.15.23:0/3876480844 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46401d7b0 con 0x55fe5c6575d0 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3876480844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 msgr2=0x55fe5c6579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe464011800 tx=0x7fe464014000 comp rx=0 tx=0).stop 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3876480844 shutdown_connections 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c662df0 0x55fe5c6651e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c6575d0 0x55fe5c6579b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3876480844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c657ef0 0x55fe5c6628b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3876480844 >> 172.21.15.23:0/3876480844 conn(0x55fe5c5b76a0 msgr2=0x55fe5c653a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3876480844 shutdown_connections 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3876480844 wait complete. 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.544+0000 7fe48d0afb80 1 -- start start 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 Processor -- start 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c65fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c660280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c545d80 con 0x55fe5c6575d0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c545a20 con 0x55fe5c657ef0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c42d6a0 con 0x55fe5c662df0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe48d0afb80 10 client.0 _ll_register_callbacks cb 0x55fe5c54f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe472a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe473400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c660280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.545+0000 7fe472000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c65fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33234/0 (socket says 172.21.15.23:33234) 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472a00700 1 -- 172.21.15.23:0/2246762842 learned_addr learned my addr 172.21.15.23:0/2246762842 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 -- 172.21.15.23:0/2246762842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 msgr2=0x55fe5c660280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c660280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 -- 172.21.15.23:0/2246762842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 msgr2=0x55fe5c662850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 -- 172.21.15.23:0/2246762842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe464017070 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472a00700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe473400700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c660280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe472000700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c65fb40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe47406a380 tx=0x7fe460012960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6fb929b00c170df server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe453e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.546+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460017070 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe5c56a380 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe460010430 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4600105f0 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 10 client.28883 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fe5c3d7b60 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55fe5c56a380 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe46001c840 con 0x55fe5c657ef0 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 10 client.28883 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:42.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 10 client.28883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 20 client.28883 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c672b40 0x55fe5c674f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 4 client.28883 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.547+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe5c5a2700 con 0x55fe5c672b40 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.548+0000 7fe472a00700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c672b40 0x55fe5c674f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.548+0000 7fe472a00700 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c672b40 0x55fe5c674f60 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e584ad8b3d2c08be server_cookie=4e9e8235d6749cda in_seq=0 out_seq=0 2024-03-30T04:14:42.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.548+0000 7fe453e00700 10 client.28883 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe453e00700 1 -- 172.21.15.23:0/2246762842 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55fe5c5a2700 con 0x55fe5c672b40 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe453e00700 10 client.28883 handle_command_reply: tid=0 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 client.28883 shutdown 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 msgr2=0x55fe5c65fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c65fb40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe47406a380 tx=0x7fe460012960 comp rx=0 tx=0).stop 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 shutdown_connections 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c672b40 0x55fe5c674f60 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c660280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c662850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/2246762842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c65fb40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe453e00700 0 client.28883 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.549+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 >> 172.21.15.23:0/2246762842 conn(0x55fe5c5b76a0 msgr2=0x55fe5c6596c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.550+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 shutdown_connections 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.550+0000 7fe48d0afb80 1 -- 172.21.15.23:0/2246762842 wait complete. 2024-03-30T04:14:42.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.550+0000 7fe48d0afb80 20 client.28883 trim_cache size 0 max 16384 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.567+0000 7fe48d0afb80 1 Processor -- start 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 -- start start 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c657ef0 0x55fe5c660280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c662df0 0x55fe5c65fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c658860 con 0x55fe5c662df0 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c6547d0 con 0x55fe5c6575d0 2024-03-30T04:14:42.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c65e090 con 0x55fe5c657ef0 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe472a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe472a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38176/0 (socket says 172.21.15.23:38176) 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe472a00700 1 -- 172.21.15.23:0/1699000374 learned_addr learned my addr 172.21.15.23:0/1699000374 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe472000700 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c657ef0 0x55fe5c660280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.568+0000 7fe473400700 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c662df0 0x55fe5c65fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 -- 172.21.15.23:0/1699000374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c657ef0 msgr2=0x55fe5c660280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c657ef0 0x55fe5c660280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 -- 172.21.15.23:0/1699000374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c662df0 msgr2=0x55fe5c65fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c662df0 0x55fe5c65fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 -- 172.21.15.23:0/1699000374 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe5c56fea0 con 0x55fe5c6575d0 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe472a00700 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe464011df0 tx=0x7fe464017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd09fc3f610d00c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe45fe00700 1 -- 172.21.15.23:0/1699000374 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46402c040 con 0x55fe5c6575d0 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe45fe00700 1 -- 172.21.15.23:0/1699000374 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe464031dc0 con 0x55fe5c6575d0 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.569+0000 7fe45fe00700 1 -- 172.21.15.23:0/1699000374 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4640315d0 con 0x55fe5c6575d0 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 -- 172.21.15.23:0/1699000374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 msgr2=0x55fe5c674f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe464011df0 tx=0x7fe464017000 comp rx=0 tx=0).stop 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 -- 172.21.15.23:0/1699000374 shutdown_connections 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c662df0 0x55fe5c65fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c6575d0 0x55fe5c674f60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/1699000374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c657ef0 0x55fe5c660280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 -- 172.21.15.23:0/1699000374 >> 172.21.15.23:0/1699000374 conn(0x55fe5c5b76a0 msgr2=0x55fe5c64c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 -- 172.21.15.23:0/1699000374 shutdown_connections 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.570+0000 7fe48d0afb80 1 -- 172.21.15.23:0/1699000374 wait complete. 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 -- start start 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 Processor -- start 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c6679c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c6c5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c6c5eb0 con 0x55fe5c6575d0 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c6c5ff0 con 0x55fe5c657ef0 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.571+0000 7fe48d0afb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe5c6c6130 con 0x55fe5c662df0 2024-03-30T04:14:42.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe48d0afb80 10 client.0 _ll_register_callbacks cb 0x55fe5c54f470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe472000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe472a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c6679c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe472000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38188/0 (socket says 172.21.15.23:38188) 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c6c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe472000700 1 -- 172.21.15.23:0/3161353679 learned_addr learned my addr 172.21.15.23:0/3161353679 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 -- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 msgr2=0x55fe5c667f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 -- 172.21.15.23:0/3161353679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 msgr2=0x55fe5c6679c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c6679c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 -- 172.21.15.23:0/3161353679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe5c42b3b0 con 0x55fe5c662df0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe472000700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe473400700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c6c5970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0cb710 tx=0x7fe46c009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0d5c50961a141c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.572+0000 7fe453e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c09b860 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe464021070 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe46c09b160 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe46c09ef90 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 10 client.28870 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55fe5c42fcd0 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe464021070 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe46c0b9050 con 0x55fe5c662df0 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 10 client.28870 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 10 client.28870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 20 client.28870 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c6c7660 0x55fe5c6c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 4 client.28870 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:14:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.573+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fe5c3f8cf0 con 0x55fe5c6c7660 2024-03-30T04:14:42.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe472a00700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c6c7660 0x55fe5c6c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe472a00700 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c6c7660 0x55fe5c6c9a50 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e01b06ce9e210d server_cookie=7549b3e79e5f37f8 in_seq=0 out_seq=0 2024-03-30T04:14:42.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe453e00700 10 client.28870 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe453e00700 1 -- 172.21.15.23:0/3161353679 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fe5c3f8cf0 con 0x55fe5c6c7660 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe453e00700 10 client.28870 handle_command_reply: tid=0 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.574+0000 7fe48d0afb80 1 client.28870 shutdown 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 msgr2=0x55fe5c6c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c6c5970 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0cb710 tx=0x7fe46c009920 comp rx=0 tx=0).stop 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 shutdown_connections 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55fe5c6c7660 0x55fe5c6c9a50 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe5c662df0 0x55fe5c6c5970 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe453e00700 0 client.28870 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55fe5c6575d0 0x55fe5c6679c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 --2- 172.21.15.23:0/3161353679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55fe5c657ef0 0x55fe5c667f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 >> 172.21.15.23:0/3161353679 conn(0x55fe5c5b76a0 msgr2=0x55fe5c65adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 shutdown_connections 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 1 -- 172.21.15.23:0/3161353679 wait complete. 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe48d0afb80 20 client.28870 trim_cache size 0 max 16384 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:14:42.579 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 msgr2=0x7fe4580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.575+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 0x7fe4580a3750 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe464005d10 tx=0x7fe464009310 comp rx=0 tx=0).stop 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 msgr2=0x7fe4741a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 0x7fe4741a9b40 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe46c00b6c0 tx=0x7fe46c009440 comp rx=0 tx=0).stop 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 shutdown_connections 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe474108700 0x7fe4741a9b40 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe4580a1290 0x7fe4580a3750 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe474100f60 0x7fe474107c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 --2- 172.21.15.23:0/4040061805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe474106080 0x7fe4741081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 >> 172.21.15.23:0/4040061805 conn(0x7fe4740fc7c0 msgr2=0x7fe4740fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 shutdown_connections 2024-03-30T04:14:42.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.576+0000 7fe479600700 1 -- 172.21.15.23:0/4040061805 wait complete. 2024-03-30T04:14:42.593 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:14:42.593 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:14:42.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:42 smithi154 ceph-mon[79161]: pgmap v399: 129 pgs: 129 active+clean; 570 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.8 MiB/s wr, 126 op/s 2024-03-30T04:14:42.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.806+0000 7fda41400700 1 -- 172.21.15.23:0/3719569630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0ff680 msgr2=0x7fda3c10dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.806+0000 7fda41400700 1 --2- 172.21.15.23:0/3719569630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0ff680 0x7fda3c10dae0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fda34009f90 tx=0x7fda3400b330 comp rx=0 tx=0).stop 2024-03-30T04:14:42.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 -- 172.21.15.23:0/3719569630 shutdown_connections 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 --2- 172.21.15.23:0/3719569630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0ff680 0x7fda3c10dae0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 --2- 172.21.15.23:0/3719569630 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c106030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 --2- 172.21.15.23:0/3719569630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0fecc0 0x7fda3c0ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 -- 172.21.15.23:0/3719569630 >> 172.21.15.23:0/3719569630 conn(0x7fda3c075c10 msgr2=0x7fda3c0fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 -- 172.21.15.23:0/3719569630 shutdown_connections 2024-03-30T04:14:42.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.807+0000 7fda41400700 1 -- 172.21.15.23:0/3719569630 wait complete. 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 Processor -- start 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 -- start start 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0fecc0 0x7fda3c1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c19e400 con 0x7fda3c105c50 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c19e570 con 0x7fda3c0ff680 2024-03-30T04:14:42.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.808+0000 7fda41400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3c19e730 con 0x7fda3c0fecc0 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda3b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38204/0 (socket says 172.21.15.23:38204) 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 -- 172.21.15.23:0/363737778 learned_addr learned my addr 172.21.15.23:0/363737778 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda3b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33258/0 (socket says 172.21.15.23:33258) 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda3a800700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0fecc0 0x7fda3c1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 -- 172.21.15.23:0/363737778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0fecc0 msgr2=0x7fda3c1994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0fecc0 0x7fda3c1994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 -- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 msgr2=0x7fda3c19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 -- 172.21.15.23:0/363737778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda34009c40 con 0x7fda3c0ff680 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda3b200700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.809+0000 7fda39e00700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e9a0 tx=0x7fda2800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.810+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28009790 con 0x7fda3c0ff680 2024-03-30T04:14:42.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.810+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda2800ce80 con 0x7fda3c0ff680 2024-03-30T04:14:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.810+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3c19ea10 con 0x7fda3c0ff680 2024-03-30T04:14:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.810+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28018680 con 0x7fda3c0ff680 2024-03-30T04:14:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.810+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3c1a6250 con 0x7fda3c0ff680 2024-03-30T04:14:42.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.811+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3c101e00 con 0x7fda3c0ff680 2024-03-30T04:14:42.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.813+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fda28010430 con 0x7fda3c0ff680 2024-03-30T04:14:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.814+0000 7fda33400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 0x7fda200a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.814+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fda28014070 con 0x7fda3c0ff680 2024-03-30T04:14:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.815+0000 7fda3a800700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 0x7fda200a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:42.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.815+0000 7fda3a800700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 0x7fda200a37d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda2c005fd0 tx=0x7fda2c009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:42.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:42.817+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fda28096410 con 0x7fda3c0ff680 2024-03-30T04:14:43.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.131+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda3c19a7e0 con 0x7fda3c0ff680 2024-03-30T04:14:43.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.132+0000 7fda33400700 1 -- 172.21.15.23:0/363737778 <== mon.2 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fda280cae60 con 0x7fda3c0ff680 2024-03-30T04:14:43.131 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:14:43.131 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:14:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 msgr2=0x7fda200a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 0x7fda200a37d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda2c005fd0 tx=0x7fda2c009310 comp rx=0 tx=0).stop 2024-03-30T04:14:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 msgr2=0x7fda3c199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e9a0 tx=0x7fda2800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 shutdown_connections 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fda3c105c50 0x7fda3c19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda3c0fecc0 0x7fda3c1994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fda200a1310 0x7fda200a37d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 --2- 172.21.15.23:0/363737778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda3c0ff680 0x7fda3c199a20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.135+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 >> 172.21.15.23:0/363737778 conn(0x7fda3c075c10 msgr2=0x7fda3c103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.136+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 shutdown_connections 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.136+0000 7fda41400700 1 -- 172.21.15.23:0/363737778 wait complete. 2024-03-30T04:14:43.135 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:14:43.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:14:43.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1232378 9.45 MB/sec execute 560 sec latency 20.360 ms 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 -- 172.21.15.23:0/1887531375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 msgr2=0x7f2e280682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1887531375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280682b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2e14009920 tx=0x7f2e14009c30 comp rx=0 tx=0).stop 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 -- 172.21.15.23:0/1887531375 shutdown_connections 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1887531375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e280687f0 0x7f2e2810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1887531375 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e28105ce0 0x7f2e281060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1887531375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280682b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 -- 172.21.15.23:0/1887531375 >> 172.21.15.23:0/1887531375 conn(0x7f2e28076d40 msgr2=0x7f2e28077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 -- 172.21.15.23:0/1887531375 shutdown_connections 2024-03-30T04:14:43.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.374+0000 7f2e2e400700 1 -- 172.21.15.23:0/1887531375 wait complete. 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.375+0000 7f2e2e400700 1 Processor -- start 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 -- start start 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e28105ce0 0x7f2e281006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281a5650 con 0x7f2e28105ce0 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281a57c0 con 0x7f2e280687f0 2024-03-30T04:14:43.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e2e400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e281a5980 con 0x7f2e28067e30 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.376+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38222/0 (socket says 172.21.15.23:38222) 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 -- 172.21.15.23:0/1846558563 learned_addr learned my addr 172.21.15.23:0/1846558563 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e27400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e27e00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e28105ce0 0x7f2e281006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 -- 172.21.15.23:0/1846558563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 msgr2=0x7f2e280ffc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280ffc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 -- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e28105ce0 msgr2=0x7f2e281006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e28105ce0 0x7f2e281006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 -- 172.21.15.23:0/1846558563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14009520 con 0x7f2e280687f0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e26a00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400fa00 tx=0x7f2e1400bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e1fe00700 1 -- 172.21.15.23:0/1846558563 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1401c070 con 0x7f2e280687f0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.377+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e281a5c00 con 0x7f2e280687f0 2024-03-30T04:14:43.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.378+0000 7f2e1fe00700 1 -- 172.21.15.23:0/1846558563 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e1400bb20 con 0x7f2e280687f0 2024-03-30T04:14:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.378+0000 7f2e1fe00700 1 -- 172.21.15.23:0/1846558563 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14004930 con 0x7f2e280687f0 2024-03-30T04:14:43.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.378+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e281a6080 con 0x7f2e280687f0 2024-03-30T04:14:43.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.381+0000 7f2e1fe00700 1 -- 172.21.15.23:0/1846558563 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2e14020070 con 0x7f2e280687f0 2024-03-30T04:14:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.382+0000 7f2e4201db80 1 Processor -- start 2024-03-30T04:14:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.382+0000 7f2e1fe00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 0x7f2e0c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.382+0000 7f2e4201db80 1 -- start start 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.382+0000 7f2e1fe00700 1 -- 172.21.15.23:0/1846558563 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2e140cdd10 con 0x7f2e280687f0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e27e00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 0x7f2e0c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 0x55d04b6319b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 0x55d04b63c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b63f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b58a990 con 0x55d04b63cdf0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b588350 con 0x55d04b6315d0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e4201db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b583c40 con 0x55d04b631ef0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 0x55d04b63c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e27400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 0x55d04b6319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 0x55d04b63c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43782/0 (socket says 172.21.15.23:43782) 2024-03-30T04:14:43.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e26a00700 1 -- 172.21.15.23:0/3372863715 learned_addr learned my addr 172.21.15.23:0/3372863715 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.383+0000 7f2e27e00700 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b63f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 -- 172.21.15.23:0/3372863715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 msgr2=0x55d04b63c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 0x55d04b63c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 -- 172.21.15.23:0/3372863715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 msgr2=0x55d04b63f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b63f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 -- 172.21.15.23:0/3372863715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d04b4053b0 con 0x55d04b6315d0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27e00700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 0x7f2e0c0a37d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e200093f0 tx=0x7f2e20009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e27400700 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 0x55d04b6319b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009cc0 tx=0x7f2e18011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7348ea4da2fb2166 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e1cc00700 1 -- 172.21.15.23:0/3372863715 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18011cf0 con 0x55d04b6315d0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e1cc00700 1 -- 172.21.15.23:0/3372863715 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e18011e50 con 0x55d04b6315d0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.384+0000 7f2e1cc00700 1 -- 172.21.15.23:0/3372863715 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18005280 con 0x55d04b6315d0 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 -- 172.21.15.23:0/3372863715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 msgr2=0x55d04b6319b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 0x55d04b6319b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2e18009cc0 tx=0x7f2e18011440 comp rx=0 tx=0).stop 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 -- 172.21.15.23:0/3372863715 shutdown_connections 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b63f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b6315d0 0x55d04b6319b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 --2- 172.21.15.23:0/3372863715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b631ef0 0x55d04b63c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 -- 172.21.15.23:0/3372863715 >> 172.21.15.23:0/3372863715 conn(0x55d04b5916a0 msgr2=0x55d04b62da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 -- 172.21.15.23:0/3372863715 shutdown_connections 2024-03-30T04:14:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.385+0000 7f2e4201db80 1 -- 172.21.15.23:0/3372863715 wait complete. 2024-03-30T04:14:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.386+0000 7f2e4201db80 1 -- start start 2024-03-30T04:14:43.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.386+0000 7f2e4201db80 1 Processor -- start 2024-03-30T04:14:43.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b63c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b639b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b51fd80 con 0x55d04b631ef0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b51fa20 con 0x55d04b63cdf0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b4076a0 con 0x55d04b6315d0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e4201db80 10 client.0 _ll_register_callbacks cb 0x55d04b529470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b639b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b63c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38228/0 (socket says 172.21.15.23:38228) 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e27e00700 1 -- 172.21.15.23:0/2298001723 learned_addr learned my addr 172.21.15.23:0/2298001723 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 -- 172.21.15.23:0/2298001723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 msgr2=0x55d04b63c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b63c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 -- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 msgr2=0x55d04b63a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e26a00700 1 -- 172.21.15.23:0/2298001723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18009920 con 0x55d04b631ef0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.387+0000 7f2e27e00700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e26a00700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b639b20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e140974e0 tx=0x7f2e1409b170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75dc758de20a1f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1409aed0 con 0x55d04b631ef0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2e1409e8a0 con 0x55d04b631ef0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d04b544380 con 0x55d04b631ef0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e140a2e60 con 0x55d04b631ef0 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e4201db80 10 client.18933 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d04b3b1b60 con 0x55d04b631ef0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e140b7030 con 0x55d04b631ef0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.388+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2e140a29d0 con 0x55d04b631ef0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 10 client.18933 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 10 client.18933 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 20 client.18933 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b64c6a0 0x55d04b64ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 4 client.18933 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d04b57c700 con 0x55d04b64c6a0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.389+0000 7f2e27400700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b64c6a0 0x55d04b64ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.390+0000 7f2e27400700 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b64c6a0 0x55d04b64ea90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fa689239e511c5c2 server_cookie=27bfa26c0d82e34a in_seq=0 out_seq=0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.390+0000 7f2e11600700 10 client.18933 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.390+0000 7f2e11600700 1 -- 172.21.15.23:0/2298001723 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d04b57c700 con 0x55d04b64c6a0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.390+0000 7f2e11600700 10 client.18933 handle_command_reply: tid=0 2024-03-30T04:14:43.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.390+0000 7f2e4201db80 1 client.18933 shutdown 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 msgr2=0x55d04b639b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b639b20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e140974e0 tx=0x7f2e1409b170 comp rx=0 tx=0).stop 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 shutdown_connections 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b63a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b64c6a0 0x55d04b64ea90 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b63c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2298001723 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b639b20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e11600700 0 client.18933 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 >> 172.21.15.23:0/2298001723 conn(0x55d04b5916a0 msgr2=0x55d04b6336c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 shutdown_connections 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 1 -- 172.21.15.23:0/2298001723 wait complete. 2024-03-30T04:14:43.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.391+0000 7f2e4201db80 20 client.18933 trim_cache size 0 max 16384 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 Processor -- start 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 -- start start 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b6265a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b634df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b632850 con 0x55d04b631ef0 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b62f160 con 0x55d04b63cdf0 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.408+0000 7f2e4201db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b62e660 con 0x55d04b6315d0 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b6265a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43798/0 (socket says 172.21.15.23:43798) 2024-03-30T04:14:43.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 -- 172.21.15.23:0/2584373220 learned_addr learned my addr 172.21.15.23:0/2584373220 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27e00700 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b634df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 -- 172.21.15.23:0/2584373220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 msgr2=0x55d04b634df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b634df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 -- 172.21.15.23:0/2584373220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 msgr2=0x55d04b6265a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b6265a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 -- 172.21.15.23:0/2584373220 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d04b549ea0 con 0x55d04b6315d0 2024-03-30T04:14:43.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e27400700 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800a010 tx=0x7f2e180096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d9d82488953e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e1d600700 1 -- 172.21.15.23:0/2584373220 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18005da0 con 0x55d04b6315d0 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e1d600700 1 -- 172.21.15.23:0/2584373220 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e180048f0 con 0x55d04b6315d0 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.409+0000 7f2e1d600700 1 -- 172.21.15.23:0/2584373220 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e18003950 con 0x55d04b6315d0 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 -- 172.21.15.23:0/2584373220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 msgr2=0x55d04b643aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800a010 tx=0x7f2e180096d0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 -- 172.21.15.23:0/2584373220 shutdown_connections 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b63cdf0 0x55d04b634df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b643aa0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 --2- 172.21.15.23:0/2584373220 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b631ef0 0x55d04b6265a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 -- 172.21.15.23:0/2584373220 >> 172.21.15.23:0/2584373220 conn(0x55d04b5916a0 msgr2=0x55d04b6533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 -- 172.21.15.23:0/2584373220 shutdown_connections 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.410+0000 7f2e4201db80 1 -- 172.21.15.23:0/2584373220 wait complete. 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 -- start start 2024-03-30T04:14:43.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 Processor -- start 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b641790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 0x55d04b641cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b6994b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b6999f0 con 0x55d04b63cdf0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b699b60 con 0x55d04b631ef0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.411+0000 7f2e4201db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d04b699d20 con 0x55d04b6315d0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e4201db80 10 client.0 _ll_register_callbacks cb 0x55d04b529470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b641790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 0x55d04b641cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b641790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43808/0 (socket says 172.21.15.23:43808) 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e27400700 1 -- 172.21.15.23:0/555461575 learned_addr learned my addr 172.21.15.23:0/555461575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e27e00700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b6994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 -- 172.21.15.23:0/555461575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 msgr2=0x55d04b641790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b641790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 -- 172.21.15.23:0/555461575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 msgr2=0x55d04b6994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b6994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 -- 172.21.15.23:0/555461575 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1801d040 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e26a00700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 0x55d04b641cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2e14098dc0 tx=0x7f2e1402a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4afbcdd08746500 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.412+0000 7f2e11600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1409b1e0 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e18011e50 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2e1409e8a0 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e140a2e60 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 10 client.28913 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55d04b409cd0 con 0x55d04b631ef0 2024-03-30T04:14:43.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e14082030 con 0x55d04b631ef0 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2e1409acd0 con 0x55d04b631ef0 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 10 client.28913 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 10 client.28913 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 20 client.28913 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b6a1040 0x55d04b6a3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 4 client.28913 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.413+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d04b3d2cf0 con 0x55d04b6a1040 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e27400700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b6a1040 0x55d04b6a3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e27400700 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b6a1040 0x55d04b6a3460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9ed267cf13f39ef0 server_cookie=cc85add7869ce912 in_seq=0 out_seq=0 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e11600700 10 client.28913 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e11600700 1 -- 172.21.15.23:0/555461575 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d04b3d2cf0 con 0x55d04b6a1040 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e11600700 10 client.28913 handle_command_reply: tid=0 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.414+0000 7f2e4201db80 1 client.28913 shutdown 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 msgr2=0x55d04b641cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 0x55d04b641cd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2e14098dc0 tx=0x7f2e1402a000 comp rx=0 tx=0).stop 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 shutdown_connections 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d04b63cdf0 0x55d04b6994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55d04b6a1040 0x55d04b6a3460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d04b6315d0 0x55d04b641790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 --2- 172.21.15.23:0/555461575 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55d04b631ef0 0x55d04b641cd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e11600700 0 client.28913 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 >> 172.21.15.23:0/555461575 conn(0x55d04b5916a0 msgr2=0x55d04b6265a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 shutdown_connections 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 1 -- 172.21.15.23:0/555461575 wait complete. 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.415+0000 7f2e4201db80 20 client.28913 trim_cache size 0 max 16384 2024-03-30T04:14:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 msgr2=0x7f2e0c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 0x7f2e0c0a37d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2e200093f0 tx=0x7f2e20009920 comp rx=0 tx=0).stop 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 msgr2=0x7f2e281001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400fa00 tx=0x7f2e1400bef0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 shutdown_connections 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2e28105ce0 0x7f2e281006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e28067e30 0x7f2e280ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2e0c0a1310 0x7f2e0c0a37d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 --2- 172.21.15.23:0/1846558563 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e280687f0 0x7f2e281001b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 >> 172.21.15.23:0/1846558563 conn(0x7f2e28076d40 msgr2=0x7f2e280fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 shutdown_connections 2024-03-30T04:14:43.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.416+0000 7f2e2e400700 1 -- 172.21.15.23:0/1846558563 wait complete. 2024-03-30T04:14:43.431 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:43 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/363737778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:43.433 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:14:43.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:43 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/363737778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 -- 172.21.15.23:0/1960264770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c101910 msgr2=0x7f5e3c105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 --2- 172.21.15.23:0/1960264770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c101910 0x7f5e3c105950 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009990 tx=0x7f5e28009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 -- 172.21.15.23:0/1960264770 shutdown_connections 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 --2- 172.21.15.23:0/1960264770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 0x7f5e3c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 --2- 172.21.15.23:0/1960264770 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c100f60 0x7f5e3c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 --2- 172.21.15.23:0/1960264770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c101910 0x7f5e3c105950 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 -- 172.21.15.23:0/1960264770 >> 172.21.15.23:0/1960264770 conn(0x7f5e3c0fc7c0 msgr2=0x7f5e3c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 -- 172.21.15.23:0/1960264770 shutdown_connections 2024-03-30T04:14:43.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.641+0000 7f5e43e00700 1 -- 172.21.15.23:0/1960264770 wait complete. 2024-03-30T04:14:43.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.642+0000 7f5e43e00700 1 Processor -- start 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.642+0000 7f5e43e00700 1 -- start start 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 0x7f5e3c199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c19dcc0 0x7f5e3c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c19e740 con 0x7f5e3c19dcc0 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c19e8b0 con 0x7f5e3c100f60 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e43e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c19ea70 con 0x7f5e3c106080 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e40a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 0x7f5e3c199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38268/0 (socket says 172.21.15.23:38268) 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e41400700 1 -- 172.21.15.23:0/3317750992 learned_addr learned my addr 172.21.15.23:0/3317750992 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.643+0000 7f5e41e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c19dcc0 0x7f5e3c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 -- 172.21.15.23:0/3317750992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 msgr2=0x7f5e3c199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 0x7f5e3c199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 -- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c19dcc0 msgr2=0x7f5e3c19e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c19dcc0 0x7f5e3c19e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 -- 172.21.15.23:0/3317750992 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280095f0 con 0x7f5e3c100f60 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e41400700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000e9a0 tx=0x7f5e3000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e3a000700 1 -- 172.21.15.23:0/3317750992 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30009790 con 0x7f5e3c100f60 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e3a000700 1 -- 172.21.15.23:0/3317750992 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e30010430 con 0x7f5e3c100f60 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1a5fd0 con 0x7f5e3c100f60 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e3a000700 1 -- 172.21.15.23:0/3317750992 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30018680 con 0x7f5e3c100f60 2024-03-30T04:14:43.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.644+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1a64a0 con 0x7f5e3c100f60 2024-03-30T04:14:43.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.647+0000 7f5e3a000700 1 -- 172.21.15.23:0/3317750992 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f5e300105a0 con 0x7f5e3c100f60 2024-03-30T04:14:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.649+0000 7f5e578e0b80 1 Processor -- start 2024-03-30T04:14:43.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e3a000700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 0x7f5e240a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 -- start start 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e3a000700 1 -- 172.21.15.23:0/3317750992 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e30014070 con 0x7f5e3c100f60 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e41e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 0x7f5e240a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da864321e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8637d990 con 0x55da8642fdf0 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8637b350 con 0x55da864245d0 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86376c40 con 0x55da86424ef0 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.650+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38278/0 (socket says 172.21.15.23:38278) 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41400700 1 -- 172.21.15.23:0/2392397056 learned_addr learned my addr 172.21.15.23:0/2392397056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41e00700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da864321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 -- 172.21.15.23:0/2392397056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 msgr2=0x55da864249b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 -- 172.21.15.23:0/2392397056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 msgr2=0x55da864321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da864321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 -- 172.21.15.23:0/2392397056 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da861f83b0 con 0x55da86424ef0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41e00700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da864321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41400700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e41e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 0x7f5e240a37d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400f590 tx=0x7f5e34009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642f8b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5e280098e0 tx=0x7f5e28005ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ceec1e5ea3e13ff server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e2e600700 1 -- 172.21.15.23:0/2392397056 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2801d070 con 0x55da86424ef0 2024-03-30T04:14:43.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e2e600700 1 -- 172.21.15.23:0/2392397056 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e2801e040 con 0x55da86424ef0 2024-03-30T04:14:43.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.651+0000 7f5e2e600700 1 -- 172.21.15.23:0/2392397056 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e280047b0 con 0x55da86424ef0 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2392397056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 msgr2=0x55da8642f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642f8b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f5e280098e0 tx=0x7f5e28005ec0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2392397056 shutdown_connections 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da864321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da864249b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2392397056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642f8b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2392397056 >> 172.21.15.23:0/2392397056 conn(0x55da863846a0 msgr2=0x55da86420a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2392397056 shutdown_connections 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.652+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2392397056 wait complete. 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.653+0000 7f5e578e0b80 1 -- start start 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.653+0000 7f5e578e0b80 1 Processor -- start 2024-03-30T04:14:43.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8642d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86312d80 con 0x55da8642fdf0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86312a20 con 0x55da864245d0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da861fa6a0 con 0x55da86424ef0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e578e0b80 10 client.0 _ll_register_callbacks cb 0x55da8631c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38284/0 (socket says 172.21.15.23:38284) 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 -- 172.21.15.23:0/3878609541 learned_addr learned my addr 172.21.15.23:0/3878609541 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41e00700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8642d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 -- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 msgr2=0x55da8642ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 -- 172.21.15.23:0/3878609541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 msgr2=0x55da8642d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8642d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.654+0000 7f5e41400700 1 -- 172.21.15.23:0/3878609541 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1a64a0 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e41400700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000c510 tx=0x7f5e3009a720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11b65a4d6eec16b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e22a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3009dd60 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e300a23b0 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da86337380 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e300b8170 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e578e0b80 10 client.28925 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f5e3c10e470 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.655+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55da86337380 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e300a2020 con 0x55da864245d0 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 10 client.28925 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 10 client.28925 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 20 client.28925 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da8643f920 0x55da86441d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 4 client.28925 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da8636f700 con 0x55da8643f920 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.656+0000 7f5e40a00700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da8643f920 0x55da86441d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.657+0000 7f5e40a00700 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da8643f920 0x55da86441d40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f7a3b44b5a98907a server_cookie=39bb0b5666df4905 in_seq=0 out_seq=0 2024-03-30T04:14:43.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.657+0000 7f5e22a00700 10 client.28925 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.657+0000 7f5e22a00700 1 -- 172.21.15.23:0/3878609541 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55da8636f700 con 0x55da8643f920 2024-03-30T04:14:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.657+0000 7f5e22a00700 10 client.28925 handle_command_reply: tid=0 2024-03-30T04:14:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 client.28925 shutdown 2024-03-30T04:14:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 msgr2=0x55da8642f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000c510 tx=0x7f5e3009a720 comp rx=0 tx=0).stop 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 shutdown_connections 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da8643f920 0x55da86441d40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8642d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da8642f7d0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e22a00700 0 client.28925 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/3878609541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da8642ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 >> 172.21.15.23:0/3878609541 conn(0x55da863846a0 msgr2=0x55da864265d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 shutdown_connections 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 1 -- 172.21.15.23:0/3878609541 wait complete. 2024-03-30T04:14:43.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.658+0000 7f5e578e0b80 20 client.28925 trim_cache size 0 max 16384 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 Processor -- start 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 -- start start 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da86441d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da864274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8641ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86425850 con 0x55da8642fdf0 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da8642d2f0 con 0x55da864245d0 2024-03-30T04:14:43.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86432430 con 0x55da86424ef0 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da86441d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e41400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da86441d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38296/0 (socket says 172.21.15.23:38296) 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e41400700 1 -- 172.21.15.23:0/2829439477 learned_addr learned my addr 172.21.15.23:0/2829439477 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da864274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.672+0000 7f5e41e00700 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8641ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 -- 172.21.15.23:0/2829439477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 msgr2=0x55da86441d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da86441d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 -- 172.21.15.23:0/2829439477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 msgr2=0x55da8641ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8641ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 -- 172.21.15.23:0/2829439477 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da861a4b60 con 0x55da86424ef0 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e40a00700 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da864274c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009590 tx=0x7f5e280058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b90eba9407d506e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e2f000700 1 -- 172.21.15.23:0/2829439477 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2801d040 con 0x55da86424ef0 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e2f000700 1 -- 172.21.15.23:0/2829439477 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e280056a0 con 0x55da86424ef0 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.673+0000 7f5e2f000700 1 -- 172.21.15.23:0/2829439477 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e28004b70 con 0x55da86424ef0 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2829439477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 msgr2=0x55da864274c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da864274c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009590 tx=0x7f5e280058e0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2829439477 shutdown_connections 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da8642fdf0 0x55da8641ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da864245d0 0x55da86441d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/2829439477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da86424ef0 0x55da864274c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2829439477 >> 172.21.15.23:0/2829439477 conn(0x55da863846a0 msgr2=0x55da864195a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2829439477 shutdown_connections 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- 172.21.15.23:0/2829439477 wait complete. 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.674+0000 7f5e578e0b80 1 -- start start 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 Processor -- start 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 0x55da864349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da8642fdf0 0x55da86434f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da864928d0 con 0x55da864245d0 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86492a40 con 0x55da86424ef0 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da86492c00 con 0x55da8642fdf0 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.675+0000 7f5e578e0b80 10 client.0 _ll_register_callbacks cb 0x55da8631c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e40a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 0x55da864349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da8642fdf0 0x55da86434f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33346/0 (socket says 172.21.15.23:33346) 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e40a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 0x55da864349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38302/0 (socket says 172.21.15.23:38302) 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 -- 172.21.15.23:0/249751004 learned_addr learned my addr 172.21.15.23:0/249751004 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 -- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da8642fdf0 msgr2=0x55da86434f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da8642fdf0 0x55da86434f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 -- 172.21.15.23:0/249751004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 msgr2=0x55da864349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 0x55da864349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 -- 172.21.15.23:0/249751004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da861f83b0 con 0x55da864245d0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e41400700 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5e300969a0 tx=0x7f5e300999c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15601fcf5e10b80f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.676+0000 7f5e22a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e300960b0 con 0x55da864245d0 2024-03-30T04:14:43.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5e300a2130 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e2801e040 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e30095ec0 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 10 client.18954 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55da8633cea0 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e2801e040 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5e300994b0 con 0x55da864245d0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 10 client.18954 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 10 client.18954 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 20 client.18954 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da86493ff0 0x55da86496410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 4 client.18954 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.677+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da861c5cf0 con 0x55da86493ff0 2024-03-30T04:14:43.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e40a00700 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da86493ff0 0x55da86496410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e40a00700 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da86493ff0 0x55da86496410 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=917a9f2acce76433 server_cookie=ce98994d4e925d4f in_seq=0 out_seq=0 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e22a00700 10 client.18954 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e22a00700 1 -- 172.21.15.23:0/249751004 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da861c5cf0 con 0x55da86493ff0 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e22a00700 10 client.18954 handle_command_reply: tid=0 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 client.18954 shutdown 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 msgr2=0x55da864344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f5e300969a0 tx=0x7f5e300999c0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 shutdown_connections 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x55da86493ff0 0x55da86496410 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55da8642fdf0 0x55da86434f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.678+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55da864245d0 0x55da864344b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e22a00700 0 client.18954 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:14:43.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e578e0b80 1 --2- 172.21.15.23:0/249751004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55da86424ef0 0x55da864349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 >> 172.21.15.23:0/249751004 conn(0x55da863846a0 msgr2=0x55da8642b260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 shutdown_connections 2024-03-30T04:14:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e578e0b80 1 -- 172.21.15.23:0/249751004 wait complete. 2024-03-30T04:14:43.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e578e0b80 20 client.18954 trim_cache size 0 max 16384 2024-03-30T04:14:43.682 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 msgr2=0x7f5e240a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 0x7f5e240a37d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400f590 tx=0x7f5e34009920 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 msgr2=0x7f5e3c1993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.679+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000e9a0 tx=0x7f5e3000ecb0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 shutdown_connections 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5e3c19dcc0 0x7f5e3c19e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5e240a1310 0x7f5e240a37d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5e3c100f60 0x7f5e3c1993e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 --2- 172.21.15.23:0/3317750992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e3c106080 0x7f5e3c199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 >> 172.21.15.23:0/3317750992 conn(0x7f5e3c0fc7c0 msgr2=0x7f5e3c0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 shutdown_connections 2024-03-30T04:14:43.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.680+0000 7f5e43e00700 1 -- 172.21.15.23:0/3317750992 wait complete. 2024-03-30T04:14:43.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:43 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/363737778' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:14:43.696 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 -- 172.21.15.23:0/1843507092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 msgr2=0x7f2460069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 --2- 172.21.15.23:0/1843507092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460069640 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f2454009920 tx=0x7f2454009c30 comp rx=0 tx=0).stop 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 -- 172.21.15.23:0/1843507092 shutdown_connections 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 --2- 172.21.15.23:0/1843507092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f246010dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 --2- 172.21.15.23:0/1843507092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460069640 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 --2- 172.21.15.23:0/1843507092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2460069c10 0x7f2460105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 -- 172.21.15.23:0/1843507092 >> 172.21.15.23:0/1843507092 conn(0x7f2460076d80 msgr2=0x7f2460077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 -- 172.21.15.23:0/1843507092 shutdown_connections 2024-03-30T04:14:43.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.915+0000 7f2467400700 1 -- 172.21.15.23:0/1843507092 wait complete. 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.916+0000 7f2467400700 1 Processor -- start 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.916+0000 7f2467400700 1 -- start start 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f246019bb90 0x7f246019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246019c610 con 0x7f246019bb90 2024-03-30T04:14:43.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246019c780 con 0x7f24601060b0 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2467400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246019c940 con 0x7f2460069260 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f245fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f245fe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38324/0 (socket says 172.21.15.23:38324) 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2465400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f246019bb90 0x7f246019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f245fe00700 1 -- 172.21.15.23:0/487284041 learned_addr learned my addr 172.21.15.23:0/487284041 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.917+0000 7f2464a00700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 -- 172.21.15.23:0/487284041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 msgr2=0x7f2460197300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460197300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 -- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f246019bb90 msgr2=0x7f246019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f246019bb90 0x7f246019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 -- 172.21.15.23:0/487284041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454009520 con 0x7f24601060b0 2024-03-30T04:14:43.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245fe00700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9f0 tx=0x7f244c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245d600700 1 -- 172.21.15.23:0/487284041 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c009690 con 0x7f24601060b0 2024-03-30T04:14:43.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245d600700 1 -- 172.21.15.23:0/487284041 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244c00ce80 con 0x7f24601060b0 2024-03-30T04:14:43.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24601a3ea0 con 0x7f24601060b0 2024-03-30T04:14:43.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f245d600700 1 -- 172.21.15.23:0/487284041 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c018420 con 0x7f24601060b0 2024-03-30T04:14:43.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.918+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24601a42d0 con 0x7f24601060b0 2024-03-30T04:14:43.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.923+0000 7f245d600700 1 -- 172.21.15.23:0/487284041 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f244c00c9c0 con 0x7f24601060b0 2024-03-30T04:14:43.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.924+0000 7f247ae57b80 1 Processor -- start 2024-03-30T04:14:43.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.924+0000 7f247ae57b80 1 -- start start 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.924+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b37999fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.924+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799ad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f247ae57b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3798f4990 con 0x55b37999fa40 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f247ae57b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3798f2350 con 0x55b3799ab210 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f247ae57b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3798edc40 con 0x55b3799a0360 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f245d600700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 0x7f24480a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2465400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b37999fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2465400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b37999fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33364/0 (socket says 172.21.15.23:33364) 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2465400700 1 -- 172.21.15.23:0/3176471070 learned_addr learned my addr 172.21.15.23:0/3176471070 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f245fe00700 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f245d600700 1 -- 172.21.15.23:0/487284041 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f244c014070 con 0x7f24601060b0 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2465400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 0x7f24480a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 -- 172.21.15.23:0/3176471070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 msgr2=0x55b3799ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 -- 172.21.15.23:0/3176471070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 msgr2=0x55b37999fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b37999fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.925+0000 7f2464a00700 1 -- 172.21.15.23:0/3176471070 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b37976f3b0 con 0x55b3799a0360 2024-03-30T04:14:43.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.926+0000 7f2465400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 0x7f24480a3750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2458003fd0 tx=0x7f2458004b40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.926+0000 7f2464a00700 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799aacd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2454005e90 tx=0x7f2454005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a037eb53ed98722b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.926+0000 7f2451c00700 1 -- 172.21.15.23:0/3176471070 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245400fe50 con 0x55b3799a0360 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.926+0000 7f2451c00700 1 -- 172.21.15.23:0/3176471070 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f245401d040 con 0x55b3799a0360 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.926+0000 7f2451c00700 1 -- 172.21.15.23:0/3176471070 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454005040 con 0x55b3799a0360 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 -- 172.21.15.23:0/3176471070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 msgr2=0x55b3799aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799aacd0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f2454005e90 tx=0x7f2454005dc0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 -- 172.21.15.23:0/3176471070 shutdown_connections 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b37999fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 --2- 172.21.15.23:0/3176471070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799aacd0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 -- 172.21.15.23:0/3176471070 >> 172.21.15.23:0/3176471070 conn(0x55b3798fb6a0 msgr2=0x55b379997a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 -- 172.21.15.23:0/3176471070 shutdown_connections 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.927+0000 7f247ae57b80 1 -- 172.21.15.23:0/3176471070 wait complete. 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.928+0000 7f247ae57b80 1 -- start start 2024-03-30T04:14:43.927 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.928+0000 7f247ae57b80 1 Processor -- start 2024-03-30T04:14:43.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b379889d80 con 0x55b37999fa40 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b379889a20 con 0x55b3799ab210 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3797716a0 con 0x55b3799a0360 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f247ae57b80 10 client.0 _ll_register_callbacks cb 0x55b379893470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f2464a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f2464a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799aabb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33380/0 (socket says 172.21.15.23:33380) 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f2465400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f2464a00700 1 -- 172.21.15.23:0/2626991149 learned_addr learned my addr 172.21.15.23:0/2626991149 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.929+0000 7f245fe00700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 -- 172.21.15.23:0/2626991149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 msgr2=0x55b3799a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 -- 172.21.15.23:0/2626991149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 msgr2=0x55b3799aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 -- 172.21.15.23:0/2626991149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246010e470 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2465400700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799a84d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2458004a60 tx=0x7f2458004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29d29383585b5b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2446000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458007de0 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2458025400 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.930+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b3798ae380 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24580255c0 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 10 client.28943 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b37971bb60 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b3798ae380 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2458019780 con 0x55b3799ab210 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 10 client.28943 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 10 client.28943 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 20 client.28943 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b3799bae50 0x55b3799bd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 4 client.28943 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b3798e6700 con 0x55b3799bae50 2024-03-30T04:14:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.931+0000 7f2464a00700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b3799bae50 0x55b3799bd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.932+0000 7f2464a00700 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b3799bae50 0x55b3799bd270 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85cb56f93a4ae3c2 server_cookie=431abc6b1426835c in_seq=0 out_seq=0 2024-03-30T04:14:43.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.932+0000 7f2446000700 10 client.28943 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f2446000700 1 -- 172.21.15.23:0/2626991149 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b3798e6700 con 0x55b3799bae50 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f2446000700 10 client.28943 handle_command_reply: tid=0 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 client.28943 shutdown 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 msgr2=0x55b3799a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799a84d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2458004a60 tx=0x7f2458004ad0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 shutdown_connections 2024-03-30T04:14:43.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b3799bae50 0x55b3799bd270 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799a84d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2626991149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f2446000700 0 client.28943 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.934+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 >> 172.21.15.23:0/2626991149 conn(0x55b3798fb6a0 msgr2=0x55b3799a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.935+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 shutdown_connections 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.935+0000 7f247ae57b80 1 -- 172.21.15.23:0/2626991149 wait complete. 2024-03-30T04:14:43.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.935+0000 7f247ae57b80 20 client.28943 trim_cache size 0 max 16384 2024-03-30T04:14:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 Processor -- start 2024-03-30T04:14:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 -- start start 2024-03-30T04:14:43.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 0x55b3799a3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b379997f80 con 0x55b3799ab210 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3799b23c0 con 0x55b37999fa40 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.949+0000 7f247ae57b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b3799a6460 con 0x55b3799a0360 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f245fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38360/0 (socket says 172.21.15.23:38360) 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 -- 172.21.15.23:0/2693821292 learned_addr learned my addr 172.21.15.23:0/2693821292 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2465400700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 0x55b3799a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 -- 172.21.15.23:0/2693821292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 msgr2=0x55b3799a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 -- 172.21.15.23:0/2693821292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 msgr2=0x55b3799a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 0x55b3799a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 -- 172.21.15.23:0/2693821292 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b3798b3ea0 con 0x55b37999fa40 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f245fe00700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2465400700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 0x55b3799a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.950+0000 7f2464a00700 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2454005b00 tx=0x7f245402d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34259f81538018b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.951+0000 7f2452600700 1 -- 172.21.15.23:0/2693821292 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540058e0 con 0x55b37999fa40 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.951+0000 7f2452600700 1 -- 172.21.15.23:0/2693821292 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2454005570 con 0x55b37999fa40 2024-03-30T04:14:43.951 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.951+0000 7f2452600700 1 -- 172.21.15.23:0/2693821292 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245400b980 con 0x55b37999fa40 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.951+0000 7f247ae57b80 1 -- 172.21.15.23:0/2693821292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 msgr2=0x55b3799b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.951+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2454005b00 tx=0x7f245402d040 comp rx=0 tx=0).stop 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 -- 172.21.15.23:0/2693821292 shutdown_connections 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b3799ab210 0x55b3799a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b37999fa40 0x55b3799b1e80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 --2- 172.21.15.23:0/2693821292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 -- 172.21.15.23:0/2693821292 >> 172.21.15.23:0/2693821292 conn(0x55b3798fb6a0 msgr2=0x55b3799aa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 -- 172.21.15.23:0/2693821292 shutdown_connections 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 -- 172.21.15.23:0/2693821292 wait complete. 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.952+0000 7f247ae57b80 1 -- start start 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 Processor -- start 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b37999c190 con 0x55b37999fa40 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b37999c2d0 con 0x55b3799ab210 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f247ae57b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b37999c490 con 0x55b3799a0360 2024-03-30T04:14:43.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.953+0000 7f2464a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43900/0 (socket says 172.21.15.23:43900) 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 -- 172.21.15.23:0/1781148423 learned_addr learned my addr 172.21.15.23:0/1781148423 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f2464a00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33394/0 (socket says 172.21.15.23:33394) 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f2465400700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f247ae57b80 10 client.0 _ll_register_callbacks cb 0x55b379893470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 -- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 msgr2=0x55b3799b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 -- 172.21.15.23:0/1781148423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 msgr2=0x55b3799afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 -- 172.21.15.23:0/1781148423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c069e70 con 0x55b3799a0360 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f245fe00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ca570 tx=0x7f244c069e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6eae181114cfb1b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:43.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.954+0000 7f2446000700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c09dcb0 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f244c0a2300 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f245401d040 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f244c0106a0 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f247ae57b80 10 client.28927 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b37976f3b0 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f245401d040 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.955+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f244c0a1e70 con 0x55b3799a0360 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 10 client.28927 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 10 client.28927 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 20 client.28927 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b379a13e20 0x55b379a16240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 4 client.28927 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b37973ccf0 con 0x55b379a13e20 2024-03-30T04:14:43.957 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f2464a00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b379a13e20 0x55b379a16240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f2464a00700 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b379a13e20 0x55b379a16240 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d5b3c4b6c3a3ed98 server_cookie=5ce24c8b2696e8b3 in_seq=0 out_seq=0 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.956+0000 7f2446000700 10 client.28927 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.958+0000 7f2446000700 1 -- 172.21.15.23:0/1781148423 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b37973ccf0 con 0x55b379a13e20 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.958+0000 7f2446000700 10 client.28927 handle_command_reply: tid=0 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.958+0000 7f247ae57b80 1 client.28927 shutdown 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.958+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 msgr2=0x55b3799b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.958+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f244c0ca570 tx=0x7f244c069e00 comp rx=0 tx=0).stop 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 shutdown_connections 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55b379a13e20 0x55b379a16240 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.959 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55b3799ab210 0x55b3799b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b37999fa40 0x55b3799afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 --2- 172.21.15.23:0/1781148423 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3799a0360 0x55b3799b02f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 >> 172.21.15.23:0/1781148423 conn(0x55b3798fb6a0 msgr2=0x55b3799a7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f2446000700 0 client.28927 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 shutdown_connections 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 1 -- 172.21.15.23:0/1781148423 wait complete. 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f247ae57b80 20 client.28927 trim_cache size 0 max 16384 2024-03-30T04:14:43.960 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.959+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 msgr2=0x7f24480a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 0x7f24480a3750 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2458003fd0 tx=0x7f2458004b40 comp rx=0 tx=0).stop 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 msgr2=0x7f2460197840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f244c00e9f0 tx=0x7f244c00ed00 comp rx=0 tx=0).stop 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 shutdown_connections 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f246019bb90 0x7f246019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2460069260 0x7f2460197300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f24480a1290 0x7f24480a3750 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 --2- 172.21.15.23:0/487284041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24601060b0 0x7f2460197840 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 >> 172.21.15.23:0/487284041 conn(0x7f2460076d80 msgr2=0x7f24600fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 shutdown_connections 2024-03-30T04:14:43.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:43.960+0000 7f2467400700 1 -- 172.21.15.23:0/487284041 wait complete. 2024-03-30T04:14:43.978 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 -- 172.21.15.23:0/2563003527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c069bf0 msgr2=0x7fdd6c105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 --2- 172.21.15.23:0/2563003527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c069bf0 0x7fdd6c105980 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdd580099c0 tx=0x7fdd58009cd0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 -- 172.21.15.23:0/2563003527 shutdown_connections 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 --2- 172.21.15.23:0/2563003527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c1060b0 0x7fdd6c10dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 --2- 172.21.15.23:0/2563003527 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c069240 0x7fdd6c069620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 --2- 172.21.15.23:0/2563003527 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c069bf0 0x7fdd6c105980 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 -- 172.21.15.23:0/2563003527 >> 172.21.15.23:0/2563003527 conn(0x7fdd6c076d80 msgr2=0x7fdd6c077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 -- 172.21.15.23:0/2563003527 shutdown_connections 2024-03-30T04:14:44.196 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.197+0000 7fdd73c00700 1 -- 172.21.15.23:0/2563003527 wait complete. 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.198+0000 7fdd73c00700 1 Processor -- start 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.198+0000 7fdd73c00700 1 -- start start 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1060b0 0x7fdd6c1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c1006f0 0x7fdd6c1a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c1a59a0 con 0x7fdd6c1060b0 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c1a5b10 con 0x7fdd6c069240 2024-03-30T04:14:44.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd73c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c1a5cd0 con 0x7fdd6c1006f0 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38386/0 (socket says 172.21.15.23:38386) 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 -- 172.21.15.23:0/3305032269 learned_addr learned my addr 172.21.15.23:0/3305032269 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd6be00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1060b0 0x7fdd6c1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c1006f0 0x7fdd6c1a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 -- 172.21.15.23:0/3305032269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c1006f0 msgr2=0x7fdd6c1a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c1006f0 0x7fdd6c1a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 -- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1060b0 msgr2=0x7fdd6c1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1060b0 0x7fdd6c1001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.199+0000 7fdd71200700 1 -- 172.21.15.23:0/3305032269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580095f0 con 0x7fdd6c069240 2024-03-30T04:14:44.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd71200700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000d980 tx=0x7fdd6000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd69600700 1 -- 172.21.15.23:0/3305032269 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60004220 con 0x7fdd6c069240 2024-03-30T04:14:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd69600700 1 -- 172.21.15.23:0/3305032269 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd60007640 con 0x7fdd6c069240 2024-03-30T04:14:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd69600700 1 -- 172.21.15.23:0/3305032269 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60005690 con 0x7fdd6c069240 2024-03-30T04:14:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c1a5fb0 con 0x7fdd6c069240 2024-03-30T04:14:44.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.200+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c1a6380 con 0x7fdd6c069240 2024-03-30T04:14:44.203 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.205+0000 7fdd8763bb80 1 Processor -- start 2024-03-30T04:14:44.204 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.205+0000 7fdd8763bb80 1 -- start start 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 0x5600fbdde220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdeba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbd2e990 con 0x5600fbdde760 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbd2c350 con 0x5600fbddde40 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.206+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbd27c40 con 0x5600fbde9610 2024-03-30T04:14:44.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.207+0000 7fdd69600700 1 -- 172.21.15.23:0/3305032269 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fdd60004750 con 0x7fdd6c069240 2024-03-30T04:14:44.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.207+0000 7fdd71200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.207+0000 7fdd71200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33442/0 (socket says 172.21.15.23:33442) 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.207+0000 7fdd71200700 1 -- 172.21.15.23:0/3424251726 learned_addr learned my addr 172.21.15.23:0/3424251726 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.207+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 0x5600fbdde220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd6be00700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdeba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 -- 172.21.15.23:0/3424251726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 msgr2=0x5600fbdeba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdeba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 -- 172.21.15.23:0/3424251726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 msgr2=0x5600fbde90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 -- 172.21.15.23:0/3424251726 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600fbba93b0 con 0x5600fbddde40 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71200700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:44.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 0x5600fbdde220 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdd64011580 tx=0x7fdd64009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6354b1bf0d9d22f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.208+0000 7fdd5e000700 1 -- 172.21.15.23:0/3424251726 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6400e040 con 0x5600fbddde40 2024-03-30T04:14:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.209+0000 7fdd5e000700 1 -- 172.21.15.23:0/3424251726 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd6400f040 con 0x5600fbddde40 2024-03-30T04:14:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.209+0000 7fdd5e000700 1 -- 172.21.15.23:0/3424251726 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004450 con 0x5600fbddde40 2024-03-30T04:14:44.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.209+0000 7fdd69600700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 0x7fdd540a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.209+0000 7fdd69600700 1 -- 172.21.15.23:0/3305032269 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd600ccda0 con 0x7fdd6c069240 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.209+0000 7fdd71200700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 0x7fdd540a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3424251726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 msgr2=0x5600fbdde220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 0x5600fbdde220 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdd64011580 tx=0x7fdd64009920 comp rx=0 tx=0).stop 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3424251726 shutdown_connections 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdeba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbddde40 0x5600fbdde220 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3424251726 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbdde760 0x5600fbde90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3424251726 >> 172.21.15.23:0/3424251726 conn(0x5600fbd356a0 msgr2=0x5600fbdd1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd71200700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 0x7fdd540a3850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd600a19c0 tx=0x7fdd600050b0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3424251726 shutdown_connections 2024-03-30T04:14:44.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.210+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3424251726 wait complete. 2024-03-30T04:14:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.211+0000 7fdd8763bb80 1 -- start start 2024-03-30T04:14:44.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.211+0000 7fdd8763bb80 1 Processor -- start 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbddde40 0x5600fbde8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 0x5600fbde68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbcc3d80 con 0x5600fbde9610 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbcc3a20 con 0x5600fbdde760 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbbab6a0 con 0x5600fbddde40 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd8763bb80 10 client.0 _ll_register_callbacks cb 0x5600fbccd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd6be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd6be00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38418/0 (socket says 172.21.15.23:38418) 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd6be00700 1 -- 172.21.15.23:0/3080163015 learned_addr learned my addr 172.21.15.23:0/3080163015 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbddde40 0x5600fbde8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.212+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 0x5600fbde68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 -- 172.21.15.23:0/3080163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbddde40 msgr2=0x5600fbde8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbddde40 0x5600fbde8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 -- 172.21.15.23:0/3080163015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 msgr2=0x5600fbde68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 0x5600fbde68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 -- 172.21.15.23:0/3080163015 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c1a6380 con 0x5600fbdde760 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 0x5600fbde68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd6be00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd580099c0 tx=0x7fdd58005800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28731772966e54d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd52a00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:44.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5801d040 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd5800bc60 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.213+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600fbce8380 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd58005a00 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 10 client.28961 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fdd64011230 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5600fbce8380 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdd5800f6b0 con 0x5600fbdde760 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 10 client.28961 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 10 client.28961 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 20 client.28961 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbdf9260 0x5600fbdfb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 4 client.28961 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:44.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.214+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600fbd20700 con 0x5600fbdf9260 2024-03-30T04:14:44.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.215+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbdf9260 0x5600fbdfb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.215+0000 7fdd52a00700 10 client.28961 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:44.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.215+0000 7fdd71c00700 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbdf9260 0x5600fbdfb680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57d681c9286d0172 server_cookie=4db8058ba20a9b7 in_seq=0 out_seq=0 2024-03-30T04:14:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.216+0000 7fdd52a00700 1 -- 172.21.15.23:0/3080163015 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5600fbd20700 con 0x5600fbdf9260 2024-03-30T04:14:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.216+0000 7fdd52a00700 10 client.28961 handle_command_reply: tid=0 2024-03-30T04:14:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 client.28961 shutdown 2024-03-30T04:14:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 msgr2=0x5600fbde6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdd580099c0 tx=0x7fdd58005800 comp rx=0 tx=0).stop 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 shutdown_connections 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbde9610 0x5600fbde68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbdf9260 0x5600fbdfb680 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbddde40 0x5600fbde8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/3080163015 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbde6300 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd52a00700 0 client.28961 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 >> 172.21.15.23:0/3080163015 conn(0x5600fbd356a0 msgr2=0x5600fbddfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 shutdown_connections 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 1 -- 172.21.15.23:0/3080163015 wait complete. 2024-03-30T04:14:44.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.217+0000 7fdd8763bb80 20 client.28961 trim_cache size 0 max 16384 2024-03-30T04:14:44.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 Processor -- start 2024-03-30T04:14:44.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 -- start start 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbdde760 0x5600fbdf02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbde9610 0x5600fbdcaf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbddf0f0 con 0x5600fbddde40 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbdcd820 con 0x5600fbde9610 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbde6680 con 0x5600fbdde760 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.234+0000 7fdd71c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33456/0 (socket says 172.21.15.23:33456) 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 -- 172.21.15.23:0/108751624 learned_addr learned my addr 172.21.15.23:0/108751624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd6be00700 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbdde760 0x5600fbdf02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbde9610 0x5600fbdcaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 -- 172.21.15.23:0/108751624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbdde760 msgr2=0x5600fbdf02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbdde760 0x5600fbdf02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 -- 172.21.15.23:0/108751624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbde9610 msgr2=0x5600fbdcaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbde9610 0x5600fbdcaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 -- 172.21.15.23:0/108751624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600fbb55b60 con 0x5600fbddde40 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd71c00700 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdd64011520 tx=0x7fdd6400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1d52902dba3c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.235+0000 7fdd5ea00700 1 -- 172.21.15.23:0/108751624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6400e030 con 0x5600fbddde40 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.236+0000 7fdd5ea00700 1 -- 172.21.15.23:0/108751624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd64023070 con 0x5600fbddde40 2024-03-30T04:14:44.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.236+0000 7fdd5ea00700 1 -- 172.21.15.23:0/108751624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64025040 con 0x5600fbddde40 2024-03-30T04:14:44.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.236+0000 7fdd8763bb80 1 -- 172.21.15.23:0/108751624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 msgr2=0x5600fbde6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.236+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdd64011520 tx=0x7fdd6400b040 comp rx=0 tx=0).stop 2024-03-30T04:14:44.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1233125 9.44 MB/sec execute 561 sec latency 25.413 ms 2024-03-30T04:14:44.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 -- 172.21.15.23:0/108751624 shutdown_connections 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbde9610 0x5600fbdcaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbde6860 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/108751624 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbdde760 0x5600fbdf02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 -- 172.21.15.23:0/108751624 >> 172.21.15.23:0/108751624 conn(0x5600fbd356a0 msgr2=0x5600fbdca5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 -- 172.21.15.23:0/108751624 shutdown_connections 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 -- 172.21.15.23:0/108751624 wait complete. 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 -- start start 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.237+0000 7fdd8763bb80 1 Processor -- start 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbdf0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdee620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbdeed00 con 0x5600fbddde40 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbdeee70 con 0x5600fbdde760 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600fbdef030 con 0x5600fbde9610 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd8763bb80 10 client.0 _ll_register_callbacks cb 0x5600fbccd470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.238+0000 7fdd71c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33460/0 (socket says 172.21.15.23:33460) 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 -- 172.21.15.23:0/2391709451 learned_addr learned my addr 172.21.15.23:0/2391709451 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd6be00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbdf0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdee620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 -- 172.21.15.23:0/2391709451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 msgr2=0x5600fbdee620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdee620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 -- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 msgr2=0x5600fbdf0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbdf0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 -- 172.21.15.23:0/2391709451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd64028040 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd71c00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdd64009960 tx=0x7fdd64003bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540bc6efd235ae2f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.239+0000 7fdd52a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64025070 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd64003c90 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd6400f040 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd6400e040 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd8763bb80 10 client.18990 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5600fbcedea0 con 0x5600fbddde40 2024-03-30T04:14:44.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.240+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd6400f040 con 0x5600fbddde40 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdd6401b020 con 0x5600fbddde40 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 10 client.18990 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 10 client.18990 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 20 client.18990 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbe4df20 0x5600fbe50310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 4 client.18990 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5600fbb76cf0 con 0x5600fbe4df20 2024-03-30T04:14:44.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.241+0000 7fdd6be00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbe4df20 0x5600fbe50310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd6be00700 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbe4df20 0x5600fbe50310 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13a0261a9fcae8f1 server_cookie=fb96300c94045ef6 in_seq=0 out_seq=0 2024-03-30T04:14:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd52a00700 10 client.18990 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:44.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd52a00700 1 -- 172.21.15.23:0/2391709451 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5600fbb76cf0 con 0x5600fbe4df20 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd52a00700 10 client.18990 handle_command_reply: tid=0 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 client.18990 shutdown 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 msgr2=0x5600fbdeff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdd64009960 tx=0x7fdd64003bf0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 shutdown_connections 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5600fbe4df20 0x5600fbe50310 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600fbde9610 0x5600fbdee620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd52a00700 0 client.18990 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5600fbddde40 0x5600fbdeff30 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 --2- 172.21.15.23:0/2391709451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5600fbdde760 0x5600fbdf0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 >> 172.21.15.23:0/2391709451 conn(0x5600fbd356a0 msgr2=0x5600fbdcaf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 shutdown_connections 2024-03-30T04:14:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.242+0000 7fdd8763bb80 1 -- 172.21.15.23:0/2391709451 wait complete. 2024-03-30T04:14:44.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd8763bb80 20 client.18990 trim_cache size 0 max 16384 2024-03-30T04:14:44.246 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 msgr2=0x7fdd540a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 0x7fdd540a3850 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd600a19c0 tx=0x7fdd600050b0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 msgr2=0x7fdd6c0ffc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000d980 tx=0x7fdd6000dc90 comp rx=0 tx=0).stop 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 shutdown_connections 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd6c1006f0 0x7fdd6c1a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd540a1390 0x7fdd540a3850 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd6c069240 0x7fdd6c0ffc70 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 --2- 172.21.15.23:0/3305032269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1060b0 0x7fdd6c1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.243+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 >> 172.21.15.23:0/3305032269 conn(0x7fdd6c076d80 msgr2=0x7fdd6c1071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.245+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 shutdown_connections 2024-03-30T04:14:44.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.245+0000 7fdd73c00700 1 -- 172.21.15.23:0/3305032269 wait complete. 2024-03-30T04:14:44.262 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:44 smithi023 ceph-mon[38254]: pgmap v400: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 199 op/s 2024-03-30T04:14:44.264 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 -- 172.21.15.23:0/1932980737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 msgr2=0x7fdca8073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 --2- 172.21.15.23:0/1932980737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8073560 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009990 tx=0x7fdc9c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 -- 172.21.15.23:0/1932980737 shutdown_connections 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 --2- 172.21.15.23:0/1932980737 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca810d3e0 0x7fdca810f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 --2- 172.21.15.23:0/1932980737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8073560 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 --2- 172.21.15.23:0/1932980737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca8073aa0 0x7fdca810cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 -- 172.21.15.23:0/1932980737 >> 172.21.15.23:0/1932980737 conn(0x7fdca80fc670 msgr2=0x7fdca80fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 -- 172.21.15.23:0/1932980737 shutdown_connections 2024-03-30T04:14:44.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.488+0000 7fdcae000700 1 -- 172.21.15.23:0/1932980737 wait complete. 2024-03-30T04:14:44.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 Processor -- start 2024-03-30T04:14:44.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 -- start start 2024-03-30T04:14:44.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8199230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca810d3e0 0x7fdca819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca819e150 con 0x7fdca8073aa0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca819e2c0 con 0x7fdca810d3e0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.490+0000 7fdcae000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdca819e480 con 0x7fdca8073180 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdcacc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8199230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33484/0 (socket says 172.21.15.23:33484) 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 -- 172.21.15.23:0/4068958526 learned_addr learned my addr 172.21.15.23:0/4068958526 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdcad600700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca810d3e0 0x7fdca819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 -- 172.21.15.23:0/4068958526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 msgr2=0x7fdca8199230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8199230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 -- 172.21.15.23:0/4068958526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca810d3e0 msgr2=0x7fdca819db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca810d3e0 0x7fdca819db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 -- 172.21.15.23:0/4068958526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc9c0095f0 con 0x7fdca8073aa0 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdcacc00700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:44.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.491+0000 7fdca7e00700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400d980 tx=0x7fdc9400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.492+0000 7fdca5600700 1 -- 172.21.15.23:0/4068958526 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94004220 con 0x7fdca8073aa0 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.492+0000 7fdca5600700 1 -- 172.21.15.23:0/4068958526 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc94007640 con 0x7fdca8073aa0 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.492+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca819e760 con 0x7fdca8073aa0 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.492+0000 7fdca5600700 1 -- 172.21.15.23:0/4068958526 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94005690 con 0x7fdca8073aa0 2024-03-30T04:14:44.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.492+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca81a5fa0 con 0x7fdca8073aa0 2024-03-30T04:14:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:44 smithi193 ceph-mon[79581]: pgmap v400: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 199 op/s 2024-03-30T04:14:44.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.495+0000 7fdca5600700 1 -- 172.21.15.23:0/4068958526 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fdc94004750 con 0x7fdca8073aa0 2024-03-30T04:14:44.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.496+0000 7fdcc3029b80 1 Processor -- start 2024-03-30T04:14:44.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 -- start start 2024-03-30T04:14:44.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdca5600700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 0x7fdc900a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdca5600700 1 -- 172.21.15.23:0/4068958526 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdc940cce70 con 0x7fdca8073aa0 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 0x56249a1f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1fdfc0 0x56249a2003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a14b990 con 0x56249a1f27a0 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a149350 con 0x56249a1f30c0 2024-03-30T04:14:44.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.497+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a144c40 con 0x56249a1fdfc0 2024-03-30T04:14:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38442/0 (socket says 172.21.15.23:38442) 2024-03-30T04:14:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 -- 172.21.15.23:0/2290319564 learned_addr learned my addr 172.21.15.23:0/2290319564 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 -- 172.21.15.23:0/2290319564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1fdfc0 msgr2=0x56249a2003b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:14:44.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1fdfc0 0x56249a2003b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdca7e00700 1 -- 172.21.15.23:0/2290319564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 msgr2=0x56249a1f2b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:14:44.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.498+0000 7fdcad600700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 0x7fdc900a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.499+0000 7fdcacc00700 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 0x56249a1f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.499+0000 7fdca7e00700 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 0x56249a1f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.500+0000 7fdca7e00700 1 -- 172.21.15.23:0/2290319564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562499fc63b0 con 0x56249a1f30c0 2024-03-30T04:14:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.500+0000 7fdcacc00700 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 0x56249a1f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:14:44.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.500+0000 7fdcad600700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 0x7fdc900a37d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdca80678e0 tx=0x7fdca0009100 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.502+0000 7fdca7e00700 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fdc940965b0 tx=0x7fdc94019f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=377426bb7765e0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.502+0000 7fdc99c00700 1 -- 172.21.15.23:0/2290319564 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc940b9050 con 0x56249a1f30c0 2024-03-30T04:14:44.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.502+0000 7fdc99c00700 1 -- 172.21.15.23:0/2290319564 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc94099680 con 0x56249a1f30c0 2024-03-30T04:14:44.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.502+0000 7fdc99c00700 1 -- 172.21.15.23:0/2290319564 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc94099290 con 0x56249a1f30c0 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.507+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2290319564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 msgr2=0x56249a1fda80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.507+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fdc940965b0 tx=0x7fdc94019f20 comp rx=0 tx=0).stop 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.508+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2290319564 shutdown_connections 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.508+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1fdfc0 0x56249a2003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.508+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f27a0 0x56249a1f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.508+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2290319564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a1fda80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.508+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2290319564 >> 172.21.15.23:0/2290319564 conn(0x56249a1526a0 msgr2=0x56249a1eec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.509+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2290319564 shutdown_connections 2024-03-30T04:14:44.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.511+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2290319564 wait complete. 2024-03-30T04:14:44.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.512+0000 7fdcc3029b80 1 -- start start 2024-03-30T04:14:44.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.512+0000 7fdcc3029b80 1 Processor -- start 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1fb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a0e0d80 con 0x56249a1f30c0 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a0e0a20 con 0x56249a1fdfc0 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.515+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562499fc86a0 con 0x56249a1f27a0 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.516+0000 7fdcc3029b80 10 client.0 _ll_register_callbacks cb 0x56249a0ea470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.516+0000 7fdcad600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.516+0000 7fdcad600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38456/0 (socket says 172.21.15.23:38456) 2024-03-30T04:14:44.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.516+0000 7fdcad600700 1 -- 172.21.15.23:0/655030156 learned_addr learned my addr 172.21.15.23:0/655030156 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdca7e00700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1fb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcacc00700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcad600700 1 -- 172.21.15.23:0/655030156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 msgr2=0x56249a1fad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcad600700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1fad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcad600700 1 -- 172.21.15.23:0/655030156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 msgr2=0x56249a1fb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcad600700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1fb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.517+0000 7fdcad600700 1 -- 172.21.15.23:0/655030156 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca81a5fa0 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.518+0000 7fdcad600700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007780 tx=0x7fdca0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd4272fed6111dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.518+0000 7fdc8e000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.518+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0015850 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.518+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56249a105380 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.518+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdca00159b0 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdca00048f0 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 10 client.28979 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562499f72b60 con 0x56249a1fdfc0 2024-03-30T04:14:44.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56249a105380 con 0x56249a1fdfc0 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdca0021d20 con 0x56249a1fdfc0 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 10 client.28979 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 10 client.28979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 20 client.28979 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a204810 0x56249a206c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 4 client.28979 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56249a13d700 con 0x56249a204810 2024-03-30T04:14:44.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.519+0000 7fdcacc00700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a204810 0x56249a206c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.520+0000 7fdcacc00700 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a204810 0x56249a206c30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6448b9c43416adc4 server_cookie=e5404e042c865672 in_seq=0 out_seq=0 2024-03-30T04:14:44.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.520+0000 7fdc8e000700 10 client.28979 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:44.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdc8e000700 1 -- 172.21.15.23:0/655030156 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56249a13d700 con 0x56249a204810 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdc8e000700 10 client.28979 handle_command_reply: tid=0 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 client.28979 shutdown 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 msgr2=0x56249a1fb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007780 tx=0x7fdca0004b40 comp rx=0 tx=0).stop 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 shutdown_connections 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1fb810 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a204810 0x56249a206c30 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/655030156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1fb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdc8e000700 0 client.28979 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 >> 172.21.15.23:0/655030156 conn(0x56249a1526a0 msgr2=0x56249a1f4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 shutdown_connections 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 1 -- 172.21.15.23:0/655030156 wait complete. 2024-03-30T04:14:44.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.521+0000 7fdcc3029b80 20 client.28979 trim_cache size 0 max 16384 2024-03-30T04:14:44.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 Processor -- start 2024-03-30T04:14:44.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 -- start start 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 0x56249a211a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a201380 con 0x56249a1fdfc0 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a1f3ff0 con 0x56249a1f30c0 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.537+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a20d800 con 0x56249a1f27a0 2024-03-30T04:14:44.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcad600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 0x56249a211a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:43984/0 (socket says 172.21.15.23:43984) 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38468/0 (socket says 172.21.15.23:38468) 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 -- 172.21.15.23:0/2826598636 learned_addr learned my addr 172.21.15.23:0/2826598636 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 -- 172.21.15.23:0/2826598636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 msgr2=0x56249a20d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 -- 172.21.15.23:0/2826598636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 msgr2=0x56249a211a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 0x56249a211a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 -- 172.21.15.23:0/2826598636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56249a10aea0 con 0x56249a1f27a0 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdca7e00700 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcad600700 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 0x56249a211a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:44.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.538+0000 7fdcacc00700 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda073814b806dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.539+0000 7fdc9a600700 1 -- 172.21.15.23:0/2826598636 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c01d040 con 0x56249a1f27a0 2024-03-30T04:14:44.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.539+0000 7fdc9a600700 1 -- 172.21.15.23:0/2826598636 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc9c00bec0 con 0x56249a1f27a0 2024-03-30T04:14:44.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.539+0000 7fdc9a600700 1 -- 172.21.15.23:0/2826598636 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c005710 con 0x56249a1f27a0 2024-03-30T04:14:44.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.539+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2826598636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 msgr2=0x56249a1ee160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.539+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c000c00 tx=0x7fdc9c005040 comp rx=0 tx=0).stop 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2826598636 shutdown_connections 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1fdfc0 0x56249a211a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1ee160 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/2826598636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1f30c0 0x56249a20d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2826598636 >> 172.21.15.23:0/2826598636 conn(0x56249a1526a0 msgr2=0x56249a1e77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2826598636 shutdown_connections 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 -- 172.21.15.23:0/2826598636 wait complete. 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 -- start start 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.540+0000 7fdcc3029b80 1 Processor -- start 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1f8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a1f7aa0 con 0x56249a1f30c0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a1f7c10 con 0x56249a1fdfc0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249a1f7dd0 con 0x56249a1f27a0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcc3029b80 10 client.0 _ll_register_callbacks cb 0x56249a0ea470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcacc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdca7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38482/0 (socket says 172.21.15.23:38482) 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 -- 172.21.15.23:0/1630289926 learned_addr learned my addr 172.21.15.23:0/1630289926 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 -- 172.21.15.23:0/1630289926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 msgr2=0x56249a1f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 -- 172.21.15.23:0/1630289926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 msgr2=0x56249a1f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.541+0000 7fdcad600700 1 -- 172.21.15.23:0/1630289926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562499fc63b0 con 0x56249a1fdfc0 2024-03-30T04:14:44.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdcad600700 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007780 tx=0x7fdca0003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48ce568421ebdf6f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdc8e000700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca0016bb0 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562499f93cf0 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdca0015850 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdca00167e0 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdcc3029b80 10 client.28985 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.542+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562499f72b60 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562499f93cf0 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdca0021ae0 con 0x56249a1fdfc0 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 10 client.28985 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 10 client.28985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 20 client.28985 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:44.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a2738c0 0x56249a275ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 4 client.28985 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:14:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562499fc8ce0 con 0x56249a2738c0 2024-03-30T04:14:44.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.543+0000 7fdcacc00700 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a2738c0 0x56249a275ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcacc00700 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a2738c0 0x56249a275ce0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74ed0a6898da3be3 server_cookie=4df43a16025362eb in_seq=0 out_seq=0 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdc8e000700 10 client.28985 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdc8e000700 1 -- 172.21.15.23:0/1630289926 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562499fc8ce0 con 0x56249a2738c0 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdc8e000700 10 client.28985 handle_command_reply: tid=0 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 client.28985 shutdown 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 msgr2=0x56249a1f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fdca0007780 tx=0x7fdca0003d60 comp rx=0 tx=0).stop 2024-03-30T04:14:44.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 shutdown_connections 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56249a1fdfc0 0x56249a1f7560 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x56249a2738c0 0x56249a275ce0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56249a1f27a0 0x56249a1f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 --2- 172.21.15.23:0/1630289926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56249a1f30c0 0x56249a1f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 >> 172.21.15.23:0/1630289926 conn(0x56249a1526a0 msgr2=0x56249a211a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdc8e000700 0 client.28985 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 shutdown_connections 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.544+0000 7fdcc3029b80 1 -- 172.21.15.23:0/1630289926 wait complete. 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcc3029b80 20 client.28985 trim_cache size 0 max 16384 2024-03-30T04:14:44.547 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 msgr2=0x7fdc900a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 0x7fdc900a37d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdca80678e0 tx=0x7fdca0009100 comp rx=0 tx=0).stop 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 msgr2=0x7fdca8199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdc9400d980 tx=0x7fdc9400dc90 comp rx=0 tx=0).stop 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 shutdown_connections 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdc900a1310 0x7fdc900a37d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdca810d3e0 0x7fdca819db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdca8073180 0x7fdca8199230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 --2- 172.21.15.23:0/4068958526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdca8073aa0 0x7fdca8199770 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:44.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.545+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 >> 172.21.15.23:0/4068958526 conn(0x7fdca80fc670 msgr2=0x7fdca80fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:44.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.546+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 shutdown_connections 2024-03-30T04:14:44.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:44.546+0000 7fdcae000700 1 -- 172.21.15.23:0/4068958526 wait complete. 2024-03-30T04:14:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:44 smithi154 ceph-mon[79161]: pgmap v400: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 199 op/s 2024-03-30T04:14:45.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1234597 9.43 MB/sec execute 562 sec latency 108.973 ms 2024-03-30T04:14:45.564 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:14:45.564 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:14:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 -- 172.21.15.23:0/3134937329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4341060b0 msgr2=0x7fc43410dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 --2- 172.21.15.23:0/3134937329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4341060b0 0x7fc43410dc30 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fc42c00b0a0 tx=0x7fc42c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 -- 172.21.15.23:0/3134937329 shutdown_connections 2024-03-30T04:14:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 --2- 172.21.15.23:0/3134937329 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4341060b0 0x7fc43410dc30 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 --2- 172.21.15.23:0/3134937329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc434069260 0x7fc434069640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 --2- 172.21.15.23:0/3134937329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc434069c10 0x7fc434105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 -- 172.21.15.23:0/3134937329 >> 172.21.15.23:0/3134937329 conn(0x7fc434076d80 msgr2=0x7fc434077190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 -- 172.21.15.23:0/3134937329 shutdown_connections 2024-03-30T04:14:45.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.787+0000 7fc43ba00700 1 -- 172.21.15.23:0/3134937329 wait complete. 2024-03-30T04:14:45.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.788+0000 7fc43ba00700 1 Processor -- start 2024-03-30T04:14:45.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 -- start start 2024-03-30T04:14:45.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 0x7fc4341994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4341060b0 0x7fc434199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 0x7fc43419e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419e850 con 0x7fc43419ddd0 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419e9c0 con 0x7fc434069260 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc43ba00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43419eb80 con 0x7fc4341060b0 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc439000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 0x7fc4341994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc433e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4341060b0 0x7fc434199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc439a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 0x7fc43419e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc439000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 0x7fc4341994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38504/0 (socket says 172.21.15.23:38504) 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.789+0000 7fc439000700 1 -- 172.21.15.23:0/1704247292 learned_addr learned my addr 172.21.15.23:0/1704247292 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 -- 172.21.15.23:0/1704247292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4341060b0 msgr2=0x7fc434199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4341060b0 0x7fc434199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 -- 172.21.15.23:0/1704247292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 msgr2=0x7fc4341994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 0x7fc4341994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 -- 172.21.15.23:0/1704247292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c009d00 con 0x7fc43419ddd0 2024-03-30T04:14:45.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc439a00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 0x7fc43419e250 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0062a0 tx=0x7fc42c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc431600700 1 -- 172.21.15.23:0/1704247292 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c00e040 con 0x7fc43419ddd0 2024-03-30T04:14:45.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc431600700 1 -- 172.21.15.23:0/1704247292 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc42c01d470 con 0x7fc43419ddd0 2024-03-30T04:14:45.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc431600700 1 -- 172.21.15.23:0/1704247292 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0052a0 con 0x7fc43419ddd0 2024-03-30T04:14:45.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.790+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4341a6080 con 0x7fc43419ddd0 2024-03-30T04:14:45.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.791+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4341a64b0 con 0x7fc43419ddd0 2024-03-30T04:14:45.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.794+0000 7fc431600700 1 -- 172.21.15.23:0/1704247292 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc42c01d5e0 con 0x7fc43419ddd0 2024-03-30T04:14:45.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.795+0000 7fc431600700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 0x7fc41c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.795+0000 7fc44f459b80 1 Processor -- start 2024-03-30T04:14:45.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.795+0000 7fc431600700 1 -- 172.21.15.23:0/1704247292 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc42c0ce3f0 con 0x7fc43419ddd0 2024-03-30T04:14:45.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc439000700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 0x7fc41c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 -- start start 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7a650 0x558e7ca7aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca88260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c9d3a10 con 0x558e7ca85e70 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c9d13d0 con 0x558e7ca7a650 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.796+0000 7fc44f459b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c9cccc0 con 0x558e7ca7af70 2024-03-30T04:14:45.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc439000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7a650 0x558e7ca7aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44016/0 (socket says 172.21.15.23:44016) 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 -- 172.21.15.23:0/499173010 learned_addr learned my addr 172.21.15.23:0/499173010 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc439a00700 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca88260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc439000700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 0x7fc41c0a37d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc428009d20 tx=0x7fc428009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 -- 172.21.15.23:0/499173010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7a650 msgr2=0x558e7ca7aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7a650 0x558e7ca7aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 -- 172.21.15.23:0/499173010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 msgr2=0x558e7ca88260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca88260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 -- 172.21.15.23:0/499173010 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e7c84e430 con 0x558e7ca7af70 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc433e00700 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc420009cb0 tx=0x7fc420011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977dc8b12837b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.797+0000 7fc426000700 1 -- 172.21.15.23:0/499173010 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420011d20 con 0x558e7ca7af70 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc426000700 1 -- 172.21.15.23:0/499173010 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc420011e80 con 0x558e7ca7af70 2024-03-30T04:14:45.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc426000700 1 -- 172.21.15.23:0/499173010 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420005280 con 0x558e7ca7af70 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 -- 172.21.15.23:0/499173010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 msgr2=0x558e7ca85930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc420009cb0 tx=0x7fc420011440 comp rx=0 tx=0).stop 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 -- 172.21.15.23:0/499173010 shutdown_connections 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca88260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7a650 0x558e7ca7aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 --2- 172.21.15.23:0/499173010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7af70 0x558e7ca85930 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 -- 172.21.15.23:0/499173010 >> 172.21.15.23:0/499173010 conn(0x558e7c9da720 msgr2=0x558e7ca76b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 -- 172.21.15.23:0/499173010 shutdown_connections 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.798+0000 7fc44f459b80 1 -- 172.21.15.23:0/499173010 wait complete. 2024-03-30T04:14:45.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.799+0000 7fc44f459b80 1 -- start start 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.799+0000 7fc44f459b80 1 Processor -- start 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca85760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca83020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c968e00 con 0x558e7ca85e70 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c968aa0 con 0x558e7ca7af70 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7c850720 con 0x558e7ca7a650 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc44f459b80 10 client.0 _ll_register_callbacks cb 0x558e7c9724f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:45.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc439000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca85760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38518/0 (socket says 172.21.15.23:38518) 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 -- 172.21.15.23:0/2860048148 learned_addr learned my addr 172.21.15.23:0/2860048148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc439a00700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca83020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 -- 172.21.15.23:0/2860048148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 msgr2=0x558e7ca85760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca85760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 -- 172.21.15.23:0/2860048148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 msgr2=0x558e7ca83020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca83020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc433e00700 1 -- 172.21.15.23:0/2860048148 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4200098e0 con 0x558e7ca7af70 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc439a00700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca83020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.800+0000 7fc439000700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc433e00700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc420011790 tx=0x7fc420005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=664e9aa5cbdd1586 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc41aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420004850 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc420004260 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e7c9c5780 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc420004420 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc44f459b80 10 client.28997 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.801+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558e7c7fabe0 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558e7c9c5780 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc4200050d0 con 0x558e7ca7af70 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 10 client.28997 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 10 client.28997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 20 client.28997 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7ca95820 0x558e7ca97c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 4 client.28997 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e7c81bd70 con 0x558e7ca95820 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.802+0000 7fc439000700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7ca95820 0x558e7ca97c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.803+0000 7fc439000700 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7ca95820 0x558e7ca97c40 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7629b5c7803c5f server_cookie=6d1c6656929590ab in_seq=0 out_seq=0 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.803+0000 7fc41aa00700 10 client.28997 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc41aa00700 1 -- 172.21.15.23:0/2860048148 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558e7c81bd70 con 0x558e7ca95820 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc41aa00700 10 client.28997 handle_command_reply: tid=0 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 client.28997 shutdown 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 msgr2=0x558e7ca82aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc420011790 tx=0x7fc420005d60 comp rx=0 tx=0).stop 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 shutdown_connections 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca85e70 0x558e7ca83020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7ca95820 0x558e7ca97c40 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca85760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 --2- 172.21.15.23:0/2860048148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca7af70 0x558e7ca82aa0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc41aa00700 0 client.28997 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 >> 172.21.15.23:0/2860048148 conn(0x558e7c9da720 msgr2=0x558e7ca7c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 shutdown_connections 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 1 -- 172.21.15.23:0/2860048148 wait complete. 2024-03-30T04:14:45.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.804+0000 7fc44f459b80 20 client.28997 trim_cache size 0 max 16384 2024-03-30T04:14:45.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 Processor -- start 2024-03-30T04:14:45.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 -- start start 2024-03-30T04:14:45.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7ca91240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7ca7d8d0 con 0x558e7ca7af70 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7ca7a470 con 0x558e7ca85e70 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc44f459b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7ca8d030 con 0x558e7ca7a650 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc439000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.822+0000 7fc439000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44036/0 (socket says 172.21.15.23:44036) 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 -- 172.21.15.23:0/61779337 learned_addr learned my addr 172.21.15.23:0/61779337 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc433e00700 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439a00700 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7ca91240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 -- 172.21.15.23:0/61779337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 msgr2=0x558e7ca91240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7ca91240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 -- 172.21.15.23:0/61779337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 msgr2=0x558e7ca8caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 -- 172.21.15.23:0/61779337 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e7c992f20 con 0x558e7ca7a650 2024-03-30T04:14:45.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc439000700 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc428009880 tx=0x7fc428014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b89c086b356c3c21 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc426a00700 1 -- 172.21.15.23:0/61779337 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc428022040 con 0x558e7ca7a650 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.823+0000 7fc426a00700 1 -- 172.21.15.23:0/61779337 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc42802f070 con 0x558e7ca7a650 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc426a00700 1 -- 172.21.15.23:0/61779337 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42802ad60 con 0x558e7ca7a650 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 -- 172.21.15.23:0/61779337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 msgr2=0x558e7ca82ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc428009880 tx=0x7fc428014000 comp rx=0 tx=0).stop 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 -- 172.21.15.23:0/61779337 shutdown_connections 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7ca91240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca82ff0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 --2- 172.21.15.23:0/61779337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 -- 172.21.15.23:0/61779337 >> 172.21.15.23:0/61779337 conn(0x558e7c9da720 msgr2=0x558e7ca81370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 -- 172.21.15.23:0/61779337 shutdown_connections 2024-03-30T04:14:45.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.824+0000 7fc44f459b80 1 -- 172.21.15.23:0/61779337 wait complete. 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 -- start start 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 Processor -- start 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca8b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7cae9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7cae9ba0 con 0x558e7ca7af70 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7cae9ce0 con 0x558e7ca85e70 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e7cae9e20 con 0x558e7ca7a650 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.825+0000 7fc44f459b80 10 client.0 _ll_register_callbacks cb 0x558e7c9724f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc439000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca8b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc439000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca8b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44048/0 (socket says 172.21.15.23:44048) 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc439000700 1 -- 172.21.15.23:0/130682814 learned_addr learned my addr 172.21.15.23:0/130682814 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:45.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc439a00700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7cae9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 -- 172.21.15.23:0/130682814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 msgr2=0x558e7ca8b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca8b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 -- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 msgr2=0x558e7cae9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7cae9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 -- 172.21.15.23:0/130682814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428017070 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc433e00700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8bdd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc420009bd0 tx=0x7fc420004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bba6d950a3215 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc41aa00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.826+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4200117c0 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e7c98d400 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc420004260 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc420004440 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 10 client.19023 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x558e7c852d50 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558e7c98d400 con 0x558e7ca7af70 2024-03-30T04:14:45.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc420003bd0 con 0x558e7ca7af70 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 10 client.19023 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 10 client.19023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 20 client.19023 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7caeb1d0 0x558e7caed5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 4 client.19023 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.827+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558e7c850d60 con 0x558e7caeb1d0 2024-03-30T04:14:45.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.828+0000 7fc439000700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7caeb1d0 0x558e7caed5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.828+0000 7fc439000700 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7caeb1d0 0x558e7caed5f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e61e8ec827e95bfe server_cookie=6de76c22b13ca273 in_seq=0 out_seq=0 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.828+0000 7fc41aa00700 10 client.19023 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.829+0000 7fc41aa00700 1 -- 172.21.15.23:0/130682814 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558e7c850d60 con 0x558e7caeb1d0 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.829+0000 7fc41aa00700 10 client.19023 handle_command_reply: tid=0 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.829+0000 7fc44f459b80 1 client.19023 shutdown 2024-03-30T04:14:45.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 msgr2=0x558e7ca8bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8bdd0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fc420009bd0 tx=0x7fc420004f60 comp rx=0 tx=0).stop 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 shutdown_connections 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558e7ca85e70 0x558e7cae9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x558e7caeb1d0 0x558e7caed5f0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e7ca7a650 0x558e7ca8b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 --2- 172.21.15.23:0/130682814 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558e7ca7af70 0x558e7ca8bdd0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 >> 172.21.15.23:0/130682814 conn(0x558e7c9da720 msgr2=0x558e7ca91ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc41aa00700 0 client.19023 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 shutdown_connections 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 1 -- 172.21.15.23:0/130682814 wait complete. 2024-03-30T04:14:45.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.830+0000 7fc44f459b80 20 client.19023 trim_cache size 0 max 16384 2024-03-30T04:14:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:45.832 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:14:45.832 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "071bca6b-a25a-48ef-8afb-eb6efb2ea8c9", 2024-03-30T04:14:45.832 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:14:45.832 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 msgr2=0x7fc41c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 0x7fc41c0a37d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc428009d20 tx=0x7fc428009260 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 msgr2=0x7fc43419e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 0x7fc43419e250 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0062a0 tx=0x7fc42c00bd50 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 shutdown_connections 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc43419ddd0 0x7fc43419e250 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc41c0a1310 0x7fc41c0a37d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc434069260 0x7fc4341994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 --2- 172.21.15.23:0/1704247292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4341060b0 0x7fc434199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 >> 172.21.15.23:0/1704247292 conn(0x7fc434076d80 msgr2=0x7fc4341071e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 shutdown_connections 2024-03-30T04:14:45.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:45.831+0000 7fc43ba00700 1 -- 172.21.15.23:0/1704247292 wait complete. 2024-03-30T04:14:45.847 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:14:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.118+0000 7f03b9e00700 1 -- 172.21.15.23:0/3334961394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 msgr2=0x7f03b410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.118+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3334961394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410f990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f03ac00b0a0 tx=0x7f03ac00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 -- 172.21.15.23:0/3334961394 shutdown_connections 2024-03-30T04:14:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3334961394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410f990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3334961394 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b40731c0 0x7f03b40735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3334961394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b4073ae0 0x7f03b410d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 -- 172.21.15.23:0/3334961394 >> 172.21.15.23:0/3334961394 conn(0x7f03b40fc7f0 msgr2=0x7f03b40fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 -- 172.21.15.23:0/3334961394 shutdown_connections 2024-03-30T04:14:46.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.119+0000 7f03b9e00700 1 -- 172.21.15.23:0/3334961394 wait complete. 2024-03-30T04:14:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.121+0000 7f03b9e00700 1 Processor -- start 2024-03-30T04:14:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.122+0000 7f03b9e00700 1 -- start start 2024-03-30T04:14:46.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.122+0000 7f03b9e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 0x7f03b410ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.122+0000 7f03b9e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 0x7f03b4109f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.122+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 0x7f03b410ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 0x7f03b410ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:44074/0 (socket says 172.21.15.23:44074) 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b2a00700 1 -- 172.21.15.23:0/3385560756 learned_addr learned my addr 172.21.15.23:0/3385560756 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b410ac00 con 0x7f03b4073ae0 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b2000700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 0x7f03b4109f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b410ad70 con 0x7f03b410d540 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.123+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b41a9d30 con 0x7f03b40731c0 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b3400700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b2000700 1 -- 172.21.15.23:0/3385560756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 msgr2=0x7f03b410ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b2000700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 0x7f03b410ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b2000700 1 -- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 msgr2=0x7f03b410a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b2000700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b2000700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac009d00 con 0x7f03b4073ae0 2024-03-30T04:14:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.124+0000 7f03b3400700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:14:46.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b2000700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 0x7f03b4109f30 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c9a0 tx=0x7f03a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b8a00700 1 -- 172.21.15.23:0/3385560756 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4004440 con 0x7f03b4073ae0 2024-03-30T04:14:46.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b8a00700 1 -- 172.21.15.23:0/3385560756 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03a400ce90 con 0x7f03b4073ae0 2024-03-30T04:14:46.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b8a00700 1 -- 172.21.15.23:0/3385560756 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003cb0 con 0x7f03b4073ae0 2024-03-30T04:14:46.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b41a9fb0 con 0x7f03b4073ae0 2024-03-30T04:14:46.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.125+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b41aa490 con 0x7f03b4073ae0 2024-03-30T04:14:46.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.130+0000 7f03b8a00700 1 -- 172.21.15.23:0/3385560756 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f03a4007650 con 0x7f03b4073ae0 2024-03-30T04:14:46.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.132+0000 7f03b8a00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 0x7f03980a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.132+0000 7f03cd8bcb80 1 Processor -- start 2024-03-30T04:14:46.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.132+0000 7f03b8a00700 1 -- 172.21.15.23:0/3385560756 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f03a40ccf30 con 0x7f03b4073ae0 2024-03-30T04:14:46.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.132+0000 7f03b2a00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 0x7f03980a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 -- start start 2024-03-30T04:14:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc7a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b2a00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 0x7f03980a37d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f039c005950 tx=0x7f039c00b500 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 0x5625efc858b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efbd3990 con 0x5625efc7aef0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efbd1350 con 0x5625efc7a5d0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efbccc40 con 0x5625efc85df0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc7a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc7a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38554/0 (socket says 172.21.15.23:38554) 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b2a00700 1 -- 172.21.15.23:0/1922243528 learned_addr learned my addr 172.21.15.23:0/1922243528 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b3400700 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.133+0000 7f03b2000700 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 0x5625efc858b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 -- 172.21.15.23:0/1922243528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 msgr2=0x5625efc881e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc881e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 -- 172.21.15.23:0/1922243528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 msgr2=0x5625efc7a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc7a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 -- 172.21.15.23:0/1922243528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625efa4e3b0 con 0x5625efc7aef0 2024-03-30T04:14:46.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03b2000700 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 0x5625efc858b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f03a40ca240 tx=0x7f03a40995e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692e7cd53152c8b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03a3e00700 1 -- 172.21.15.23:0/1922243528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a409a300 con 0x5625efc7aef0 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03a3e00700 1 -- 172.21.15.23:0/1922243528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03a4099c00 con 0x5625efc7aef0 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.134+0000 7f03a3e00700 1 -- 172.21.15.23:0/1922243528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4007db0 con 0x5625efc7aef0 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1922243528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 msgr2=0x5625efc858b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 0x5625efc858b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f03a40ca240 tx=0x7f03a40995e0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1922243528 shutdown_connections 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc7a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1922243528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7aef0 0x5625efc858b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1922243528 >> 172.21.15.23:0/1922243528 conn(0x5625efbda6a0 msgr2=0x5625efc76a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1922243528 shutdown_connections 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.135+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1922243528 wait complete. 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 -- start start 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 Processor -- start 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc82bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc83110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efb68d80 con 0x5625efc7a5d0 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efb68a20 con 0x5625efc85df0 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.136+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efa506a0 con 0x5625efc7aef0 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03cd8bcb80 10 client.0 _ll_register_callbacks cb 0x5625efb72470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38568/0 (socket says 172.21.15.23:38568) 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 -- 172.21.15.23:0/1574712961 learned_addr learned my addr 172.21.15.23:0/1574712961 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b2000700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc83110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b2a00700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc82bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 -- 172.21.15.23:0/1574712961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 msgr2=0x5625efc83110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc83110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 -- 172.21.15.23:0/1574712961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 msgr2=0x5625efc82bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc82bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b3400700 1 -- 172.21.15.23:0/1574712961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b41aa490 con 0x5625efc85df0 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.137+0000 7f03b2a00700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc82bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03b3400700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009440 tx=0x7f03ac009470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7bdc40ed8317510 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03a0c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac00e040 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f03ac0074e0 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625efb8d380 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03ac004e60 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03cd8bcb80 10 client.29012 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.138+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5625ef9fab60 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5625efb8d380 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03ac019050 con 0x5625efc85df0 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 10 client.29012 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 10 client.29012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 20 client.29012 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efc959c0 0x5625efc97de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 4 client.29012 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625efbc5700 con 0x5625efc959c0 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.139+0000 7f03b2a00700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efc959c0 0x5625efc97de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.140+0000 7f03b2a00700 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efc959c0 0x5625efc97de0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d47641bcedd6220e server_cookie=e751cdb839ee5e99 in_seq=0 out_seq=0 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.140+0000 7f03a0c00700 10 client.29012 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03a0c00700 1 -- 172.21.15.23:0/1574712961 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5625efbc5700 con 0x5625efc959c0 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03a0c00700 10 client.29012 handle_command_reply: tid=0 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 client.29012 shutdown 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 msgr2=0x5625efc83650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009440 tx=0x7f03ac009470 comp rx=0 tx=0).stop 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 shutdown_connections 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc85df0 0x5625efc83650 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc82bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efc959c0 0x5625efc97de0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1574712961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc83110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 >> 172.21.15.23:0/1574712961 conn(0x5625efbda6a0 msgr2=0x5625efc7c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03a0c00700 0 client.29012 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 shutdown_connections 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1574712961 wait complete. 2024-03-30T04:14:46.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.141+0000 7f03cd8bcb80 20 client.29012 trim_cache size 0 max 16384 2024-03-30T04:14:46.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 Processor -- start 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 -- start start 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc8ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 0x5625efc6ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc814a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc85b80 con 0x5625efc7a5d0 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc841d0 con 0x5625efc7aef0 2024-03-30T04:14:46.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.159+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc7c4d0 con 0x5625efc85df0 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 0x5625efc6ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc8ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc814a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b2000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 0x5625efc6ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38576/0 (socket says 172.21.15.23:38576) 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b2000700 1 -- 172.21.15.23:0/1610374694 learned_addr learned my addr 172.21.15.23:0/1610374694 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 -- 172.21.15.23:0/1610374694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 msgr2=0x5625efc6ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 0x5625efc6ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 -- 172.21.15.23:0/1610374694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 msgr2=0x5625efc8ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc8ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 -- 172.21.15.23:0/1610374694 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625efb92ea0 con 0x5625efc85df0 2024-03-30T04:14:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.160+0000 7f03b3400700 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc814a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009cd0 tx=0x7f03ac009590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9437a3aad0d83402 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03a9200700 1 -- 172.21.15.23:0/1610374694 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac00e030 con 0x5625efc85df0 2024-03-30T04:14:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03a9200700 1 -- 172.21.15.23:0/1610374694 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03ac007960 con 0x5625efc85df0 2024-03-30T04:14:46.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03a9200700 1 -- 172.21.15.23:0/1610374694 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0059c0 con 0x5625efc85df0 2024-03-30T04:14:46.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1610374694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 msgr2=0x5625efc814a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc814a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009cd0 tx=0x7f03ac009590 comp rx=0 tx=0).stop 2024-03-30T04:14:46.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1610374694 shutdown_connections 2024-03-30T04:14:46.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc85df0 0x5625efc814a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc7a5d0 0x5625efc8ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/1610374694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7aef0 0x5625efc6ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1610374694 >> 172.21.15.23:0/1610374694 conn(0x5625efbda6a0 msgr2=0x5625efc6f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1610374694 shutdown_connections 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.161+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/1610374694 wait complete. 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 -- start start 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 Processor -- start 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc8c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 0x5625efc8adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc8b490 con 0x5625efc85df0 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc8b600 con 0x5625efc7a5d0 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.162+0000 7f03cd8bcb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625efc8b7c0 con 0x5625efc7aef0 2024-03-30T04:14:46.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03cd8bcb80 10 client.0 _ll_register_callbacks cb 0x5625efb72470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc8c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b2a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc8c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:38582/0 (socket says 172.21.15.23:38582) 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 0x5625efc8adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b2a00700 1 -- 172.21.15.23:0/2277208314 learned_addr learned my addr 172.21.15.23:0/2277208314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b2000700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 -- 172.21.15.23:0/2277208314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 msgr2=0x5625efc8cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc8cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 -- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 msgr2=0x5625efc8c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc8c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 -- 172.21.15.23:0/2277208314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625ef9fab60 con 0x5625efc85df0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03b3400700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 0x5625efc8adb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009fd0 tx=0x7f03ac009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3965951312a68eaa server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.163+0000 7f03a0c00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:14:46.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac00e070 con 0x5625efc85df0 2024-03-30T04:14:46.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03a4099c00 con 0x5625efc85df0 2024-03-30T04:14:46.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f03ac005600 con 0x5625efc85df0 2024-03-30T04:14:46.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03ac0057c0 con 0x5625efc85df0 2024-03-30T04:14:46.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03cd8bcb80 10 client.19038 fetch_fsmap learned FSMap version 30 2024-03-30T04:14:46.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5625efa52cd0 con 0x5625efc85df0 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.164+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03a4099c00 con 0x5625efc85df0 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f03ac00b4d0 con 0x5625efc85df0 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 10 client.19038 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 10 client.19038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 20 client.19038 populate_metadata read hostname 'smithi023' 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efcea760 0x5625efcecb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:14:46.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 4 client.19038 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5625efa1bcf0 con 0x5625efcea760 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03b2a00700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efcea760 0x5625efcecb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.165+0000 7f03b2a00700 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efcea760 0x5625efcecb80 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f29e48e52135b16 server_cookie=641fbef82fe7caa2 in_seq=0 out_seq=0 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03a0c00700 10 client.19038 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03a0c00700 1 -- 172.21.15.23:0/2277208314 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5625efa1bcf0 con 0x5625efcea760 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03a0c00700 10 client.19038 handle_command_reply: tid=0 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 client.19038 shutdown 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 msgr2=0x5625efc8adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 0x5625efc8adb0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009fd0 tx=0x7f03ac009b80 comp rx=0 tx=0).stop 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 shutdown_connections 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625efc85df0 0x5625efc8adb0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5625efc7a5d0 0x5625efc8c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5625efcea760 0x5625efcecb80 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 --2- 172.21.15.23:0/2277208314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5625efc7aef0 0x5625efc8cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 >> 172.21.15.23:0/2277208314 conn(0x5625efbda6a0 msgr2=0x5625efc7def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03a0c00700 0 client.19038 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 shutdown_connections 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.166+0000 7f03cd8bcb80 1 -- 172.21.15.23:0/2277208314 wait complete. 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03cd8bcb80 20 client.19038 trim_cache size 0 max 16384 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout: "071bca6b-a25a-48ef-8afb-eb6efb2ea8c9": { 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:14:46.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "071bca6b-a25a-48ef-8afb-eb6efb2ea8c9", 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 msgr2=0x7f03980a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 0x7f03980a37d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f039c005950 tx=0x7f039c00b500 comp rx=0 tx=0).stop 2024-03-30T04:14:46.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 msgr2=0x7f03b4109f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 0x7f03b4109f30 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c9a0 tx=0x7f03a400ccb0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 shutdown_connections 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f03980a1310 0x7f03980a37d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f03b410d540 0x7f03b410a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03b40731c0 0x7f03b410ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 --2- 172.21.15.23:0/3385560756 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f03b4073ae0 0x7f03b4109f30 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 >> 172.21.15.23:0/3385560756 conn(0x7f03b40fc7f0 msgr2=0x7f03b40fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 shutdown_connections 2024-03-30T04:14:46.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:14:46.167+0000 7f03b9e00700 1 -- 172.21.15.23:0/3385560756 wait complete. 2024-03-30T04:14:46.185 INFO:tasks.cephfs.filesystem:scrub status for tag:071bca6b-a25a-48ef-8afb-eb6efb2ea8c9 - {'path': '/', 'tag': '071bca6b-a25a-48ef-8afb-eb6efb2ea8c9', 'options': 'recursive,force'} 2024-03-30T04:14:46.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1236076 9.43 MB/sec execute 563 sec latency 38.426 ms 2024-03-30T04:14:46.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:46 smithi193 ceph-mon[79581]: pgmap v401: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:46.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:46 smithi023 ceph-mon[38254]: pgmap v401: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:46.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:46 smithi154 ceph-mon[79161]: pgmap v401: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:47.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1237369 9.43 MB/sec execute 564 sec latency 28.785 ms 2024-03-30T04:14:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:14:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:14:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:14:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:47 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:14:47.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:14:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:14:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:47 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:47.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:47 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:14:48.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1239462 9.43 MB/sec execute 565 sec latency 20.623 ms 2024-03-30T04:14:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:48 smithi193 ceph-mon[79581]: pgmap v402: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:48 smithi023 ceph-mon[38254]: pgmap v402: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:48.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:48 smithi154 ceph-mon[79161]: pgmap v402: 129 pgs: 129 active+clean; 569 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-03-30T04:14:49.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1240166 9.42 MB/sec execute 566 sec latency 30.559 ms 2024-03-30T04:14:49.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:49 smithi193 ceph-mon[79581]: pgmap v403: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:14:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:49 smithi023 ceph-mon[38254]: pgmap v403: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:14:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:49 smithi154 ceph-mon[79161]: pgmap v403: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 204 op/s 2024-03-30T04:14:50.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1241842 9.42 MB/sec execute 567 sec latency 37.990 ms 2024-03-30T04:14:51.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1243171 9.42 MB/sec execute 568 sec latency 49.417 ms 2024-03-30T04:14:52.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1244478 9.41 MB/sec execute 569 sec latency 33.051 ms 2024-03-30T04:14:52.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:52 smithi154 ceph-mon[79161]: pgmap v404: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 145 op/s 2024-03-30T04:14:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:52 smithi193 ceph-mon[79581]: pgmap v404: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 145 op/s 2024-03-30T04:14:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:52 smithi023 ceph-mon[38254]: pgmap v404: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 3.1 MiB/s wr, 145 op/s 2024-03-30T04:14:53.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1246649 9.42 MB/sec execute 570 sec latency 19.888 ms 2024-03-30T04:14:54.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1247736 9.41 MB/sec execute 571 sec latency 28.993 ms 2024-03-30T04:14:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:54 smithi154 ceph-mon[79161]: pgmap v405: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 213 op/s 2024-03-30T04:14:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:54 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:14:54.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:54 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:14:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:54 smithi193 ceph-mon[79581]: pgmap v405: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 213 op/s 2024-03-30T04:14:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:54 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:14:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:54 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:14:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:54 smithi023 ceph-mon[38254]: pgmap v405: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.5 MiB/s wr, 213 op/s 2024-03-30T04:14:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:54 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:14:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:54 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:14:55.235 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1249583 9.41 MB/sec execute 572 sec latency 63.532 ms 2024-03-30T04:14:56.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1250459 9.41 MB/sec execute 573 sec latency 64.412 ms 2024-03-30T04:14:56.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:56 smithi193 ceph-mon[79581]: pgmap v406: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:56 smithi023 ceph-mon[38254]: pgmap v406: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:56.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:56 smithi154 ceph-mon[79161]: pgmap v406: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:57.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1251808 9.40 MB/sec execute 574 sec latency 37.828 ms 2024-03-30T04:14:58.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1253739 9.40 MB/sec execute 575 sec latency 20.738 ms 2024-03-30T04:14:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:14:58 smithi193 ceph-mon[79581]: pgmap v407: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:14:58 smithi023 ceph-mon[38254]: pgmap v407: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:14:58 smithi154 ceph-mon[79161]: pgmap v407: 129 pgs: 129 active+clean; 568 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-03-30T04:14:59.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1254538 9.39 MB/sec execute 576 sec latency 32.425 ms 2024-03-30T04:15:00.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1256565 9.39 MB/sec execute 577 sec latency 20.980 ms 2024-03-30T04:15:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:00 smithi193 ceph-mon[79581]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-03-30T04:15:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:00 smithi023 ceph-mon[38254]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-03-30T04:15:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:00 smithi154 ceph-mon[79161]: pgmap v408: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 209 op/s 2024-03-30T04:15:01.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1257367 9.39 MB/sec execute 578 sec latency 65.817 ms 2024-03-30T04:15:02.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1257405 9.37 MB/sec execute 579 sec latency 93.217 ms 2024-03-30T04:15:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:02 smithi193 ceph-mon[79581]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:15:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:02 smithi023 ceph-mon[38254]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:15:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:02 smithi154 ceph-mon[79161]: pgmap v409: 129 pgs: 129 active+clean; 570 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.9 MiB/s wr, 136 op/s 2024-03-30T04:15:03.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1258465 9.36 MB/sec execute 580 sec latency 134.911 ms 2024-03-30T04:15:04.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1260217 9.36 MB/sec execute 581 sec latency 21.494 ms 2024-03-30T04:15:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:04 smithi193 ceph-mon[79581]: pgmap v410: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 227 op/s 2024-03-30T04:15:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:04 smithi023 ceph-mon[38254]: pgmap v410: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 227 op/s 2024-03-30T04:15:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:04 smithi154 ceph-mon[79161]: pgmap v410: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 227 op/s 2024-03-30T04:15:05.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1261488 9.36 MB/sec execute 582 sec latency 22.540 ms 2024-03-30T04:15:06.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1262722 9.35 MB/sec execute 583 sec latency 82.093 ms 2024-03-30T04:15:06.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:06 smithi193 ceph-mon[79581]: pgmap v411: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:06.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:06 smithi023 ceph-mon[38254]: pgmap v411: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:06.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:06 smithi154 ceph-mon[79161]: pgmap v411: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:07.236 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1264438 9.36 MB/sec execute 584 sec latency 39.734 ms 2024-03-30T04:15:08.243 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1265723 9.35 MB/sec execute 585 sec latency 29.389 ms 2024-03-30T04:15:08.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:08 smithi193 ceph-mon[79581]: pgmap v412: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:08 smithi023 ceph-mon[38254]: pgmap v412: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:08 smithi154 ceph-mon[79161]: pgmap v412: 129 pgs: 129 active+clean; 565 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 159 op/s 2024-03-30T04:15:09.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1267833 9.36 MB/sec execute 586 sec latency 23.487 ms 2024-03-30T04:15:10.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1268626 9.34 MB/sec execute 587 sec latency 32.346 ms 2024-03-30T04:15:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:10 smithi193 ceph-mon[79581]: pgmap v413: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 228 op/s 2024-03-30T04:15:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:10 smithi193 ceph-mon[79581]: osdmap e104: 12 total, 12 up, 12 in 2024-03-30T04:15:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:10 smithi023 ceph-mon[38254]: pgmap v413: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 228 op/s 2024-03-30T04:15:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:10 smithi023 ceph-mon[38254]: osdmap e104: 12 total, 12 up, 12 in 2024-03-30T04:15:10.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:10 smithi154 ceph-mon[79161]: pgmap v413: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 228 op/s 2024-03-30T04:15:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:10 smithi154 ceph-mon[79161]: osdmap e104: 12 total, 12 up, 12 in 2024-03-30T04:15:11.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1269825 9.34 MB/sec execute 588 sec latency 124.849 ms 2024-03-30T04:15:12.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1271516 9.34 MB/sec execute 589 sec latency 28.172 ms 2024-03-30T04:15:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:12 smithi193 ceph-mon[79581]: pgmap v415: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-30T04:15:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:12 smithi023 ceph-mon[38254]: pgmap v415: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-30T04:15:12.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:12 smithi154 ceph-mon[79161]: pgmap v415: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.4 MiB/s wr, 191 op/s 2024-03-30T04:15:13.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1272813 9.34 MB/sec execute 590 sec latency 29.443 ms 2024-03-30T04:15:14.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1274908 9.34 MB/sec execute 591 sec latency 18.843 ms 2024-03-30T04:15:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:14 smithi193 ceph-mon[79581]: pgmap v416: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:14.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:14 smithi023 ceph-mon[38254]: pgmap v416: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:14 smithi154 ceph-mon[79161]: pgmap v416: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:15.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1275495 9.33 MB/sec execute 592 sec latency 59.907 ms 2024-03-30T04:15:16.186 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:15:16.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1276641 9.32 MB/sec execute 593 sec latency 42.107 ms 2024-03-30T04:15:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 -- 172.21.15.23:0/4209054867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34101910 msgr2=0x7fed34105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 --2- 172.21.15.23:0/4209054867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34101910 0x7fed34105950 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fed20009990 tx=0x7fed20009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 -- 172.21.15.23:0/4209054867 shutdown_connections 2024-03-30T04:15:16.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 --2- 172.21.15.23:0/4209054867 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34106080 0x7fed3410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 --2- 172.21.15.23:0/4209054867 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed34100f60 0x7fed34101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 --2- 172.21.15.23:0/4209054867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34101910 0x7fed34105950 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 -- 172.21.15.23:0/4209054867 >> 172.21.15.23:0/4209054867 conn(0x7fed340fc7c0 msgr2=0x7fed340febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 -- 172.21.15.23:0/4209054867 shutdown_connections 2024-03-30T04:15:16.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.423+0000 7fed38e00700 1 -- 172.21.15.23:0/4209054867 wait complete. 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.424+0000 7fed38e00700 1 Processor -- start 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 -- start start 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34106080 0x7fed34199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed3419dd20 0x7fed3419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3419e7a0 con 0x7fed3419dd20 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3419e910 con 0x7fed34100f60 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed38e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3419ead0 con 0x7fed34106080 2024-03-30T04:15:16.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50624/0 (socket says 172.21.15.23:50624) 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed31400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34106080 0x7fed34199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed31e00700 1 -- 172.21.15.23:0/3493677747 learned_addr learned my addr 172.21.15.23:0/3493677747 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.425+0000 7fed32800700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed3419dd20 0x7fed3419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 -- 172.21.15.23:0/3493677747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34106080 msgr2=0x7fed34199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34106080 0x7fed34199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 -- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed3419dd20 msgr2=0x7fed3419e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed3419dd20 0x7fed3419e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 -- 172.21.15.23:0/3493677747 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed24009520 con 0x7fed34100f60 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed31e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fed2400eb30 tx=0x7fed2400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed2aa00700 1 -- 172.21.15.23:0/3493677747 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed24009880 con 0x7fed34100f60 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed2aa00700 1 -- 172.21.15.23:0/3493677747 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed240044e0 con 0x7fed34100f60 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed200095f0 con 0x7fed34100f60 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed2aa00700 1 -- 172.21.15.23:0/3493677747 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed24003e90 con 0x7fed34100f60 2024-03-30T04:15:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.426+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed341a6320 con 0x7fed34100f60 2024-03-30T04:15:16.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.429+0000 7fed2aa00700 1 -- 172.21.15.23:0/3493677747 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fed24004000 con 0x7fed34100f60 2024-03-30T04:15:16.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed2aa00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 0x7fed180a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed2aa00700 1 -- 172.21.15.23:0/3493677747 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fed24014070 con 0x7fed34100f60 2024-03-30T04:15:16.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed4c91db80 1 Processor -- start 2024-03-30T04:15:16.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed31400700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 0x7fed180a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed4c91db80 1 -- start start 2024-03-30T04:15:16.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.431+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb765a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed4c91db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb75b3990 con 0x555eb765a5d0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed4c91db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb75b1350 con 0x555eb765aef0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed4c91db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb75acc40 con 0x555eb7665df0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50638/0 (socket says 172.21.15.23:50638) 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed32800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 -- 172.21.15.23:0/1288751488 learned_addr learned my addr 172.21.15.23:0/1288751488 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31e00700 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb765a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 0x7fed180a3750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fed3419a990 tx=0x7fed2000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 -- 172.21.15.23:0/1288751488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 msgr2=0x555eb76681e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76681e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 -- 172.21.15.23:0/1288751488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 msgr2=0x555eb765a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb765a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.432+0000 7fed31400700 1 -- 172.21.15.23:0/1288751488 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555eb742e3b0 con 0x555eb765aef0 2024-03-30T04:15:16.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.433+0000 7fed31400700 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fed20009990 tx=0x7fed20020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=926b885c59df21f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.433+0000 7fed1f400700 1 -- 172.21.15.23:0/1288751488 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20005660 con 0x555eb765aef0 2024-03-30T04:15:16.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.433+0000 7fed1f400700 1 -- 172.21.15.23:0/1288751488 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed2001e040 con 0x555eb765aef0 2024-03-30T04:15:16.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.433+0000 7fed1f400700 1 -- 172.21.15.23:0/1288751488 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2000f8e0 con 0x555eb765aef0 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 -- 172.21.15.23:0/1288751488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 msgr2=0x555eb76658b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fed20009990 tx=0x7fed20020000 comp rx=0 tx=0).stop 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 -- 172.21.15.23:0/1288751488 shutdown_connections 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76681e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb765a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1288751488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb76658b0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 -- 172.21.15.23:0/1288751488 >> 172.21.15.23:0/1288751488 conn(0x555eb75ba6a0 msgr2=0x555eb7656a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 -- 172.21.15.23:0/1288751488 shutdown_connections 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.434+0000 7fed4c91db80 1 -- 172.21.15.23:0/1288751488 wait complete. 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.435+0000 7fed4c91db80 1 -- start start 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.435+0000 7fed4c91db80 1 Processor -- start 2024-03-30T04:15:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb7662ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb7548d80 con 0x555eb765a5d0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb7548a20 con 0x555eb765aef0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb74306a0 con 0x555eb7665df0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed4c91db80 10 client.0 _ll_register_callbacks cb 0x555eb7552470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb7662ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54388/0 (socket says 172.21.15.23:54388) 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed31e00700 1 -- 172.21.15.23:0/381558925 learned_addr learned my addr 172.21.15.23:0/381558925 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.436+0000 7fed32800700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 -- 172.21.15.23:0/381558925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 msgr2=0x555eb76630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 -- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 msgr2=0x555eb7662ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb7662ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 -- 172.21.15.23:0/381558925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3410e470 con 0x555eb765a5d0 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31400700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb7662ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed31e00700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fed24095fe0 tx=0x7fed24099cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86ed9528942d3b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed13e00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed240af070 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fed2409e090 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.437+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555eb756d380 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed240a26e0 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 10 client.19050 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555eb73dab60 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed2407f080 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed24099750 con 0x555eb765a5d0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 10 client.19050 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 10 client.19050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 20 client.19050 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb7675900 0x555eb7677cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 4 client.19050 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.438+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555eb75a5700 con 0x555eb7675900 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.439+0000 7fed31400700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb7675900 0x555eb7677cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.439+0000 7fed31400700 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb7675900 0x555eb7677cf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8873a81f2078cd78 server_cookie=73f22c26f09a7773 in_seq=0 out_seq=0 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.439+0000 7fed13e00700 10 client.19050 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed13e00700 1 -- 172.21.15.23:0/381558925 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555eb75a5700 con 0x555eb7675900 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed13e00700 10 client.19050 handle_command_reply: tid=0 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 client.19050 shutdown 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 msgr2=0x555eb76657c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fed24095fe0 tx=0x7fed24099cb0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 shutdown_connections 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb7675900 0x555eb7677cf0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765a5d0 0x555eb76657c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed13e00700 0 client.19050 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 --2- 172.21.15.23:0/381558925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765aef0 0x555eb7662ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 >> 172.21.15.23:0/381558925 conn(0x555eb75ba6a0 msgr2=0x555eb765c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 shutdown_connections 2024-03-30T04:15:16.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.440+0000 7fed4c91db80 1 -- 172.21.15.23:0/381558925 wait complete. 2024-03-30T04:15:16.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.441+0000 7fed4c91db80 20 client.19050 trim_cache size 0 max 16384 2024-03-30T04:15:16.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.459+0000 7fed4c91db80 1 Processor -- start 2024-03-30T04:15:16.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.459+0000 7fed4c91db80 1 -- start start 2024-03-30T04:15:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 0x555eb766ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb7665df0 0x555eb764f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb765d590 con 0x555eb7665df0 2024-03-30T04:15:16.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb76636a0 con 0x555eb765a5d0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed4c91db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb765b850 con 0x555eb765aef0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50656/0 (socket says 172.21.15.23:50656) 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31e00700 1 -- 172.21.15.23:0/1943391744 learned_addr learned my addr 172.21.15.23:0/1943391744 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 0x555eb766ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed32800700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb7665df0 0x555eb764f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 -- 172.21.15.23:0/1943391744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 msgr2=0x555eb76630c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 -- 172.21.15.23:0/1943391744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb7665df0 msgr2=0x555eb764f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb7665df0 0x555eb764f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31400700 1 -- 172.21.15.23:0/1943391744 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555eb7572ea0 con 0x555eb765aef0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.460+0000 7fed31e00700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.461+0000 7fed31400700 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 0x555eb766ca10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fed20000c00 tx=0x7fed20009ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c38575e9d005411 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.461+0000 7fed1fe00700 1 -- 172.21.15.23:0/1943391744 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2003b3c0 con 0x555eb765aef0 2024-03-30T04:15:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.461+0000 7fed1fe00700 1 -- 172.21.15.23:0/1943391744 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed20035040 con 0x555eb765aef0 2024-03-30T04:15:16.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.461+0000 7fed1fe00700 1 -- 172.21.15.23:0/1943391744 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2000f730 con 0x555eb765aef0 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- 172.21.15.23:0/1943391744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 msgr2=0x555eb766ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 0x555eb766ca10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fed20000c00 tx=0x7fed20009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- 172.21.15.23:0/1943391744 shutdown_connections 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb7665df0 0x555eb764f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb76630c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1943391744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb765aef0 0x555eb766ca10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- 172.21.15.23:0/1943391744 >> 172.21.15.23:0/1943391744 conn(0x555eb75ba6a0 msgr2=0x555eb764ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- 172.21.15.23:0/1943391744 shutdown_connections 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- 172.21.15.23:0/1943391744 wait complete. 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 -- start start 2024-03-30T04:15:16.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.462+0000 7fed4c91db80 1 Processor -- start 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 0x555eb766c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb76c2c50 con 0x555eb765aef0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb76c2dc0 con 0x555eb765a5d0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555eb76c2f80 con 0x555eb7665df0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed4c91db80 10 client.0 _ll_register_callbacks cb 0x555eb7552470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50666/0 (socket says 172.21.15.23:50666) 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed31e00700 1 -- 172.21.15.23:0/1212531531 learned_addr learned my addr 172.21.15.23:0/1212531531 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed31400700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 0x555eb766c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.463+0000 7fed32800700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 -- 172.21.15.23:0/1212531531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 msgr2=0x555eb76c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 -- 172.21.15.23:0/1212531531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 msgr2=0x555eb766c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 0x555eb766c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 -- 172.21.15.23:0/1212531531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555eb73dab60 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31400700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 0x555eb766c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed32800700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed31e00700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fed24099f60 tx=0x7fed2400ea80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b84e23aec2dba7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed13e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed240af070 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fed2409e090 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed2001e040 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed2409a4c0 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed4c91db80 10 client.29036 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.464+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x555eb742e3b0 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed2001e040 con 0x555eb765a5d0 2024-03-30T04:15:16.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed240a2250 con 0x555eb765a5d0 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 10 client.29036 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 10 client.29036 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 20 client.29036 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb76ca330 0x555eb76cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 4 client.29036 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.465+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555eb73fbcf0 con 0x555eb76ca330 2024-03-30T04:15:16.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed31400700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb76ca330 0x555eb76cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed31400700 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb76ca330 0x555eb76cc720 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e93c96eb6cdd12b server_cookie=7739d3170dbd8c30 in_seq=0 out_seq=0 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed13e00700 10 client.29036 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed13e00700 1 -- 172.21.15.23:0/1212531531 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555eb73fbcf0 con 0x555eb76ca330 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed13e00700 10 client.29036 handle_command_reply: tid=0 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.466+0000 7fed4c91db80 1 client.29036 shutdown 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 msgr2=0x555eb766c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fed24099f60 tx=0x7fed2400ea80 comp rx=0 tx=0).stop 2024-03-30T04:15:16.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 shutdown_connections 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x555eb76ca330 0x555eb76cc720 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555eb7665df0 0x555eb76c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x555eb765a5d0 0x555eb766c1c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 --2- 172.21.15.23:0/1212531531 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555eb765aef0 0x555eb766c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed13e00700 0 client.29036 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 >> 172.21.15.23:0/1212531531 conn(0x555eb75ba6a0 msgr2=0x555eb7661330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 shutdown_connections 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 1 -- 172.21.15.23:0/1212531531 wait complete. 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed4c91db80 20 client.29036 trim_cache size 0 max 16384 2024-03-30T04:15:16.470 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 msgr2=0x7fed180a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 0x7fed180a3750 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fed3419a990 tx=0x7fed2000b260 comp rx=0 tx=0).stop 2024-03-30T04:15:16.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.467+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 msgr2=0x7fed34199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fed2400eb30 tx=0x7fed2400ee40 comp rx=0 tx=0).stop 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 shutdown_connections 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed3419dd20 0x7fed3419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fed180a1290 0x7fed180a3750 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed34100f60 0x7fed34199440 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 --2- 172.21.15.23:0/3493677747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed34106080 0x7fed34199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 >> 172.21.15.23:0/3493677747 conn(0x7fed340fc7c0 msgr2=0x7fed340feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 shutdown_connections 2024-03-30T04:15:16.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.468+0000 7fed38e00700 1 -- 172.21.15.23:0/3493677747 wait complete. 2024-03-30T04:15:16.485 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:16 smithi023 ceph-mon[38254]: pgmap v417: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:16.487 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:15:16.487 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:15:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:16 smithi193 ceph-mon[79581]: pgmap v417: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:16 smithi154 ceph-mon[79161]: pgmap v417: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 3.9 MiB/s wr, 166 op/s 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 -- 172.21.15.23:0/3401866404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 msgr2=0x7fb7e410d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3401866404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e410d040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009990 tx=0x7fb7d0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 -- 172.21.15.23:0/3401866404 shutdown_connections 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3401866404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e410f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3401866404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e40ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3401866404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e410d040 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 -- 172.21.15.23:0/3401866404 >> 172.21.15.23:0/3401866404 conn(0x7fb7e40fb5f0 msgr2=0x7fb7e40fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 -- 172.21.15.23:0/3401866404 shutdown_connections 2024-03-30T04:15:16.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.702+0000 7fb7eb800700 1 -- 172.21.15.23:0/3401866404 wait complete. 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.703+0000 7fb7eb800700 1 Processor -- start 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 -- start start 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419e2c0 con 0x7fb7e4100380 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419e430 con 0x7fb7e40ff9d0 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7eb800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419e5f0 con 0x7fb7e410d580 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e9800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e3e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e9800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34960/0 (socket says 172.21.15.23:34960) 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e41993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50680/0 (socket says 172.21.15.23:50680) 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.704+0000 7fb7e9800700 1 -- 172.21.15.23:0/3704940028 learned_addr learned my addr 172.21.15.23:0/3704940028 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 -- 172.21.15.23:0/3704940028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 msgr2=0x7fb7e41993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e41993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 -- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 msgr2=0x7fb7e4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:16.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 -- 172.21.15.23:0/3704940028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d00095f0 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e9800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009fd0 tx=0x7fb7dc00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc018070 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7dc00f040 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e419e870 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00cd00 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.705+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e41a6110 con 0x7fb7e410d580 2024-03-30T04:15:16.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.707+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e41087a0 con 0x7fb7e410d580 2024-03-30T04:15:16.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.709+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fb7dc004ad0 con 0x7fb7e410d580 2024-03-30T04:15:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.710+0000 7fb7e1600700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 0x7fb7cc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.710+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fb7dc0ccac0 con 0x7fb7e410d580 2024-03-30T04:15:16.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.710+0000 7fb7e8e00700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 0x7fb7cc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:16.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.711+0000 7fb7e8e00700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 0x7fb7cc0a3750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009a10 tx=0x7fb7d8009190 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:16.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:16.714+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fb7dc095bc0 con 0x7fb7e410d580 2024-03-30T04:15:17.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.014+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7e4069810 con 0x7fb7e410d580 2024-03-30T04:15:17.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.015+0000 7fb7e1600700 1 -- 172.21.15.23:0/3704940028 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+10924 (secure 0 0 0) 0x7fb7dc00ce60 con 0x7fb7e410d580 2024-03-30T04:15:17.015 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:15:17.015 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:05:39.192768+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:15:17.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.018+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 msgr2=0x7fb7cc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.018+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 0x7fb7cc0a3750 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8009a10 tx=0x7fb7d8009190 comp rx=0 tx=0).stop 2024-03-30T04:15:17.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.018+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 msgr2=0x7fb7e419dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.018+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009fd0 tx=0x7fb7dc00ef90 comp rx=0 tx=0).stop 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 shutdown_connections 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7e410d580 0x7fb7e419dc80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fb7cc0a1290 0x7fb7cc0a3750 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7e40ff9d0 0x7fb7e41993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 --2- 172.21.15.23:0/3704940028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4100380 0x7fb7e4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 >> 172.21.15.23:0/3704940028 conn(0x7fb7e40fb5f0 msgr2=0x7fb7e40fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 shutdown_connections 2024-03-30T04:15:17.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.019+0000 7fb7eb800700 1 -- 172.21.15.23:0/3704940028 wait complete. 2024-03-30T04:15:17.020 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 30 2024-03-30T04:15:17.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:15:17.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1278564 9.33 MB/sec execute 594 sec latency 21.752 ms 2024-03-30T04:15:17.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 -- 172.21.15.23:0/712802505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6640692b0 msgr2=0x7ff664069690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 --2- 172.21.15.23:0/712802505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6640692b0 0x7ff664069690 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6580098f0 tx=0x7ff658009c00 comp rx=0 tx=0).stop 2024-03-30T04:15:17.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 -- 172.21.15.23:0/712802505 shutdown_connections 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 --2- 172.21.15.23:0/712802505 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff66410d750 0x7ff66410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 --2- 172.21.15.23:0/712802505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff6640692b0 0x7ff664069690 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 --2- 172.21.15.23:0/712802505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff664069bd0 0x7ff66410d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 -- 172.21.15.23:0/712802505 >> 172.21.15.23:0/712802505 conn(0x7ff664076d50 msgr2=0x7ff664077160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 -- 172.21.15.23:0/712802505 shutdown_connections 2024-03-30T04:15:17.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.250+0000 7ff66c400700 1 -- 172.21.15.23:0/712802505 wait complete. 2024-03-30T04:15:17.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 Processor -- start 2024-03-30T04:15:17.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 -- start start 2024-03-30T04:15:17.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 0x7ff664199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 0x7ff664199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 0x7ff66419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66419e350 con 0x7ff6640692b0 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66419e4c0 con 0x7ff66410d750 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.252+0000 7ff66c400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66419e680 con 0x7ff664069bd0 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 0x7ff664199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 0x7ff664199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 0x7ff664199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34972/0 (socket says 172.21.15.23:34972) 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669000700 1 -- 172.21.15.23:0/3889140626 learned_addr learned my addr 172.21.15.23:0/3889140626 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff66a400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 0x7ff66419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 -- 172.21.15.23:0/3889140626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 msgr2=0x7ff664199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 0x7ff664199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 -- 172.21.15.23:0/3889140626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 msgr2=0x7ff66419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 0x7ff66419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 -- 172.21.15.23:0/3889140626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c009d00 con 0x7ff6640692b0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff66a400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 0x7ff66419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.253+0000 7ff669a00700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 0x7ff664199430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff658009fd0 tx=0x7ff65800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.254+0000 7ff662000700 1 -- 172.21.15.23:0/3889140626 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65801c070 con 0x7ff6640692b0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.254+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff658009520 con 0x7ff6640692b0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.254+0000 7ff662000700 1 -- 172.21.15.23:0/3889140626 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff65800baa0 con 0x7ff6640692b0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.254+0000 7ff662000700 1 -- 172.21.15.23:0/3889140626 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580038e0 con 0x7ff6640692b0 2024-03-30T04:15:17.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.254+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6641a5fc0 con 0x7ff6640692b0 2024-03-30T04:15:17.256 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.257+0000 7ff662000700 1 -- 172.21.15.23:0/3889140626 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff658020030 con 0x7ff6640692b0 2024-03-30T04:15:17.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.258+0000 7ff67ff41b80 1 Processor -- start 2024-03-30T04:15:17.258 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.258+0000 7ff67ff41b80 1 -- start start 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebdaa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff662000700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 0x7ff64c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebcf8990 con 0x5595ebd9f5d0 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebcf6350 con 0x5595ebd9fef0 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebcf1c40 con 0x5595ebdaadf0 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff662000700 1 -- 172.21.15.23:0/3889140626 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff6580cdd90 con 0x7ff6640692b0 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff66a400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff66a400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54466/0 (socket says 172.21.15.23:54466) 2024-03-30T04:15:17.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff66a400700 1 -- 172.21.15.23:0/2174979068 learned_addr learned my addr 172.21.15.23:0/2174979068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff669000700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 0x7ff64c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff669a00700 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebdaa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.259+0000 7ff66a400700 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 -- 172.21.15.23:0/2174979068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 msgr2=0x5595ebdad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 -- 172.21.15.23:0/2174979068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 msgr2=0x5595ebdaa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebdaa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 -- 172.21.15.23:0/2174979068 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595ebb733b0 con 0x5595ebd9f5d0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff669000700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 0x7ff64c0a37d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff66419a980 tx=0x7ff650009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff66a400700 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff65c01a730 tx=0x7ff65c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7312301791dfab92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff656a00700 1 -- 172.21.15.23:0/2174979068 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c00e040 con 0x5595ebd9f5d0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff656a00700 1 -- 172.21.15.23:0/2174979068 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff65c00f040 con 0x5595ebd9f5d0 2024-03-30T04:15:17.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.260+0000 7ff656a00700 1 -- 172.21.15.23:0/2174979068 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65c004310 con 0x5595ebd9f5d0 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2174979068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 msgr2=0x5595ebd9f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff65c01a730 tx=0x7ff65c0099e0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2174979068 shutdown_connections 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebd9f9b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2174979068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebdaa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.261+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2174979068 >> 172.21.15.23:0/2174979068 conn(0x5595ebcff6a0 msgr2=0x5595ebd9ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.262+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2174979068 shutdown_connections 2024-03-30T04:15:17.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.262+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2174979068 wait complete. 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.262+0000 7ff67ff41b80 1 -- start start 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.262+0000 7ff67ff41b80 1 Processor -- start 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 0x5595ebda8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebc8dd80 con 0x5595ebdaadf0 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebc8da20 con 0x5595ebd9f5d0 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebb756a0 con 0x5595ebd9fef0 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.263+0000 7ff67ff41b80 10 client.0 _ll_register_callbacks cb 0x5595ebc97470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:34992/0 (socket says 172.21.15.23:34992) 2024-03-30T04:15:17.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 -- 172.21.15.23:0/379625981 learned_addr learned my addr 172.21.15.23:0/379625981 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff66a400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 0x5595ebda8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669a00700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 -- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 msgr2=0x5595ebdaa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaa7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 -- 172.21.15.23:0/379625981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 msgr2=0x5595ebda8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 0x5595ebda8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 -- 172.21.15.23:0/379625981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c01a3e0 con 0x5595ebd9fef0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669a00700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff66a400700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 0x5595ebda8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff669000700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff65000ebb0 tx=0x7ff6500047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=589256787c8e09c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.264+0000 7ff64b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65001cbc0 con 0x5595ebd9fef0 2024-03-30T04:15:17.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff650023070 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595ebcb2380 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff65001cd20 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff67ff41b80 10 client.29029 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5595ebb1fb60 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.265+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5595ebcb2380 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff65001d410 con 0x5595ebd9fef0 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 10 client.29029 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 10 client.29029 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 20 client.29029 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebdba8e0 0x5595ebdbcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 4 client.29029 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:17.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.266+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595ebcea700 con 0x5595ebdba8e0 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.267+0000 7ff669a00700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebdba8e0 0x5595ebdbcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.268+0000 7ff64b400700 10 client.29029 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.268+0000 7ff669a00700 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebdba8e0 0x5595ebdbcd00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=22415e52da2a21c8 server_cookie=149c39b9e6b606c in_seq=0 out_seq=0 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff64b400700 1 -- 172.21.15.23:0/379625981 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5595ebcea700 con 0x5595ebdba8e0 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff64b400700 10 client.29029 handle_command_reply: tid=0 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 client.29029 shutdown 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 msgr2=0x5595ebda7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff65000ebb0 tx=0x7ff6500047c0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 shutdown_connections 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebdaadf0 0x5595ebda8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebdba8e0 0x5595ebdbcd00 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/379625981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9fef0 0x5595ebda7b40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff64b400700 0 client.29029 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.269+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 >> 172.21.15.23:0/379625981 conn(0x5595ebcff6a0 msgr2=0x5595ebda16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.270+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 shutdown_connections 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.270+0000 7ff67ff41b80 1 -- 172.21.15.23:0/379625981 wait complete. 2024-03-30T04:15:17.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.270+0000 7ff67ff41b80 20 client.29029 trim_cache size 0 max 16384 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 Processor -- start 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 -- start start 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebda8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdaa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebdbd090 con 0x5595ebd9f5d0 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebda6090 con 0x5595ebd9fef0 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.286+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebd9bf20 con 0x5595ebdaadf0 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50732/0 (socket says 172.21.15.23:50732) 2024-03-30T04:15:17.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 -- 172.21.15.23:0/2905611371 learned_addr learned my addr 172.21.15.23:0/2905611371 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff66a400700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdaa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669a00700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebda8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 -- 172.21.15.23:0/2905611371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 msgr2=0x5595ebdaa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdaa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 -- 172.21.15.23:0/2905611371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 msgr2=0x5595ebda8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebda8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 -- 172.21.15.23:0/2905611371 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595ebcb7ea0 con 0x5595ebd9fef0 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff66a400700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdaa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.287+0000 7ff669000700 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff650013010 tx=0x7ff650009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903ba1da3511b681 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.288+0000 7ff657400700 1 -- 172.21.15.23:0/2905611371 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650011ab0 con 0x5595ebd9fef0 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.288+0000 7ff657400700 1 -- 172.21.15.23:0/2905611371 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff650021070 con 0x5595ebd9fef0 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.288+0000 7ff657400700 1 -- 172.21.15.23:0/2905611371 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6500286e0 con 0x5595ebd9fef0 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2905611371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 msgr2=0x5595ebd94f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7ff650013010 tx=0x7ff650009af0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2905611371 shutdown_connections 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebdaadf0 0x5595ebdaa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9f5d0 0x5595ebda8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/2905611371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebd9fef0 0x5595ebd94f70 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2905611371 >> 172.21.15.23:0/2905611371 conn(0x5595ebcff6a0 msgr2=0x5595ebd945a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2905611371 shutdown_connections 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- 172.21.15.23:0/2905611371 wait complete. 2024-03-30T04:15:17.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 -- start start 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.289+0000 7ff67ff41b80 1 Processor -- start 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9fef0 0x5595ebdafd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebdaadf0 0x5595ebe0d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebdb0420 con 0x5595ebd9fef0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebe0dc10 con 0x5595ebdaadf0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595ebe0dd50 con 0x5595ebd9f5d0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff67ff41b80 10 client.0 _ll_register_callbacks cb 0x5595ebc97470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff669a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff669000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9fef0 0x5595ebdafd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff669a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35010/0 (socket says 172.21.15.23:35010) 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff669a00700 1 -- 172.21.15.23:0/1340835735 learned_addr learned my addr 172.21.15.23:0/1340835735 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.290+0000 7ff66a400700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebdaadf0 0x5595ebe0d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 -- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebdaadf0 msgr2=0x5595ebe0d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebdaadf0 0x5595ebe0d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 -- 172.21.15.23:0/1340835735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9fef0 msgr2=0x5595ebdafd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9fef0 0x5595ebdafd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 -- 172.21.15.23:0/1340835735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595ebb733b0 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff669a00700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff658005c10 tx=0x7ff65809a9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f48476ce789d8d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff64b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6580998c0 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff65c00f040 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff65809b1f0 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff658098f30 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff67ff41b80 10 client.29035 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.291+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5595ebb1fb60 con 0x5595ebd9f5d0 2024-03-30T04:15:17.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff65c00f040 con 0x5595ebd9f5d0 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff658099150 con 0x5595ebd9f5d0 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 10 client.29035 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 10 client.29035 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 20 client.29035 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebe0f200 0x5595ebe11620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 4 client.29035 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5595ebb40cf0 con 0x5595ebe0f200 2024-03-30T04:15:17.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.292+0000 7ff669000700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebe0f200 0x5595ebe11620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff64b400700 10 client.29035 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:17.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff669000700 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebe0f200 0x5595ebe11620 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9e909f21d77158d server_cookie=fc70cb1c308e20d1 in_seq=0 out_seq=0 2024-03-30T04:15:17.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff64b400700 1 -- 172.21.15.23:0/1340835735 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5595ebb40cf0 con 0x5595ebe0f200 2024-03-30T04:15:17.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff64b400700 10 client.29035 handle_command_reply: tid=0 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 client.29035 shutdown 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 msgr2=0x5595ebdaf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff658005c10 tx=0x7ff65809a9e0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 shutdown_connections 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5595ebdaadf0 0x5595ebe0d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x5595ebe0f200 0x5595ebe11620 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5595ebd9f5d0 0x5595ebdaf800 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 --2- 172.21.15.23:0/1340835735 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5595ebd9fef0 0x5595ebdafd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff64b400700 0 client.29035 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 >> 172.21.15.23:0/1340835735 conn(0x5595ebcff6a0 msgr2=0x5595ebda61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 shutdown_connections 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.293+0000 7ff67ff41b80 1 -- 172.21.15.23:0/1340835735 wait complete. 2024-03-30T04:15:17.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff67ff41b80 20 client.29035 trim_cache size 0 max 16384 2024-03-30T04:15:17.298 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:17.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 msgr2=0x7ff64c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 0x7ff64c0a37d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff66419a980 tx=0x7ff650009260 comp rx=0 tx=0).stop 2024-03-30T04:15:17.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 msgr2=0x7ff664199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 0x7ff664199430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff658009fd0 tx=0x7ff65800fa00 comp rx=0 tx=0).stop 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 shutdown_connections 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff64c0a1310 0x7ff64c0a37d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff66410d750 0x7ff66419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff6640692b0 0x7ff664199430 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 --2- 172.21.15.23:0/3889140626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff664069bd0 0x7ff664199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 >> 172.21.15.23:0/3889140626 conn(0x7ff664076d50 msgr2=0x7ff66410edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 shutdown_connections 2024-03-30T04:15:17.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.294+0000 7ff66c400700 1 -- 172.21.15.23:0/3889140626 wait complete. 2024-03-30T04:15:17.311 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:17 smithi023 ceph-mon[38254]: osdmap e105: 12 total, 12 up, 12 in 2024-03-30T04:15:17.311 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:17 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3704940028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:17.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:15:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:17 smithi193 ceph-mon[79581]: osdmap e105: 12 total, 12 up, 12 in 2024-03-30T04:15:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:17 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3704940028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:17.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 -- 172.21.15.23:0/2378502261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 msgr2=0x7f544010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 --2- 172.21.15.23:0/2378502261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f544010dc00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f543800b0a0 tx=0x7f543800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 -- 172.21.15.23:0/2378502261 shutdown_connections 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 --2- 172.21.15.23:0/2378502261 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f544010dc00 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 --2- 172.21.15.23:0/2378502261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f5440101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 --2- 172.21.15.23:0/2378502261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440101910 0x7f5440105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 -- 172.21.15.23:0/2378502261 >> 172.21.15.23:0/2378502261 conn(0x7f54400fc7c0 msgr2=0x7f54400febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 -- 172.21.15.23:0/2378502261 shutdown_connections 2024-03-30T04:15:17.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.524+0000 7f5448200700 1 -- 172.21.15.23:0/2378502261 wait complete. 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 Processor -- start 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 -- start start 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f54400735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440074d20 0x7f5440073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54401a9f80 con 0x7f5440106080 2024-03-30T04:15:17.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54401aa0f0 con 0x7f5440100f60 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.525+0000 7f5448200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54401aa2b0 con 0x7f5440074d20 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50764/0 (socket says 172.21.15.23:50764) 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 -- 172.21.15.23:0/1234727636 learned_addr learned my addr 172.21.15.23:0/1234727636 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5444e00700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f54400735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5446200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440074d20 0x7f5440073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 -- 172.21.15.23:0/1234727636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440074d20 msgr2=0x7f5440073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440074d20 0x7f5440073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 -- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 msgr2=0x7f54400735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f54400735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 -- 172.21.15.23:0/1234727636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5438009d00 con 0x7f5440100f60 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f5445800700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f543400e9a0 tx=0x7f543400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.526+0000 7f543e000700 1 -- 172.21.15.23:0/1234727636 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434009790 con 0x7f5440100f60 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.527+0000 7f543e000700 1 -- 172.21.15.23:0/1234727636 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5434010430 con 0x7f5440100f60 2024-03-30T04:15:17.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.527+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54401aa590 con 0x7f5440100f60 2024-03-30T04:15:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.527+0000 7f543e000700 1 -- 172.21.15.23:0/1234727636 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434018680 con 0x7f5440100f60 2024-03-30T04:15:17.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.527+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54401aaaa0 con 0x7f5440100f60 2024-03-30T04:15:17.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.530+0000 7f543e000700 1 -- 172.21.15.23:0/1234727636 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f54340105a0 con 0x7f5440100f60 2024-03-30T04:15:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.531+0000 7f543e000700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 0x7f54280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.531+0000 7f543e000700 1 -- 172.21.15.23:0/1234727636 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5434014070 con 0x7f5440100f60 2024-03-30T04:15:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.532+0000 7f5444e00700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 0x7f54280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.532+0000 7f545bde1b80 1 Processor -- start 2024-03-30T04:15:17.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.532+0000 7f545bde1b80 1 -- start start 2024-03-30T04:15:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.532+0000 7f5444e00700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 0x7f54280a3750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5440074790 tx=0x7f542c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9c3df0 0x56374b9c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b911990 con 0x56374b9b8ef0 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b90f350 con 0x56374b9c3df0 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f545bde1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b90ac40 con 0x56374b9b85d0 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5446200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9c3df0 0x56374b9c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54524/0 (socket says 172.21.15.23:54524) 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 -- 172.21.15.23:0/3959607853 learned_addr learned my addr 172.21.15.23:0/3959607853 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5445800700 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 -- 172.21.15.23:0/3959607853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 msgr2=0x56374b9b89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9b89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 -- 172.21.15.23:0/3959607853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9c3df0 msgr2=0x56374b9c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9c3df0 0x56374b9c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.533+0000 7f5444e00700 1 -- 172.21.15.23:0/3959607853 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56374b78c3b0 con 0x56374b9b8ef0 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.534+0000 7f5445800700 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.534+0000 7f5444e00700 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f542c011cd0 tx=0x7f542c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc24046770c7018 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.534+0000 7f5432a00700 1 -- 172.21.15.23:0/3959607853 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c004030 con 0x56374b9b8ef0 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.534+0000 7f5432a00700 1 -- 172.21.15.23:0/3959607853 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f542c00e840 con 0x56374b9b8ef0 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.534+0000 7f5432a00700 1 -- 172.21.15.23:0/3959607853 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c003b20 con 0x56374b9b8ef0 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 -- 172.21.15.23:0/3959607853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 msgr2=0x56374b9c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f542c011cd0 tx=0x7f542c014000 comp rx=0 tx=0).stop 2024-03-30T04:15:17.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 -- 172.21.15.23:0/3959607853 shutdown_connections 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9c3df0 0x56374b9c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9b89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 --2- 172.21.15.23:0/3959607853 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9b8ef0 0x56374b9c38b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 -- 172.21.15.23:0/3959607853 >> 172.21.15.23:0/3959607853 conn(0x56374b9186a0 msgr2=0x56374b9b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 -- 172.21.15.23:0/3959607853 shutdown_connections 2024-03-30T04:15:17.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.535+0000 7f545bde1b80 1 -- 172.21.15.23:0/3959607853 wait complete. 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.536+0000 7f545bde1b80 1 -- start start 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.536+0000 7f545bde1b80 1 Processor -- start 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b8a6d80 con 0x56374b9c3df0 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b8a6a20 con 0x56374b9b8ef0 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b78e6a0 con 0x56374b9b85d0 2024-03-30T04:15:17.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f545bde1b80 10 client.0 _ll_register_callbacks cb 0x56374b8b0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5444e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5444e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50778/0 (socket says 172.21.15.23:50778) 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5444e00700 1 -- 172.21.15.23:0/569583097 learned_addr learned my addr 172.21.15.23:0/569583097 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5446200700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 -- 172.21.15.23:0/569583097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 msgr2=0x56374b9c0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 -- 172.21.15.23:0/569583097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 msgr2=0x56374b9c12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5445800700 1 -- 172.21.15.23:0/569583097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542c017070 con 0x56374b9b85d0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5444e00700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.537+0000 7f5446200700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5445800700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c38a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f54340ca5f0 tx=0x7f543409a030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76781d4cb5b8a106 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5427400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54340996a0 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f54340a2430 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56374b8cb380 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54340a1fa0 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f545bde1b80 10 client.29047 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56374b738b60 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.538+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56374b8cb380 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5434094bc0 con 0x56374b9b85d0 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 10 client.29047 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 10 client.29047 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 20 client.29047 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374b9d3a60 0x56374b9d5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 4 client.29047 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56374b903700 con 0x56374b9d3a60 2024-03-30T04:15:17.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.539+0000 7f5444e00700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374b9d3a60 0x56374b9d5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.540+0000 7f5444e00700 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374b9d3a60 0x56374b9d5e80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fb878f6913964524 server_cookie=2ff3a5730a1860ba in_seq=0 out_seq=0 2024-03-30T04:15:17.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.540+0000 7f5427400700 10 client.29047 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.540+0000 7f5427400700 1 -- 172.21.15.23:0/569583097 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56374b903700 con 0x56374b9d3a60 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.540+0000 7f5427400700 10 client.29047 handle_command_reply: tid=0 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 client.29047 shutdown 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 msgr2=0x56374b9c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c38a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f54340ca5f0 tx=0x7f543409a030 comp rx=0 tx=0).stop 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 shutdown_connections 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374b9d3a60 0x56374b9d5e80 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c38a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 --2- 172.21.15.23:0/569583097 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f5427400700 0 client.29047 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 >> 172.21.15.23:0/569583097 conn(0x56374b9186a0 msgr2=0x56374b9ba650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 shutdown_connections 2024-03-30T04:15:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 1 -- 172.21.15.23:0/569583097 wait complete. 2024-03-30T04:15:17.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.541+0000 7f545bde1b80 20 client.29047 trim_cache size 0 max 16384 2024-03-30T04:15:17.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 Processor -- start 2024-03-30T04:15:17.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 -- start start 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9bb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9d6250 con 0x56374b9c3df0 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9bf0e0 con 0x56374b9b8ef0 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.556+0000 7f545bde1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9b83c0 con 0x56374b9b85d0 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5446200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9bb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50792/0 (socket says 172.21.15.23:50792) 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 -- 172.21.15.23:0/2147103835 learned_addr learned my addr 172.21.15.23:0/2147103835 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5445800700 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 -- 172.21.15.23:0/2147103835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 msgr2=0x56374b9c0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 -- 172.21.15.23:0/2147103835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 msgr2=0x56374b9bb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9bb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 -- 172.21.15.23:0/2147103835 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56374b8d0ea0 con 0x56374b9b8ef0 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5445800700 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5444e00700 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f542c00efd0 tx=0x7f542c029040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97975fa647b640ab server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.557+0000 7f5433400700 1 -- 172.21.15.23:0/2147103835 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c02c040 con 0x56374b9b8ef0 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f5433400700 1 -- 172.21.15.23:0/2147103835 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f542c0056e0 con 0x56374b9b8ef0 2024-03-30T04:15:17.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f5433400700 1 -- 172.21.15.23:0/2147103835 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c0036f0 con 0x56374b9b8ef0 2024-03-30T04:15:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 -- 172.21.15.23:0/2147103835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 msgr2=0x56374b9c1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f542c00efd0 tx=0x7f542c029040 comp rx=0 tx=0).stop 2024-03-30T04:15:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 -- 172.21.15.23:0/2147103835 shutdown_connections 2024-03-30T04:15:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9bb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b85d0 0x56374b9c0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 --2- 172.21.15.23:0/2147103835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b8ef0 0x56374b9c1270 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 -- 172.21.15.23:0/2147103835 >> 172.21.15.23:0/2147103835 conn(0x56374b9186a0 msgr2=0x56374b9c0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 -- 172.21.15.23:0/2147103835 shutdown_connections 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.558+0000 7f545bde1b80 1 -- 172.21.15.23:0/2147103835 wait complete. 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.559+0000 7f545bde1b80 1 -- start start 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.559+0000 7f545bde1b80 1 Processor -- start 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.559+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 0x56374b9ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9c8f70 con 0x56374b9c3df0 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9c90e0 con 0x56374b9b85d0 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56374b9c92a0 con 0x56374b9b8ef0 2024-03-30T04:15:17.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f545bde1b80 10 client.0 _ll_register_callbacks cb 0x56374b8b0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5445800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5445800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50796/0 (socket says 172.21.15.23:50796) 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5445800700 1 -- 172.21.15.23:0/1117026110 learned_addr learned my addr 172.21.15.23:0/1117026110 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5446200700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 0x56374b9ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 -- 172.21.15.23:0/1117026110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 msgr2=0x56374b9ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 -- 172.21.15.23:0/1117026110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 msgr2=0x56374b9c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5444e00700 1 -- 172.21.15.23:0/1117026110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56374b738b60 con 0x56374b9b8ef0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5445800700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.560+0000 7f5446200700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5444e00700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 0x56374b9ca840 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f542c005950 tx=0x7f542c005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1088d998cee883d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5427400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f542c02c070 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f542c031cf0 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56374b759cf0 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f542c005220 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f545bde1b80 10 client.29053 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x56374b78c3b0 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.561+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56374b759cf0 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f542c003b60 con 0x56374b9b8ef0 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 10 client.29053 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 10 client.29053 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 20 client.29053 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374ba28560 0x56374ba2a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 4 client.29053 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56374b78ece0 con 0x56374ba28560 2024-03-30T04:15:17.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5445800700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374ba28560 0x56374ba2a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5445800700 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374ba28560 0x56374ba2a950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1bf1036c686d6b2 server_cookie=c2d8ba7c3a16b432 in_seq=0 out_seq=0 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5427400700 10 client.29053 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5427400700 1 -- 172.21.15.23:0/1117026110 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56374b78ece0 con 0x56374ba28560 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.562+0000 7f5427400700 10 client.29053 handle_command_reply: tid=0 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 client.29053 shutdown 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 msgr2=0x56374b9ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 0x56374b9ca840 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f542c005950 tx=0x7f542c005d50 comp rx=0 tx=0).stop 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 shutdown_connections 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x56374ba28560 0x56374ba2a950 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56374b9c3df0 0x56374b9c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f5427400700 0 client.29053 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56374b9b85d0 0x56374b9ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 --2- 172.21.15.23:0/1117026110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56374b9b8ef0 0x56374b9ca840 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 >> 172.21.15.23:0/1117026110 conn(0x56374b9186a0 msgr2=0x56374b9ad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 shutdown_connections 2024-03-30T04:15:17.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 1 -- 172.21.15.23:0/1117026110 wait complete. 2024-03-30T04:15:17.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.563+0000 7f545bde1b80 20 client.29053 trim_cache size 0 max 16384 2024-03-30T04:15:17.567 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 msgr2=0x7f54280a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 0x7f54280a3750 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5440074790 tx=0x7f542c009260 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 msgr2=0x7f54400730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f543400e9a0 tx=0x7f543400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 shutdown_connections 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5440074d20 0x7f5440073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f54280a1290 0x7f54280a3750 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5440100f60 0x7f54400730b0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 --2- 172.21.15.23:0/1234727636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5440106080 0x7f54400735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 >> 172.21.15.23:0/1234727636 conn(0x7f54400fc7c0 msgr2=0x7f54400fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 shutdown_connections 2024-03-30T04:15:17.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.564+0000 7f5448200700 1 -- 172.21.15.23:0/1234727636 wait complete. 2024-03-30T04:15:17.581 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:15:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:17 smithi154 ceph-mon[79161]: osdmap e105: 12 total, 12 up, 12 in 2024-03-30T04:15:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:17 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3704940028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:17.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 -- 172.21.15.23:0/1769575751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab281060a0 msgr2=0x7fab2810dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 --2- 172.21.15.23:0/1769575751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab281060a0 0x7fab2810dc20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fab20009fc0 tx=0x7fab2000b330 comp rx=0 tx=0).stop 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 -- 172.21.15.23:0/1769575751 shutdown_connections 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 --2- 172.21.15.23:0/1769575751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab281060a0 0x7fab2810dc20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 --2- 172.21.15.23:0/1769575751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab280692e0 0x7fab280696c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 --2- 172.21.15.23:0/1769575751 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab28069c90 0x7fab28105970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 -- 172.21.15.23:0/1769575751 >> 172.21.15.23:0/1769575751 conn(0x7fab28076d50 msgr2=0x7fab28077160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 -- 172.21.15.23:0/1769575751 shutdown_connections 2024-03-30T04:15:17.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.790+0000 7fab2ea00700 1 -- 172.21.15.23:0/1769575751 wait complete. 2024-03-30T04:15:17.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 Processor -- start 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 -- start start 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 0x7fab2819e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2819e840 con 0x7fab280692e0 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2819e9b0 con 0x7fab281060a0 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.791+0000 7fab2ea00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2819eb70 con 0x7fab2819ddc0 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50818/0 (socket says 172.21.15.23:50818) 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54572/0 (socket says 172.21.15.23:54572) 2024-03-30T04:15:17.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 -- 172.21.15.23:0/2611462362 learned_addr learned my addr 172.21.15.23:0/2611462362 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab2ca00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 0x7fab2819e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 -- 172.21.15.23:0/2611462362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 msgr2=0x7fab2819e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 0x7fab2819e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 -- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 msgr2=0x7fab28199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 -- 172.21.15.23:0/2611462362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab20009c40 con 0x7fab281060a0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab2ca00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 0x7fab2819e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27e00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.792+0000 7fab27400700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e970 tx=0x7fab1400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.793+0000 7fab24c00700 1 -- 172.21.15.23:0/2611462362 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14009790 con 0x7fab281060a0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.793+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab281a60d0 con 0x7fab281060a0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.793+0000 7fab24c00700 1 -- 172.21.15.23:0/2611462362 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab1400eeb0 con 0x7fab281060a0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.793+0000 7fab24c00700 1 -- 172.21.15.23:0/2611462362 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab140055c0 con 0x7fab281060a0 2024-03-30T04:15:17.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.793+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab281a65b0 con 0x7fab281060a0 2024-03-30T04:15:17.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.796+0000 7fab24c00700 1 -- 172.21.15.23:0/2611462362 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fab14004750 con 0x7fab281060a0 2024-03-30T04:15:17.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 Processor -- start 2024-03-30T04:15:17.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 -- start start 2024-03-30T04:15:17.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab24c00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 0x7fab0c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d768ef0 0x55ab0d7738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d6c1990 con 0x55ab0d773df0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d6bf350 con 0x55ab0d768ef0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab42527b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d6bac40 con 0x55ab0d7685d0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.798+0000 7fab24c00700 1 -- 172.21.15.23:0/2611462362 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab14014070 con 0x7fab281060a0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab2ca00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 0x7fab0c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35096/0 (socket says 172.21.15.23:35096) 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 -- 172.21.15.23:0/645983762 learned_addr learned my addr 172.21.15.23:0/645983762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27400700 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d768ef0 0x55ab0d7738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab2ca00700 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 -- 172.21.15.23:0/645983762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d768ef0 msgr2=0x55ab0d7738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d768ef0 0x55ab0d7738b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 -- 172.21.15.23:0/645983762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 msgr2=0x55ab0d7761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7761e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 -- 172.21.15.23:0/645983762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab0d53c3b0 con 0x55ab0d7685d0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.799+0000 7fab27e00700 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fab180115a0 tx=0x7fab180118b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9766d3516206b56 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab2ca00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 0x7fab0c0a3750 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fab20009790 tx=0x7fab20009720 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab1d200700 1 -- 172.21.15.23:0/645983762 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab18004030 con 0x55ab0d7685d0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab1d200700 1 -- 172.21.15.23:0/645983762 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab18003680 con 0x55ab0d7685d0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab1d200700 1 -- 172.21.15.23:0/645983762 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab180196f0 con 0x55ab0d7685d0 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab42527b80 1 -- 172.21.15.23:0/645983762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 msgr2=0x55ab0d7689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.800+0000 7fab42527b80 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fab180115a0 tx=0x7fab180118b0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 -- 172.21.15.23:0/645983762 shutdown_connections 2024-03-30T04:15:17.802 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d7685d0 0x55ab0d7689b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 --2- 172.21.15.23:0/645983762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d768ef0 0x55ab0d7738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 -- 172.21.15.23:0/645983762 >> 172.21.15.23:0/645983762 conn(0x55ab0d6c86a0 msgr2=0x55ab0d764a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 -- 172.21.15.23:0/645983762 shutdown_connections 2024-03-30T04:15:17.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 -- 172.21.15.23:0/645983762 wait complete. 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.801+0000 7fab42527b80 1 -- start start 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 Processor -- start 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d656d80 con 0x55ab0d773df0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d656a20 con 0x55ab0d7685d0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d53e6a0 con 0x55ab0d768ef0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab42527b80 10 client.0 _ll_register_callbacks cb 0x55ab0d660470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.802+0000 7fab27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab2ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50838/0 (socket says 172.21.15.23:50838) 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35108/0 (socket says 172.21.15.23:35108) 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 -- 172.21.15.23:0/1757332609 learned_addr learned my addr 172.21.15.23:0/1757332609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 -- 172.21.15.23:0/1757332609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 msgr2=0x55ab0d770aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 -- 172.21.15.23:0/1757332609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 msgr2=0x55ab0d7710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 -- 172.21.15.23:0/1757332609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab18009d90 con 0x55ab0d7685d0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27400700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab2ca00700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab27e00700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fab18003d90 tx=0x7fab18004240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf8d99b214bf772 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab11600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.803+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab18003df0 con 0x55ab0d7685d0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab0d67b380 con 0x55ab0d7685d0 2024-03-30T04:15:17.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab18005040 con 0x55ab0d7685d0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab18005200 con 0x55ab0d7685d0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 10 client.29084 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ab0d4e8b60 con 0x55ab0d7685d0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ab0d67b380 con 0x55ab0d7685d0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fab18019450 con 0x55ab0d7685d0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 10 client.29084 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 10 client.29084 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 20 client.29084 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7837f0 0x55ab0d785c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 4 client.29084 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.804+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab0d6b3700 con 0x55ab0d7837f0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.805+0000 7fab27400700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7837f0 0x55ab0d785c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.805+0000 7fab27400700 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7837f0 0x55ab0d785c10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=708fd954a5cccba1 server_cookie=569778508a5b5830 in_seq=0 out_seq=0 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.805+0000 7fab11600700 10 client.29084 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:17.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.806+0000 7fab11600700 1 -- 172.21.15.23:0/1757332609 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ab0d6b3700 con 0x55ab0d7837f0 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.806+0000 7fab11600700 10 client.29084 handle_command_reply: tid=0 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.806+0000 7fab42527b80 1 client.29084 shutdown 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 msgr2=0x55ab0d7737b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fab18003d90 tx=0x7fab18004240 comp rx=0 tx=0).stop 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 shutdown_connections 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7837f0 0x55ab0d785c10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d773df0 0x55ab0d7710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d7737b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 --2- 172.21.15.23:0/1757332609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d770aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab11600700 0 client.29084 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 >> 172.21.15.23:0/1757332609 conn(0x55ab0d6c86a0 msgr2=0x55ab0d76a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 shutdown_connections 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 1 -- 172.21.15.23:0/1757332609 wait complete. 2024-03-30T04:15:17.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.807+0000 7fab42527b80 20 client.29084 trim_cache size 0 max 16384 2024-03-30T04:15:17.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.823+0000 7fab42527b80 1 Processor -- start 2024-03-30T04:15:17.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 -- start start 2024-03-30T04:15:17.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d771070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d768ef0 0x55ab0d77aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d7692d0 con 0x55ab0d768ef0 2024-03-30T04:15:17.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d785fa0 con 0x55ab0d7685d0 2024-03-30T04:15:17.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab42527b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d77afb0 con 0x55ab0d773df0 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab2ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab2ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35120/0 (socket says 172.21.15.23:35120) 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab2ca00700 1 -- 172.21.15.23:0/4161922306 learned_addr learned my addr 172.21.15.23:0/4161922306 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab27400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d768ef0 0x55ab0d77aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.824+0000 7fab27e00700 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d771070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 -- 172.21.15.23:0/4161922306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 msgr2=0x55ab0d771070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d771070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 -- 172.21.15.23:0/4161922306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d768ef0 msgr2=0x55ab0d77aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.826 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d768ef0 0x55ab0d77aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 -- 172.21.15.23:0/4161922306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab0d680ea0 con 0x55ab0d773df0 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab2ca00700 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fab20004cd0 tx=0x7fab2001e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2b243281ffb443 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab1dc00700 1 -- 172.21.15.23:0/4161922306 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2000e050 con 0x55ab0d773df0 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab1dc00700 1 -- 172.21.15.23:0/4161922306 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab2002d070 con 0x55ab0d773df0 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.825+0000 7fab1dc00700 1 -- 172.21.15.23:0/4161922306 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200054b0 con 0x55ab0d773df0 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- 172.21.15.23:0/4161922306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 msgr2=0x55ab0d75df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fab20004cd0 tx=0x7fab2001e000 comp rx=0 tx=0).stop 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- 172.21.15.23:0/4161922306 shutdown_connections 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d773df0 0x55ab0d75df70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d7685d0 0x55ab0d771070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 --2- 172.21.15.23:0/4161922306 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d768ef0 0x55ab0d77aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- 172.21.15.23:0/4161922306 >> 172.21.15.23:0/4161922306 conn(0x55ab0d6c86a0 msgr2=0x55ab0d75d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- 172.21.15.23:0/4161922306 shutdown_connections 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- 172.21.15.23:0/4161922306 wait complete. 2024-03-30T04:15:17.827 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.826+0000 7fab42527b80 1 -- start start 2024-03-30T04:15:17.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 Processor -- start 2024-03-30T04:15:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 0x55ab0d77a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d779190 con 0x55ab0d7685d0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d779300 con 0x55ab0d773df0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab0d7794c0 con 0x55ab0d768ef0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.827+0000 7fab42527b80 10 client.0 _ll_register_callbacks cb 0x55ab0d660470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 0x55ab0d77a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:35122/0 (socket says 172.21.15.23:35122) 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab2ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50864/0 (socket says 172.21.15.23:50864) 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 -- 172.21.15.23:0/2792150165 learned_addr learned my addr 172.21.15.23:0/2792150165 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 -- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 msgr2=0x55ab0d778a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 -- 172.21.15.23:0/2792150165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 msgr2=0x55ab0d77a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 0x55ab0d77a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 -- 172.21.15.23:0/2792150165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2810e460 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab2ca00700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27e00700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 0x55ab0d77a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab27400700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c5b0 tx=0x7fab1400cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe19a3bd923765b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab11600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.828+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14099f90 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab1405fe40 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab18003680 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab1409ddc0 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 10 client.29071 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55ab0d540cd0 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab1407c020 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fab1409a1f0 con 0x55ab0d768ef0 2024-03-30T04:15:17.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 10 client.29071 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 10 client.29071 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 20 client.29071 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7e1180 0x55ab0d7e35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 4 client.29071 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ab0d509cf0 con 0x55ab0d7e1180 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.829+0000 7fab27e00700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7e1180 0x55ab0d7e35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab27e00700 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7e1180 0x55ab0d7e35a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f25997db669f02ad server_cookie=9fa3c7a474835258 in_seq=0 out_seq=0 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab11600700 10 client.29071 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab11600700 1 -- 172.21.15.23:0/2792150165 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ab0d509cf0 con 0x55ab0d7e1180 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab11600700 10 client.29071 handle_command_reply: tid=0 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 client.29071 shutdown 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 msgr2=0x55ab0d77a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c5b0 tx=0x7fab1400cf10 comp rx=0 tx=0).stop 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 shutdown_connections 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ab0d7e1180 0x55ab0d7e35a0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ab0d773df0 0x55ab0d778a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ab0d7685d0 0x55ab0d77a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 --2- 172.21.15.23:0/2792150165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ab0d768ef0 0x55ab0d77a810 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 >> 172.21.15.23:0/2792150165 conn(0x55ab0d6c86a0 msgr2=0x55ab0d76f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab11600700 0 client.29071 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 shutdown_connections 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.830+0000 7fab42527b80 1 -- 172.21.15.23:0/2792150165 wait complete. 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab42527b80 20 client.29071 trim_cache size 0 max 16384 2024-03-30T04:15:17.835 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:17.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 msgr2=0x7fab0c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 0x7fab0c0a3750 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fab20009790 tx=0x7fab20009720 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 msgr2=0x7fab28199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e970 tx=0x7fab1400ed30 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 shutdown_connections 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab2819ddc0 0x7fab2819e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab280692e0 0x7fab28199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab0c0a1290 0x7fab0c0a3750 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 --2- 172.21.15.23:0/2611462362 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab281060a0 0x7fab28199a70 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 >> 172.21.15.23:0/2611462362 conn(0x7fab28076d50 msgr2=0x7fab2810c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.831+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 shutdown_connections 2024-03-30T04:15:17.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:17.832+0000 7fab2ea00700 1 -- 172.21.15.23:0/2611462362 wait complete. 2024-03-30T04:15:17.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 -- 172.21.15.23:0/2216135724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 msgr2=0x7fd51c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 --2- 172.21.15.23:0/2216135724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c0735c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009990 tx=0x7fd50c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 -- 172.21.15.23:0/2216135724 shutdown_connections 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 --2- 172.21.15.23:0/2216135724 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c10d530 0x7fd51c10f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 --2- 172.21.15.23:0/2216135724 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c0735c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 --2- 172.21.15.23:0/2216135724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c073b00 0x7fd51c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.079+0000 7fd524800700 1 -- 172.21.15.23:0/2216135724 >> 172.21.15.23:0/2216135724 conn(0x7fd51c0fc7c0 msgr2=0x7fd51c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.080+0000 7fd524800700 1 -- 172.21.15.23:0/2216135724 shutdown_connections 2024-03-30T04:15:18.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.080+0000 7fd524800700 1 -- 172.21.15.23:0/2216135724 wait complete. 2024-03-30T04:15:18.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 Processor -- start 2024-03-30T04:15:18.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 -- start start 2024-03-30T04:15:18.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c073b00 0x7fd51c199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 0x7fd51c19dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19e310 con 0x7fd51c0731e0 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19e480 con 0x7fd51c10d530 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd524800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19e640 con 0x7fd51c073b00 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd521e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd521400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c073b00 0x7fd51c199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd522800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 0x7fd51c19dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd521e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55442/0 (socket says 172.21.15.23:55442) 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.081+0000 7fd521e00700 1 -- 172.21.15.23:0/2635630951 learned_addr learned my addr 172.21.15.23:0/2635630951 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 -- 172.21.15.23:0/2635630951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c073b00 msgr2=0x7fd51c199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c073b00 0x7fd51c199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 -- 172.21.15.23:0/2635630951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 msgr2=0x7fd51c19dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 0x7fd51c19dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 -- 172.21.15.23:0/2635630951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0095f0 con 0x7fd51c0731e0 2024-03-30T04:15:18.083 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd522800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 0x7fd51c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd521e00700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd50c000c00 tx=0x7fd50c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd516a00700 1 -- 172.21.15.23:0/2635630951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c01d070 con 0x7fd51c0731e0 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd516a00700 1 -- 172.21.15.23:0/2635630951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd50c022470 con 0x7fd51c0731e0 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd516a00700 1 -- 172.21.15.23:0/2635630951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c017650 con 0x7fd51c0731e0 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.082+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c19e8c0 con 0x7fd51c0731e0 2024-03-30T04:15:18.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.083+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c1a6160 con 0x7fd51c0731e0 2024-03-30T04:15:18.085 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.086+0000 7fd516a00700 1 -- 172.21.15.23:0/2635630951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd50c0225e0 con 0x7fd51c0731e0 2024-03-30T04:15:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 Processor -- start 2024-03-30T04:15:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 -- start start 2024-03-30T04:15:18.087 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd516a00700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 0x7fd5040a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd516a00700 1 -- 172.21.15.23:0/2635630951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd50c0ce460 con 0x7fd51c0731e0 2024-03-30T04:15:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 0x55cbbc95e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.088 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd522800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 0x7fd5040a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc969df0 0x55cbbc96c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc8b7990 con 0x55cbbc95e5d0 2024-03-30T04:15:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc8b5350 con 0x55cbbc95eef0 2024-03-30T04:15:18.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.088+0000 7fd538418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc8b0c40 con 0x55cbbc969df0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50942/0 (socket says 172.21.15.23:50942) 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 -- 172.21.15.23:0/3561605218 learned_addr learned my addr 172.21.15.23:0/3561605218 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521e00700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 0x55cbbc95e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd522800700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc969df0 0x55cbbc96c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 -- 172.21.15.23:0/3561605218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc969df0 msgr2=0x55cbbc96c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc969df0 0x55cbbc96c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 -- 172.21.15.23:0/3561605218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 msgr2=0x55cbbc95e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 0x55cbbc95e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 -- 172.21.15.23:0/3561605218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbbc7323b0 con 0x55cbbc95eef0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521e00700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 0x55cbbc95e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd522800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 0x7fd5040a37d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd518007a80 tx=0x7fd518007e40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd521400700 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd510009d20 tx=0x7fd510011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abd1be3dff00893 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.089+0000 7fd50b000700 1 -- 172.21.15.23:0/3561605218 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd510019630 con 0x55cbbc95eef0 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd50b000700 1 -- 172.21.15.23:0/3561605218 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd510019c50 con 0x55cbbc95eef0 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd50b000700 1 -- 172.21.15.23:0/3561605218 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51000ec30 con 0x55cbbc95eef0 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 -- 172.21.15.23:0/3561605218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 msgr2=0x55cbbc9698b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd510009d20 tx=0x7fd510011720 comp rx=0 tx=0).stop 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 -- 172.21.15.23:0/3561605218 shutdown_connections 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc969df0 0x55cbbc96c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95e5d0 0x55cbbc95e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 --2- 172.21.15.23:0/3561605218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc9698b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 -- 172.21.15.23:0/3561605218 >> 172.21.15.23:0/3561605218 conn(0x55cbbc8be6a0 msgr2=0x55cbbc95aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.090+0000 7fd538418b80 1 -- 172.21.15.23:0/3561605218 shutdown_connections 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.091+0000 7fd538418b80 1 -- 172.21.15.23:0/3561605218 wait complete. 2024-03-30T04:15:18.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.091+0000 7fd538418b80 1 -- start start 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.091+0000 7fd538418b80 1 Processor -- start 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc969df0 0x55cbbc9670c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc84cd80 con 0x55cbbc969df0 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc84ca20 con 0x55cbbc95eef0 2024-03-30T04:15:18.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc7346a0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd538418b80 10 client.0 _ll_register_callbacks cb 0x55cbbc856470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd522800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc969df0 0x55cbbc9670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd521400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd521e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd521400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50958/0 (socket says 172.21.15.23:50958) 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd521400700 1 -- 172.21.15.23:0/455189792 learned_addr learned my addr 172.21.15.23:0/455189792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.092+0000 7fd521e00700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60678/0 (socket says 172.21.15.23:60678) 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 -- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 msgr2=0x55cbbc966a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 -- 172.21.15.23:0/455189792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc969df0 msgr2=0x55cbbc9670c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc969df0 0x55cbbc9670c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 -- 172.21.15.23:0/455189792 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd510009980 con 0x55cbbc95e5d0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521400700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd521e00700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0cb770 tx=0x7fd50c09b130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=875eb356972770e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd4ff400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c09ac40 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd50c09b9e0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cbbc871380 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd50c09b2e0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd538418b80 10 client.29083 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.093+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cbbc6deb60 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd50c07e020 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd50c09a7d0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 10 client.29083 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 10 client.29083 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 20 client.29083 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9795e0 0x55cbbc97ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 4 client.29083 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cbbc8a9700 con 0x55cbbc9795e0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.094+0000 7fd521400700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9795e0 0x55cbbc97ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.095+0000 7fd4ff400700 10 client.29083 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.095+0000 7fd521400700 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9795e0 0x55cbbc97ba00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d53350cf6332f5a9 server_cookie=75ee562f8d00f971 in_seq=0 out_seq=0 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.095+0000 7fd4ff400700 1 -- 172.21.15.23:0/455189792 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55cbbc8a9700 con 0x55cbbc9795e0 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.095+0000 7fd4ff400700 10 client.29083 handle_command_reply: tid=0 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 client.29083 shutdown 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 msgr2=0x55cbbc9697a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0cb770 tx=0x7fd50c09b130 comp rx=0 tx=0).stop 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 shutdown_connections 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc969df0 0x55cbbc9670c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9795e0 0x55cbbc97ba00 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc9697a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 --2- 172.21.15.23:0/455189792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc95eef0 0x55cbbc966a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 >> 172.21.15.23:0/455189792 conn(0x55cbbc8be6a0 msgr2=0x55cbbc9606c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd4ff400700 0 client.29083 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 shutdown_connections 2024-03-30T04:15:18.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 1 -- 172.21.15.23:0/455189792 wait complete. 2024-03-30T04:15:18.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.096+0000 7fd538418b80 20 client.29083 trim_cache size 0 max 16384 2024-03-30T04:15:18.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 Processor -- start 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 -- start start 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc970a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc961df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc961930 con 0x55cbbc95eef0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc95e3f0 con 0x55cbbc969df0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.113+0000 7fd538418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc970fb0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60694/0 (socket says 172.21.15.23:60694) 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 -- 172.21.15.23:0/2003257603 learned_addr learned my addr 172.21.15.23:0/2003257603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521400700 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc970a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd522800700 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc961df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 -- 172.21.15.23:0/2003257603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 msgr2=0x55cbbc961df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc961df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 -- 172.21.15.23:0/2003257603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 msgr2=0x55cbbc970a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc970a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 -- 172.21.15.23:0/2003257603 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cbbc876ea0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.114+0000 7fd521e00700 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd50c09afe0 tx=0x7fd50c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ababf7c61f82ae84 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.115+0000 7fd50ba00700 1 -- 172.21.15.23:0/2003257603 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c09ac40 con 0x55cbbc95e5d0 2024-03-30T04:15:18.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.115+0000 7fd50ba00700 1 -- 172.21.15.23:0/2003257603 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd50c07e030 con 0x55cbbc95e5d0 2024-03-30T04:15:18.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.115+0000 7fd50ba00700 1 -- 172.21.15.23:0/2003257603 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0a36d0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.115+0000 7fd538418b80 1 -- 172.21.15.23:0/2003257603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 msgr2=0x55cbbc967060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.115+0000 7fd538418b80 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd50c09afe0 tx=0x7fd50c005fd0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 -- 172.21.15.23:0/2003257603 shutdown_connections 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc961df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc967060 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 --2- 172.21.15.23:0/2003257603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc970a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 -- 172.21.15.23:0/2003257603 >> 172.21.15.23:0/2003257603 conn(0x55cbbc8be6a0 msgr2=0x55cbbc9535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 -- 172.21.15.23:0/2003257603 shutdown_connections 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 -- 172.21.15.23:0/2003257603 wait complete. 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 -- start start 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.116+0000 7fd538418b80 1 Processor -- start 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc96e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc96e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc96eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc9cc4f0 con 0x55cbbc95eef0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc9cc660 con 0x55cbbc969df0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cbbc9cc820 con 0x55cbbc95e5d0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd538418b80 10 client.0 _ll_register_callbacks cb 0x55cbbc856470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd522800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc96eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd521400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc96e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd522800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc96eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:50980/0 (socket says 172.21.15.23:50980) 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd522800700 1 -- 172.21.15.23:0/92887939 learned_addr learned my addr 172.21.15.23:0/92887939 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.117+0000 7fd521e00700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc96e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 -- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 msgr2=0x55cbbc96eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc96eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 -- 172.21.15.23:0/92887939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 msgr2=0x55cbbc96e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc96e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 -- 172.21.15.23:0/92887939 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cbbc6deb60 con 0x55cbbc95e5d0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd521e00700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc96e420 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0977e0 tx=0x7fd50c005d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bae698040679eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd4ff400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.118+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c09ac40 con 0x55cbbc95e5d0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fd50c07f080 con 0x55cbbc95e5d0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd510019c50 con 0x55cbbc95e5d0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd50c09b9e0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 10 client.29089 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:18.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cbbc736cd0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd510019c50 con 0x55cbbc95e5d0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd50c0a10a0 con 0x55cbbc95e5d0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 10 client.29089 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 10 client.29089 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 20 client.29089 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9cdad0 0x55cbbc9cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 4 client.29089 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.119+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cbbc6ffcf0 con 0x55cbbc9cdad0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd521400700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9cdad0 0x55cbbc9cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd4ff400700 10 client.29089 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd521400700 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9cdad0 0x55cbbc9cfec0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=65ce338df1fe41b8 server_cookie=fcaa306c20006e0 in_seq=0 out_seq=0 2024-03-30T04:15:18.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd4ff400700 1 -- 172.21.15.23:0/92887939 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cbbc6ffcf0 con 0x55cbbc9cdad0 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd4ff400700 10 client.29089 handle_command_reply: tid=0 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.120+0000 7fd538418b80 1 client.29089 shutdown 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 msgr2=0x55cbbc96e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc96e420 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0977e0 tx=0x7fd50c005d40 comp rx=0 tx=0).stop 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 shutdown_connections 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55cbbc969df0 0x55cbbc96eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55cbbc9cdad0 0x55cbbc9cfec0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55cbbc95e5d0 0x55cbbc96e420 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 --2- 172.21.15.23:0/92887939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55cbbc95eef0 0x55cbbc96e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 >> 172.21.15.23:0/92887939 conn(0x55cbbc8be6a0 msgr2=0x55cbbc961df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd4ff400700 0 client.29089 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 shutdown_connections 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 1 -- 172.21.15.23:0/92887939 wait complete. 2024-03-30T04:15:18.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.121+0000 7fd538418b80 20 client.29089 trim_cache size 0 max 16384 2024-03-30T04:15:18.125 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:18.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 msgr2=0x7fd5040a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 0x7fd5040a37d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd518007a80 tx=0x7fd518007e40 comp rx=0 tx=0).stop 2024-03-30T04:15:18.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 msgr2=0x7fd51c1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd50c000c00 tx=0x7fd50c005740 comp rx=0 tx=0).stop 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 shutdown_connections 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd5040a1310 0x7fd5040a37d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd51c10d530 0x7fd51c19dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd51c0731e0 0x7fd51c1993f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 --2- 172.21.15.23:0/2635630951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd51c073b00 0x7fd51c199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 >> 172.21.15.23:0/2635630951 conn(0x7fd51c0fc7c0 msgr2=0x7fd51c0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 shutdown_connections 2024-03-30T04:15:18.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.122+0000 7fd524800700 1 -- 172.21.15.23:0/2635630951 wait complete. 2024-03-30T04:15:18.140 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:15:18.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1279517 9.32 MB/sec execute 595 sec latency 25.660 ms 2024-03-30T04:15:18.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 -- 172.21.15.23:0/659871326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb54100f60 msgr2=0x7fcb54101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 --2- 172.21.15.23:0/659871326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb54100f60 0x7fcb54101340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009990 tx=0x7fcb44009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 -- 172.21.15.23:0/659871326 shutdown_connections 2024-03-30T04:15:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 --2- 172.21.15.23:0/659871326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb5410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 --2- 172.21.15.23:0/659871326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb54100f60 0x7fcb54101340 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 --2- 172.21.15.23:0/659871326 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54101910 0x7fcb54105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 -- 172.21.15.23:0/659871326 >> 172.21.15.23:0/659871326 conn(0x7fcb540fc7c0 msgr2=0x7fcb540febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 -- 172.21.15.23:0/659871326 shutdown_connections 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.353+0000 7fcb59c00700 1 -- 172.21.15.23:0/659871326 wait complete. 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.354+0000 7fcb59c00700 1 Processor -- start 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 -- start start 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 0x7fcb5419e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5419e790 con 0x7fcb54100f60 2024-03-30T04:15:18.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5419e900 con 0x7fcb54106080 2024-03-30T04:15:18.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb59c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5419eac0 con 0x7fcb5419dd10 2024-03-30T04:15:18.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51006/0 (socket says 172.21.15.23:51006) 2024-03-30T04:15:18.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55500/0 (socket says 172.21.15.23:55500) 2024-03-30T04:15:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb52000700 1 -- 172.21.15.23:0/2959840562 learned_addr learned my addr 172.21.15.23:0/2959840562 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.355+0000 7fcb53400700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 0x7fcb5419e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 -- 172.21.15.23:0/2959840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 msgr2=0x7fcb5419e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 0x7fcb5419e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 -- 172.21.15.23:0/2959840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 msgr2=0x7fcb54199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 -- 172.21.15.23:0/2959840562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40009520 con 0x7fcb54100f60 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52000700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb53400700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 0x7fcb5419e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098e0 tx=0x7fcb44005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb4b400700 1 -- 172.21.15.23:0/2959840562 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4401d070 con 0x7fcb54100f60 2024-03-30T04:15:18.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb4b400700 1 -- 172.21.15.23:0/2959840562 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb44022470 con 0x7fcb54100f60 2024-03-30T04:15:18.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.356+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb440095f0 con 0x7fcb54100f60 2024-03-30T04:15:18.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.357+0000 7fcb4b400700 1 -- 172.21.15.23:0/2959840562 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb44017650 con 0x7fcb54100f60 2024-03-30T04:15:18.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.357+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb541a61f0 con 0x7fcb54100f60 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.360+0000 7fcb4b400700 1 -- 172.21.15.23:0/2959840562 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fcb440225e0 con 0x7fcb54100f60 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb6d6b1b80 1 Processor -- start 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb4b400700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 0x7fcb380a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb6d6b1b80 1 -- start start 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb4b400700 1 -- 172.21.15.23:0/2959840562 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fcb440ce380 con 0x7fcb54100f60 2024-03-30T04:15:18.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe419b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.361+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fd9a990 con 0x55be0fe415d0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fd98350 con 0x55be0fe41ef0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fd93c40 con 0x55be0fe4cdf0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51010/0 (socket says 172.21.15.23:51010) 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 -- 172.21.15.23:0/2765334504 learned_addr learned my addr 172.21.15.23:0/2765334504 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb53400700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 0x7fcb380a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb53400700 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 -- 172.21.15.23:0/2765334504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 msgr2=0x55be0fe4f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 -- 172.21.15.23:0/2765334504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 msgr2=0x55be0fe419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.362+0000 7fcb52000700 1 -- 172.21.15.23:0/2765334504 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be0fc153b0 con 0x55be0fe41ef0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb52000700 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcb40011790 tx=0x7fcb40011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de3d22c9867c9f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb53400700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 0x7fcb380a3750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009fd0 tx=0x7fcb4c009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb3fe00700 1 -- 172.21.15.23:0/2765334504 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb40004020 con 0x55be0fe41ef0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb3fe00700 1 -- 172.21.15.23:0/2765334504 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb40003680 con 0x55be0fe41ef0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb3fe00700 1 -- 172.21.15.23:0/2765334504 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4000f700 con 0x55be0fe41ef0 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2765334504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 msgr2=0x55be0fe4c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.363+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcb40011790 tx=0x7fcb40011aa0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2765334504 shutdown_connections 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2765334504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe4c8b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2765334504 >> 172.21.15.23:0/2765334504 conn(0x55be0fda16a0 msgr2=0x55be0fe3da90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2765334504 shutdown_connections 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2765334504 wait complete. 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.364+0000 7fcb6d6b1b80 1 -- start start 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 Processor -- start 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe49a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fd2fd80 con 0x55be0fe415d0 2024-03-30T04:15:18.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fd2fa20 con 0x55be0fe41ef0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fc176a0 con 0x55be0fe4cdf0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb6d6b1b80 10 client.0 _ll_register_callbacks cb 0x55be0fd39470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe49a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55528/0 (socket says 172.21.15.23:55528) 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb52a00700 1 -- 172.21.15.23:0/1251392722 learned_addr learned my addr 172.21.15.23:0/1251392722 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.365+0000 7fcb52000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe49a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51012/0 (socket says 172.21.15.23:51012) 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb53400700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 -- 172.21.15.23:0/1251392722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 msgr2=0x55be0fe4a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 -- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 msgr2=0x55be0fe49a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe49a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 -- 172.21.15.23:0/1251392722 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb40011440 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb52a00700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400b690 tx=0x7fcb4409b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=512e2cf9185dc4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4409b900 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fcb4409b200 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.366+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be0fd54380 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb4409f030 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 10 client.19143 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55be0fbc1b60 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be0fd54380 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb440b3d20 con 0x55be0fe415d0 2024-03-30T04:15:18.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 10 client.19143 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 10 client.19143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 20 client.19143 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0fe5c560 0x55be0fe5e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 4 client.19143 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be0fd8c700 con 0x55be0fe5c560 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.367+0000 7fcb52000700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0fe5c560 0x55be0fe5e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.368+0000 7fcb52000700 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0fe5c560 0x55be0fe5e950 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e750aa5d2497df server_cookie=eaaf4a9389d848c in_seq=0 out_seq=0 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.368+0000 7fcb3cc00700 10 client.19143 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb3cc00700 1 -- 172.21.15.23:0/1251392722 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55be0fd8c700 con 0x55be0fe5c560 2024-03-30T04:15:18.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb3cc00700 10 client.19143 handle_command_reply: tid=0 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 client.19143 shutdown 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 msgr2=0x55be0fe4c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400b690 tx=0x7fcb4409b090 comp rx=0 tx=0).stop 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 shutdown_connections 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0fe5c560 0x55be0fe5e950 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe4cdf0 0x55be0fe4a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe4c6f0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/1251392722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe49a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb3cc00700 0 client.19143 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 >> 172.21.15.23:0/1251392722 conn(0x55be0fda16a0 msgr2=0x55be0fe43530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 shutdown_connections 2024-03-30T04:15:18.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.369+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/1251392722 wait complete. 2024-03-30T04:15:18.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.370+0000 7fcb6d6b1b80 20 client.19143 trim_cache size 0 max 16384 2024-03-30T04:15:18.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.383+0000 7fcb6d6b1b80 1 Processor -- start 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.383+0000 7fcb6d6b1b80 1 -- start start 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe41ef0 0x55be0fe4a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 0x55be0fe53a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fe53fb0 con 0x55be0fe415d0 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fe4afb0 con 0x55be0fe4cdf0 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fe41c00 con 0x55be0fe41ef0 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55534/0 (socket says 172.21.15.23:55534) 2024-03-30T04:15:18.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52a00700 1 -- 172.21.15.23:0/2980962858 learned_addr learned my addr 172.21.15.23:0/2980962858 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb53400700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 0x55be0fe53a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52000700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe41ef0 0x55be0fe4a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52a00700 1 -- 172.21.15.23:0/2980962858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe41ef0 msgr2=0x55be0fe4a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.384+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe41ef0 0x55be0fe4a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb52a00700 1 -- 172.21.15.23:0/2980962858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 msgr2=0x55be0fe53a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 0x55be0fe53a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb52a00700 1 -- 172.21.15.23:0/2980962858 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be0fd59ea0 con 0x55be0fe415d0 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb53400700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 0x55be0fe53a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb52a00700 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcb44097c70 tx=0x7fcb4409b100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b108bd30a559444d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb48c00700 1 -- 172.21.15.23:0/2980962858 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4409b720 con 0x55be0fe415d0 2024-03-30T04:15:18.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb48c00700 1 -- 172.21.15.23:0/2980962858 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb4409ad40 con 0x55be0fe415d0 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.385+0000 7fcb48c00700 1 -- 172.21.15.23:0/2980962858 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440a35f0 con 0x55be0fe415d0 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2980962858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 msgr2=0x55be0fe5e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fcb44097c70 tx=0x7fcb4409b100 comp rx=0 tx=0).stop 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2980962858 shutdown_connections 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe4cdf0 0x55be0fe53a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe415d0 0x55be0fe5e950 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/2980962858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe41ef0 0x55be0fe4a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2980962858 >> 172.21.15.23:0/2980962858 conn(0x55be0fda16a0 msgr2=0x55be0fe365a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2980962858 shutdown_connections 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/2980962858 wait complete. 2024-03-30T04:15:18.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.386+0000 7fcb6d6b1b80 1 -- start start 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 Processor -- start 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe51b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe4cdf0 0x55be0feaf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0fe52420 con 0x55be0fe4cdf0 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0feaf840 con 0x55be0fe41ef0 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be0feaf980 con 0x55be0fe415d0 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb6d6b1b80 10 client.0 _ll_register_callbacks cb 0x55be0fd39470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.387+0000 7fcb52a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60746/0 (socket says 172.21.15.23:60746) 2024-03-30T04:15:18.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 -- 172.21.15.23:0/3754007141 learned_addr learned my addr 172.21.15.23:0/3754007141 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52000700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe51b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb53400700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe4cdf0 0x55be0feaf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 -- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 msgr2=0x55be0fe51b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe51b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 -- 172.21.15.23:0/3754007141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe4cdf0 msgr2=0x55be0feaf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe4cdf0 0x55be0feaf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 -- 172.21.15.23:0/3754007141 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c008d90 con 0x55be0fe415d0 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb52a00700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fcb44099550 tx=0x7fcb44022c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3f0ea618b970e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:18.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb3cc00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb440a35f0 con 0x55be0fe415d0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fcb4409b5d0 con 0x55be0fe415d0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.388+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb40003680 con 0x55be0fe415d0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb440a10b0 con 0x55be0fe415d0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 10 client.29107 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55be0fbc1b60 con 0x55be0fe415d0 2024-03-30T04:15:18.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb40003680 con 0x55be0fe415d0 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb440ad070 con 0x55be0fe415d0 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 10 client.29107 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 10 client.29107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 20 client.29107 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0feb0c00 0x55be0feb2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 4 client.29107 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55be0fbe2cf0 con 0x55be0feb0c00 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.389+0000 7fcb52000700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0feb0c00 0x55be0feb2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb52000700 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0feb0c00 0x55be0feb2ff0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76d92e187d5f9ebc server_cookie=cb8b095e7bff7217 in_seq=0 out_seq=0 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb3cc00700 10 client.29107 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:18.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb3cc00700 1 -- 172.21.15.23:0/3754007141 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55be0fbe2cf0 con 0x55be0feb0c00 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb3cc00700 10 client.29107 handle_command_reply: tid=0 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 client.29107 shutdown 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 msgr2=0x55be0fe51600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fcb44099550 tx=0x7fcb44022c60 comp rx=0 tx=0).stop 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 shutdown_connections 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be0fe4cdf0 0x55be0feaf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55be0feb0c00 0x55be0feb2ff0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be0fe415d0 0x55be0fe51600 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 --2- 172.21.15.23:0/3754007141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be0fe41ef0 0x55be0fe51b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb3cc00700 0 client.29107 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 >> 172.21.15.23:0/3754007141 conn(0x55be0fda16a0 msgr2=0x55be0fe44df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 shutdown_connections 2024-03-30T04:15:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.390+0000 7fcb6d6b1b80 1 -- 172.21.15.23:0/3754007141 wait complete. 2024-03-30T04:15:18.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb6d6b1b80 20 client.29107 trim_cache size 0 max 16384 2024-03-30T04:15:18.401 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 msgr2=0x7fcb380a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 0x7fcb380a3750 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c009fd0 tx=0x7fcb4c009920 comp rx=0 tx=0).stop 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 msgr2=0x7fcb54199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098e0 tx=0x7fcb44005740 comp rx=0 tx=0).stop 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 shutdown_connections 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcb5419dd10 0x7fcb5419e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb54100f60 0x7fcb54199430 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcb380a1290 0x7fcb380a3750 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 --2- 172.21.15.23:0/2959840562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb54106080 0x7fcb54199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:18.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.391+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 >> 172.21.15.23:0/2959840562 conn(0x7fcb540fc7c0 msgr2=0x7fcb540fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:18.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.392+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 shutdown_connections 2024-03-30T04:15:18.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:18.392+0000 7fcb59c00700 1 -- 172.21.15.23:0/2959840562 wait complete. 2024-03-30T04:15:18.407 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:18 smithi023 ceph-mon[38254]: pgmap v419: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 104 op/s 2024-03-30T04:15:18.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:18 smithi193 ceph-mon[79581]: pgmap v419: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 104 op/s 2024-03-30T04:15:18.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:18 smithi154 ceph-mon[79161]: pgmap v419: 129 pgs: 129 active+clean; 570 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.4 MiB/s wr, 104 op/s 2024-03-30T04:15:19.237 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1281666 9.32 MB/sec execute 596 sec latency 22.682 ms 2024-03-30T04:15:19.410 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:15:19.411 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:15:19.470 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:15:19.471 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.471 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.471 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.471 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:19 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:15:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:19 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 -- 172.21.15.23:0/813380262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff418100f50 msgr2=0x7ff418101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 --2- 172.21.15.23:0/813380262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff418100f50 0x7ff418101330 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff408009990 tx=0x7ff408009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 -- 172.21.15.23:0/813380262 shutdown_connections 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 --2- 172.21.15.23:0/813380262 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff41810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 --2- 172.21.15.23:0/813380262 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff418100f50 0x7ff418101330 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 --2- 172.21.15.23:0/813380262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418101900 0x7ff418105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.633 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 -- 172.21.15.23:0/813380262 >> 172.21.15.23:0/813380262 conn(0x7ff4180fc7f0 msgr2=0x7ff4180fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 -- 172.21.15.23:0/813380262 shutdown_connections 2024-03-30T04:15:19.634 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.633+0000 7ff41dc00700 1 -- 172.21.15.23:0/813380262 wait complete. 2024-03-30T04:15:19.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.634+0000 7ff41dc00700 1 Processor -- start 2024-03-30T04:15:19.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 -- start start 2024-03-30T04:15:19.635 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418100f50 0x7ff418199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff41819dcd0 0x7ff41819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819e750 con 0x7ff41819dcd0 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819e8c0 con 0x7ff418106070 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff41dc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819ea80 con 0x7ff418100f50 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff416000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff416000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51044/0 (socket says 172.21.15.23:51044) 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff416000700 1 -- 172.21.15.23:0/2437700275 learned_addr learned my addr 172.21.15.23:0/2437700275 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff416a00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418100f50 0x7ff418199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.635+0000 7ff417400700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff41819dcd0 0x7ff41819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 -- 172.21.15.23:0/2437700275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418100f50 msgr2=0x7ff418199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418100f50 0x7ff418199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 -- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff41819dcd0 msgr2=0x7ff41819e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff41819dcd0 0x7ff41819e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 -- 172.21.15.23:0/2437700275 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080095f0 con 0x7ff418106070 2024-03-30T04:15:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff416000700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff40400e970 tx=0x7ff40400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff40f400700 1 -- 172.21.15.23:0/2437700275 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff404009790 con 0x7ff418106070 2024-03-30T04:15:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.636+0000 7ff40f400700 1 -- 172.21.15.23:0/2437700275 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40400c7e0 con 0x7ff418106070 2024-03-30T04:15:19.637 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.637+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4181a5fe0 con 0x7ff418106070 2024-03-30T04:15:19.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.637+0000 7ff40f400700 1 -- 172.21.15.23:0/2437700275 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040054e0 con 0x7ff418106070 2024-03-30T04:15:19.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.637+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4181a6460 con 0x7ff418106070 2024-03-30T04:15:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.642+0000 7ff40f400700 1 -- 172.21.15.23:0/2437700275 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff40400c950 con 0x7ff418106070 2024-03-30T04:15:19.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.642+0000 7ff4316f6b80 1 Processor -- start 2024-03-30T04:15:19.642 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.642+0000 7ff4316f6b80 1 -- start start 2024-03-30T04:15:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b638a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 0x55974b646260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b591a10 con 0x55974b643e70 2024-03-30T04:15:19.643 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b58f3d0 con 0x55974b638f70 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b58acc0 con 0x55974b638650 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff40f400700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 0x7ff3fc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff416a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff416a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51052/0 (socket says 172.21.15.23:51052) 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff416a00700 1 -- 172.21.15.23:0/893944900 learned_addr learned my addr 172.21.15.23:0/893944900 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff417400700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b638a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff40f400700 1 -- 172.21.15.23:0/2437700275 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff404014070 con 0x7ff418106070 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.643+0000 7ff416000700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 0x55974b646260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 -- 172.21.15.23:0/893944900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 msgr2=0x55974b638a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b638a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 -- 172.21.15.23:0/893944900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 msgr2=0x55974b646260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 0x55974b646260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 -- 172.21.15.23:0/893944900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55974b40c430 con 0x55974b638f70 2024-03-30T04:15:19.644 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416000700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 0x55974b646260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff417400700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 0x7ff3fc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.645 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff416a00700 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff408005ec0 tx=0x7ff408005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3ae607e2bcb23eb server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff403e00700 1 -- 172.21.15.23:0/893944900 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40801d070 con 0x55974b638f70 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff417400700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 0x7ff3fc0a3750 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff410009ec0 tx=0x7ff410004b40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff403e00700 1 -- 172.21.15.23:0/893944900 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40801e040 con 0x55974b638f70 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.644+0000 7ff403e00700 1 -- 172.21.15.23:0/893944900 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408004920 con 0x55974b638f70 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 -- 172.21.15.23:0/893944900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 msgr2=0x55974b643930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff408005ec0 tx=0x7ff408005ef0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.646 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 -- 172.21.15.23:0/893944900 shutdown_connections 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b643e70 0x55974b646260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b638a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/893944900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b638f70 0x55974b643930 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 -- 172.21.15.23:0/893944900 >> 172.21.15.23:0/893944900 conn(0x55974b598720 msgr2=0x55974b634b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 -- 172.21.15.23:0/893944900 shutdown_connections 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.645+0000 7ff4316f6b80 1 -- 172.21.15.23:0/893944900 wait complete. 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.646+0000 7ff4316f6b80 1 -- start start 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.646+0000 7ff4316f6b80 1 Processor -- start 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b640b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b641250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b526e00 con 0x55974b638f70 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b526aa0 con 0x55974b643e70 2024-03-30T04:15:19.647 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b40e720 con 0x55974b638650 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff416a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff416000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b640b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff416a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60784/0 (socket says 172.21.15.23:60784) 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff416a00700 1 -- 172.21.15.23:0/3030271549 learned_addr learned my addr 172.21.15.23:0/3030271549 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff417400700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b641250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.647+0000 7ff4316f6b80 10 client.0 _ll_register_callbacks cb 0x55974b5304f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 -- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 msgr2=0x55974b641250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b641250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 -- 172.21.15.23:0/3030271549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 msgr2=0x55974b640b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b640b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 -- 172.21.15.23:0/3030271549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41810e460 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff416a00700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff408005920 tx=0x7ff40800b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdddf694178f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff400c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40801d030 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff4080047a0 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.648+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55974b54b400 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff40800bda0 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 10 client.29119 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55974b3b8be0 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55974b54b400 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff40800f9f0 con 0x55974b638650 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 10 client.29119 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:19.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 10 client.29119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 20 client.29119 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b655880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 4 client.29119 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55974b583780 con 0x55974b653490 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.649+0000 7ff416000700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b655880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.650+0000 7ff416000700 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b655880 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e76a5b4c54049e server_cookie=1316aa061d8e8cf7 in_seq=0 out_seq=0 2024-03-30T04:15:19.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.650+0000 7ff400c00700 10 client.29119 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff400c00700 1 -- 172.21.15.23:0/3030271549 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55974b583780 con 0x55974b653490 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff400c00700 10 client.29119 handle_command_reply: tid=0 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 client.29119 shutdown 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 msgr2=0x55974b6438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7ff408005920 tx=0x7ff40800b900 comp rx=0 tx=0).stop 2024-03-30T04:15:19.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 shutdown_connections 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b641250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b655880 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff400c00700 0 client.29119 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6438e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3030271549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b640b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 >> 172.21.15.23:0/3030271549 conn(0x55974b598720 msgr2=0x55974b63a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 shutdown_connections 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.652+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3030271549 wait complete. 2024-03-30T04:15:19.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.653+0000 7ff4316f6b80 20 client.29119 trim_cache size 0 max 16384 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 Processor -- start 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 -- start start 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 0x55974b64aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 0x55974b64f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b64f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b655f40 con 0x55974b638650 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b63a1f0 con 0x55974b643e70 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.671+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b639920 con 0x55974b638f70 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 0x55974b64aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 0x55974b64f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 0x55974b64aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55580/0 (socket says 172.21.15.23:55580) 2024-03-30T04:15:19.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416a00700 1 -- 172.21.15.23:0/3932794536 learned_addr learned my addr 172.21.15.23:0/3932794536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff417400700 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b64f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 -- 172.21.15.23:0/3932794536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 msgr2=0x55974b64f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b64f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 -- 172.21.15.23:0/3932794536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 msgr2=0x55974b64aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 0x55974b64aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 -- 172.21.15.23:0/3932794536 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55974b65a3b0 con 0x55974b638f70 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff416000700 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 0x55974b64f1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ca660 tx=0x7ff404096360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637d66796db13845 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff40cc00700 1 -- 172.21.15.23:0/3932794536 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40409a720 con 0x55974b638f70 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.672+0000 7ff40cc00700 1 -- 172.21.15.23:0/3932794536 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40409a020 con 0x55974b638f70 2024-03-30T04:15:19.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff40cc00700 1 -- 172.21.15.23:0/3932794536 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040056f0 con 0x55974b638f70 2024-03-30T04:15:19.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3932794536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 msgr2=0x55974b64f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 0x55974b64f1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ca660 tx=0x7ff404096360 comp rx=0 tx=0).stop 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3932794536 shutdown_connections 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b64f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638650 0x55974b64aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/3932794536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638f70 0x55974b64f1e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3932794536 >> 172.21.15.23:0/3932794536 conn(0x55974b598720 msgr2=0x55974b62dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3932794536 shutdown_connections 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.673+0000 7ff4316f6b80 1 -- 172.21.15.23:0/3932794536 wait complete. 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.674+0000 7ff4316f6b80 1 -- start start 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.674+0000 7ff4316f6b80 1 Processor -- start 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.674+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.674+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b649920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.674+0000 7ff4316f6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b6a71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b64a0c0 con 0x55974b638f70 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b64a230 con 0x55974b643e70 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff4316f6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55974b6a76e0 con 0x55974b638650 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff4316f6b80 10 client.0 _ll_register_callbacks cb 0x55974b5304f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60796/0 (socket says 172.21.15.23:60796) 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 -- 172.21.15.23:0/2554405296 learned_addr learned my addr 172.21.15.23:0/2554405296 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416000700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b649920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff417400700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b6a71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 -- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 msgr2=0x55974b6a71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b6a71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 -- 172.21.15.23:0/2554405296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 msgr2=0x55974b649920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b649920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 -- 172.21.15.23:0/2554405296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4181a6460 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff416a00700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff40800fe50 tx=0x7ff40800fc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6da248e33c36f351 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.675+0000 7ff400c00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40801d040 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff40801e040 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff40800b970 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff408005bc0 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 10 client.29125 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55974b550f20 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff40801e040 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff408004870 con 0x55974b638650 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 10 client.29125 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 10 client.29125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:19.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 20 client.29125 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b6aab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 4 client.29125 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.676+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55974b3d9d70 con 0x55974b653490 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.677+0000 7ff416000700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b6aab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.677+0000 7ff416000700 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b6aab20 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eafdd4cdd1608b server_cookie=f184612e0e026b2b in_seq=0 out_seq=0 2024-03-30T04:15:19.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.677+0000 7ff400c00700 10 client.29125 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.678+0000 7ff400c00700 1 -- 172.21.15.23:0/2554405296 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55974b3d9d70 con 0x55974b653490 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.678+0000 7ff400c00700 10 client.29125 handle_command_reply: tid=0 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.678+0000 7ff4316f6b80 1 client.29125 shutdown 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.678+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 msgr2=0x55974b6493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.678+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff40800fe50 tx=0x7ff40800fc20 comp rx=0 tx=0).stop 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 shutdown_connections 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55974b643e70 0x55974b6a71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55974b653490 0x55974b6aab20 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55974b638650 0x55974b6493e0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 --2- 172.21.15.23:0/2554405296 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55974b638f70 0x55974b649920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff400c00700 0 client.29125 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 >> 172.21.15.23:0/2554405296 conn(0x55974b598720 msgr2=0x55974b64fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 shutdown_connections 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 1 -- 172.21.15.23:0/2554405296 wait complete. 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff4316f6b80 20 client.29125 trim_cache size 0 max 16384 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "e2dc0f8e-1456-413f-98f4-925dad489a25", 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:15:19.682 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 msgr2=0x7ff3fc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 0x7ff3fc0a3750 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff410009ec0 tx=0x7ff410004b40 comp rx=0 tx=0).stop 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 msgr2=0x7ff418199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.679+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7ff40400e970 tx=0x7ff40400ec80 comp rx=0 tx=0).stop 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 shutdown_connections 2024-03-30T04:15:19.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff41819dcd0 0x7ff41819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff418100f50 0x7ff418199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff3fc0a1290 0x7ff3fc0a3750 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 --2- 172.21.15.23:0/2437700275 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff418106070 0x7ff418199980 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 >> 172.21.15.23:0/2437700275 conn(0x7ff4180fc7f0 msgr2=0x7ff4180fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 shutdown_connections 2024-03-30T04:15:19.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.680+0000 7ff41dc00700 1 -- 172.21.15.23:0/2437700275 wait complete. 2024-03-30T04:15:19.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:15:19.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.685 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.685 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.685 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:19 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:19.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:15:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 -- 172.21.15.23:0/3621586482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa744105ce0 msgr2=0x7fa7441060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 --2- 172.21.15.23:0/3621586482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa744105ce0 0x7fa7441060c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fa738009990 tx=0x7fa738009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 -- 172.21.15.23:0/3621586482 shutdown_connections 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 --2- 172.21.15.23:0/3621586482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa7440687f0 0x7fa74410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 --2- 172.21.15.23:0/3621586482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa744105ce0 0x7fa7441060c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 --2- 172.21.15.23:0/3621586482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744067e30 0x7fa7440682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 -- 172.21.15.23:0/3621586482 >> 172.21.15.23:0/3621586482 conn(0x7fa744076d40 msgr2=0x7fa744077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 -- 172.21.15.23:0/3621586482 shutdown_connections 2024-03-30T04:15:19.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.903+0000 7fa74c200700 1 -- 172.21.15.23:0/3621586482 wait complete. 2024-03-30T04:15:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.904+0000 7fa74c200700 1 Processor -- start 2024-03-30T04:15:19.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 -- start start 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 0x7fa74410a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744105ce0 0x7fa74410aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441a99f0 con 0x7fa7440687f0 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441a9b60 con 0x7fa744067e30 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74c200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7441a9d20 con 0x7fa744105ce0 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa749800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa748e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 0x7fa74410a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa749800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:51112/0 (socket says 172.21.15.23:51112) 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa749800700 1 -- 172.21.15.23:0/2838910980 learned_addr learned my addr 172.21.15.23:0/2838910980 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.905+0000 7fa74a200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744105ce0 0x7fa74410aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 -- 172.21.15.23:0/2838910980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744105ce0 msgr2=0x7fa74410aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744105ce0 0x7fa74410aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 -- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 msgr2=0x7fa74410a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 0x7fa74410a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 -- 172.21.15.23:0/2838910980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380095f0 con 0x7fa744067e30 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa748e00700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 0x7fa74410a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa749800700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa738005910 tx=0x7fa738004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa742000700 1 -- 172.21.15.23:0/2838910980 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73801d070 con 0x7fa744067e30 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa742000700 1 -- 172.21.15.23:0/2838910980 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa738022470 con 0x7fa744067e30 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa742000700 1 -- 172.21.15.23:0/2838910980 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa738017650 con 0x7fa744067e30 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.906+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7441a9fa0 con 0x7fa744067e30 2024-03-30T04:15:19.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.907+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7441aa310 con 0x7fa744067e30 2024-03-30T04:15:19.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.910+0000 7fa742000700 1 -- 172.21.15.23:0/2838910980 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa7380225e0 con 0x7fa744067e30 2024-03-30T04:15:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 Processor -- start 2024-03-30T04:15:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 -- start start 2024-03-30T04:15:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa742000700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 0x7fa72c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa742000700 1 -- 172.21.15.23:0/2838910980 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa7380ce2f0 con 0x7fa744067e30 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef61fdf0 0x5571ef6221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef56d990 con 0x5571ef61fdf0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef56b350 con 0x5571ef614ef0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.911+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef566c40 con 0x5571ef6145d0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60810/0 (socket says 172.21.15.23:60810) 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 -- 172.21.15.23:0/1838613077 learned_addr learned my addr 172.21.15.23:0/1838613077 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa74a200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 0x7fa72c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa748e00700 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 -- 172.21.15.23:0/1838613077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 msgr2=0x5571ef61f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 -- 172.21.15.23:0/1838613077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef61fdf0 msgr2=0x5571ef6221e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:19.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef61fdf0 0x5571ef6221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.912+0000 7fa749800700 1 -- 172.21.15.23:0/1838613077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571ef3e83b0 con 0x5571ef6145d0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa74a200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 0x7fa72c0a37d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7440fe700 tx=0x7fa73c009920 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa749800700 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa73809afe0 tx=0x7fa738022c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=549ce867404981cf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa736a00700 1 -- 172.21.15.23:0/1838613077 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73809b750 con 0x5571ef6145d0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa736a00700 1 -- 172.21.15.23:0/1838613077 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7380aee90 con 0x5571ef6145d0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa736a00700 1 -- 172.21.15.23:0/1838613077 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7380228c0 con 0x5571ef6145d0 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1838613077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 msgr2=0x5571ef6149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.913+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa73809afe0 tx=0x7fa738022c90 comp rx=0 tx=0).stop 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1838613077 shutdown_connections 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef61fdf0 0x5571ef6221e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef6149b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1838613077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1838613077 >> 172.21.15.23:0/1838613077 conn(0x5571ef5746a0 msgr2=0x5571ef610a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1838613077 shutdown_connections 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1838613077 wait complete. 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 -- start start 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.914+0000 7fa75fc84b80 1 Processor -- start 2024-03-30T04:15:19.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef61d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef502d80 con 0x5571ef6145d0 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef502a20 con 0x5571ef614ef0 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef3ea6a0 con 0x5571ef61fdf0 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa75fc84b80 10 client.0 _ll_register_callbacks cb 0x5571ef50c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55614/0 (socket says 172.21.15.23:55614) 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 -- 172.21.15.23:0/1847076309 learned_addr learned my addr 172.21.15.23:0/1847076309 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa748e00700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa74a200700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef61d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 -- 172.21.15.23:0/1847076309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 msgr2=0x5571ef61d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef61d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 -- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 msgr2=0x5571ef61cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.915+0000 7fa749800700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa749800700 1 -- 172.21.15.23:0/1847076309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7441aa310 con 0x5571ef6145d0 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa74a200700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef61d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa748e00700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa749800700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa738097ac0 tx=0x7fa7380cb570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477f443b4ab539fd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa72b400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:19.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7380b7070 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa73809b050 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5571ef527380 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7380a10d0 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa75fc84b80 10 client.19173 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.916+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5571ef394b60 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5571ef527380 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa73807c070 con 0x5571ef6145d0 2024-03-30T04:15:19.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 10 client.19173 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 10 client.19173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 20 client.19173 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef62f1b0 0x5571ef6315a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 4 client.19173 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571ef55f700 con 0x5571ef62f1b0 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.917+0000 7fa748e00700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef62f1b0 0x5571ef6315a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.918+0000 7fa748e00700 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef62f1b0 0x5571ef6315a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c90b9be58e24df8d server_cookie=d03ccf324f9e2230 in_seq=0 out_seq=0 2024-03-30T04:15:19.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.918+0000 7fa72b400700 10 client.19173 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa72b400700 1 -- 172.21.15.23:0/1847076309 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5571ef55f700 con 0x5571ef62f1b0 2024-03-30T04:15:19.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa72b400700 10 client.19173 handle_command_reply: tid=0 2024-03-30T04:15:19.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 client.19173 shutdown 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 msgr2=0x5571ef61f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fa738097ac0 tx=0x7fa7380cb570 comp rx=0 tx=0).stop 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 shutdown_connections 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef62f1b0 0x5571ef6315a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef61d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef61f810 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/1847076309 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef61cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 >> 172.21.15.23:0/1847076309 conn(0x5571ef5746a0 msgr2=0x5571ef6166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa72b400700 0 client.19173 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 shutdown_connections 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.919+0000 7fa75fc84b80 1 -- 172.21.15.23:0/1847076309 wait complete. 2024-03-30T04:15:19.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.920+0000 7fa75fc84b80 20 client.19173 trim_cache size 0 max 16384 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.937+0000 7fa75fc84b80 1 Processor -- start 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.937+0000 7fa75fc84b80 1 -- start start 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 0x5571ef61b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef61fdf0 0x5571ef62af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef61bec0 con 0x5571ef614ef0 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef611660 con 0x5571ef61fdf0 2024-03-30T04:15:19.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef61da60 con 0x5571ef6145d0 2024-03-30T04:15:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa748e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 0x5571ef61b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa74a200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef61fdf0 0x5571ef62af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:60826/0 (socket says 172.21.15.23:60826) 2024-03-30T04:15:19.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 -- 172.21.15.23:0/2677220454 learned_addr learned my addr 172.21.15.23:0/2677220454 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 -- 172.21.15.23:0/2677220454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef61fdf0 msgr2=0x5571ef62af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef61fdf0 0x5571ef62af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 -- 172.21.15.23:0/2677220454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 msgr2=0x5571ef61b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 0x5571ef61b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa749800700 1 -- 172.21.15.23:0/2677220454 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571ef52cea0 con 0x5571ef6145d0 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.938+0000 7fa748e00700 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 0x5571ef61b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.939+0000 7fa749800700 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa738000c00 tx=0x7fa73809a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d09eecbaa11743e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.939+0000 7fa737400700 1 -- 172.21.15.23:0/2677220454 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73809a7c0 con 0x5571ef6145d0 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.939+0000 7fa737400700 1 -- 172.21.15.23:0/2677220454 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa73809b750 con 0x5571ef6145d0 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.939+0000 7fa737400700 1 -- 172.21.15.23:0/2677220454 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7380a3000 con 0x5571ef6145d0 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- 172.21.15.23:0/2677220454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 msgr2=0x5571ef61b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fa738000c00 tx=0x7fa73809a560 comp rx=0 tx=0).stop 2024-03-30T04:15:19.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- 172.21.15.23:0/2677220454 shutdown_connections 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef61fdf0 0x5571ef62af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef6145d0 0x5571ef61b440 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/2677220454 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef614ef0 0x5571ef61b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- 172.21.15.23:0/2677220454 >> 172.21.15.23:0/2677220454 conn(0x5571ef5746a0 msgr2=0x5571ef617ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- 172.21.15.23:0/2677220454 shutdown_connections 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- 172.21.15.23:0/2677220454 wait complete. 2024-03-30T04:15:19.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 -- start start 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.940+0000 7fa75fc84b80 1 Processor -- start 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef67c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef67c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef6247c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef624d00 con 0x5571ef6145d0 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef624e70 con 0x5571ef614ef0 2024-03-30T04:15:19.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571ef625030 con 0x5571ef61fdf0 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa75fc84b80 10 client.0 _ll_register_callbacks cb 0x5571ef50c470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa749800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef67c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa748e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef67c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa749800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef67c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55628/0 (socket says 172.21.15.23:55628) 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa749800700 1 -- 172.21.15.23:0/902253636 learned_addr learned my addr 172.21.15.23:0/902253636 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.941+0000 7fa74a200700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef6247c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 -- 172.21.15.23:0/902253636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 msgr2=0x5571ef6247c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef6247c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 -- 172.21.15.23:0/902253636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 msgr2=0x5571ef67c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef67c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 -- 172.21.15.23:0/902253636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa730021a00 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa748e00700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef67c680 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x5571ef631db0 tx=0x7fa730012ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1938be96d3eeb82c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa72b400700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730017040 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7380aee90 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa7300210b0 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7300122d0 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa75fc84b80 10 client.29162 fetch_fsmap learned FSMap version 30 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.942+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x5571ef394b60 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7380aee90 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 30) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa730010430 con 0x5571ef614ef0 2024-03-30T04:15:19.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 10 client.29162 fetch_fsmap finished waiting for FSMap version 30 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 10 client.29162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 20 client.29162 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef683f40 0x5571ef686360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 4 client.29162 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5571ef3b5cf0 con 0x5571ef683f40 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.943+0000 7fa749800700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef683f40 0x5571ef686360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa749800700 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef683f40 0x5571ef686360 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdeef866303417bd server_cookie=acc9db943db8f335 in_seq=0 out_seq=0 2024-03-30T04:15:19.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa72b400700 10 client.29162 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa72b400700 1 -- 172.21.15.23:0/902253636 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5571ef3b5cf0 con 0x5571ef683f40 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa72b400700 10 client.29162 handle_command_reply: tid=0 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 client.29162 shutdown 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 msgr2=0x5571ef67c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef67c680 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x5571ef631db0 tx=0x7fa730012ce0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 shutdown_connections 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x5571ef683f40 0x5571ef686360 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5571ef61fdf0 0x5571ef6247c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5571ef6145d0 0x5571ef67c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 --2- 172.21.15.23:0/902253636 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5571ef614ef0 0x5571ef67c680 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa72b400700 0 client.29162 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 >> 172.21.15.23:0/902253636 conn(0x5571ef5746a0 msgr2=0x5571ef61b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 shutdown_connections 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 1 -- 172.21.15.23:0/902253636 wait complete. 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.944+0000 7fa75fc84b80 20 client.29162 trim_cache size 0 max 16384 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stdout: "e2dc0f8e-1456-413f-98f4-925dad489a25": { 2024-03-30T04:15:19.950 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:15:19.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "e2dc0f8e-1456-413f-98f4-925dad489a25", 2024-03-30T04:15:19.951 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:15:19.951 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:15:19.951 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:15:19.951 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 msgr2=0x7fa72c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 0x7fa72c0a37d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa7440fe700 tx=0x7fa73c009920 comp rx=0 tx=0).stop 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 msgr2=0x7fa74410a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa738005910 tx=0x7fa738004730 comp rx=0 tx=0).stop 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 shutdown_connections 2024-03-30T04:15:19.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa744105ce0 0x7fa74410aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa72c0a1310 0x7fa72c0a37d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa744067e30 0x7fa74410a040 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 --2- 172.21.15.23:0/2838910980 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7440687f0 0x7fa74410a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 >> 172.21.15.23:0/2838910980 conn(0x7fa744076d40 msgr2=0x7fa7440fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 shutdown_connections 2024-03-30T04:15:19.954 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:19.945+0000 7fa74c200700 1 -- 172.21.15.23:0/2838910980 wait complete. 2024-03-30T04:15:19.963 INFO:tasks.cephfs.filesystem:scrub status for tag:e2dc0f8e-1456-413f-98f4-925dad489a25 - {'path': '/', 'tag': 'e2dc0f8e-1456-413f-98f4-925dad489a25', 'options': 'recursive,force'} 2024-03-30T04:15:20.238 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1282575 9.32 MB/sec execute 597 sec latency 26.938 ms 2024-03-30T04:15:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:20 smithi193 ceph-mon[79581]: pgmap v420: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 188 op/s 2024-03-30T04:15:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:20 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:15:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:20 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:15:20.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:20 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:20 smithi023 ceph-mon[38254]: pgmap v420: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 188 op/s 2024-03-30T04:15:20.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:20 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:15:20.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:20 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:15:20.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:20 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:20.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:20 smithi154 ceph-mon[79161]: pgmap v420: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.5 MiB/s wr, 188 op/s 2024-03-30T04:15:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:20 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:15:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:20 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:15:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:20 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:15:21.238 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1283677 9.31 MB/sec execute 598 sec latency 47.687 ms 2024-03-30T04:15:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:15:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:21.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:21 smithi193 ceph-mon[79581]: pgmap v421: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:21 smithi023 ceph-mon[38254]: pgmap v421: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:21 smithi154 ceph-mon[79161]: pgmap v421: 129 pgs: 129 active+clean; 571 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 4.1 MiB/s wr, 170 op/s 2024-03-30T04:15:22.238 INFO:tasks.workunit.client.0.smithi023.stdout: 1 1284927 9.30 MB/sec execute 599 sec latency 24.899 ms 2024-03-30T04:15:23.238 INFO:tasks.workunit.client.0.smithi023.stdout: 1 cleanup 600 sec 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: 0 cleanup 600 sec 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Operation Count AvgLat MaxLat 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: ---------------------------------------- 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: NTCreateX 177684 1.163 433.202 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Close 130494 0.005 0.530 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Rename 7528 5.323 108.134 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Unlink 35896 1.743 78.289 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Deltree 4 90.004 113.004 2024-03-30T04:15:23.504 INFO:tasks.workunit.client.0.smithi023.stdout: Mkdir 2 0.028 0.042 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Qpathinfo 161055 1.098 235.075 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Qfileinfo 28214 0.002 0.060 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Qfsinfo 29556 0.323 10.792 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Sfileinfo 14464 0.045 1.760 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Find 62284 0.272 196.564 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: WriteX 88519 0.136 152.246 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: ReadX 278798 0.029 60.114 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: LockX 580 0.776 4.748 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: UnlockX 580 0.729 11.237 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: Flush 12449 4.895 211.315 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-30T04:15:23.505 INFO:tasks.workunit.client.0.smithi023.stdout:Throughput 9.30409 MB/sec 1 clients 1 procs max_latency=433.212 ms 2024-03-30T04:15:23.506 INFO:tasks.workunit.client.0.smithi023.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-30T04:15:23.506 INFO:tasks.workunit.client.0.smithi023.stdout: 2024-03-30T04:15:23.506 INFO:tasks.workunit.client.0.smithi023.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-30T04:15:23.508 INFO:tasks.workunit.client.0.smithi023.stdout:9 of 10 processes prepared for launch 0 sec 2024-03-30T04:15:23.608 INFO:tasks.workunit.client.0.smithi023.stdout:10 of 10 processes prepared for launch 0 sec 2024-03-30T04:15:23.609 INFO:tasks.workunit.client.0.smithi023.stdout:releasing clients 2024-03-30T04:15:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:23 smithi154 ceph-mon[79161]: pgmap v422: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:23.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:23 smithi193 ceph-mon[79581]: pgmap v422: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:23.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:23 smithi023 ceph-mon[38254]: pgmap v422: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:24.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 375 134.09 MB/sec warmup 1 sec latency 153.908 ms 2024-03-30T04:15:25.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 760 101.80 MB/sec warmup 2 sec latency 241.192 ms 2024-03-30T04:15:25.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:25 smithi154 ceph-mon[79161]: pgmap v423: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:25.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:25 smithi193 ceph-mon[79581]: pgmap v423: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:25.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:25 smithi023 ceph-mon[38254]: pgmap v423: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-03-30T04:15:26.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 943 70.34 MB/sec warmup 3 sec latency 565.304 ms 2024-03-30T04:15:26.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:26 smithi154 ceph-mon[79161]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:15:26.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:26 smithi193 ceph-mon[79581]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:15:26.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:26 smithi023 ceph-mon[38254]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-30T04:15:27.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 1588 63.79 MB/sec warmup 4 sec latency 196.959 ms 2024-03-30T04:15:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:28 smithi193 ceph-mon[79581]: pgmap v424: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 165 op/s 2024-03-30T04:15:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:28 smithi023 ceph-mon[38254]: pgmap v424: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 165 op/s 2024-03-30T04:15:28.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 2242 54.30 MB/sec warmup 5 sec latency 258.637 ms 2024-03-30T04:15:28.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:28 smithi154 ceph-mon[79161]: pgmap v424: 129 pgs: 129 active+clean; 573 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.9 MiB/s wr, 165 op/s 2024-03-30T04:15:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:29 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:15:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:29 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:15:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:29 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:15:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:29 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:15:29.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 2794 53.73 MB/sec warmup 6 sec latency 215.766 ms 2024-03-30T04:15:29.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:29 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:15:29.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:29 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:15:30.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:30 smithi193 ceph-mon[79581]: pgmap v425: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 289 op/s 2024-03-30T04:15:30.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:30 smithi023 ceph-mon[38254]: pgmap v425: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 289 op/s 2024-03-30T04:15:30.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 3429 52.99 MB/sec warmup 7 sec latency 223.233 ms 2024-03-30T04:15:30.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:30 smithi154 ceph-mon[79161]: pgmap v425: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 289 op/s 2024-03-30T04:15:31.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 3757 48.48 MB/sec warmup 8 sec latency 176.807 ms 2024-03-30T04:15:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:32 smithi193 ceph-mon[79581]: pgmap v426: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.4 MiB/s wr, 217 op/s 2024-03-30T04:15:32.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:32 smithi023 ceph-mon[38254]: pgmap v426: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.4 MiB/s wr, 217 op/s 2024-03-30T04:15:32.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 3930 43.73 MB/sec warmup 9 sec latency 312.921 ms 2024-03-30T04:15:32.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:32 smithi154 ceph-mon[79161]: pgmap v426: 129 pgs: 129 active+clean; 652 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 9.4 MiB/s wr, 217 op/s 2024-03-30T04:15:33.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:33 smithi023 ceph-mon[38254]: pgmap v427: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 23 MiB/s wr, 420 op/s 2024-03-30T04:15:33.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 4176 40.07 MB/sec warmup 10 sec latency 273.841 ms 2024-03-30T04:15:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:33 smithi154 ceph-mon[79161]: pgmap v427: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 23 MiB/s wr, 420 op/s 2024-03-30T04:15:33.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:33 smithi193 ceph-mon[79581]: pgmap v427: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 23 MiB/s wr, 420 op/s 2024-03-30T04:15:34.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 4594 37.92 MB/sec warmup 11 sec latency 224.650 ms 2024-03-30T04:15:35.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 5308 37.57 MB/sec warmup 12 sec latency 192.120 ms 2024-03-30T04:15:35.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:35 smithi154 ceph-mon[79161]: pgmap v428: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:35.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:35 smithi193 ceph-mon[79581]: pgmap v428: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:35.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:35 smithi023 ceph-mon[38254]: pgmap v428: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:36.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 5972 37.19 MB/sec warmup 13 sec latency 474.489 ms 2024-03-30T04:15:37.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 6586 38.03 MB/sec warmup 14 sec latency 1297.123 ms 2024-03-30T04:15:38.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:38 smithi193 ceph-mon[79581]: pgmap v429: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:38.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:38 smithi023 ceph-mon[38254]: pgmap v429: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:38.609 INFO:tasks.workunit.client.0.smithi023.stdout: 10 6990 37.64 MB/sec warmup 15 sec latency 2150.630 ms 2024-03-30T04:15:38.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:38 smithi154 ceph-mon[79161]: pgmap v429: 129 pgs: 129 active+clean; 757 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 22 MiB/s wr, 353 op/s 2024-03-30T04:15:39.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 7338 36.13 MB/sec warmup 16 sec latency 180.329 ms 2024-03-30T04:15:40.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:40 smithi193 ceph-mon[79581]: pgmap v430: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 23 MiB/s wr, 493 op/s 2024-03-30T04:15:40.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:40 smithi023 ceph-mon[38254]: pgmap v430: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 23 MiB/s wr, 493 op/s 2024-03-30T04:15:40.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 7579 34.41 MB/sec warmup 17 sec latency 244.351 ms 2024-03-30T04:15:40.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:40 smithi154 ceph-mon[79161]: pgmap v430: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 23 MiB/s wr, 493 op/s 2024-03-30T04:15:41.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 7994 33.56 MB/sec warmup 18 sec latency 260.381 ms 2024-03-30T04:15:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:42 smithi193 ceph-mon[79581]: pgmap v431: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 342 op/s 2024-03-30T04:15:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:42 smithi023 ceph-mon[38254]: pgmap v431: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 342 op/s 2024-03-30T04:15:42.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 8678 33.41 MB/sec warmup 19 sec latency 192.790 ms 2024-03-30T04:15:42.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:42 smithi154 ceph-mon[79161]: pgmap v431: 129 pgs: 129 active+clean; 758 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 16 MiB/s wr, 342 op/s 2024-03-30T04:15:43.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 9329 33.42 MB/sec warmup 20 sec latency 668.050 ms 2024-03-30T04:15:44.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:44 smithi193 ceph-mon[79581]: pgmap v432: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 479 op/s 2024-03-30T04:15:44.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:44 smithi023 ceph-mon[38254]: pgmap v432: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 479 op/s 2024-03-30T04:15:44.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 9904 33.97 MB/sec warmup 21 sec latency 1575.860 ms 2024-03-30T04:15:44.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:44 smithi154 ceph-mon[79161]: pgmap v432: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 17 MiB/s wr, 479 op/s 2024-03-30T04:15:45.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:45 smithi193 ceph-mon[79581]: pgmap v433: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:45.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:45 smithi023 ceph-mon[38254]: pgmap v433: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:45.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 10404 33.90 MB/sec warmup 22 sec latency 55.606 ms 2024-03-30T04:15:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:45 smithi154 ceph-mon[79161]: pgmap v433: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:46.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 10660 32.89 MB/sec warmup 23 sec latency 158.942 ms 2024-03-30T04:15:47.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 10867 31.71 MB/sec warmup 24 sec latency 221.908 ms 2024-03-30T04:15:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:48 smithi193 ceph-mon[79581]: pgmap v434: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:48 smithi023 ceph-mon[38254]: pgmap v434: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:48.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 11164 30.91 MB/sec warmup 25 sec latency 271.226 ms 2024-03-30T04:15:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:48 smithi154 ceph-mon[79161]: pgmap v434: 129 pgs: 129 active+clean; 771 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.5 MiB/s wr, 276 op/s 2024-03-30T04:15:49.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 11566 30.61 MB/sec warmup 26 sec latency 169.784 ms 2024-03-30T04:15:49.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 -- 172.21.15.23:0/2278145200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c106090 msgr2=0x7fc76c10dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 --2- 172.21.15.23:0/2278145200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c106090 0x7fc76c10dc10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc76400b0a0 tx=0x7fc76400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 -- 172.21.15.23:0/2278145200 shutdown_connections 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 --2- 172.21.15.23:0/2278145200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c106090 0x7fc76c10dc10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 --2- 172.21.15.23:0/2278145200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c100f70 0x7fc76c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 --2- 172.21.15.23:0/2278145200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c101920 0x7fc76c105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 -- 172.21.15.23:0/2278145200 >> 172.21.15.23:0/2278145200 conn(0x7fc76c0fc7f0 msgr2=0x7fc76c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 -- 172.21.15.23:0/2278145200 shutdown_connections 2024-03-30T04:15:50.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.211+0000 7fc770c00700 1 -- 172.21.15.23:0/2278145200 wait complete. 2024-03-30T04:15:50.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 Processor -- start 2024-03-30T04:15:50.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 -- start start 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c100f70 0x7fc76c197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 0x7fc76c197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c19c630 con 0x7fc76c106090 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c19c7a0 con 0x7fc76c19bbb0 2024-03-30T04:15:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.213+0000 7fc770c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc76c19c960 con 0x7fc76c100f70 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43310/0 (socket says 172.21.15.23:43310) 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 -- 172.21.15.23:0/3406451186 learned_addr learned my addr 172.21.15.23:0/3406451186 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc769e00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c100f70 0x7fc76c197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc769400700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 0x7fc76c197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 -- 172.21.15.23:0/3406451186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c100f70 msgr2=0x7fc76c197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c100f70 0x7fc76c197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 -- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 msgr2=0x7fc76c197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 0x7fc76c197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 -- 172.21.15.23:0/3406451186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764009d00 con 0x7fc76c19bbb0 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc769400700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 0x7fc76c197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.214+0000 7fc76a800700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc76400b070 tx=0x7fc76400bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.215+0000 7fc762a00700 1 -- 172.21.15.23:0/3406451186 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76400e040 con 0x7fc76c19bbb0 2024-03-30T04:15:50.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.215+0000 7fc762a00700 1 -- 172.21.15.23:0/3406451186 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc76401d480 con 0x7fc76c19bbb0 2024-03-30T04:15:50.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.215+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc76c1a3e60 con 0x7fc76c19bbb0 2024-03-30T04:15:50.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.215+0000 7fc762a00700 1 -- 172.21.15.23:0/3406451186 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640052a0 con 0x7fc76c19bbb0 2024-03-30T04:15:50.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.215+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c1a4230 con 0x7fc76c19bbb0 2024-03-30T04:15:50.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.219+0000 7fc762a00700 1 -- 172.21.15.23:0/3406451186 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc76401d5f0 con 0x7fc76c19bbb0 2024-03-30T04:15:50.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.220+0000 7fc7846fdb80 1 Processor -- start 2024-03-30T04:15:50.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.220+0000 7fc7846fdb80 1 -- start start 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc762a00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 0x7fc7500a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc42d1990 con 0x55afc43785d0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc42cf350 con 0x55afc4378ef0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc42cac40 con 0x55afc4383df0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc762a00700 1 -- 172.21.15.23:0/3406451186 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc7640ce190 con 0x7fc76c19bbb0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc76a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc769400700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 0x7fc7500a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc76a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57476/0 (socket says 172.21.15.23:57476) 2024-03-30T04:15:50.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.221+0000 7fc76a800700 1 -- 172.21.15.23:0/1227750831 learned_addr learned my addr 172.21.15.23:0/1227750831 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 -- 172.21.15.23:0/1227750831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 msgr2=0x55afc43838b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 -- 172.21.15.23:0/1227750831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 msgr2=0x55afc43789b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 -- 172.21.15.23:0/1227750831 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afc414c3b0 con 0x55afc4383df0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc76a800700 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc7640979c0 tx=0x7fc76400b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b540259c37dce3c server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc757000700 1 -- 172.21.15.23:0/1227750831 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640a1070 con 0x55afc4383df0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc769400700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 0x7fc7500a3750 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc76c198820 tx=0x7fc75c009870 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc757000700 1 -- 172.21.15.23:0/1227750831 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc76409b650 con 0x55afc4383df0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.222+0000 7fc757000700 1 -- 172.21.15.23:0/1227750831 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76409b1b0 con 0x55afc4383df0 2024-03-30T04:15:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 -- 172.21.15.23:0/1227750831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 msgr2=0x55afc43861e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc7640979c0 tx=0x7fc76400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 -- 172.21.15.23:0/1227750831 shutdown_connections 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43861e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/1227750831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.224+0000 7fc7846fdb80 1 -- 172.21.15.23:0/1227750831 >> 172.21.15.23:0/1227750831 conn(0x55afc42d86a0 msgr2=0x55afc4374a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.225+0000 7fc7846fdb80 1 -- 172.21.15.23:0/1227750831 shutdown_connections 2024-03-30T04:15:50.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.225+0000 7fc7846fdb80 1 -- 172.21.15.23:0/1227750831 wait complete. 2024-03-30T04:15:50.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.226+0000 7fc7846fdb80 1 -- start start 2024-03-30T04:15:50.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.226+0000 7fc7846fdb80 1 Processor -- start 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 0x55afc4380b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 0x55afc4381130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4266d80 con 0x55afc43785d0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4266a20 con 0x55afc4383df0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc414e6a0 con 0x55afc4378ef0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc7846fdb80 10 client.0 _ll_register_callbacks cb 0x55afc4270470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.227+0000 7fc769e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35324/0 (socket says 172.21.15.23:35324) 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 -- 172.21.15.23:0/2026660923 learned_addr learned my addr 172.21.15.23:0/2026660923 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769400700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 0x55afc4380b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc76a800700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 0x55afc4381130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 -- 172.21.15.23:0/2026660923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 msgr2=0x55afc4380b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 0x55afc4380b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 -- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 msgr2=0x55afc4381130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 0x55afc4381130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769e00700 1 -- 172.21.15.23:0/2026660923 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc76c1a4230 con 0x55afc43785d0 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc76a800700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 0x55afc4381130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.228+0000 7fc769400700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 0x55afc4380b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc769e00700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc7580116a0 tx=0x7fc758011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47aac2f599214b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc74b400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:50.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758004460 con 0x55afc43785d0 2024-03-30T04:15:50.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc758003680 con 0x55afc43785d0 2024-03-30T04:15:50.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55afc428b380 con 0x55afc43785d0 2024-03-30T04:15:50.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc758003840 con 0x55afc43785d0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc7846fdb80 10 client.19188 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.229+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55afc40f8b60 con 0x55afc43785d0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55afc428b380 con 0x55afc43785d0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc75800fbd0 con 0x55afc43785d0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 10 client.19188 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 10 client.19188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 20 client.19188 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc439bdf0 0x55afc439e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 4 client.19188 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.230+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afc42c3700 con 0x55afc439bdf0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.231+0000 7fc769400700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc439bdf0 0x55afc439e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.231+0000 7fc769400700 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc439bdf0 0x55afc439e210 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c8576446a1b98fc server_cookie=52dc9a7456cf9ec5 in_seq=0 out_seq=0 2024-03-30T04:15:50.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.231+0000 7fc74b400700 10 client.19188 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc74b400700 1 -- 172.21.15.23:0/2026660923 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55afc42c3700 con 0x55afc439bdf0 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc74b400700 10 client.19188 handle_command_reply: tid=0 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 client.19188 shutdown 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 msgr2=0x55afc4383810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc7580116a0 tx=0x7fc758011a60 comp rx=0 tx=0).stop 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 shutdown_connections 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4383df0 0x55afc4381130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc4383810 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc439bdf0 0x55afc439e210 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/2026660923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4378ef0 0x55afc4380b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 >> 172.21.15.23:0/2026660923 conn(0x55afc42d86a0 msgr2=0x55afc437a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc74b400700 0 client.19188 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:50.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 shutdown_connections 2024-03-30T04:15:50.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.232+0000 7fc7846fdb80 1 -- 172.21.15.23:0/2026660923 wait complete. 2024-03-30T04:15:50.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.233+0000 7fc7846fdb80 20 client.19188 trim_cache size 0 max 16384 2024-03-30T04:15:50.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.251+0000 7fc7846fdb80 1 Processor -- start 2024-03-30T04:15:50.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 -- start start 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 0x55afc43861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc4378ef0 0x55afc43930d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc43994d0 con 0x55afc4378ef0 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4382100 con 0x55afc43785d0 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4379850 con 0x55afc4383df0 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc769400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc4378ef0 0x55afc43930d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc76a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc76a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57490/0 (socket says 172.21.15.23:57490) 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.252+0000 7fc76a800700 1 -- 172.21.15.23:0/3263327529 learned_addr learned my addr 172.21.15.23:0/3263327529 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc769e00700 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 0x55afc43861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 -- 172.21.15.23:0/3263327529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 msgr2=0x55afc43861e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 0x55afc43861e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 -- 172.21.15.23:0/3263327529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc4378ef0 msgr2=0x55afc43930d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc4378ef0 0x55afc43930d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 -- 172.21.15.23:0/3263327529 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afc4290ea0 con 0x55afc4383df0 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc769e00700 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 0x55afc43861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:50.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc76a800700 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc7640cb5b0 tx=0x7fc76409b7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd6c4a06ab31ca84 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc757a00700 1 -- 172.21.15.23:0/3263327529 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7640a1030 con 0x55afc4383df0 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc757a00700 1 -- 172.21.15.23:0/3263327529 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7640a3340 con 0x55afc4383df0 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.253+0000 7fc757a00700 1 -- 172.21.15.23:0/3263327529 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76409b1b0 con 0x55afc4383df0 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3263327529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 msgr2=0x55afc437f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc7640cb5b0 tx=0x7fc76409b7c0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3263327529 shutdown_connections 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc437f430 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc43785d0 0x55afc43861e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3263327529 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc4378ef0 0x55afc43930d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3263327529 >> 172.21.15.23:0/3263327529 conn(0x55afc42d86a0 msgr2=0x55afc436d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3263327529 shutdown_connections 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.254+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3263327529 wait complete. 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 -- start start 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 Processor -- start 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43932c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4391d00 con 0x55afc43785d0 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc4391e70 con 0x55afc4378ef0 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afc43eec20 con 0x55afc4383df0 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.255+0000 7fc7846fdb80 10 client.0 _ll_register_callbacks cb 0x55afc4270470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc769e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43932c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc769400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc769400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43342/0 (socket says 172.21.15.23:43342) 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc769400700 1 -- 172.21.15.23:0/3530220046 learned_addr learned my addr 172.21.15.23:0/3530220046 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 -- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 msgr2=0x55afc43910b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 -- 172.21.15.23:0/3530220046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 msgr2=0x55afc43932c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43932c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 -- 172.21.15.23:0/3530220046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afc414c3b0 con 0x55afc4383df0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc769400700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc76a800700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43915f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55afc436e790 tx=0x7fc7640ce120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d630ba00d95923a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.256+0000 7fc74b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76409ed80 con 0x55afc4383df0 2024-03-30T04:15:50.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc76409af50 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc76409b650 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7640a2eb0 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc7846fdb80 10 client.29161 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55afc4150cd0 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.257+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc76409b650 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc764099eb0 con 0x55afc4383df0 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 10 client.29161 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 10 client.29161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 20 client.29161 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc43efe10 0x55afc43f2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 4 client.29161 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.258+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55afc4119cf0 con 0x55afc43efe10 2024-03-30T04:15:50.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.259+0000 7fc769e00700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc43efe10 0x55afc43f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.259+0000 7fc74b400700 10 client.29161 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.259+0000 7fc769e00700 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc43efe10 0x55afc43f2200 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daed457b3327b941 server_cookie=4e186793100f8032 in_seq=0 out_seq=0 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc74b400700 1 -- 172.21.15.23:0/3530220046 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55afc4119cf0 con 0x55afc43efe10 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc74b400700 10 client.29161 handle_command_reply: tid=0 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 client.29161 shutdown 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 msgr2=0x55afc43915f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43915f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x55afc436e790 tx=0x7fc7640ce120 comp rx=0 tx=0).stop 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 shutdown_connections 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55afc43efe10 0x55afc43f2200 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55afc4383df0 0x55afc43915f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55afc43785d0 0x55afc43932c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 --2- 172.21.15.23:0/3530220046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55afc4378ef0 0x55afc43910b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 >> 172.21.15.23:0/3530220046 conn(0x55afc42d86a0 msgr2=0x55afc437f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc74b400700 0 client.29161 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 shutdown_connections 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 1 -- 172.21.15.23:0/3530220046 wait complete. 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.260+0000 7fc7846fdb80 20 client.29161 trim_cache size 0 max 16384 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:15:50.267 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 msgr2=0x7fc7500a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 0x7fc7500a3750 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc76c198820 tx=0x7fc75c009870 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 msgr2=0x7fc76c19c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc76400b070 tx=0x7fc76400bda0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 shutdown_connections 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc7500a1290 0x7fc7500a3750 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c19bbb0 0x7fc76c19c030 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc76c100f70 0x7fc76c197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 --2- 172.21.15.23:0/3406451186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc76c106090 0x7fc76c197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 >> 172.21.15.23:0/3406451186 conn(0x7fc76c0fc7f0 msgr2=0x7fc76c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 shutdown_connections 2024-03-30T04:15:50.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.261+0000 7fc770c00700 1 -- 172.21.15.23:0/3406451186 wait complete. 2024-03-30T04:15:50.280 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:15:50.280 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:15:50.353 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:50 smithi023 ceph-mon[38254]: pgmap v435: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 399 op/s 2024-03-30T04:15:50.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:50 smithi193 ceph-mon[79581]: pgmap v435: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 399 op/s 2024-03-30T04:15:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 -- 172.21.15.23:0/3980943569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0101810 msgr2=0x7fa4e0105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/3980943569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0101810 0x7fa4e0105850 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009990 tx=0x7fa4cc009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 -- 172.21.15.23:0/3980943569 shutdown_connections 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/3980943569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/3980943569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e0101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/3980943569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0101810 0x7fa4e0105850 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 -- 172.21.15.23:0/3980943569 >> 172.21.15.23:0/3980943569 conn(0x7fa4e00fc760 msgr2=0x7fa4e00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.536+0000 7fa4e7e00700 1 -- 172.21.15.23:0/3980943569 shutdown_connections 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.537+0000 7fa4e7e00700 1 -- 172.21.15.23:0/3980943569 wait complete. 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.538+0000 7fa4e7e00700 1 Processor -- start 2024-03-30T04:15:50.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.538+0000 7fa4e7e00700 1 -- start start 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01a5960 con 0x7fa4e0105f80 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01a5ad0 con 0x7fa4e0100e60 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e7e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01a5c90 con 0x7fa4e0073b30 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e4a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57518/0 (socket says 172.21.15.23:57518) 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 -- 172.21.15.23:0/1681118806 learned_addr learned my addr 172.21.15.23:0/1681118806 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5400700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 -- 172.21.15.23:0/1681118806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 msgr2=0x7fa4e00730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e00730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 -- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 msgr2=0x7fa4e00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.539+0000 7fa4e5e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e5e00700 1 -- 172.21.15.23:0/1681118806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc0095f0 con 0x7fa4e0073b30 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e5400700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e4a00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e5e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800ef30 tx=0x7fa4d800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8018070 con 0x7fa4e0073b30 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4d800f040 con 0x7fa4e0073b30 2024-03-30T04:15:50.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01a5f10 con 0x7fa4e0073b30 2024-03-30T04:15:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d800cdb0 con 0x7fa4e0073b30 2024-03-30T04:15:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.540+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01a6390 con 0x7fa4e0073b30 2024-03-30T04:15:50.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.541+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0109250 con 0x7fa4e0073b30 2024-03-30T04:15:50.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.544+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa4d8003a00 con 0x7fa4e0073b30 2024-03-30T04:15:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.545+0000 7fa4de000700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 0x7fa4c80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.546+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa4d80ccd30 con 0x7fa4e0073b30 2024-03-30T04:15:50.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.546+0000 7fa4e5400700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 0x7fa4c80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:50.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.546+0000 7fa4e5400700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 0x7fa4c80a3750 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009cb0 tx=0x7fa4d4009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:50.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.547+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa4d8095dd0 con 0x7fa4e0073b30 2024-03-30T04:15:50.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 12237 30.39 MB/sec warmup 27 sec latency 1014.169 ms 2024-03-30T04:15:50.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:50 smithi154 ceph-mon[79161]: pgmap v435: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 399 op/s 2024-03-30T04:15:50.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.839+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4e0102330 con 0x7fa4e0073b30 2024-03-30T04:15:50.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.840+0000 7fa4de000700 1 -- 172.21.15.23:0/1681118806 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10931 (secure 0 0 0) 0x7fa4d80993f0 con 0x7fa4e0073b30 2024-03-30T04:15:50.839 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:15:50.839 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:15:25.335802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:15:50.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 msgr2=0x7fa4c80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 0x7fa4c80a3750 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009cb0 tx=0x7fa4d4009260 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 msgr2=0x7fa4e01a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800ef30 tx=0x7fa4d800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 shutdown_connections 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4e0073b30 0x7fa4e01a5360 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa4c80a1290 0x7fa4c80a3750 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa4e0100e60 0x7fa4e00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 --2- 172.21.15.23:0/1681118806 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa4e0105f80 0x7fa4e00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 >> 172.21.15.23:0/1681118806 conn(0x7fa4e00fc760 msgr2=0x7fa4e00fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 shutdown_connections 2024-03-30T04:15:50.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:50.843+0000 7fa4e7e00700 1 -- 172.21.15.23:0/1681118806 wait complete. 2024-03-30T04:15:50.843 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-30T04:15:50.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 -- 172.21.15.23:0/300326816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754067e30 msgr2=0x7fe7540682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 --2- 172.21.15.23:0/300326816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754067e30 0x7fe7540682b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe7400099c0 tx=0x7fe740009cd0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 -- 172.21.15.23:0/300326816 shutdown_connections 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 --2- 172.21.15.23:0/300326816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe75410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 --2- 172.21.15.23:0/300326816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754105ce0 0x7fe7541060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 --2- 172.21.15.23:0/300326816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754067e30 0x7fe7540682b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 -- 172.21.15.23:0/300326816 >> 172.21.15.23:0/300326816 conn(0x7fe754076d40 msgr2=0x7fe754077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 -- 172.21.15.23:0/300326816 shutdown_connections 2024-03-30T04:15:51.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.101+0000 7fe75a800700 1 -- 172.21.15.23:0/300326816 wait complete. 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.102+0000 7fe75a800700 1 Processor -- start 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.102+0000 7fe75a800700 1 -- start start 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754067e30 0x7fe754197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754105ce0 0x7fe75419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419c180 con 0x7fe754067e30 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419c2f0 con 0x7fe7540687f0 2024-03-30T04:15:51.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe75a800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75419c4b0 con 0x7fe754105ce0 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43382/0 (socket says 172.21.15.23:43382) 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 -- 172.21.15.23:0/1743505764 learned_addr learned my addr 172.21.15.23:0/1743505764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe753400700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754067e30 0x7fe754197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe753e00700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754105ce0 0x7fe75419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 -- 172.21.15.23:0/1743505764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754105ce0 msgr2=0x7fe75419bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754105ce0 0x7fe75419bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 -- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754067e30 msgr2=0x7fe754197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754067e30 0x7fe754197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.103+0000 7fe752a00700 1 -- 172.21.15.23:0/1743505764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7400095f0 con 0x7fe7540687f0 2024-03-30T04:15:51.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe752a00700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe74000b580 tx=0x7fe740005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe74be00700 1 -- 172.21.15.23:0/1743505764 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74001d070 con 0x7fe7540687f0 2024-03-30T04:15:51.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe74be00700 1 -- 172.21.15.23:0/1743505764 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe74000fb30 con 0x7fe7540687f0 2024-03-30T04:15:51.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75419c730 con 0x7fe7540687f0 2024-03-30T04:15:51.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe74be00700 1 -- 172.21.15.23:0/1743505764 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740022ab0 con 0x7fe7540687f0 2024-03-30T04:15:51.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.104+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7541a3fd0 con 0x7fe7540687f0 2024-03-30T04:15:51.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.108+0000 7fe74be00700 1 -- 172.21.15.23:0/1743505764 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe74000fca0 con 0x7fe7540687f0 2024-03-30T04:15:51.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.109+0000 7fe74be00700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 0x7fe7380a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.109+0000 7fe76e306b80 1 Processor -- start 2024-03-30T04:15:51.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.109+0000 7fe74be00700 1 -- 172.21.15.23:0/1743505764 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe7400ce4c0 con 0x7fe7540687f0 2024-03-30T04:15:51.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.109+0000 7fe753400700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 0x7fe7380a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 -- start start 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b16cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06a60990 con 0x55be06b075d0 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06a5e350 con 0x55be06b07ef0 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe76e306b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06a59c40 con 0x55be06b17200 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe753400700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 0x7fe7380a37d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe744007c60 tx=0x7fe7440073d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe752a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b16cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b079b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35378/0 (socket says 172.21.15.23:35378) 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe753400700 1 -- 172.21.15.23:0/495871561 learned_addr learned my addr 172.21.15.23:0/495871561 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.110+0000 7fe753e00700 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 -- 172.21.15.23:0/495871561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 msgr2=0x55be06b195f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b195f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 -- 172.21.15.23:0/495871561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 msgr2=0x55be06b079b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b079b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 -- 172.21.15.23:0/495871561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be068db3b0 con 0x55be06b07ef0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe752a00700 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b16cc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe7400cb830 tx=0x7fe7400057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c499bb6781b47 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe748c00700 1 -- 172.21.15.23:0/495871561 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74009b980 con 0x55be06b07ef0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe748c00700 1 -- 172.21.15.23:0/495871561 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe740095e90 con 0x55be06b07ef0 2024-03-30T04:15:51.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.111+0000 7fe748c00700 1 -- 172.21.15.23:0/495871561 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74009a770 con 0x55be06b07ef0 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 -- 172.21.15.23:0/495871561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 msgr2=0x55be06b16cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b16cc0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe7400cb830 tx=0x7fe7400057f0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 -- 172.21.15.23:0/495871561 shutdown_connections 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b195f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b079b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 --2- 172.21.15.23:0/495871561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b16cc0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 -- 172.21.15.23:0/495871561 >> 172.21.15.23:0/495871561 conn(0x55be06a676a0 msgr2=0x55be06b03a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 -- 172.21.15.23:0/495871561 shutdown_connections 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.112+0000 7fe76e306b80 1 -- 172.21.15.23:0/495871561 wait complete. 2024-03-30T04:15:51.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.113+0000 7fe76e306b80 1 -- start start 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.113+0000 7fe76e306b80 1 Processor -- start 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b13ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b14530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be069f5d80 con 0x55be06b075d0 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be069f5a20 con 0x55be06b07ef0 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be068dd6a0 con 0x55be06b17200 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe76e306b80 10 client.0 _ll_register_callbacks cb 0x55be069ff470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35392/0 (socket says 172.21.15.23:35392) 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe753e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b14530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.114+0000 7fe753400700 1 -- 172.21.15.23:0/1849313599 learned_addr learned my addr 172.21.15.23:0/1849313599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 -- 172.21.15.23:0/1849313599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 msgr2=0x55be06b14530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b14530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 -- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 msgr2=0x55be06b13ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b13ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 -- 172.21.15.23:0/1849313599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7541a3fd0 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753e00700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe753400700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe7440116c0 tx=0x7fe7440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7915caeb275c532 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe73d600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74401e830 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe744017070 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.115+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be06a1a380 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe744010ec0 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 10 client.19212 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55be06887b60 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be06a1a380 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fe744028990 con 0x55be06b075d0 2024-03-30T04:15:51.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 10 client.19212 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 10 client.19212 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 20 client.19212 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b26b30 0x55be06b28f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 4 client.19212 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.116+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be06a52700 con 0x55be06b26b30 2024-03-30T04:15:51.118 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.117+0000 7fe752a00700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b26b30 0x55be06b28f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.117+0000 7fe752a00700 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b26b30 0x55be06b28f50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8e40b8fb3d604703 server_cookie=3823de519522b4f1 in_seq=0 out_seq=0 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.117+0000 7fe73d600700 10 client.19212 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe73d600700 1 -- 172.21.15.23:0/1849313599 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55be06a52700 con 0x55be06b26b30 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe73d600700 10 client.19212 handle_command_reply: tid=0 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 client.19212 shutdown 2024-03-30T04:15:51.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 msgr2=0x55be06b16be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fe7440116c0 tx=0x7fe7440047c0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 shutdown_connections 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b26b30 0x55be06b28f50 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b17200 0x55be06b14530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b16be0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 --2- 172.21.15.23:0/1849313599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b07ef0 0x55be06b13ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe73d600700 0 client.19212 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 >> 172.21.15.23:0/1849313599 conn(0x55be06a676a0 msgr2=0x55be06b096c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.118+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 shutdown_connections 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.119+0000 7fe76e306b80 1 -- 172.21.15.23:0/1849313599 wait complete. 2024-03-30T04:15:51.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.119+0000 7fe76e306b80 20 client.19212 trim_cache size 0 max 16384 2024-03-30T04:15:51.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.134+0000 7fe76e306b80 1 Processor -- start 2024-03-30T04:15:51.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.134+0000 7fe76e306b80 1 -- start start 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b075d0 0x55be06b144f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b0a5e0 con 0x55be06b17200 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b24270 con 0x55be06b075d0 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe76e306b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b292e0 con 0x55be06b07ef0 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe753e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe753e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35406/0 (socket says 172.21.15.23:35406) 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.135+0000 7fe753e00700 1 -- 172.21.15.23:0/925771317 learned_addr learned my addr 172.21.15.23:0/925771317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 -- 172.21.15.23:0/925771317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 msgr2=0x55be06b1dbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1dbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 -- 172.21.15.23:0/925771317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b075d0 msgr2=0x55be06b144f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b075d0 0x55be06b144f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 -- 172.21.15.23:0/925771317 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be06a1fea0 con 0x55be06b17200 2024-03-30T04:15:51.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe753e00700 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e00 tx=0x7fe74c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327bf106042a74d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe749600700 1 -- 172.21.15.23:0/925771317 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c004920 con 0x55be06b17200 2024-03-30T04:15:51.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe749600700 1 -- 172.21.15.23:0/925771317 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe74c00f040 con 0x55be06b17200 2024-03-30T04:15:51.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.136+0000 7fe749600700 1 -- 172.21.15.23:0/925771317 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74c020970 con 0x55be06b17200 2024-03-30T04:15:51.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 -- 172.21.15.23:0/925771317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 msgr2=0x55be06afcf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fe74c009e00 tx=0x7fe74c009e30 comp rx=0 tx=0).stop 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 -- 172.21.15.23:0/925771317 shutdown_connections 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b17200 0x55be06afcf70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b075d0 0x55be06b144f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 --2- 172.21.15.23:0/925771317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 -- 172.21.15.23:0/925771317 >> 172.21.15.23:0/925771317 conn(0x55be06a676a0 msgr2=0x55be06afc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 -- 172.21.15.23:0/925771317 shutdown_connections 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.137+0000 7fe76e306b80 1 -- 172.21.15.23:0/925771317 wait complete. 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 -- start start 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 Processor -- start 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b1bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 0x55be06b79980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b79ec0 con 0x55be06b075d0 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b7a000 con 0x55be06b17200 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be06b7a140 con 0x55be06b07ef0 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.138+0000 7fe76e306b80 10 client.0 _ll_register_callbacks cb 0x55be069ff470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b1bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe753e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 0x55be06b79980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe753400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b1bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35408/0 (socket says 172.21.15.23:35408) 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe753400700 1 -- 172.21.15.23:0/2941364094 learned_addr learned my addr 172.21.15.23:0/2941364094 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 -- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 msgr2=0x55be06b79980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 0x55be06b79980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 -- 172.21.15.23:0/2941364094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 msgr2=0x55be06b1bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b1bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 -- 172.21.15.23:0/2941364094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c009ab0 con 0x55be06b07ef0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe753e00700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 0x55be06b79980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe752a00700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1c300 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe7400cb860 tx=0x7fe740009cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f30634146caec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe73d600700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7400b1030 con 0x55be06b07ef0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe7400b2070 con 0x55be06b07ef0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.139+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe740095e90 con 0x55be06b07ef0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe74009b300 con 0x55be06b07ef0 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 10 client.29182 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55be068dfcd0 con 0x55be06b07ef0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe74007e020 con 0x55be06b07ef0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fe74009b4e0 con 0x55be06b07ef0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 10 client.29182 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 10 client.29182 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 20 client.29182 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b7b530 0x55be06b7d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 4 client.29182 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55be068a8cf0 con 0x55be06b7b530 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.140+0000 7fe753400700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b7b530 0x55be06b7d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe753400700 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b7b530 0x55be06b7d950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=75d7b3c271583ed8 server_cookie=372bd7c26215a19d in_seq=0 out_seq=0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe73d600700 10 client.29182 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe73d600700 1 -- 172.21.15.23:0/2941364094 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55be068a8cf0 con 0x55be06b7b530 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe73d600700 10 client.29182 handle_command_reply: tid=0 2024-03-30T04:15:51.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 client.29182 shutdown 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 msgr2=0x55be06b1c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1c300 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe7400cb860 tx=0x7fe740009cd0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 shutdown_connections 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55be06b17200 0x55be06b79980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55be06b075d0 0x55be06b1bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x55be06b7b530 0x55be06b7d950 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 --2- 172.21.15.23:0/2941364094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be06b07ef0 0x55be06b1c300 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe73d600700 0 client.29182 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.141+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 >> 172.21.15.23:0/2941364094 conn(0x55be06a676a0 msgr2=0x55be06afcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 shutdown_connections 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe76e306b80 1 -- 172.21.15.23:0/2941364094 wait complete. 2024-03-30T04:15:51.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe76e306b80 20 client.29182 trim_cache size 0 max 16384 2024-03-30T04:15:51.149 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:51.149 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 msgr2=0x7fe7380a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 0x7fe7380a37d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe744007c60 tx=0x7fe7440073d0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 msgr2=0x7fe7541977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe74000b580 tx=0x7fe740005740 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 shutdown_connections 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe754105ce0 0x7fe75419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe754067e30 0x7fe754197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.142+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe7380a1310 0x7fe7380a37d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.143+0000 7fe75a800700 1 --2- 172.21.15.23:0/1743505764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7540687f0 0x7fe7541977a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.143+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 >> 172.21.15.23:0/1743505764 conn(0x7fe754076d40 msgr2=0x7fe7540fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.143+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 shutdown_connections 2024-03-30T04:15:51.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.143+0000 7fe75a800700 1 -- 172.21.15.23:0/1743505764 wait complete. 2024-03-30T04:15:51.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:15:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 -- 172.21.15.23:0/3134637041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828067e30 msgr2=0x7f18280682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 --2- 172.21.15.23:0/3134637041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828067e30 0x7f18280682b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f18140099c0 tx=0x7f1814009cd0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 -- 172.21.15.23:0/3134637041 shutdown_connections 2024-03-30T04:15:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 --2- 172.21.15.23:0/3134637041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18280687f0 0x7f182810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 --2- 172.21.15.23:0/3134637041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1828105ce0 0x7f18281060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 --2- 172.21.15.23:0/3134637041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828067e30 0x7f18280682b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 -- 172.21.15.23:0/3134637041 >> 172.21.15.23:0/3134637041 conn(0x7f1828076d40 msgr2=0x7f1828077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.415+0000 7f182ce00700 1 -- 172.21.15.23:0/3134637041 shutdown_connections 2024-03-30T04:15:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.416+0000 7f182ce00700 1 -- 172.21.15.23:0/3134637041 wait complete. 2024-03-30T04:15:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 Processor -- start 2024-03-30T04:15:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 -- start start 2024-03-30T04:15:51.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18280687f0 0x7f18281999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828105ce0 0x7f182819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182819e380 con 0x7f18280687f0 2024-03-30T04:15:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182819e4f0 con 0x7f1828105ce0 2024-03-30T04:15:51.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f182ce00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182819e6b0 con 0x7f1828067e30 2024-03-30T04:15:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.417+0000 7f1825e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57572/0 (socket says 172.21.15.23:57572) 2024-03-30T04:15:51.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 -- 172.21.15.23:0/3904296947 learned_addr learned my addr 172.21.15.23:0/3904296947 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1826800700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828105ce0 0x7f182819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825400700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18280687f0 0x7f18281999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 -- 172.21.15.23:0/3904296947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828105ce0 msgr2=0x7f182819dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828105ce0 0x7f182819dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 -- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18280687f0 msgr2=0x7f18281999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18280687f0 0x7f18281999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 -- 172.21.15.23:0/3904296947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140095f0 con 0x7f1828067e30 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.418+0000 7f1825e00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f181800e9a0 tx=0x7f181800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.419+0000 7f181ea00700 1 -- 172.21.15.23:0/3904296947 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818009790 con 0x7f1828067e30 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.419+0000 7f181ea00700 1 -- 172.21.15.23:0/3904296947 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1818010430 con 0x7f1828067e30 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.419+0000 7f181ea00700 1 -- 172.21.15.23:0/3904296947 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818018680 con 0x7f1828067e30 2024-03-30T04:15:51.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.419+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182819e990 con 0x7f1828067e30 2024-03-30T04:15:51.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.419+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18281a61d0 con 0x7f1828067e30 2024-03-30T04:15:51.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.423+0000 7f181ea00700 1 -- 172.21.15.23:0/3904296947 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f18180105a0 con 0x7f1828067e30 2024-03-30T04:15:51.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 Processor -- start 2024-03-30T04:15:51.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 -- start start 2024-03-30T04:15:51.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 0x556d7aee89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 0x556d7aef38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7ae41990 con 0x556d7aee8ef0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7ae3f350 con 0x556d7aee85d0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.424+0000 7f18408c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7ae3ac40 con 0x556d7aef3df0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1826800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 0x556d7aee89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1826800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 0x556d7aee89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43432/0 (socket says 172.21.15.23:43432) 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1826800700 1 -- 172.21.15.23:0/2340740284 learned_addr learned my addr 172.21.15.23:0/2340740284 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f181ea00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 0x7f180c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825e00700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 0x556d7aef38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f181ea00700 1 -- 172.21.15.23:0/3904296947 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1818014070 con 0x7f1828067e30 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1826800700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 0x7f180c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825e00700 1 -- 172.21.15.23:0/2340740284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 msgr2=0x556d7aef61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825400700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825e00700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825e00700 1 -- 172.21.15.23:0/2340740284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 msgr2=0x556d7aee89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.425+0000 7f1825e00700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 0x556d7aee89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1825e00700 1 -- 172.21.15.23:0/2340740284 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d7acbc3b0 con 0x556d7aee8ef0 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1825400700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1825e00700 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 0x556d7aef38b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f18180ca570 tx=0x7f181808f680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b583215695832b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1813400700 1 -- 172.21.15.23:0/2340740284 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181809a630 con 0x556d7aee8ef0 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1813400700 1 -- 172.21.15.23:0/2340740284 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f181801c1c0 con 0x556d7aee8ef0 2024-03-30T04:15:51.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.426+0000 7f1813400700 1 -- 172.21.15.23:0/2340740284 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180109a0 con 0x556d7aee8ef0 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 -- 172.21.15.23:0/2340740284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 msgr2=0x556d7aef38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f1826800700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 0x7f180c0a37d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f182000fb80 tx=0x7f1820004b40 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 0x556d7aef38b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f18180ca570 tx=0x7f181808f680 comp rx=0 tx=0).stop 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 -- 172.21.15.23:0/2340740284 shutdown_connections 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee85d0 0x556d7aee89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2340740284 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee8ef0 0x556d7aef38b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 -- 172.21.15.23:0/2340740284 >> 172.21.15.23:0/2340740284 conn(0x556d7ae486a0 msgr2=0x556d7aee4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.430 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 -- 172.21.15.23:0/2340740284 shutdown_connections 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.427+0000 7f18408c6b80 1 -- 172.21.15.23:0/2340740284 wait complete. 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.428+0000 7f18408c6b80 1 -- start start 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.428+0000 7f18408c6b80 1 Processor -- start 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aef37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7add6d80 con 0x556d7aee85d0 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7add6a20 con 0x556d7aee8ef0 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7acbe6a0 con 0x556d7aef3df0 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f18408c6b80 10 client.0 _ll_register_callbacks cb 0x556d7ade0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f1826800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f1826800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57584/0 (socket says 172.21.15.23:57584) 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f1826800700 1 -- 172.21.15.23:0/2480461810 learned_addr learned my addr 172.21.15.23:0/2480461810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f1825400700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.429+0000 7f1825e00700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aef37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 -- 172.21.15.23:0/2480461810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 msgr2=0x556d7aef0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 -- 172.21.15.23:0/2480461810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 msgr2=0x556d7aef37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aef37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 -- 172.21.15.23:0/2480461810 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18281a61d0 con 0x556d7aef3df0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1826800700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1820018c70 tx=0x7f1820004ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6508de6ebe901f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1807e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000e040 con 0x556d7aef3df0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1820003e10 con 0x556d7aef3df0 2024-03-30T04:15:51.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.430+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d7adfb380 con 0x556d7aef3df0 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1820007990 con 0x556d7aef3df0 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 10 client.29191 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556d7ac68b60 con 0x556d7aef3df0 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d7adfb380 con 0x556d7aef3df0 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f182001a470 con 0x556d7aef3df0 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 10 client.29191 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 10 client.29191 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 20 client.29191 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af03a00 0x556d7af05e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 4 client.29191 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.431+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d7ae33700 con 0x556d7af03a00 2024-03-30T04:15:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.432+0000 7f1825e00700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af03a00 0x556d7af05e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.432+0000 7f1825e00700 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af03a00 0x556d7af05e20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=484a1ef0d34353c4 server_cookie=cb558a7a0c99d653 in_seq=0 out_seq=0 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.432+0000 7f1807e00700 10 client.29191 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f1807e00700 1 -- 172.21.15.23:0/2480461810 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556d7ae33700 con 0x556d7af03a00 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f1807e00700 10 client.29191 handle_command_reply: tid=0 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 client.29191 shutdown 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 msgr2=0x556d7aef1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1820018c70 tx=0x7f1820004ad0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 shutdown_connections 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af03a00 0x556d7af05e20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aef1230 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aef37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f1807e00700 0 client.29191 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 --2- 172.21.15.23:0/2480461810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.433+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 >> 172.21.15.23:0/2480461810 conn(0x556d7ae486a0 msgr2=0x556d7aeea670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.434+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 shutdown_connections 2024-03-30T04:15:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.434+0000 7f18408c6b80 1 -- 172.21.15.23:0/2480461810 wait complete. 2024-03-30T04:15:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.434+0000 7f18408c6b80 20 client.29191 trim_cache size 0 max 16384 2024-03-30T04:15:51.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.451+0000 7f18408c6b80 1 Processor -- start 2024-03-30T04:15:51.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.451+0000 7f18408c6b80 1 -- start start 2024-03-30T04:15:51.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aee3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aedd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7aee9570 con 0x556d7aee85d0 2024-03-30T04:15:51.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7aeddae0 con 0x556d7aee8ef0 2024-03-30T04:15:51.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f18408c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7aef1570 con 0x556d7aef3df0 2024-03-30T04:15:51.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f1825400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f1825400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43444/0 (socket says 172.21.15.23:43444) 2024-03-30T04:15:51.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.452+0000 7f1825400700 1 -- 172.21.15.23:0/1248701766 learned_addr learned my addr 172.21.15.23:0/1248701766 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825400700 1 -- 172.21.15.23:0/1248701766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 msgr2=0x556d7aedd5a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:51.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825400700 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aedd5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825400700 1 -- 172.21.15.23:0/1248701766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 msgr2=0x556d7aee3f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825e00700 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aee3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825400700 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aee3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825400700 1 -- 172.21.15.23:0/1248701766 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d7ae00ea0 con 0x556d7aee8ef0 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.453+0000 7f1825e00700 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aee3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.454+0000 7f1825400700 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f18140099c0 tx=0x7f1814005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f06793e47ea690b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.454+0000 7f1813e00700 1 -- 172.21.15.23:0/1248701766 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181401d030 con 0x556d7aee8ef0 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.454+0000 7f1813e00700 1 -- 172.21.15.23:0/1248701766 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f181401e040 con 0x556d7aee8ef0 2024-03-30T04:15:51.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.454+0000 7f1813e00700 1 -- 172.21.15.23:0/1248701766 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18140038c0 con 0x556d7aee8ef0 2024-03-30T04:15:51.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.455+0000 7f18408c6b80 1 -- 172.21.15.23:0/1248701766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 msgr2=0x556d7aeeb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.455+0000 7f18408c6b80 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f18140099c0 tx=0x7f1814005fd0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 -- 172.21.15.23:0/1248701766 shutdown_connections 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aef3df0 0x556d7aedd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aee85d0 0x556d7aee3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 --2- 172.21.15.23:0/1248701766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aeeb5c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 -- 172.21.15.23:0/1248701766 >> 172.21.15.23:0/1248701766 conn(0x556d7ae486a0 msgr2=0x556d7af0a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 -- 172.21.15.23:0/1248701766 shutdown_connections 2024-03-30T04:15:51.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 -- 172.21.15.23:0/1248701766 wait complete. 2024-03-30T04:15:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.456+0000 7f18408c6b80 1 -- start start 2024-03-30T04:15:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.457+0000 7f18408c6b80 1 Processor -- start 2024-03-30T04:15:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.457+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.457+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f18408c6b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 0x556d7aef9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f18408c6b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7aef9c70 con 0x556d7aef3df0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1825e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f18408c6b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7aef9de0 con 0x556d7aee8ef0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f18408c6b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d7af57800 con 0x556d7aee85d0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1825400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1826800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 0x556d7aef9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1825400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43448/0 (socket says 172.21.15.23:43448) 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1825e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57602/0 (socket says 172.21.15.23:57602) 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f1825400700 1 -- 172.21.15.23:0/957585256 learned_addr learned my addr 172.21.15.23:0/957585256 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.458+0000 7f18408c6b80 10 client.0 _ll_register_callbacks cb 0x556d7ade0470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 -- 172.21.15.23:0/957585256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 msgr2=0x556d7aef9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 -- 172.21.15.23:0/957585256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 msgr2=0x556d7aef9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 0x556d7aef9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 -- 172.21.15.23:0/957585256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182810e550 con 0x556d7aee85d0 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825400700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1826800700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 0x556d7aef9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1825e00700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1818099c30 tx=0x7f18180963a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7c5c43ab2986849 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1807e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180a23b0 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f181809dd60 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f181801c1c0 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.459+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1818010880 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 10 client.29194 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556d7acc0cd0 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18180b8020 con 0x556d7aee85d0 2024-03-30T04:15:51.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f181809a190 con 0x556d7aee85d0 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 10 client.29194 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 10 client.29194 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 20 client.29194 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af58c50 0x556d7af5b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 4 client.29194 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556d7ac89cf0 con 0x556d7af58c50 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.460+0000 7f1825400700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af58c50 0x556d7af5b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.461+0000 7f1825400700 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af58c50 0x556d7af5b040 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b71a60f3ed3fab8a server_cookie=8efca8629f35722f in_seq=0 out_seq=0 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.461+0000 7f1807e00700 10 client.29194 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.461+0000 7f1807e00700 1 -- 172.21.15.23:0/957585256 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556d7ac89cf0 con 0x556d7af58c50 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.461+0000 7f1807e00700 10 client.29194 handle_command_reply: tid=0 2024-03-30T04:15:51.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.461+0000 7f18408c6b80 1 client.29194 shutdown 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 msgr2=0x556d7aefaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f1818099c30 tx=0x7f18180963a0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 shutdown_connections 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x556d7af58c50 0x556d7af5b040 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x556d7aef3df0 0x556d7aef9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556d7aee85d0 0x556d7aefaab0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 --2- 172.21.15.23:0/957585256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x556d7aee8ef0 0x556d7aef9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 >> 172.21.15.23:0/957585256 conn(0x556d7ae486a0 msgr2=0x556d7aeef3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f1807e00700 0 client.29194 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 shutdown_connections 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 1 -- 172.21.15.23:0/957585256 wait complete. 2024-03-30T04:15:51.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.462+0000 7f18408c6b80 20 client.29194 trim_cache size 0 max 16384 2024-03-30T04:15:51.464 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:51.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 msgr2=0x7f180c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 0x7f180c0a37d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f182000fb80 tx=0x7f1820004b40 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 msgr2=0x7f1828199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f181800e9a0 tx=0x7f181800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 shutdown_connections 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f180c0a1310 0x7f180c0a37d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1828105ce0 0x7f182819dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1828067e30 0x7f1828199460 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 --2- 172.21.15.23:0/3904296947 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18280687f0 0x7f18281999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 >> 172.21.15.23:0/3904296947 conn(0x7f1828076d40 msgr2=0x7f1828104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 shutdown_connections 2024-03-30T04:15:51.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.463+0000 7f182ce00700 1 -- 172.21.15.23:0/3904296947 wait complete. 2024-03-30T04:15:51.482 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:51 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1681118806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:51.486 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:15:51.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:51 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1681118806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:51.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 12902 30.70 MB/sec warmup 28 sec latency 1629.156 ms 2024-03-30T04:15:51.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:51 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1681118806' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:15:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 -- 172.21.15.23:0/499229432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc924105ce0 msgr2=0x7fc9241060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 --2- 172.21.15.23:0/499229432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc924105ce0 0x7fc9241060c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc914009990 tx=0x7fc914009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 -- 172.21.15.23:0/499229432 shutdown_connections 2024-03-30T04:15:51.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 --2- 172.21.15.23:0/499229432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9240687f0 0x7fc92410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 --2- 172.21.15.23:0/499229432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc924105ce0 0x7fc9241060c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 --2- 172.21.15.23:0/499229432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924067e30 0x7fc9240682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 -- 172.21.15.23:0/499229432 >> 172.21.15.23:0/499229432 conn(0x7fc924076d40 msgr2=0x7fc924077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 -- 172.21.15.23:0/499229432 shutdown_connections 2024-03-30T04:15:51.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.740+0000 7fc92ac00700 1 -- 172.21.15.23:0/499229432 wait complete. 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 Processor -- start 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 -- start start 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9240687f0 0x7fc924114530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 0x7fc924110970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc924111080 con 0x7fc9240687f0 2024-03-30T04:15:51.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241111f0 con 0x7fc924067e30 2024-03-30T04:15:51.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.742+0000 7fc92ac00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241113b0 con 0x7fc924105ce0 2024-03-30T04:15:51.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43456/0 (socket says 172.21.15.23:43456) 2024-03-30T04:15:51.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 -- 172.21.15.23:0/4279364210 learned_addr learned my addr 172.21.15.23:0/4279364210 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923400700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9240687f0 0x7fc924114530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc928c00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 0x7fc924110970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 -- 172.21.15.23:0/4279364210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 msgr2=0x7fc924110970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 0x7fc924110970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 -- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9240687f0 msgr2=0x7fc924114530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9240687f0 0x7fc924114530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc923e00700 1 -- 172.21.15.23:0/4279364210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9140095f0 con 0x7fc924067e30 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.743+0000 7fc928c00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 0x7fc924110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc923e00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc914004730 tx=0x7fc914004760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc920c00700 1 -- 172.21.15.23:0/4279364210 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91401d070 con 0x7fc924067e30 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc920c00700 1 -- 172.21.15.23:0/4279364210 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc914005da0 con 0x7fc924067e30 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9241115d0 con 0x7fc924067e30 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc920c00700 1 -- 172.21.15.23:0/4279364210 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9140173f0 con 0x7fc924067e30 2024-03-30T04:15:51.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.744+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9241aa630 con 0x7fc924067e30 2024-03-30T04:15:51.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.748+0000 7fc920c00700 1 -- 172.21.15.23:0/4279364210 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fc914022670 con 0x7fc924067e30 2024-03-30T04:15:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.749+0000 7fc920c00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 0x7fc9080a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.749+0000 7fc920c00700 1 -- 172.21.15.23:0/4279364210 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc9140ce4d0 con 0x7fc924067e30 2024-03-30T04:15:51.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.749+0000 7fc923400700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 0x7fc9080a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.750+0000 7fc93e7ebb80 1 Processor -- start 2024-03-30T04:15:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc923400700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 0x7fc9080a3750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc9240ff1c0 tx=0x7fc910009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 -- start start 2024-03-30T04:15:51.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff8229b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 0x55ebff82d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff77b990 con 0x55ebff8225d0 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff779350 con 0x55ebff822ef0 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.751+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff774c40 con 0x55ebff82ddf0 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc923e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff8229b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc923e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff8229b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35490/0 (socket says 172.21.15.23:35490) 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57622/0 (socket says 172.21.15.23:57622) 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc923e00700 1 -- 172.21.15.23:0/2759919041 learned_addr learned my addr 172.21.15.23:0/2759919041 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc923400700 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 0x55ebff82d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 -- 172.21.15.23:0/2759919041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 msgr2=0x55ebff82d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 0x55ebff82d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 -- 172.21.15.23:0/2759919041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 msgr2=0x55ebff8229b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.752+0000 7fc928c00700 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff8229b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc928c00700 1 -- 172.21.15.23:0/2759919041 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebff5f63b0 con 0x55ebff82ddf0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc923400700 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 0x55ebff82d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc928c00700 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007e70 tx=0x7fc91c00c5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eee00451aefb3e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc919200700 1 -- 172.21.15.23:0/2759919041 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00cd40 con 0x55ebff82ddf0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc919200700 1 -- 172.21.15.23:0/2759919041 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc91c00f900 con 0x55ebff82ddf0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.753+0000 7fc919200700 1 -- 172.21.15.23:0/2759919041 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91c00e830 con 0x55ebff82ddf0 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/2759919041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 msgr2=0x55ebff8301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007e70 tx=0x7fc91c00c5a0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/2759919041 shutdown_connections 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff8301e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff8229b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/2759919041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff822ef0 0x55ebff82d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/2759919041 >> 172.21.15.23:0/2759919041 conn(0x55ebff7826a0 msgr2=0x55ebff81ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/2759919041 shutdown_connections 2024-03-30T04:15:51.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.754+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/2759919041 wait complete. 2024-03-30T04:15:51.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.755+0000 7fc93e7ebb80 1 -- start start 2024-03-30T04:15:51.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.755+0000 7fc93e7ebb80 1 Processor -- start 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.755+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.755+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff82ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.755+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff82ddf0 0x55ebff82b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff710d80 con 0x55ebff8225d0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff710a20 con 0x55ebff82ddf0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff5f86a0 con 0x55ebff822ef0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc93e7ebb80 10 client.0 _ll_register_callbacks cb 0x55ebff71a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35494/0 (socket says 172.21.15.23:35494) 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 -- 172.21.15.23:0/1922757172 learned_addr learned my addr 172.21.15.23:0/1922757172 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc928c00700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff82ddf0 0x55ebff82b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 -- 172.21.15.23:0/1922757172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 msgr2=0x55ebff82ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff82ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 -- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff82ddf0 msgr2=0x55ebff82b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff82ddf0 0x55ebff82b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.756+0000 7fc923e00700 1 -- 172.21.15.23:0/1922757172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c007aa0 con 0x55ebff8225d0 2024-03-30T04:15:51.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.757+0000 7fc923e00700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc9140cb8f0 tx=0x7fc914005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4dfd52632dc42a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.757+0000 7fc90d600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.757+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91409aaa0 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.757+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc91409b3b0 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.757+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebff735380 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.758+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9140ce330 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.758+0000 7fc93e7ebb80 10 client.19245 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.758+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ebff5a2b60 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.758+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ebff735380 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.758+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc9140a1170 con 0x55ebff8225d0 2024-03-30T04:15:51.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 10 client.19245 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 10 client.19245 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 20 client.19245 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff83d720 0x55ebff83fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 4 client.19245 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebff76d700 con 0x55ebff83d720 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.759+0000 7fc923400700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff83d720 0x55ebff83fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.760+0000 7fc90d600700 10 client.19245 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:51.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.760+0000 7fc923400700 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff83d720 0x55ebff83fb40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54b1a580630bf61e server_cookie=8db106dc8bb12e1f in_seq=0 out_seq=0 2024-03-30T04:15:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.763+0000 7fc90d600700 1 -- 172.21.15.23:0/1922757172 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ebff76d700 con 0x55ebff83d720 2024-03-30T04:15:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.763+0000 7fc90d600700 10 client.19245 handle_command_reply: tid=0 2024-03-30T04:15:51.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.763+0000 7fc93e7ebb80 1 client.19245 shutdown 2024-03-30T04:15:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 msgr2=0x55ebff82d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc9140cb8f0 tx=0x7fc914005f10 comp rx=0 tx=0).stop 2024-03-30T04:15:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 shutdown_connections 2024-03-30T04:15:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff83d720 0x55ebff83fb40 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff82ddf0 0x55ebff82b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff8225d0 0x55ebff82d7a0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1922757172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff82ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc90d600700 0 client.19245 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 >> 172.21.15.23:0/1922757172 conn(0x55ebff7826a0 msgr2=0x55ebff824670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.764+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 shutdown_connections 2024-03-30T04:15:51.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.765+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1922757172 wait complete. 2024-03-30T04:15:51.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.765+0000 7fc93e7ebb80 20 client.19245 trim_cache size 0 max 16384 2024-03-30T04:15:51.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 Processor -- start 2024-03-30T04:15:51.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 -- start start 2024-03-30T04:15:51.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff822ef0 0x55ebff834aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff82aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff83a4b0 con 0x55ebff822ef0 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff81f660 con 0x55ebff8225d0 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.783+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff823640 con 0x55ebff82ddf0 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43496/0 (socket says 172.21.15.23:43496) 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 -- 172.21.15.23:0/301103786 learned_addr learned my addr 172.21.15.23:0/301103786 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923400700 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff822ef0 0x55ebff834aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc928c00700 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff82aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 -- 172.21.15.23:0/301103786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 msgr2=0x55ebff82aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff82aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 -- 172.21.15.23:0/301103786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff822ef0 msgr2=0x55ebff834aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff822ef0 0x55ebff834aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 -- 172.21.15.23:0/301103786 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebff73aea0 con 0x55ebff8225d0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.784+0000 7fc923e00700 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc9140cb8a0 tx=0x7fc91409f110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84c88f718362df0e server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.785+0000 7fc919c00700 1 -- 172.21.15.23:0/301103786 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9140ce330 con 0x55ebff8225d0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.785+0000 7fc919c00700 1 -- 172.21.15.23:0/301103786 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc91409b3b0 con 0x55ebff8225d0 2024-03-30T04:15:51.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.785+0000 7fc919c00700 1 -- 172.21.15.23:0/301103786 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9140a3570 con 0x55ebff8225d0 2024-03-30T04:15:51.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.785+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/301103786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 msgr2=0x55ebff83fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.786 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.785+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc9140cb8a0 tx=0x7fc91409f110 comp rx=0 tx=0).stop 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/301103786 shutdown_connections 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff82ddf0 0x55ebff82aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff83fb40 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/301103786 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff822ef0 0x55ebff834aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/301103786 >> 172.21.15.23:0/301103786 conn(0x55ebff7826a0 msgr2=0x55ebff825e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/301103786 shutdown_connections 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/301103786 wait complete. 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 -- start start 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.786+0000 7fc93e7ebb80 1 Processor -- start 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff832fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 0x55ebff833500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff890530 con 0x55ebff82ddf0 2024-03-30T04:15:51.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff890670 con 0x55ebff8225d0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebff890800 con 0x55ebff822ef0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc93e7ebb80 10 client.0 _ll_register_callbacks cb 0x55ebff71a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc923e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43508/0 (socket says 172.21.15.23:43508) 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc923e00700 1 -- 172.21.15.23:0/1522131598 learned_addr learned my addr 172.21.15.23:0/1522131598 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc923400700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff832fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.787+0000 7fc928c00700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 0x55ebff833500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 -- 172.21.15.23:0/1522131598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 msgr2=0x55ebff832fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff832fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 -- 172.21.15.23:0/1522131598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 msgr2=0x55ebff833500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 0x55ebff833500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 -- 172.21.15.23:0/1522131598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebff5a2b60 con 0x55ebff8225d0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc928c00700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 0x55ebff833500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc923e00700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc914006010 tx=0x7fc9140970d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87d6cca32ed0e6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:51.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc90d600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:51.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9140ce330 con 0x55ebff8225d0 2024-03-30T04:15:51.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fc91409f1e0 con 0x55ebff8225d0 2024-03-30T04:15:51.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.788+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc91c00f900 con 0x55ebff8225d0 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.789+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9140a37a0 con 0x55ebff8225d0 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.789+0000 7fc93e7ebb80 10 client.29225 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.789+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ebff5facd0 con 0x55ebff8225d0 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.789+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9140a9020 con 0x55ebff8225d0 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.789+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fc91409a4d0 con 0x55ebff8225d0 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 10 client.29225 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:51.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 10 client.29225 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 20 client.29225 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff891af0 0x55ebff893ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 4 client.29225 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ebff5c3cf0 con 0x55ebff891af0 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc923400700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff891af0 0x55ebff893ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc923400700 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff891af0 0x55ebff893ee0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b3490a5bb36cd7b0 server_cookie=60dd6b73cdfed57d in_seq=0 out_seq=0 2024-03-30T04:15:51.791 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.790+0000 7fc90d600700 10 client.29225 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.792+0000 7fc90d600700 1 -- 172.21.15.23:0/1522131598 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ebff5c3cf0 con 0x55ebff891af0 2024-03-30T04:15:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.792+0000 7fc90d600700 10 client.29225 handle_command_reply: tid=0 2024-03-30T04:15:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.792+0000 7fc93e7ebb80 1 client.29225 shutdown 2024-03-30T04:15:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 msgr2=0x55ebff832a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc914006010 tx=0x7fc9140970d0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 shutdown_connections 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x55ebff891af0 0x55ebff893ee0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebff82ddf0 0x55ebff833500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55ebff8225d0 0x55ebff832a80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 --2- 172.21.15.23:0/1522131598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ebff822ef0 0x55ebff832fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 >> 172.21.15.23:0/1522131598 conn(0x55ebff7826a0 msgr2=0x55ebff829340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc90d600700 0 client.29225 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 shutdown_connections 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 1 -- 172.21.15.23:0/1522131598 wait complete. 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.793+0000 7fc93e7ebb80 20 client.29225 trim_cache size 0 max 16384 2024-03-30T04:15:51.793 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:51.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 msgr2=0x7fc9080a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 0x7fc9080a3750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc9240ff1c0 tx=0x7fc910009260 comp rx=0 tx=0).stop 2024-03-30T04:15:51.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 msgr2=0x7fc924113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:51.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc914004730 tx=0x7fc914004760 comp rx=0 tx=0).stop 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 shutdown_connections 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc924105ce0 0x7fc924110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fc9080a1290 0x7fc9080a3750 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc924067e30 0x7fc924113ff0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 --2- 172.21.15.23:0/4279364210 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc9240687f0 0x7fc924114530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 >> 172.21.15.23:0/4279364210 conn(0x7fc924076d40 msgr2=0x7fc924102920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 shutdown_connections 2024-03-30T04:15:51.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:51.794+0000 7fc92ac00700 1 -- 172.21.15.23:0/4279364210 wait complete. 2024-03-30T04:15:51.814 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:15:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 -- 172.21.15.23:0/3788588932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 msgr2=0x7f373410dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 --2- 172.21.15.23:0/3788588932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f373410dbf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3730009f90 tx=0x7f373000b330 comp rx=0 tx=0).stop 2024-03-30T04:15:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 -- 172.21.15.23:0/3788588932 shutdown_connections 2024-03-30T04:15:52.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 --2- 172.21.15.23:0/3788588932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f373410dbf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 --2- 172.21.15.23:0/3788588932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3734100f50 0x7f3734101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 --2- 172.21.15.23:0/3788588932 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734101900 0x7f3734105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.058+0000 7f373c800700 1 -- 172.21.15.23:0/3788588932 >> 172.21.15.23:0/3788588932 conn(0x7f37340fc7f0 msgr2=0x7f37340fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.059+0000 7f373c800700 1 -- 172.21.15.23:0/3788588932 shutdown_connections 2024-03-30T04:15:52.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.059+0000 7f373c800700 1 -- 172.21.15.23:0/3788588932 wait complete. 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 Processor -- start 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 -- start start 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f3734199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f373419ddf0 0x7f373419e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419e870 con 0x7f3734106070 2024-03-30T04:15:52.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419e9e0 con 0x7f3734100f50 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373c800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373419eba0 con 0x7f373419ddf0 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f3739e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f3739e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43518/0 (socket says 172.21.15.23:43518) 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f3739e00700 1 -- 172.21.15.23:0/2923972645 learned_addr learned my addr 172.21.15.23:0/2923972645 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f3739400700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f3734199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.060+0000 7f373a800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f373419ddf0 0x7f373419e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 -- 172.21.15.23:0/2923972645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f373419ddf0 msgr2=0x7f373419e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f373419ddf0 0x7f373419e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 -- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 msgr2=0x7f3734199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f3734199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 -- 172.21.15.23:0/2923972645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3728009520 con 0x7f3734100f50 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f3739e00700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f372800eb30 tx=0x7f372800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f372ea00700 1 -- 172.21.15.23:0/2923972645 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728009880 con 0x7f3734100f50 2024-03-30T04:15:52.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37240095f0 con 0x7f3734100f50 2024-03-30T04:15:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f372ea00700 1 -- 172.21.15.23:0/2923972645 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37280044e0 con 0x7f3734100f50 2024-03-30T04:15:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f372ea00700 1 -- 172.21.15.23:0/2923972645 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728003d80 con 0x7f3734100f50 2024-03-30T04:15:52.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.061+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3730009c40 con 0x7f3734100f50 2024-03-30T04:15:52.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.065+0000 7f372ea00700 1 -- 172.21.15.23:0/2923972645 <== mon.2 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3728003f20 con 0x7f3734100f50 2024-03-30T04:15:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.066+0000 7f3750236b80 1 Processor -- start 2024-03-30T04:15:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.066+0000 7f372ea00700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 0x7f371c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 -- start start 2024-03-30T04:15:52.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f372ea00700 1 -- 172.21.15.23:0/2923972645 <== mon.2 v2:172.21.15.154:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3728014070 con 0x7f3734100f50 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f373a800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 0x7f371c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031ec5df0 0x55c031ec81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031e13990 con 0x55c031ebaef0 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031e11350 con 0x55c031ec5df0 2024-03-30T04:15:52.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3750236b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031e0cc40 con 0x55c031eba5d0 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3739e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3739e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57656/0 (socket says 172.21.15.23:57656) 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.067+0000 7f3739e00700 1 -- 172.21.15.23:0/667037137 learned_addr learned my addr 172.21.15.23:0/667037137 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f373a800700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031ec5df0 0x55c031ec81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739400700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 -- 172.21.15.23:0/667037137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031ec5df0 msgr2=0x55c031ec81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031ec5df0 0x55c031ec81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 -- 172.21.15.23:0/667037137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 msgr2=0x55c031ec58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 -- 172.21.15.23:0/667037137 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c031c8e3b0 con 0x55c031eba5d0 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739400700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:52.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f373a800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 0x7f371c0a3750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3730009790 tx=0x7f3730009700 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3739e00700 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3728017040 tx=0x7f372809a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=256b3ec7b220e5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3723000700 1 -- 172.21.15.23:0/667037137 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372809a330 con 0x55c031eba5d0 2024-03-30T04:15:52.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3723000700 1 -- 172.21.15.23:0/667037137 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f372809e160 con 0x55c031eba5d0 2024-03-30T04:15:52.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.068+0000 7f3723000700 1 -- 172.21.15.23:0/667037137 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37280cd1e0 con 0x55c031eba5d0 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 -- 172.21.15.23:0/667037137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 msgr2=0x55c031eba9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3728017040 tx=0x7f372809a230 comp rx=0 tx=0).stop 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 -- 172.21.15.23:0/667037137 shutdown_connections 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031ec5df0 0x55c031ec81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031eba5d0 0x55c031eba9b0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 --2- 172.21.15.23:0/667037137 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.069+0000 7f3750236b80 1 -- 172.21.15.23:0/667037137 >> 172.21.15.23:0/667037137 conn(0x55c031e1a6a0 msgr2=0x55c031eb6a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.070+0000 7f3750236b80 1 -- 172.21.15.23:0/667037137 shutdown_connections 2024-03-30T04:15:52.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.070+0000 7f3750236b80 1 -- 172.21.15.23:0/667037137 wait complete. 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.070+0000 7f3750236b80 1 -- start start 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 Processor -- start 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031ec3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031da8d80 con 0x55c031ebaef0 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031da8a20 con 0x55c031eba5d0 2024-03-30T04:15:52.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031c906a0 con 0x55c031ec5df0 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.071+0000 7f3750236b80 10 client.0 _ll_register_callbacks cb 0x55c031db2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.072+0000 7f3739400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.072+0000 7f3739400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35536/0 (socket says 172.21.15.23:35536) 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.072+0000 7f3739400700 1 -- 172.21.15.23:0/136004871 learned_addr learned my addr 172.21.15.23:0/136004871 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 -- 172.21.15.23:0/136004871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 msgr2=0x55c031ec3200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031ec3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 -- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 msgr2=0x55c031ec5840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec5840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 -- 172.21.15.23:0/136004871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3730009c40 con 0x55c031ebaef0 2024-03-30T04:15:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3739400700 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f37240138b0 tx=0x7f3724013c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155f6dc882c3ad08 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.073+0000 7f3717400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.074+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724004520 con 0x55c031ebaef0 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.074+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3724004830 con 0x55c031ebaef0 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.074+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c031dcd380 con 0x55c031ebaef0 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.075+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37240049f0 con 0x55c031ebaef0 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.075+0000 7f3750236b80 10 client.19260 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:52.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.075+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c031c3ab60 con 0x55c031ebaef0 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c031dcd380 con 0x55c031ebaef0 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3724003cc0 con 0x55c031ebaef0 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 10 client.19260 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 10 client.19260 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 20 client.19260 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031ed5a40 0x55c031ed7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 4 client.19260 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:52.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c031e05700 con 0x55c031ed5a40 2024-03-30T04:15:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.076+0000 7f3739e00700 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031ed5a40 0x55c031ed7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.077+0000 7f3717400700 10 client.19260 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.077+0000 7f3739e00700 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031ed5a40 0x55c031ed7e60 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73b08ee2ca293588 server_cookie=2e985ea7f07c07a8 in_seq=0 out_seq=0 2024-03-30T04:15:52.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.078+0000 7f3717400700 1 -- 172.21.15.23:0/136004871 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c031e05700 con 0x55c031ed5a40 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.078+0000 7f3717400700 10 client.19260 handle_command_reply: tid=0 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.078+0000 7f3750236b80 1 client.19260 shutdown 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 msgr2=0x55c031ec2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f37240138b0 tx=0x7f3724013c70 comp rx=0 tx=0).stop 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 shutdown_connections 2024-03-30T04:15:52.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031ed5a40 0x55c031ed7e60 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031ec3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 --2- 172.21.15.23:0/136004871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ec2ac0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 >> 172.21.15.23:0/136004871 conn(0x55c031e1a6a0 msgr2=0x55c031ebc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3717400700 0 client.19260 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 shutdown_connections 2024-03-30T04:15:52.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.079+0000 7f3750236b80 1 -- 172.21.15.23:0/136004871 wait complete. 2024-03-30T04:15:52.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.080+0000 7f3750236b80 20 client.19260 trim_cache size 0 max 16384 2024-03-30T04:15:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 Processor -- start 2024-03-30T04:15:52.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 -- start start 2024-03-30T04:15:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ed7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031ebd570 con 0x55c031ebaef0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031ec3800 con 0x55c031eba5d0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3750236b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031eb2480 con 0x55c031ec5df0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f373a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f373a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57682/0 (socket says 172.21.15.23:57682) 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f373a800700 1 -- 172.21.15.23:0/3326838174 learned_addr learned my addr 172.21.15.23:0/3326838174 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.101+0000 7f3739400700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ed7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f3739e00700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 -- 172.21.15.23:0/3326838174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 msgr2=0x55c031ec3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 -- 172.21.15.23:0/3326838174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 msgr2=0x55c031ed7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ed7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 -- 172.21.15.23:0/3326838174 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c031dd2ea0 con 0x55c031ec5df0 2024-03-30T04:15:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f3739400700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ed7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:52.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f3739e00700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.102+0000 7f373a800700 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3730007770 tx=0x7f373001d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aee96e0402c899d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.103+0000 7f3723a00700 1 -- 172.21.15.23:0/3326838174 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730022040 con 0x55c031ec5df0 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.103+0000 7f3723a00700 1 -- 172.21.15.23:0/3326838174 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3730008ed0 con 0x55c031ec5df0 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.103+0000 7f3723a00700 1 -- 172.21.15.23:0/3326838174 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373000e040 con 0x55c031ec5df0 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.104+0000 7f3750236b80 1 -- 172.21.15.23:0/3326838174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 msgr2=0x55c031eb5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.104+0000 7f3750236b80 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3730007770 tx=0x7f373001d000 comp rx=0 tx=0).stop 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 -- 172.21.15.23:0/3326838174 shutdown_connections 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ec5df0 0x55c031eb5f10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ec3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 --2- 172.21.15.23:0/3326838174 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ebaef0 0x55c031ed7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 -- 172.21.15.23:0/3326838174 >> 172.21.15.23:0/3326838174 conn(0x55c031e1a6a0 msgr2=0x55c031eaf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 -- 172.21.15.23:0/3326838174 shutdown_connections 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 -- 172.21.15.23:0/3326838174 wait complete. 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.105+0000 7f3750236b80 1 -- start start 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.106+0000 7f3750236b80 1 Processor -- start 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ecac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ebaef0 0x55c031ecb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 0x55c031f228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031f22df0 con 0x55c031ec5df0 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031f22f60 con 0x55c031eba5d0 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c031f23120 con 0x55c031ebaef0 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3750236b80 10 client.0 _ll_register_callbacks cb 0x55c031db2470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:52.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3739e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ecac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3739e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ecac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43560/0 (socket says 172.21.15.23:43560) 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.107+0000 7f3739e00700 1 -- 172.21.15.23:0/95235474 learned_addr learned my addr 172.21.15.23:0/95235474 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 0x55c031f228b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f3739400700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ebaef0 0x55c031ecb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 -- 172.21.15.23:0/95235474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ebaef0 msgr2=0x55c031ecb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ebaef0 0x55c031ecb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 -- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 msgr2=0x55c031ecac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ecac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 -- 172.21.15.23:0/95235474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c031c3ab60 con 0x55c031ec5df0 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f373a800700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 0x55c031f228b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3730003ce0 tx=0x7f37300094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9088d33ecba8ad03 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.108+0000 7f3717400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373000e040 con 0x55c031ec5df0 2024-03-30T04:15:52.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f373000b990 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f372809e160 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f373000bb70 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3750236b80 10 client.19266 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.109+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f373410e460 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f372809e160 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3730048070 con 0x55c031ec5df0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 10 client.19266 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 10 client.19266 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 20 client.19266 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031f2a550 0x55c031f2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 4 client.19266 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c031c5bcf0 con 0x55c031f2a550 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.110+0000 7f3739e00700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031f2a550 0x55c031f2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.112+0000 7f3739e00700 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031f2a550 0x55c031f2c940 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e24405a8369acb3 server_cookie=8cc5ecfe797f1902 in_seq=0 out_seq=0 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.112+0000 7f3717400700 10 client.19266 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.112+0000 7f3717400700 1 -- 172.21.15.23:0/95235474 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c031c5bcf0 con 0x55c031f2a550 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.112+0000 7f3717400700 10 client.19266 handle_command_reply: tid=0 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.112+0000 7f3750236b80 1 client.19266 shutdown 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 msgr2=0x55c031f228b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 0x55c031f228b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f3730003ce0 tx=0x7f37300094e0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 shutdown_connections 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c031ec5df0 0x55c031f228b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c031eba5d0 0x55c031ecac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x55c031f2a550 0x55c031f2c940 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 --2- 172.21.15.23:0/95235474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c031ebaef0 0x55c031ecb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3717400700 0 client.19266 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 >> 172.21.15.23:0/95235474 conn(0x55c031e1a6a0 msgr2=0x55c031ebddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 shutdown_connections 2024-03-30T04:15:52.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.113+0000 7f3750236b80 1 -- 172.21.15.23:0/95235474 wait complete. 2024-03-30T04:15:52.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f3750236b80 20 client.19266 trim_cache size 0 max 16384 2024-03-30T04:15:52.114 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 msgr2=0x7f371c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 0x7f371c0a3750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3730009790 tx=0x7f3730009700 comp rx=0 tx=0).stop 2024-03-30T04:15:52.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 msgr2=0x7f3734199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f372800eb30 tx=0x7f372800ee40 comp rx=0 tx=0).stop 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 shutdown_connections 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f373419ddf0 0x7f373419e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f371c0a1290 0x7f371c0a3750 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3734100f50 0x7f3734199510 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 --2- 172.21.15.23:0/2923972645 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3734106070 0x7f3734199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 >> 172.21.15.23:0/2923972645 conn(0x7f37340fc7f0 msgr2=0x7f373410d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.114+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 shutdown_connections 2024-03-30T04:15:52.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.115+0000 7f373c800700 1 -- 172.21.15.23:0/2923972645 wait complete. 2024-03-30T04:15:52.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 -- 172.21.15.23:0/1640930369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898067e30 msgr2=0x7f18980682b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 --2- 172.21.15.23:0/1640930369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898067e30 0x7f18980682b0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f18840099c0 tx=0x7f1884009cd0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 -- 172.21.15.23:0/1640930369 shutdown_connections 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 --2- 172.21.15.23:0/1640930369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f189810db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 --2- 172.21.15.23:0/1640930369 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898105ce0 0x7f18981060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 --2- 172.21.15.23:0/1640930369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898067e30 0x7f18980682b0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 -- 172.21.15.23:0/1640930369 >> 172.21.15.23:0/1640930369 conn(0x7f1898076d40 msgr2=0x7f1898077150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 -- 172.21.15.23:0/1640930369 shutdown_connections 2024-03-30T04:15:52.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.379+0000 7f189e200700 1 -- 172.21.15.23:0/1640930369 wait complete. 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.380+0000 7f189e200700 1 Processor -- start 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 -- start start 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 0x7f1898197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898105ce0 0x7f189819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189819c1a0 con 0x7f18980687f0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189819c310 con 0x7f1898105ce0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f189e200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189819c4d0 con 0x7f1898067e30 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f1896a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f1896a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35556/0 (socket says 172.21.15.23:35556) 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f1897e00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898105ce0 0x7f189819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.381+0000 7f1896a00700 1 -- 172.21.15.23:0/752072124 learned_addr learned my addr 172.21.15.23:0/752072124 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1897400700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 0x7f1898197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 -- 172.21.15.23:0/752072124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 msgr2=0x7f1898197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 0x7f1898197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 -- 172.21.15.23:0/752072124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898105ce0 msgr2=0x7f189819bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898105ce0 0x7f189819bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 -- 172.21.15.23:0/752072124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18840095f0 con 0x7f18980687f0 2024-03-30T04:15:52.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1897400700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 0x7f1898197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f1896a00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f188400b580 tx=0x7f188400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.382+0000 7f188fe00700 1 -- 172.21.15.23:0/752072124 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188401c070 con 0x7f18980687f0 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.383+0000 7f188fe00700 1 -- 172.21.15.23:0/752072124 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188400fd80 con 0x7f18980687f0 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.383+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f189819c750 con 0x7f18980687f0 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.383+0000 7f188fe00700 1 -- 172.21.15.23:0/752072124 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1884022c40 con 0x7f18980687f0 2024-03-30T04:15:52.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.383+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18981a3ff0 con 0x7f18980687f0 2024-03-30T04:15:52.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.386+0000 7f188fe00700 1 -- 172.21.15.23:0/752072124 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f1884020070 con 0x7f18980687f0 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.387+0000 7f18b1d9fb80 1 Processor -- start 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.387+0000 7f188fe00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 0x7f187c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 -- start start 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f188fe00700 1 -- 172.21.15.23:0/752072124 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f18840cde60 con 0x7f18980687f0 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f1897e00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 0x7f187c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972bea81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bdf3990 con 0x55972be9a5d0 2024-03-30T04:15:52.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bdf1350 con 0x55972be9aef0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bdecc40 con 0x55972bea5df0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f1897400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f1897400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35562/0 (socket says 172.21.15.23:35562) 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.388+0000 7f1897400700 1 -- 172.21.15.23:0/462505021 learned_addr learned my addr 172.21.15.23:0/462505021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1896a00700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897e00700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972bea81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 -- 172.21.15.23:0/462505021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 msgr2=0x55972bea81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972bea81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 -- 172.21.15.23:0/462505021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 msgr2=0x55972bea58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 -- 172.21.15.23:0/462505021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55972bc6e3b0 con 0x55972be9a5d0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1896a00700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897e00700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 0x7f187c0a3750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1890009550 tx=0x7f1890009bf0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.389+0000 7f1897400700 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f18880115f0 tx=0x7f1888011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0fa779bf3c534d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.390+0000 7f1883e00700 1 -- 172.21.15.23:0/462505021 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188800f680 con 0x55972be9a5d0 2024-03-30T04:15:52.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.390+0000 7f1883e00700 1 -- 172.21.15.23:0/462505021 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188800fca0 con 0x55972be9a5d0 2024-03-30T04:15:52.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.390+0000 7f1883e00700 1 -- 172.21.15.23:0/462505021 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1888019d10 con 0x55972be9a5d0 2024-03-30T04:15:52.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.390+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/462505021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 msgr2=0x55972be9a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.390+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f18880115f0 tx=0x7f1888011900 comp rx=0 tx=0).stop 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/462505021 shutdown_connections 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972bea81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972be9a9b0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/462505021 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/462505021 >> 172.21.15.23:0/462505021 conn(0x55972bdfa6a0 msgr2=0x55972be96a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/462505021 shutdown_connections 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/462505021 wait complete. 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 -- start start 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.391+0000 7f18b1d9fb80 1 Processor -- start 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9a5d0 0x55972bea57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972bea3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bd88d80 con 0x55972bea5df0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bd88a20 con 0x55972be9aef0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bc706a0 con 0x55972be9a5d0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f18b1d9fb80 10 client.0 _ll_register_callbacks cb 0x55972bd92470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f1896a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f1896a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43592/0 (socket says 172.21.15.23:43592) 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f1896a00700 1 -- 172.21.15.23:0/1307984422 learned_addr learned my addr 172.21.15.23:0/1307984422 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.392+0000 7f1897e00700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972bea3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1897400700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9a5d0 0x55972bea57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 -- 172.21.15.23:0/1307984422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9a5d0 msgr2=0x55972bea57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9a5d0 0x55972bea57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 -- 172.21.15.23:0/1307984422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 msgr2=0x55972bea3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972bea3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 -- 172.21.15.23:0/1307984422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1888009dd0 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1897e00700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972bea3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1896a00700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f188409ab50 tx=0x7f188409a5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a57d0ad652b35acc server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1880c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.393+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188409a2b0 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f188409e9f0 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55972bdad380 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18840a2fb0 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 10 client.29246 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55972bc1ab60 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1884064020 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f18840a2c20 con 0x55972be9aef0 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 10 client.29246 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:52.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 10 client.29246 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 20 client.29246 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972beb99f0 0x55972bebbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 4 client.29246 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.394+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55972bde5700 con 0x55972beb99f0 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.395+0000 7f1897400700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972beb99f0 0x55972bebbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.395+0000 7f1897400700 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972beb99f0 0x55972bebbe10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=915e69dcd579118a server_cookie=6b9068fbfd79036c in_seq=0 out_seq=0 2024-03-30T04:15:52.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.395+0000 7f1880c00700 10 client.29246 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f1880c00700 1 -- 172.21.15.23:0/1307984422 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55972bde5700 con 0x55972beb99f0 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f1880c00700 10 client.29246 handle_command_reply: tid=0 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 client.29246 shutdown 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 msgr2=0x55972bea2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f188409ab50 tx=0x7f188409a5a0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.398 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 shutdown_connections 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972bea3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972beb99f0 0x55972bebbe10 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9a5d0 0x55972bea57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/1307984422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972bea2b30 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 >> 172.21.15.23:0/1307984422 conn(0x55972bdfa6a0 msgr2=0x55972be9c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f1880c00700 0 client.29246 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 shutdown_connections 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.396+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/1307984422 wait complete. 2024-03-30T04:15:52.399 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.397+0000 7f18b1d9fb80 20 client.29246 trim_cache size 0 max 16384 2024-03-30T04:15:52.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.411+0000 7f18b1d9fb80 1 Processor -- start 2024-03-30T04:15:52.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.411+0000 7f18b1d9fb80 1 -- start start 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 0x55972be95f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 0x55972bea23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972beb0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972be97660 con 0x55972bea5df0 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972bea2900 con 0x55972be9a5d0 2024-03-30T04:15:52.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972be9d970 con 0x55972be9aef0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1897400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 0x55972be95f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1897400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 0x55972be95f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43594/0 (socket says 172.21.15.23:43594) 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1897400700 1 -- 172.21.15.23:0/215142787 learned_addr learned my addr 172.21.15.23:0/215142787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1896a00700 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 0x55972bea23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1897e00700 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972beb0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1896a00700 1 -- 172.21.15.23:0/215142787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 msgr2=0x55972be95f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1896a00700 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 0x55972be95f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1896a00700 1 -- 172.21.15.23:0/215142787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 msgr2=0x55972beb0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.412+0000 7f1896a00700 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972beb0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.413+0000 7f1896a00700 1 -- 172.21.15.23:0/215142787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55972bdb2ea0 con 0x55972be9aef0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.413+0000 7f1896a00700 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 0x55972bea23c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1884009990 tx=0x7f188409b400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2465567a608ea67 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.413+0000 7f188d200700 1 -- 172.21.15.23:0/215142787 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18840a2d20 con 0x55972be9aef0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.413+0000 7f188d200700 1 -- 172.21.15.23:0/215142787 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1884064070 con 0x55972be9aef0 2024-03-30T04:15:52.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.413+0000 7f188d200700 1 -- 172.21.15.23:0/215142787 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188409be30 con 0x55972be9aef0 2024-03-30T04:15:52.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/215142787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 msgr2=0x55972bea23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 0x55972bea23c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f1884009990 tx=0x7f188409b400 comp rx=0 tx=0).stop 2024-03-30T04:15:52.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/215142787 shutdown_connections 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972bea5df0 0x55972beb0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9a5d0 0x55972be95f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/215142787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972be9aef0 0x55972bea23c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/215142787 >> 172.21.15.23:0/215142787 conn(0x55972bdfa6a0 msgr2=0x55972bec0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/215142787 shutdown_connections 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/215142787 wait complete. 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 -- start start 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.414+0000 7f18b1d9fb80 1 Processor -- start 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972beb0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972beaee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972beaf540 con 0x55972be9a5d0 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972beaf6b0 con 0x55972be9aef0 2024-03-30T04:15:52.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55972beaf870 con 0x55972bea5df0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f18b1d9fb80 10 client.0 _ll_register_callbacks cb 0x55972bd92470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1897400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972beb0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1897e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972beaee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43598/0 (socket says 172.21.15.23:43598) 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 -- 172.21.15.23:0/2043271383 learned_addr learned my addr 172.21.15.23:0/2043271383 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 -- 172.21.15.23:0/2043271383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 msgr2=0x55972beaee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972beaee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 -- 172.21.15.23:0/2043271383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 msgr2=0x55972beb0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972beb0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.415+0000 7f1896a00700 1 -- 172.21.15.23:0/2043271383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55972bc1ab60 con 0x55972be9aef0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1896a00700 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1884009fd0 tx=0x7f188409be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=125cb893414129a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1880c00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:52.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18840b32f0 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f1884065080 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f188800fca0 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18840b34e0 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f18b1d9fb80 10 client.29252 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55972bc72cd0 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.416+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f188800fca0 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f188400f840 con 0x55972be9aef0 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 10 client.29252 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 10 client.29252 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 20 client.29252 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972bf0e510 0x55972bf10900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 4 client.29252 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:15:52.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55972bc3bcf0 con 0x55972bf0e510 2024-03-30T04:15:52.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.417+0000 7f1897400700 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972bf0e510 0x55972bf10900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f1897400700 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972bf0e510 0x55972bf10900 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eff0b4b9882a06c server_cookie=6683e0231dd3acd in_seq=0 out_seq=0 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f1880c00700 10 client.29252 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f1880c00700 1 -- 172.21.15.23:0/2043271383 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55972bc3bcf0 con 0x55972bf0e510 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f1880c00700 10 client.29252 handle_command_reply: tid=0 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 client.29252 shutdown 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 msgr2=0x55972beb0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f1884009fd0 tx=0x7f188409be00 comp rx=0 tx=0).stop 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 shutdown_connections 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55972bf0e510 0x55972bf10900 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55972bea5df0 0x55972beaee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55972be9a5d0 0x55972beb0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f1880c00700 0 client.29252 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 --2- 172.21.15.23:0/2043271383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55972be9aef0 0x55972beb0e60 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 >> 172.21.15.23:0/2043271383 conn(0x55972bdfa6a0 msgr2=0x55972be8f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.418+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 shutdown_connections 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f18b1d9fb80 1 -- 172.21.15.23:0/2043271383 wait complete. 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f18b1d9fb80 20 client.29252 trim_cache size 0 max 16384 2024-03-30T04:15:52.421 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 msgr2=0x7f187c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 0x7f187c0a3750 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1890009550 tx=0x7f1890009bf0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 msgr2=0x7f18981977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f188400b580 tx=0x7f188400f690 comp rx=0 tx=0).stop 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.419+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 shutdown_connections 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f187c0a1290 0x7f187c0a3750 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1898105ce0 0x7f189819bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1898067e30 0x7f1898197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 --2- 172.21.15.23:0/752072124 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f18980687f0 0x7f18981977c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:52.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 >> 172.21.15.23:0/752072124 conn(0x7f1898076d40 msgr2=0x7f18980feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:52.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 shutdown_connections 2024-03-30T04:15:52.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:52.420+0000 7f189e200700 1 -- 172.21.15.23:0/752072124 wait complete. 2024-03-30T04:15:52.435 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:52 smithi023 ceph-mon[38254]: pgmap v436: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 260 op/s 2024-03-30T04:15:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:52 smithi193 ceph-mon[79581]: pgmap v436: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 260 op/s 2024-03-30T04:15:52.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 13567 31.37 MB/sec warmup 29 sec latency 195.332 ms 2024-03-30T04:15:52.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:52 smithi154 ceph-mon[79161]: pgmap v436: 129 pgs: 129 active+clean; 776 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 260 op/s 2024-03-30T04:15:53.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-30T04:15:53.439 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-30T04:15:53.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:53 smithi193 ceph-mon[79581]: pgmap v437: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.9 MiB/s wr, 460 op/s 2024-03-30T04:15:53.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:53 smithi023 ceph-mon[38254]: pgmap v437: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.9 MiB/s wr, 460 op/s 2024-03-30T04:15:53.610 INFO:tasks.workunit.client.0.smithi023.stdout: 10 13601 30.37 MB/sec warmup 30 sec latency 950.256 ms 2024-03-30T04:15:53.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:53 smithi154 ceph-mon[79161]: pgmap v437: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 7.9 MiB/s wr, 460 op/s 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 -- 172.21.15.23:0/532620871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 msgr2=0x7f7374100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 --2- 172.21.15.23:0/532620871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f7374100fd0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f73600099c0 tx=0x7f7360009cd0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 -- 172.21.15.23:0/532620871 shutdown_connections 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 --2- 172.21.15.23:0/532620871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374101510 0x7f737410d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 --2- 172.21.15.23:0/532620871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374106c10 0x7f7374106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 --2- 172.21.15.23:0/532620871 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f7374100fd0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.753+0000 7f737a800700 1 -- 172.21.15.23:0/532620871 >> 172.21.15.23:0/532620871 conn(0x7f73740fc620 msgr2=0x7f73740fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.754+0000 7f737a800700 1 -- 172.21.15.23:0/532620871 shutdown_connections 2024-03-30T04:15:53.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.754+0000 7f737a800700 1 -- 172.21.15.23:0/532620871 wait complete. 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.754+0000 7f737a800700 1 Processor -- start 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 -- start start 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374101510 0x7f73740735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 0x7f7374073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73741a5330 con 0x7f7374100b50 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73741a54a0 con 0x7f7374106c10 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f737a800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73741a5660 con 0x7f7374101510 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f7379400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f7379400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35592/0 (socket says 172.21.15.23:35592) 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.755+0000 7f7379400700 1 -- 172.21.15.23:0/2328192056 learned_addr learned my addr 172.21.15.23:0/2328192056 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379e00700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 0x7f7374073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7378a00700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374101510 0x7f73740735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 -- 172.21.15.23:0/2328192056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374101510 msgr2=0x7f73740735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374101510 0x7f73740735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 -- 172.21.15.23:0/2328192056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 msgr2=0x7f7374073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 0x7f7374073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 -- 172.21.15.23:0/2328192056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73600095f0 con 0x7f7374100b50 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379e00700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 0x7f7374073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7379400700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f736800e9a0 tx=0x7f736800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7372000700 1 -- 172.21.15.23:0/2328192056 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368009790 con 0x7f7374100b50 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73741a5940 con 0x7f7374100b50 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7372000700 1 -- 172.21.15.23:0/2328192056 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736800cbd0 con 0x7f7374100b50 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f7372000700 1 -- 172.21.15.23:0/2328192056 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7368018680 con 0x7f7374100b50 2024-03-30T04:15:53.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.756+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73741a5e20 con 0x7f7374100b50 2024-03-30T04:15:53.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.759+0000 7f7372000700 1 -- 172.21.15.23:0/2328192056 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f736800cd40 con 0x7f7374100b50 2024-03-30T04:15:53.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 Processor -- start 2024-03-30T04:15:53.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f7372000700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 0x7f735c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f7372000700 1 -- 172.21.15.23:0/2328192056 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f7368014070 con 0x7f7374100b50 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 -- start start 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f7378a00700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 0x7f735c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e83c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e91430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642ddca10 con 0x562642e84140 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642dda3d0 con 0x562642e83820 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.761+0000 7f738f981b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642dd5cc0 con 0x562642e8f040 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35600/0 (socket says 172.21.15.23:35600) 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 -- 172.21.15.23:0/2274340072 learned_addr learned my addr 172.21.15.23:0/2274340072 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7379400700 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e83c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7379e00700 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e91430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 0x7f735c0a3750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7360009990 tx=0x7f7360005220 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 -- 172.21.15.23:0/2274340072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 msgr2=0x562642e91430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e91430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 -- 172.21.15.23:0/2274340072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 msgr2=0x562642e83c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e83c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 -- 172.21.15.23:0/2274340072 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562642c57430 con 0x562642e84140 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.762+0000 7f7378a00700 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f73600099c0 tx=0x7f7360027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44845cb989e1570 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.763+0000 7f7366600700 1 -- 172.21.15.23:0/2274340072 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736000bc20 con 0x562642e84140 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.763+0000 7f7366600700 1 -- 172.21.15.23:0/2274340072 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736001e030 con 0x562642e84140 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.763+0000 7f7366600700 1 -- 172.21.15.23:0/2274340072 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360027700 con 0x562642e84140 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.763+0000 7f738f981b80 1 -- 172.21.15.23:0/2274340072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 msgr2=0x562642e8eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.763+0000 7f738f981b80 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f73600099c0 tx=0x7f7360027000 comp rx=0 tx=0).stop 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 -- 172.21.15.23:0/2274340072 shutdown_connections 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e91430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e83c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 --2- 172.21.15.23:0/2274340072 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642e8eb00 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 -- 172.21.15.23:0/2274340072 >> 172.21.15.23:0/2274340072 conn(0x562642de3720 msgr2=0x562642e7fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 -- 172.21.15.23:0/2274340072 shutdown_connections 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 -- 172.21.15.23:0/2274340072 wait complete. 2024-03-30T04:15:53.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 -- start start 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.764+0000 7f738f981b80 1 Processor -- start 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e8c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642d71e00 con 0x562642e8f040 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642d71aa0 con 0x562642e83820 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642c59720 con 0x562642e84140 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f738f981b80 10 client.0 _ll_register_callbacks cb 0x562642d7b4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7379400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7378a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7379e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e8c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7379400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43630/0 (socket says 172.21.15.23:43630) 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7379400700 1 -- 172.21.15.23:0/1278607068 learned_addr learned my addr 172.21.15.23:0/1278607068 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:53.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.765+0000 7f7378a00700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57808/0 (socket says 172.21.15.23:57808) 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 -- 172.21.15.23:0/1278607068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 msgr2=0x562642e8bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 -- 172.21.15.23:0/1278607068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 msgr2=0x562642e8c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e8c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 -- 172.21.15.23:0/1278607068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7360020040 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7378a00700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f7379400700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7368096620 tx=0x7f736809a120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafd63df575d9dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f735aa00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736809c070 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f73680a25a0 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562642d96400 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7368099610 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f738f981b80 10 client.29264 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.766+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562642c03be0 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562642d96400 con 0x562642e83820 2024-03-30T04:15:53.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f73680a2110 con 0x562642e83820 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 10 client.29264 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 10 client.29264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 20 client.29264 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642e9eba0 0x562642ea0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 4 client.29264 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562642dce780 con 0x562642e9eba0 2024-03-30T04:15:53.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.767+0000 7f7378a00700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642e9eba0 0x562642ea0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.768+0000 7f7378a00700 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642e9eba0 0x562642ea0f90 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=603fb3e6eb7a9f19 server_cookie=80a71e4bb4d1b3d5 in_seq=0 out_seq=0 2024-03-30T04:15:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.768+0000 7f735aa00700 10 client.29264 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.768+0000 7f735aa00700 1 -- 172.21.15.23:0/1278607068 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x562642dce780 con 0x562642e9eba0 2024-03-30T04:15:53.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.768+0000 7f735aa00700 10 client.29264 handle_command_reply: tid=0 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 client.29264 shutdown 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 msgr2=0x562642e8e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f7368096620 tx=0x7f736809a120 comp rx=0 tx=0).stop 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 shutdown_connections 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e8c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e8e990 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642e9eba0 0x562642ea0f90 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 --2- 172.21.15.23:0/1278607068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e8bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f735aa00700 0 client.29264 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 >> 172.21.15.23:0/1278607068 conn(0x562642de3720 msgr2=0x562642e85910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 shutdown_connections 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 1 -- 172.21.15.23:0/1278607068 wait complete. 2024-03-30T04:15:53.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.769+0000 7f738f981b80 20 client.29264 trim_cache size 0 max 16384 2024-03-30T04:15:53.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.803+0000 7f738f981b80 1 Processor -- start 2024-03-30T04:15:53.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 -- start start 2024-03-30T04:15:53.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.803 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e7f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e7fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e7f1e0 con 0x562642e8f040 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e852c0 con 0x562642e83820 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.804+0000 7f738f981b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e847c0 con 0x562642e84140 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.805+0000 7f7379400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.805+0000 7f7379400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:43632/0 (socket says 172.21.15.23:43632) 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.805+0000 7f7379400700 1 -- 172.21.15.23:0/141253144 learned_addr learned my addr 172.21.15.23:0/141253144 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:53.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.805+0000 7f7379e00700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e7fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.805+0000 7f7378a00700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e7f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 -- 172.21.15.23:0/141253144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 msgr2=0x562642e7f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e7f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 -- 172.21.15.23:0/141253144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 msgr2=0x562642e7fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e7fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 -- 172.21.15.23:0/141253144 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562642d9bf20 con 0x562642e83820 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7378a00700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e7f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7379400700 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f736800c5b0 tx=0x7f7368010bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8201d7b286288da server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7367000700 1 -- 172.21.15.23:0/141253144 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736809c020 con 0x562642e83820 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7367000700 1 -- 172.21.15.23:0/141253144 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73680b2030 con 0x562642e83820 2024-03-30T04:15:53.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.806+0000 7f7367000700 1 -- 172.21.15.23:0/141253144 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73680b8500 con 0x562642e83820 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.807+0000 7f738f981b80 1 -- 172.21.15.23:0/141253144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 msgr2=0x562642e95c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.807+0000 7f738f981b80 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f736800c5b0 tx=0x7f7368010bb0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 -- 172.21.15.23:0/141253144 shutdown_connections 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e8f040 0x562642e7fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642e95c90 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 --2- 172.21.15.23:0/141253144 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e84140 0x562642e7f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 -- 172.21.15.23:0/141253144 >> 172.21.15.23:0/141253144 conn(0x562642de3720 msgr2=0x562642e8a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 -- 172.21.15.23:0/141253144 shutdown_connections 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 -- 172.21.15.23:0/141253144 wait complete. 2024-03-30T04:15:53.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.808+0000 7f738f981b80 1 -- start start 2024-03-30T04:15:53.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.809+0000 7f738f981b80 1 Processor -- start 2024-03-30T04:15:53.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.809+0000 7f738f981b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642eebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e94be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e952a0 con 0x562642e84140 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e95410 con 0x562642e83820 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562642e955d0 con 0x562642e8f040 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f738f981b80 10 client.0 _ll_register_callbacks cb 0x562642d7b4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f7378a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f7378a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35626/0 (socket says 172.21.15.23:35626) 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.810+0000 7f7378a00700 1 -- 172.21.15.23:0/3875578593 learned_addr learned my addr 172.21.15.23:0/3875578593 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7379400700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642eebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 -- 172.21.15.23:0/3875578593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 msgr2=0x562642e94be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e94be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 -- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 msgr2=0x562642eebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:53.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642eebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 -- 172.21.15.23:0/3875578593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562642c03be0 con 0x562642e84140 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7379400700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642eebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f7378a00700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73600058e0 tx=0x7f7360003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9820edaad362e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.811+0000 7f735aa00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.812+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736000bea0 con 0x562642e84140 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.812+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f736001e030 con 0x562642e84140 2024-03-30T04:15:53.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.812+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f736003eaa0 con 0x562642e84140 2024-03-30T04:15:53.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.814+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f736003ec60 con 0x562642e84140 2024-03-30T04:15:53.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.814+0000 7f738f981b80 10 client.19302 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:53.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.814+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562642c5bd50 con 0x562642e84140 2024-03-30T04:15:53.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.815+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f736001e030 con 0x562642e84140 2024-03-30T04:15:53.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.815+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f7360048070 con 0x562642e84140 2024-03-30T04:15:53.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 10 client.19302 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 10 client.19302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 20 client.19302 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642ef4800 0x562642ef6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 4 client.19302 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.816+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562642c24d70 con 0x562642ef4800 2024-03-30T04:15:53.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.817+0000 7f7379400700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642ef4800 0x562642ef6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:53.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.817+0000 7f735aa00700 10 client.19302 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:53.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.817+0000 7f7379400700 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642ef4800 0x562642ef6c20 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdd4352e76920171 server_cookie=41b4b858f893dd13 in_seq=0 out_seq=0 2024-03-30T04:15:53.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f735aa00700 1 -- 172.21.15.23:0/3875578593 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562642c24d70 con 0x562642ef4800 2024-03-30T04:15:53.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f735aa00700 10 client.19302 handle_command_reply: tid=0 2024-03-30T04:15:53.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 client.19302 shutdown 2024-03-30T04:15:53.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 msgr2=0x562642eec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f73600058e0 tx=0x7f7360003e90 comp rx=0 tx=0).stop 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 shutdown_connections 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x562642ef4800 0x562642ef6c20 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562642e8f040 0x562642e94be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x562642e83820 0x562642eebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f735aa00700 0 client.19302 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 --2- 172.21.15.23:0/3875578593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x562642e84140 0x562642eec250 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.819+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 >> 172.21.15.23:0/3875578593 conn(0x562642de3720 msgr2=0x562642e7f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.820+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 shutdown_connections 2024-03-30T04:15:53.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.820+0000 7f738f981b80 1 -- 172.21.15.23:0/3875578593 wait complete. 2024-03-30T04:15:53.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.822+0000 7f738f981b80 20 client.19302 trim_cache size 0 max 16384 2024-03-30T04:15:53.821 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stdout: "return_code": 0, 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_tag": "22f31e55-e366-463d-8285-656358f15f47", 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": "asynchronous" 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 msgr2=0x7f735c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 0x7f735c0a3750 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7360009990 tx=0x7f7360005220 comp rx=0 tx=0).stop 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 msgr2=0x7f73740730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:53.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f736800e9a0 tx=0x7f736800ed60 comp rx=0 tx=0).stop 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 shutdown_connections 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f735c0a1290 0x7f735c0a3750 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7374106c10 0x7f7374073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7374100b50 0x7f73740730b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 --2- 172.21.15.23:0/2328192056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7374101510 0x7f73740735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:53.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.824+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 >> 172.21.15.23:0/2328192056 conn(0x7f73740fc620 msgr2=0x7f73740fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:53.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.826+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 shutdown_connections 2024-03-30T04:15:53.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:53.826+0000 7f737a800700 1 -- 172.21.15.23:0/2328192056 wait complete. 2024-03-30T04:15:53.858 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:15:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 -- 172.21.15.23:0/531326583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80073180 msgr2=0x7fae80073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 --2- 172.21.15.23:0/531326583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80073180 0x7fae80073560 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fae70009990 tx=0x7fae70009ca0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 -- 172.21.15.23:0/531326583 shutdown_connections 2024-03-30T04:15:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 --2- 172.21.15.23:0/531326583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae8010d3e0 0x7fae8010f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 --2- 172.21.15.23:0/531326583 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae80073180 0x7fae80073560 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 --2- 172.21.15.23:0/531326583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073aa0 0x7fae8010cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 -- 172.21.15.23:0/531326583 >> 172.21.15.23:0/531326583 conn(0x7fae800fc670 msgr2=0x7fae800fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 -- 172.21.15.23:0/531326583 shutdown_connections 2024-03-30T04:15:54.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.155+0000 7fae84e00700 1 -- 172.21.15.23:0/531326583 wait complete. 2024-03-30T04:15:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 Processor -- start 2024-03-30T04:15:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 -- start start 2024-03-30T04:15:54.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae80073aa0 0x7fae801998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae8010d3e0 0x7fae8019dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019e240 con 0x7fae8010d3e0 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.157+0000 7fae84e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019e3b0 con 0x7fae80073aa0 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae84e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8019e570 con 0x7fae80073180 2024-03-30T04:15:54.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:57848/0 (socket says 172.21.15.23:57848) 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 -- 172.21.15.23:0/2920634787 learned_addr learned my addr 172.21.15.23:0/2920634787 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7fe00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae8010d3e0 0x7fae8019dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 -- 172.21.15.23:0/2920634787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae80073aa0 msgr2=0x7fae801998b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae80073aa0 0x7fae801998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 -- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae8010d3e0 msgr2=0x7fae8019dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae8010d3e0 0x7fae8019dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.158+0000 7fae7f400700 1 -- 172.21.15.23:0/2920634787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae700095f0 con 0x7fae80073180 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae7f400700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fae70005770 tx=0x7fae700057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae77e00700 1 -- 172.21.15.23:0/2920634787 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7001d070 con 0x7fae80073180 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8019e7f0 con 0x7fae80073180 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae77e00700 1 -- 172.21.15.23:0/2920634787 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae70021410 con 0x7fae80073180 2024-03-30T04:15:54.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae77e00700 1 -- 172.21.15.23:0/2920634787 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70017650 con 0x7fae80073180 2024-03-30T04:15:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.159+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae801a6090 con 0x7fae80073180 2024-03-30T04:15:54.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.162+0000 7fae77e00700 1 -- 172.21.15.23:0/2920634787 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fae7000fa00 con 0x7fae80073180 2024-03-30T04:15:54.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.164+0000 7fae99fc2b80 1 Processor -- start 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.164+0000 7fae99fc2b80 1 -- start start 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.164+0000 7fae77e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 0x7fae640a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.164+0000 7fae77e00700 1 -- 172.21.15.23:0/2920634787 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fae700ce100 con 0x7fae80073180 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7fe00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 0x7fae640a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a210c0 0x564d28a2ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28979990 con 0x564d28a207a0 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28977350 con 0x564d28a210c0 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28972c40 con 0x564d28a2bfc0 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35658/0 (socket says 172.21.15.23:35658) 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 -- 172.21.15.23:0/3912701746 learned_addr learned my addr 172.21.15.23:0/3912701746 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:54.166 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7fe00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 0x7fae640a3790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae800678e0 tx=0x7fae7800b040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 -- 172.21.15.23:0/3912701746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 msgr2=0x564d28a2e3b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2e3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 -- 172.21.15.23:0/3912701746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a210c0 msgr2=0x564d28a2ba80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a210c0 0x564d28a2ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.165+0000 7fae7f400700 1 -- 172.21.15.23:0/3912701746 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d287f43b0 con 0x564d28a207a0 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.166+0000 7fae7f400700 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fae700978d0 tx=0x7fae7009a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5f123165cc4c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.166+0000 7fae74c00700 1 -- 172.21.15.23:0/3912701746 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7009b560 con 0x564d28a207a0 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.166+0000 7fae74c00700 1 -- 172.21.15.23:0/3912701746 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae7009ae60 con 0x564d28a207a0 2024-03-30T04:15:54.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.166+0000 7fae74c00700 1 -- 172.21.15.23:0/3912701746 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7009b0c0 con 0x564d28a207a0 2024-03-30T04:15:54.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 -- 172.21.15.23:0/3912701746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 msgr2=0x564d28a20b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fae700978d0 tx=0x7fae7009a9f0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 -- 172.21.15.23:0/3912701746 shutdown_connections 2024-03-30T04:15:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a207a0 0x564d28a20b80 secure :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fae700978d0 tx=0x7fae7009a9f0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.174 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/3912701746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a210c0 0x564d28a2ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 -- 172.21.15.23:0/3912701746 >> 172.21.15.23:0/3912701746 conn(0x564d289806a0 msgr2=0x564d28a1cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 -- 172.21.15.23:0/3912701746 shutdown_connections 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.167+0000 7fae99fc2b80 1 -- 172.21.15.23:0/3912701746 wait complete. 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.168+0000 7fae99fc2b80 1 -- start start 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.168+0000 7fae99fc2b80 1 Processor -- start 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a352e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a40010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d2890ed80 con 0x564d28a210c0 2024-03-30T04:15:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d2890ea20 con 0x564d28a3dc20 2024-03-30T04:15:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d287f66a0 con 0x564d28a2bfc0 2024-03-30T04:15:54.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae99fc2b80 10 client.0 _ll_register_callbacks cb 0x564d28918470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35670/0 (socket says 172.21.15.23:35670) 2024-03-30T04:15:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.169+0000 7fae7f400700 1 -- 172.21.15.23:0/799154250 learned_addr learned my addr 172.21.15.23:0/799154250 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:54.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7fe00700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a40010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 -- 172.21.15.23:0/799154250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 msgr2=0x564d28a352e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-30T04:15:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a352e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 -- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 msgr2=0x564d28a40010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a40010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 -- 172.21.15.23:0/799154250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae801a6090 con 0x564d28a210c0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae7f400700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fae700058b0 tx=0x7fae7009ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab21a347ae6265d server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae69600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7009b0c0 con 0x564d28a210c0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.170+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fae7009b220 con 0x564d28a210c0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d28933380 con 0x564d28a210c0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae700b9140 con 0x564d28a210c0 2024-03-30T04:15:54.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 10 client.19314 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564d287a0b60 con 0x564d28a210c0 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564d28933380 con 0x564d28a210c0 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fae7009fc40 con 0x564d28a210c0 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 10 client.19314 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 10 client.19314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 20 client.19314 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a440f0 0x564d28a464e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 4 client.19314 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:15:54.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.171+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d2896b700 con 0x564d28a440f0 2024-03-30T04:15:54.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.173+0000 7fae7ea00700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a440f0 0x564d28a464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.175+0000 7fae7ea00700 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a440f0 0x564d28a464e0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8435253c696cca7f server_cookie=486466124d42080e in_seq=0 out_seq=0 2024-03-30T04:15:54.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.175+0000 7fae69600700 10 client.19314 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.177+0000 7fae69600700 1 -- 172.21.15.23:0/799154250 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564d2896b700 con 0x564d28a440f0 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.177+0000 7fae69600700 10 client.19314 handle_command_reply: tid=0 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.177+0000 7fae99fc2b80 1 client.19314 shutdown 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.177+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 msgr2=0x564d28a34da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fae700058b0 tx=0x7fae7009ed40 comp rx=0 tx=0).stop 2024-03-30T04:15:54.184 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 shutdown_connections 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a40010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a34da0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a440f0 0x564d28a464e0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/799154250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae69600700 0 client.19314 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 >> 172.21.15.23:0/799154250 conn(0x564d289806a0 msgr2=0x564d28a22890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 shutdown_connections 2024-03-30T04:15:54.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 1 -- 172.21.15.23:0/799154250 wait complete. 2024-03-30T04:15:54.186 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.178+0000 7fae99fc2b80 20 client.19314 trim_cache size 0 max 16384 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.198+0000 7fae99fc2b80 1 Processor -- start 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.198+0000 7fae99fc2b80 1 -- start start 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a34bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a22130 con 0x564d28a210c0 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a21a20 con 0x564d28a3dc20 2024-03-30T04:15:54.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a26fa0 con 0x564d28a2bfc0 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35676/0 (socket says 172.21.15.23:35676) 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae7f400700 1 -- 172.21.15.23:0/45415425 learned_addr learned my addr 172.21.15.23:0/45415425 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae7fe00700 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a34bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.199+0000 7fae7ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 -- 172.21.15.23:0/45415425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 msgr2=0x564d28a2ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 -- 172.21.15.23:0/45415425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 msgr2=0x564d28a34bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a34bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 -- 172.21.15.23:0/45415425 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d28938ea0 con 0x564d28a210c0 2024-03-30T04:15:54.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7fe00700 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a34bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:54.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae7f400700 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae7009ab60 tx=0x7fae70099410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95438621b32a907b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.200+0000 7fae75600700 1 -- 172.21.15.23:0/45415425 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700cbbb0 con 0x564d28a210c0 2024-03-30T04:15:54.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae75600700 1 -- 172.21.15.23:0/45415425 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae7009a7c0 con 0x564d28a210c0 2024-03-30T04:15:54.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae75600700 1 -- 172.21.15.23:0/45415425 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700a3020 con 0x564d28a210c0 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 -- 172.21.15.23:0/45415425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 msgr2=0x564d28a39bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fae7009ab60 tx=0x7fae70099410 comp rx=0 tx=0).stop 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 -- 172.21.15.23:0/45415425 shutdown_connections 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a3dc20 0x564d28a34bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39bd0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/45415425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a2bfc0 0x564d28a2ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 -- 172.21.15.23:0/45415425 >> 172.21.15.23:0/45415425 conn(0x564d289806a0 msgr2=0x564d28a16010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.201+0000 7fae99fc2b80 1 -- 172.21.15.23:0/45415425 shutdown_connections 2024-03-30T04:15:54.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.202+0000 7fae99fc2b80 1 -- 172.21.15.23:0/45415425 wait complete. 2024-03-30T04:15:54.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.202+0000 7fae99fc2b80 1 -- start start 2024-03-30T04:15:54.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.202+0000 7fae99fc2b80 1 Processor -- start 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 0x564d28a37df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a3dc20 0x564d28a38330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a388b0 con 0x564d28a210c0 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a389f0 con 0x564d28a2bfc0 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d28a38b80 con 0x564d28a3dc20 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae99fc2b80 10 client.0 _ll_register_callbacks cb 0x564d28918470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae7ea00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 0x564d28a37df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae7f400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35688/0 (socket says 172.21.15.23:35688) 2024-03-30T04:15:54.205 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.203+0000 7fae7f400700 1 -- 172.21.15.23:0/1260949750 learned_addr learned my addr 172.21.15.23:0/1260949750 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7fe00700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a3dc20 0x564d28a38330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7ea00700 1 -- 172.21.15.23:0/1260949750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a3dc20 msgr2=0x564d28a38330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7ea00700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a3dc20 0x564d28a38330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7ea00700 1 -- 172.21.15.23:0/1260949750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 msgr2=0x564d28a39a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7ea00700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7ea00700 1 -- 172.21.15.23:0/1260949750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae78009040 con 0x564d28a2bfc0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.204+0000 7fae7f400700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae7ea00700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 0x564d28a37df0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x564d28a46dc0 tx=0x7fae6c014f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ace13323393f09 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae69600700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c01d440 con 0x564d28a2bfc0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fae6c013410 con 0x564d28a2bfc0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae7009ae60 con 0x564d28a2bfc0 2024-03-30T04:15:54.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae6c01d5a0 con 0x564d28a2bfc0 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae99fc2b80 10 client.29279 fetch_fsmap learned FSMap version 31 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564d287a0b60 con 0x564d28a2bfc0 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.205+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae7009ae60 con 0x564d28a2bfc0 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fae6c013a70 con 0x564d28a2bfc0 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 10 client.29279 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 10 client.29279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 20 client.29279 populate_metadata read hostname 'smithi023' 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a994e0 0x564d28a9b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 4 client.29279 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d287c1cf0 con 0x564d28a994e0 2024-03-30T04:15:54.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.206+0000 7fae7f400700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a994e0 0x564d28a9b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:15:54.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae7f400700 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a994e0 0x564d28a9b900 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c8c7fed6614448f server_cookie=af24ced276918b in_seq=0 out_seq=0 2024-03-30T04:15:54.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae69600700 10 client.29279 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:54.208 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae69600700 1 -- 172.21.15.23:0/1260949750 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564d287c1cf0 con 0x564d28a994e0 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae69600700 10 client.29279 handle_command_reply: tid=0 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae99fc2b80 1 client.29279 shutdown 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 msgr2=0x564d28a37df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.207+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 0x564d28a37df0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x564d28a46dc0 tx=0x7fae6c014f10 comp rx=0 tx=0).stop 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 shutdown_connections 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x564d28a994e0 0x564d28a9b900 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d28a3dc20 0x564d28a38330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x564d28a210c0 0x564d28a39a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae69600700 0 client.29279 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 --2- 172.21.15.23:0/1260949750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x564d28a2bfc0 0x564d28a37df0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 >> 172.21.15.23:0/1260949750 conn(0x564d289806a0 msgr2=0x564d28a29c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 shutdown_connections 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 1 -- 172.21.15.23:0/1260949750 wait complete. 2024-03-30T04:15:54.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.208+0000 7fae99fc2b80 20 client.29279 trim_cache size 0 max 16384 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": { 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "22f31e55-e366-463d-8285-656358f15f47": { 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "path": "/", 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tag": "22f31e55-e366-463d-8285-656358f15f47", 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: "options": "recursive,force" 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:15:54.210 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 msgr2=0x7fae640a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 0x7fae640a3790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae800678e0 tx=0x7fae7800b040 comp rx=0 tx=0).stop 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 msgr2=0x7fae80199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fae70005770 tx=0x7fae700057a0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 shutdown_connections 2024-03-30T04:15:54.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fae8010d3e0 0x7fae8019dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fae80073180 0x7fae80199370 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fae640a12d0 0x7fae640a3790 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 --2- 172.21.15.23:0/2920634787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae80073aa0 0x7fae801998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 >> 172.21.15.23:0/2920634787 conn(0x7fae800fc670 msgr2=0x7fae800fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 shutdown_connections 2024-03-30T04:15:54.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:15:54.209+0000 7fae84e00700 1 -- 172.21.15.23:0/2920634787 wait complete. 2024-03-30T04:15:54.228 INFO:tasks.cephfs.filesystem:scrub status for tag:22f31e55-e366-463d-8285-656358f15f47 - {'path': '/', 'tag': '22f31e55-e366-463d-8285-656358f15f47', 'options': 'recursive,force'} 2024-03-30T04:15:54.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:54.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:54 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 13781 29.86 MB/sec warmup 31 sec latency 1558.340 ms 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:54.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:54 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:54.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:54.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:54.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:54.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:54.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:54 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 13960 29.24 MB/sec warmup 32 sec latency 93.594 ms 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub queued for path: / 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: active paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:55 smithi154 ceph-mon[79161]: pgmap v438: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:55.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub queued for path: / 2024-03-30T04:15:55.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: active paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:55 smithi193 ceph-mon[79581]: pgmap v438: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:55.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub queued for path: / 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: active paths [/] 2024-03-30T04:15:55.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:55.773 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:55 smithi023 ceph-mon[38254]: pgmap v438: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:56.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 14255 28.77 MB/sec warmup 33 sec latency 76.106 ms 2024-03-30T04:15:57.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:57 smithi193 ceph-mon[79581]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:57 smithi023 ceph-mon[38254]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:57.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 14592 28.27 MB/sec warmup 34 sec latency 239.667 ms 2024-03-30T04:15:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:57 smithi154 ceph-mon[79161]: scrub summary: idle+waiting paths [/] 2024-03-30T04:15:58.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:15:58 smithi193 ceph-mon[79581]: pgmap v439: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:15:58 smithi023 ceph-mon[38254]: pgmap v439: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:58.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 15205 28.16 MB/sec warmup 35 sec latency 259.677 ms 2024-03-30T04:15:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:15:58 smithi154 ceph-mon[79161]: pgmap v439: 129 pgs: 129 active+clean; 787 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 6.2 MiB/s wr, 323 op/s 2024-03-30T04:15:59.611 INFO:tasks.workunit.client.0.smithi023.stdout: 10 15918 28.46 MB/sec warmup 36 sec latency 187.840 ms 2024-03-30T04:16:00.302 INFO:tasks.workunit.client.0.smithi023.stdout:[16299] write failed on handle 13377 (Resource temporarily unavailable) 2024-03-30T04:16:00.303 INFO:tasks.workunit.client.0.smithi023.stdout:Child failed with status 1 2024-03-30T04:16:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:16:00.352 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-30T04:16:00.353 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-30T04:16:00.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:00 smithi193 ceph-mon[79581]: pgmap v440: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 600 op/s 2024-03-30T04:16:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:00 smithi023 ceph-mon[38254]: pgmap v440: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 600 op/s 2024-03-30T04:16:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:00 smithi154 ceph-mon[79161]: pgmap v440: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 13 MiB/s wr, 600 op/s 2024-03-30T04:16:01.364 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-30T04:16:01.579 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=99130478e84d41748b55014704480009 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_470100a4ccbcf7769052bf09a5413881ccf1b97f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-30T04:16:01.583 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-30T04:16:01.594 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-30T04:16:01.594 INFO:teuthology.task.exec:Executing custom commands... 2024-03-30T04:16:01.594 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:16:01.594 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-30T04:16:01.669 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-30T04:16:01.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-30T04:16:01.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 -- 172.21.15.23:0/464850718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee01018a0 msgr2=0x7f5ee01058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:01.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 --2- 172.21.15.23:0/464850718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee01018a0 0x7f5ee01058e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0098f0 tx=0x7f5ecc009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:01.937 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 -- 172.21.15.23:0/464850718 shutdown_connections 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 --2- 172.21.15.23:0/464850718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0106010 0x7f5ee010db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 --2- 172.21.15.23:0/464850718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0100ef0 0x7f5ee01012d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 --2- 172.21.15.23:0/464850718 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee01018a0 0x7f5ee01058e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 -- 172.21.15.23:0/464850718 >> 172.21.15.23:0/464850718 conn(0x7f5ee00fc750 msgr2=0x7f5ee00feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 -- 172.21.15.23:0/464850718 shutdown_connections 2024-03-30T04:16:01.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.938+0000 7f5ee6400700 1 -- 172.21.15.23:0/464850718 wait complete. 2024-03-30T04:16:01.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.939+0000 7f5ee6400700 1 Processor -- start 2024-03-30T04:16:01.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 -- start start 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0100ef0 0x7f5ee01993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0106010 0x7f5ee0199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee019e730 con 0x7f5ee019dcb0 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee019e8a0 con 0x7f5ee0106010 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5ee6400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee019ea60 con 0x7f5ee0100ef0 2024-03-30T04:16:01.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5edfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5edfe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37230/0 (socket says 172.21.15.23:37230) 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5edfe00700 1 -- 172.21.15.23:0/2224303200 learned_addr learned my addr 172.21.15.23:0/2224303200 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5edf400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0100ef0 0x7f5ee01993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.940+0000 7f5edea00700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0106010 0x7f5ee0199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 -- 172.21.15.23:0/2224303200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0100ef0 msgr2=0x7f5ee01993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0100ef0 0x7f5ee01993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 -- 172.21.15.23:0/2224303200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0106010 msgr2=0x7f5ee0199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:01.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0106010 0x7f5ee0199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 -- 172.21.15.23:0/2224303200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed00095f0 con 0x7f5ee019dcb0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5edfe00700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800ef30 tx=0x7f5ed800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8018070 con 0x7f5ee019dcb0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ed800f040 con 0x7f5ee019dcb0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed800cd00 con 0x7f5ee019dcb0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc009520 con 0x7f5ee019dcb0 2024-03-30T04:16:01.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.941+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee01a6260 con 0x7f5ee019dcb0 2024-03-30T04:16:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.943+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee01023c0 con 0x7f5ee019dcb0 2024-03-30T04:16:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.944+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f5ed8003a00 con 0x7f5ee019dcb0 2024-03-30T04:16:01.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.946+0000 7f5ed7e00700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 0x7f5ec40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.946+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f5ed80ccb50 con 0x7f5ee019dcb0 2024-03-30T04:16:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.946+0000 7f5edf400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 0x7f5ec40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:01.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.947+0000 7f5edf400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 0x7f5ec40a37d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0005fd0 tx=0x7f5ed0009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:01.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:01.950+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f5ed8095bf0 con 0x7f5ee019dcb0 2024-03-30T04:16:02.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.200+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ee0101910 con 0x7f5ec40a1310 2024-03-30T04:16:02.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.210+0000 7f5ed7e00700 1 -- 172.21.15.23:0/2224303200 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f5ee0101910 con 0x7f5ec40a1310 2024-03-30T04:16:02.209 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-30T00:00:00", "created": "2024-03-30T04:01:57", "first": "2024-03-30T04:02:00", "last": "2024-03-30T04:16:00", "last_pruned": "2024-03-30T04:16:00", "created_count": 15, "pruned_count": 9, "active": true} 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 msgr2=0x7f5ec40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 0x7f5ec40a37d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0005fd0 tx=0x7f5ed0009310 comp rx=0 tx=0).stop 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 msgr2=0x7f5ee019e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5ed800ef30 tx=0x7f5ed800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 shutdown_connections 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ee019dcb0 0x7f5ee019e130 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee0100ef0 0x7f5ee01993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f5ec40a1310 0x7f5ec40a37d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 --2- 172.21.15.23:0/2224303200 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee0106010 0x7f5ee0199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 >> 172.21.15.23:0/2224303200 conn(0x7f5ee00fc750 msgr2=0x7f5ee00feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:02.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 shutdown_connections 2024-03-30T04:16:02.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.213+0000 7f5ee6400700 1 -- 172.21.15.23:0/2224303200 wait complete. 2024-03-30T04:16:02.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-30T04:16:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 -- 172.21.15.23:0/1047852053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 msgr2=0x7fa54010d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 --2- 172.21.15.23:0/1047852053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54010d760 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b0a0 tx=0x7fa53800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 -- 172.21.15.23:0/1047852053 shutdown_connections 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 --2- 172.21.15.23:0/1047852053 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54010d760 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 --2- 172.21.15.23:0/1047852053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 0x7fa540102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 --2- 172.21.15.23:0/1047852053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 0x7fa54010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.429+0000 7fa547400700 1 -- 172.21.15.23:0/1047852053 >> 172.21.15.23:0/1047852053 conn(0x7fa5400fb680 msgr2=0x7fa5400fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.430+0000 7fa547400700 1 -- 172.21.15.23:0/1047852053 shutdown_connections 2024-03-30T04:16:02.429 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.430+0000 7fa547400700 1 -- 172.21.15.23:0/1047852053 wait complete. 2024-03-30T04:16:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 Processor -- start 2024-03-30T04:16:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 -- start start 2024-03-30T04:16:02.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 0x7fa540199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 0x7fa5401999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019e380 con 0x7fa54010b310 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019e4f0 con 0x7fa540102860 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa547400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54019e6b0 con 0x7fa540101f40 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.431+0000 7fa545400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37250/0 (socket says 172.21.15.23:37250) 2024-03-30T04:16:02.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 -- 172.21.15.23:0/2063114792 learned_addr learned my addr 172.21.15.23:0/2063114792 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa53fe00700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 0x7fa5401999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa544a00700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 0x7fa540199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 -- 172.21.15.23:0/2063114792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 msgr2=0x7fa540199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 0x7fa540199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 -- 172.21.15.23:0/2063114792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 msgr2=0x7fa5401999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 0x7fa5401999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 -- 172.21.15.23:0/2063114792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538009d00 con 0x7fa54010b310 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa545400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b070 tx=0x7fa5380078c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800e040 con 0x7fa54010b310 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa53801d470 con 0x7fa54010b310 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538005280 con 0x7fa54010b310 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.432+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54019e930 con 0x7fa54010b310 2024-03-30T04:16:02.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.433+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5401a61d0 con 0x7fa54010b310 2024-03-30T04:16:02.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.434+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa540069710 con 0x7fa54010b310 2024-03-30T04:16:02.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.436+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fa53801d5e0 con 0x7fa54010b310 2024-03-30T04:16:02.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.437+0000 7fa53d600700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 0x7fa5280a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:02.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.437+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fa5380ce780 con 0x7fa54010b310 2024-03-30T04:16:02.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.437+0000 7fa544a00700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 0x7fa5280a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:02.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.438+0000 7fa544a00700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 0x7fa5280a3750 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa534009c60 tx=0x7fa534009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:02.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.439+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fa538097820 con 0x7fa54010b310 2024-03-30T04:16:02.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:02 smithi193 ceph-mon[79581]: pgmap v441: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 476 op/s 2024-03-30T04:16:02.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:02 smithi023 ceph-mon[38254]: pgmap v441: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 476 op/s 2024-03-30T04:16:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:02 smithi154 ceph-mon[79161]: pgmap v441: 129 pgs: 129 active+clean; 778 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 12 MiB/s wr, 476 op/s 2024-03-30T04:16:02.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.686+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 --> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa540110c40 con 0x7fa5280a1290 2024-03-30T04:16:02.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.696+0000 7fa53d600700 1 -- 172.21.15.23:0/2063114792 <== mgr.15627 v2:172.21.15.23:6800/250950136 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fa540110c40 con 0x7fa5280a1290 2024-03-30T04:16:02.695 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 msgr2=0x7fa5280a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 0x7fa5280a3750 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa534009c60 tx=0x7fa534009260 comp rx=0 tx=0).stop 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 msgr2=0x7fa54019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fa53800b070 tx=0x7fa5380078c0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 shutdown_connections 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa54010b310 0x7fa54019dd40 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa540101f40 0x7fa540199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fa5280a1290 0x7fa5280a3750 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 --2- 172.21.15.23:0/2063114792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa540102860 0x7fa5401999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:02.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 >> 172.21.15.23:0/2063114792 conn(0x7fa5400fb680 msgr2=0x7fa5400fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:02.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 shutdown_connections 2024-03-30T04:16:02.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:02.699+0000 7fa547400700 1 -- 172.21.15.23:0/2063114792 wait complete. 2024-03-30T04:16:02.716 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-30T04:16:02.728 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-30T04:16:03.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:03 smithi193 ceph-mon[79581]: from='client.19326 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:03.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:03 smithi023 ceph-mon[38254]: from='client.19326 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:03 smithi154 ceph-mon[79161]: from='client.19326 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:04.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:04 smithi193 ceph-mon[79581]: pgmap v442: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 13 MiB/s wr, 598 op/s 2024-03-30T04:16:04.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:04 smithi193 ceph-mon[79581]: from='client.19332 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:04 smithi023 ceph-mon[38254]: pgmap v442: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 13 MiB/s wr, 598 op/s 2024-03-30T04:16:04.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:04 smithi023 ceph-mon[38254]: from='client.19332 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:04 smithi154 ceph-mon[79161]: pgmap v442: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 13 MiB/s wr, 598 op/s 2024-03-30T04:16:04.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:04 smithi154 ceph-mon[79161]: from='client.19332 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-30T04:16:05.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:05 smithi193 ceph-mon[79581]: pgmap v443: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:05.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:05 smithi023 ceph-mon[38254]: pgmap v443: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:05.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:05 smithi154 ceph-mon[79161]: pgmap v443: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:08.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:08 smithi193 ceph-mon[79581]: pgmap v444: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:08.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:08 smithi023 ceph-mon[38254]: pgmap v444: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:08.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:08 smithi154 ceph-mon[79161]: pgmap v444: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 8.5 MiB/s wr, 398 op/s 2024-03-30T04:16:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:16:09.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:16:09.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:16:09.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:09 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:16:09.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:16:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:16:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:16:09.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:09 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:16:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:16:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:16:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:16:09.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:09 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:16:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:10 smithi193 ceph-mon[79581]: pgmap v445: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.5 MiB/s wr, 408 op/s 2024-03-30T04:16:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:10 smithi193 ceph-mon[79581]: scrub summary: idle 2024-03-30T04:16:10.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:10 smithi193 ceph-mon[79581]: osdmap e106: 12 total, 12 up, 12 in 2024-03-30T04:16:10.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:10 smithi023 ceph-mon[38254]: pgmap v445: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.5 MiB/s wr, 408 op/s 2024-03-30T04:16:10.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:10 smithi023 ceph-mon[38254]: scrub summary: idle 2024-03-30T04:16:10.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:10 smithi023 ceph-mon[38254]: osdmap e106: 12 total, 12 up, 12 in 2024-03-30T04:16:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:10 smithi154 ceph-mon[79161]: pgmap v445: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 8.5 MiB/s wr, 408 op/s 2024-03-30T04:16:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:10 smithi154 ceph-mon[79161]: scrub summary: idle 2024-03-30T04:16:10.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:10 smithi154 ceph-mon[79161]: osdmap e106: 12 total, 12 up, 12 in 2024-03-30T04:16:12.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:12 smithi193 ceph-mon[79581]: pgmap v447: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-03-30T04:16:12.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:12 smithi023 ceph-mon[38254]: pgmap v447: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-03-30T04:16:12.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:12 smithi154 ceph-mon[79161]: pgmap v447: 129 pgs: 129 active+clean; 774 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.5 MiB/s wr, 158 op/s 2024-03-30T04:16:14.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:14 smithi193 ceph-mon[79581]: pgmap v448: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:14.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:14 smithi023 ceph-mon[38254]: pgmap v448: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:14.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:14 smithi154 ceph-mon[79161]: pgmap v448: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:16.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:16 smithi193 ceph-mon[79581]: pgmap v449: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:16.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:16 smithi193 ceph-mon[79581]: osdmap e107: 12 total, 12 up, 12 in 2024-03-30T04:16:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:16 smithi023 ceph-mon[38254]: pgmap v449: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:16.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:16 smithi023 ceph-mon[38254]: osdmap e107: 12 total, 12 up, 12 in 2024-03-30T04:16:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:16 smithi154 ceph-mon[79161]: pgmap v449: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 46 KiB/s wr, 21 op/s 2024-03-30T04:16:16.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:16 smithi154 ceph-mon[79161]: osdmap e107: 12 total, 12 up, 12 in 2024-03-30T04:16:17.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:17 smithi193 ceph-mon[79581]: pgmap v451: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-30T04:16:17.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:17 smithi023 ceph-mon[38254]: pgmap v451: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-30T04:16:17.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:17 smithi154 ceph-mon[79161]: pgmap v451: 129 pgs: 129 active+clean; 773 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 12 op/s 2024-03-30T04:16:19.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:19 smithi154 ceph-mon[79161]: pgmap v452: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 56 KiB/s wr, 20 op/s 2024-03-30T04:16:19.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:19 smithi193 ceph-mon[79581]: pgmap v452: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 56 KiB/s wr, 20 op/s 2024-03-30T04:16:19.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:19 smithi023 ceph-mon[38254]: pgmap v452: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 56 KiB/s wr, 20 op/s 2024-03-30T04:16:20.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:20 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:16:20.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:20 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:16:20.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:20 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-30T04:16:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:21 smithi154 ceph-mon[79161]: pgmap v453: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-03-30T04:16:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:21 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:16:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:21 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:16:21.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:21 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:16:21.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:21 smithi193 ceph-mon[79581]: pgmap v453: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-03-30T04:16:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:21 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:16:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:21 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:16:21.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:21 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:16:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:21 smithi023 ceph-mon[38254]: pgmap v453: 129 pgs: 129 active+clean; 769 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 50 KiB/s wr, 18 op/s 2024-03-30T04:16:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:21 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-30T04:16:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:21 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-30T04:16:21.772 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:21 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' 2024-03-30T04:16:23.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:23 smithi154 ceph-mon[79161]: pgmap v454: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:23.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:23 smithi193 ceph-mon[79581]: pgmap v454: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:23.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:23 smithi023 ceph-mon[38254]: pgmap v454: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:24.229 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-30T04:16:24.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 -- 172.21.15.23:0/666031286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 msgr2=0x7f65bc101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 --2- 172.21.15.23:0/666031286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc101340 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009990 tx=0x7f65b0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 -- 172.21.15.23:0/666031286 shutdown_connections 2024-03-30T04:16:24.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 --2- 172.21.15.23:0/666031286 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc106080 0x7f65bc10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 --2- 172.21.15.23:0/666031286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc101340 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 --2- 172.21.15.23:0/666031286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc101910 0x7f65bc105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 -- 172.21.15.23:0/666031286 >> 172.21.15.23:0/666031286 conn(0x7f65bc0fc7c0 msgr2=0x7f65bc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 -- 172.21.15.23:0/666031286 shutdown_connections 2024-03-30T04:16:24.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.447+0000 7f65c3a00700 1 -- 172.21.15.23:0/666031286 wait complete. 2024-03-30T04:16:24.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 Processor -- start 2024-03-30T04:16:24.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 -- start start 2024-03-30T04:16:24.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc106080 0x7f65bc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc1a59f0 con 0x7f65bc073b30 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc1a5b30 con 0x7f65bc100f60 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.448+0000 7f65c3a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc1a5cf0 con 0x7f65bc106080 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56118/0 (socket says 172.21.15.23:56118) 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 -- 172.21.15.23:0/444013513 learned_addr learned my addr 172.21.15.23:0/444013513 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65bbe00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc106080 0x7f65bc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 -- 172.21.15.23:0/444013513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc106080 msgr2=0x7f65bc0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc106080 0x7f65bc0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 -- 172.21.15.23:0/444013513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 msgr2=0x7f65bc0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 -- 172.21.15.23:0/444013513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00095f0 con 0x7f65bc073b30 2024-03-30T04:16:24.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1000700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.449+0000 7f65c1a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f65b400ef30 tx=0x7f65b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.450+0000 7f65b9600700 1 -- 172.21.15.23:0/444013513 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4018070 con 0x7f65bc073b30 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.450+0000 7f65b9600700 1 -- 172.21.15.23:0/444013513 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65b400f040 con 0x7f65bc073b30 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.450+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc1a5f70 con 0x7f65bc073b30 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.450+0000 7f65b9600700 1 -- 172.21.15.23:0/444013513 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b400cdb0 con 0x7f65bc073b30 2024-03-30T04:16:24.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.450+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc1a6450 con 0x7f65bc073b30 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.453+0000 7f65b9600700 1 -- 172.21.15.23:0/444013513 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f65b4003a00 con 0x7f65bc073b30 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65d74b5b80 1 Processor -- start 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65b9600700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 0x7f65a40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65d74b5b80 1 -- start start 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65b9600700 1 -- 172.21.15.23:0/444013513 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f65b40ccc50 con 0x7f65bc073b30 2024-03-30T04:16:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65bbe00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 0x7f65a40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05b79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.454+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f0510990 con 0x55c0f05b7ef0 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f050e350 con 0x55c0f05c2df0 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f0509c40 con 0x55c0f05b75d0 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05b79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05b79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48258/0 (socket says 172.21.15.23:48258) 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1000700 1 -- 172.21.15.23:0/4232170236 learned_addr learned my addr 172.21.15.23:0/4232170236 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 -- 172.21.15.23:0/4232170236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 msgr2=0x55c0f05b79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05b79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 -- 172.21.15.23:0/4232170236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 msgr2=0x55c0f05c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 -- 172.21.15.23:0/4232170236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0f038b3b0 con 0x55c0f05c2df0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65bbe00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 0x7f65a40a3750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f65bc074790 tx=0x7f65a80136e0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.455+0000 7f65c1a00700 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c51e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f65b40c9f60 tx=0x7f65b4099800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42c464144bfc74dd server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65ae000700 1 -- 172.21.15.23:0/4232170236 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4004ad0 con 0x55c0f05c2df0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65ae000700 1 -- 172.21.15.23:0/4232170236 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65b401b320 con 0x55c0f05c2df0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65ae000700 1 -- 172.21.15.23:0/4232170236 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4099c10 con 0x55c0f05c2df0 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 -- 172.21.15.23:0/4232170236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 msgr2=0x55c0f05c51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c51e0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f65b40c9f60 tx=0x7f65b4099800 comp rx=0 tx=0).stop 2024-03-30T04:16:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 -- 172.21.15.23:0/4232170236 shutdown_connections 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c51e0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05b79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/4232170236 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.456+0000 7f65d74b5b80 1 -- 172.21.15.23:0/4232170236 >> 172.21.15.23:0/4232170236 conn(0x55c0f05176a0 msgr2=0x55c0f05b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.457+0000 7f65d74b5b80 1 -- 172.21.15.23:0/4232170236 shutdown_connections 2024-03-30T04:16:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.457+0000 7f65d74b5b80 1 -- 172.21.15.23:0/4232170236 wait complete. 2024-03-30T04:16:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.457+0000 7f65d74b5b80 1 -- start start 2024-03-30T04:16:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.457+0000 7f65d74b5b80 1 Processor -- start 2024-03-30T04:16:24.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b7ef0 0x55c0f05bfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f04a5d80 con 0x55c0f05b75d0 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f04a5a20 con 0x55c0f05c2df0 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f038d6a0 con 0x55c0f05b7ef0 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65d74b5b80 10 client.0 _ll_register_callbacks cb 0x55c0f04af470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65c1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b7ef0 0x55c0f05bfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65c1000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56148/0 (socket says 172.21.15.23:56148) 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65c1000700 1 -- 172.21.15.23:0/1577516849 learned_addr learned my addr 172.21.15.23:0/1577516849 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.458+0000 7f65c1a00700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 -- 172.21.15.23:0/1577516849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b7ef0 msgr2=0x55c0f05bfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b7ef0 0x55c0f05bfb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 -- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 msgr2=0x55c0f05c02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 -- 172.21.15.23:0/1577516849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65bc1a6450 con 0x55c0f05b75d0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1a00700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65c1000700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f65b0005280 tx=0x7f65b00052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859d98f2c4d4a232 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b001d070 con 0x55c0f05b75d0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f65b0004910 con 0x55c0f05b75d0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c0f04ca380 con 0x55c0f05b75d0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65b0004ad0 con 0x55c0f05b75d0 2024-03-30T04:16:24.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65d74b5b80 10 client.19344 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.459+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c0f0337b60 con 0x55c0f05b75d0 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c0f04ca380 con 0x55c0f05b75d0 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f65b000fc70 con 0x55c0f05b75d0 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 10 client.19344 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 10 client.19344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 20 client.19344 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f05d2ad0 0x55c0f05d4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 4 client.19344 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0f0502700 con 0x55c0f05d2ad0 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.460+0000 7f65bbe00700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f05d2ad0 0x55c0f05d4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.461+0000 7f65bbe00700 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f05d2ad0 0x55c0f05d4ef0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=490fcfbdb5e156d1 server_cookie=31ae9e643787ab2a in_seq=0 out_seq=0 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.461+0000 7f65a2a00700 10 client.19344 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65a2a00700 1 -- 172.21.15.23:0/1577516849 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55c0f0502700 con 0x55c0f05d2ad0 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65a2a00700 10 client.19344 handle_command_reply: tid=0 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 client.19344 shutdown 2024-03-30T04:16:24.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 msgr2=0x55c0f05c2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f65b0005280 tx=0x7f65b00052b0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 shutdown_connections 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05c02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b75d0 0x55c0f05c2890 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f05d2ad0 0x55c0f05d4ef0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1577516849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b7ef0 0x55c0f05bfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65a2a00700 0 client.19344 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:24.469 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 >> 172.21.15.23:0/1577516849 conn(0x55c0f05176a0 msgr2=0x55c0f05b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 shutdown_connections 2024-03-30T04:16:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1577516849 wait complete. 2024-03-30T04:16:24.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.462+0000 7f65d74b5b80 20 client.19344 trim_cache size 0 max 16384 2024-03-30T04:16:24.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 Processor -- start 2024-03-30T04:16:24.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 -- start start 2024-03-30T04:16:24.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f05b9180 con 0x55c0f05b7ef0 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f05d52b0 con 0x55c0f05c2df0 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.480+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f05ca030 con 0x55c0f05b75d0 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56152/0 (socket says 172.21.15.23:56152) 2024-03-30T04:16:24.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 -- 172.21.15.23:0/3007948912 learned_addr learned my addr 172.21.15.23:0/3007948912 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65c1a00700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65c1000700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 -- 172.21.15.23:0/3007948912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 msgr2=0x55c0f05c0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 -- 172.21.15.23:0/3007948912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 msgr2=0x55c0f05ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05ac5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 -- 172.21.15.23:0/3007948912 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0f04cfea0 con 0x55c0f05b7ef0 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65c1a00700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.481+0000 7f65bbe00700 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009fd0 tx=0x7f65a801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbfbfa5c482da4b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65aea00700 1 -- 172.21.15.23:0/3007948912 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8023040 con 0x55c0f05b7ef0 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65aea00700 1 -- 172.21.15.23:0/3007948912 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65a8024040 con 0x55c0f05b7ef0 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65aea00700 1 -- 172.21.15.23:0/3007948912 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8018070 con 0x55c0f05b7ef0 2024-03-30T04:16:24.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 -- 172.21.15.23:0/3007948912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 msgr2=0x55c0f05c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009fd0 tx=0x7f65a801b000 comp rx=0 tx=0).stop 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 -- 172.21.15.23:0/3007948912 shutdown_connections 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05c2df0 0x55c0f05ac5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/3007948912 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c9a70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 -- 172.21.15.23:0/3007948912 >> 172.21.15.23:0/3007948912 conn(0x55c0f05176a0 msgr2=0x55c0f05acf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 -- 172.21.15.23:0/3007948912 shutdown_connections 2024-03-30T04:16:24.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.482+0000 7f65d74b5b80 1 -- 172.21.15.23:0/3007948912 wait complete. 2024-03-30T04:16:24.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 -- start start 2024-03-30T04:16:24.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 Processor -- start 2024-03-30T04:16:24.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 0x55c0f05c8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f05c8d20 con 0x55c0f05c2df0 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f05c8e90 con 0x55c0f05b7ef0 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0f06268d0 con 0x55c0f05b75d0 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.483+0000 7f65d74b5b80 10 client.0 _ll_register_callbacks cb 0x55c0f04af470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65bbe00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35466/0 (socket says 172.21.15.23:35466) 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 0x55c0f05c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65bbe00700 1 -- 172.21.15.23:0/1044323669 learned_addr learned my addr 172.21.15.23:0/1044323669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1000700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 -- 172.21.15.23:0/1044323669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 msgr2=0x55c0f05c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 -- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 msgr2=0x55c0f05c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 -- 172.21.15.23:0/1044323669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0f038b3b0 con 0x55c0f05c2df0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65bbe00700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65c1a00700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 0x55c0f05c8610 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f65b40c9f90 tx=0x7f65b4009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283ffce460e8b423 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65a2a00700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4004ad0 con 0x55c0f05c2df0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f65b409a0b0 con 0x55c0f05c2df0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.484+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65b401b320 con 0x55c0f05c2df0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65b40b2290 con 0x55c0f05c2df0 2024-03-30T04:16:24.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 10 client.19350 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f65a802da00 con 0x55c0f05c2df0 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65b401b320 con 0x55c0f05c2df0 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f65b40a1910 con 0x55c0f05c2df0 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 10 client.19350 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 10 client.19350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 20 client.19350 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f0627c40 0x55c0f062a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 4 client.19350 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.485+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c0f0358cf0 con 0x55c0f0627c40 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65c1000700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f0627c40 0x55c0f062a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65c1000700 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f0627c40 0x55c0f062a060 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d799efa798256a8f server_cookie=6f2d6122dbd4b767 in_seq=0 out_seq=0 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65a2a00700 10 client.19350 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65a2a00700 1 -- 172.21.15.23:0/1044323669 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c0f0358cf0 con 0x55c0f0627c40 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65a2a00700 10 client.19350 handle_command_reply: tid=0 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 client.19350 shutdown 2024-03-30T04:16:24.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 msgr2=0x55c0f05c8610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 0x55c0f05c8610 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f65b40c9f90 tx=0x7f65b4009ed0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 shutdown_connections 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c0f05c2df0 0x55c0f05c8610 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.486+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55c0f0627c40 0x55c0f062a060 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c0f05b75d0 0x55c0f05c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 1 --2- 172.21.15.23:0/1044323669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55c0f05b7ef0 0x55c0f05c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 >> 172.21.15.23:0/1044323669 conn(0x55c0f05176a0 msgr2=0x55c0f05b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65a2a00700 0 client.19350 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 shutdown_connections 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 1 -- 172.21.15.23:0/1044323669 wait complete. 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65d74b5b80 20 client.19350 trim_cache size 0 max 16384 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:24.496 INFO:teuthology.orchestra.run.smithi023.stdout: "status": "no active scrubs running", 2024-03-30T04:16:24.497 INFO:teuthology.orchestra.run.smithi023.stdout: "scrubs": {} 2024-03-30T04:16:24.497 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:24.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 msgr2=0x7f65a40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 0x7f65a40a3750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f65bc074790 tx=0x7f65a80136e0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 msgr2=0x7f65bc1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f65b400ef30 tx=0x7f65b400c3a0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 shutdown_connections 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65bc073b30 0x7f65bc1a5420 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f65a40a1290 0x7f65a40a3750 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f65bc100f60 0x7f65bc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.487+0000 7f65c3a00700 1 --2- 172.21.15.23:0/444013513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65bc106080 0x7f65bc0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.488+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 >> 172.21.15.23:0/444013513 conn(0x7f65bc0fc7c0 msgr2=0x7f65bc0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.488+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 shutdown_connections 2024-03-30T04:16:24.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.488+0000 7f65c3a00700 1 -- 172.21.15.23:0/444013513 wait complete. 2024-03-30T04:16:24.505 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-30T04:16:24.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:16:24.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.708+0000 7f0882600700 1 -- 172.21.15.23:0/2619999578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 msgr2=0x7f087c101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 --2- 172.21.15.23:0/2619999578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c101240 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f086c009990 tx=0x7f086c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 -- 172.21.15.23:0/2619999578 shutdown_connections 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 --2- 172.21.15.23:0/2619999578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c105f80 0x7f087c10db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 --2- 172.21.15.23:0/2619999578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c101240 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 --2- 172.21.15.23:0/2619999578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c101810 0x7f087c105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 -- 172.21.15.23:0/2619999578 >> 172.21.15.23:0/2619999578 conn(0x7f087c0fc700 msgr2=0x7f087c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 -- 172.21.15.23:0/2619999578 shutdown_connections 2024-03-30T04:16:24.709 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.709+0000 7f0882600700 1 -- 172.21.15.23:0/2619999578 wait complete. 2024-03-30T04:16:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.710+0000 7f0882600700 1 Processor -- start 2024-03-30T04:16:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.710+0000 7f0882600700 1 -- start start 2024-03-30T04:16:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c105f80 0x7f087c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c19dc50 0x7f087c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c19e6d0 con 0x7f087c100e60 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c19e840 con 0x7f087c19dc50 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f0882600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087c19ea00 con 0x7f087c105f80 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f087b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f087b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56162/0 (socket says 172.21.15.23:56162) 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f087b400700 1 -- 172.21.15.23:0/4190731146 learned_addr learned my addr 172.21.15.23:0/4190731146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f087aa00700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c105f80 0x7f087c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.711+0000 7f087be00700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c19dc50 0x7f087c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 -- 172.21.15.23:0/4190731146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c105f80 msgr2=0x7f087c1998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c105f80 0x7f087c1998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 -- 172.21.15.23:0/4190731146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c19dc50 msgr2=0x7f087c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c19dc50 0x7f087c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 -- 172.21.15.23:0/4190731146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086c0095f0 con 0x7f087c100e60 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f087b400700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f086c000c00 tx=0x7f086c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c01d070 con 0x7f087c100e60 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f086c022470 con 0x7f087c100e60 2024-03-30T04:16:24.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087c1a5f00 con 0x7f087c100e60 2024-03-30T04:16:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086c017650 con 0x7f087c100e60 2024-03-30T04:16:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.712+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087c1a62c0 con 0x7f087c100e60 2024-03-30T04:16:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.713+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f087c102330 con 0x7f087c100e60 2024-03-30T04:16:24.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.715+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f086c0225e0 con 0x7f087c100e60 2024-03-30T04:16:24.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.717+0000 7f0873e00700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 0x7f08600a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:24.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.718+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f086c0ce400 con 0x7f087c100e60 2024-03-30T04:16:24.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.718+0000 7f087aa00700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 0x7f08600a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:24.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.718+0000 7f087aa00700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 0x7f08600a3750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087c19a8c0 tx=0x7f08680073d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:24.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.722+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f086c0974a0 con 0x7f087c100e60 2024-03-30T04:16:24.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.987+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f087c067a10 con 0x7f087c100e60 2024-03-30T04:16:24.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.988+0000 7f0873e00700 1 -- 172.21.15.23:0/4190731146 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10931 (secure 0 0 0) 0x7f086c09aac0 con 0x7f087c100e60 2024-03-30T04:16:24.987 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:24.987 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:15:25.335802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:16:24.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 msgr2=0x7f08600a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 0x7f08600a3750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f087c19a8c0 tx=0x7f08680073d0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 msgr2=0x7f087c199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:24.989 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f086c000c00 tx=0x7f086c005740 comp rx=0 tx=0).stop 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 shutdown_connections 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f08600a1290 0x7f08600a3750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f087c19dc50 0x7f087c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.990+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f087c100e60 0x7f087c199370 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.991+0000 7f0882600700 1 --2- 172.21.15.23:0/4190731146 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f087c105f80 0x7f087c1998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.991+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 >> 172.21.15.23:0/4190731146 conn(0x7f087c0fc700 msgr2=0x7f087c0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.991+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 shutdown_connections 2024-03-30T04:16:24.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:24.991+0000 7f0882600700 1 -- 172.21.15.23:0/4190731146 wait complete. 2024-03-30T04:16:24.991 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-30T04:16:25.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-30T04:16:25.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.210+0000 7fab7b400700 1 -- 172.21.15.23:0/2494317851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74100f60 msgr2=0x7fab74101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.209 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.210+0000 7fab7b400700 1 --2- 172.21.15.23:0/2494317851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74100f60 0x7fab74101340 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fab68009990 tx=0x7fab68009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.210+0000 7fab7b400700 1 -- 172.21.15.23:0/2494317851 shutdown_connections 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.210+0000 7fab7b400700 1 --2- 172.21.15.23:0/2494317851 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74106080 0x7fab7410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.211+0000 7fab7b400700 1 --2- 172.21.15.23:0/2494317851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74100f60 0x7fab74101340 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.211+0000 7fab7b400700 1 --2- 172.21.15.23:0/2494317851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab74101910 0x7fab74105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.211+0000 7fab7b400700 1 -- 172.21.15.23:0/2494317851 >> 172.21.15.23:0/2494317851 conn(0x7fab740fc7c0 msgr2=0x7fab740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.211+0000 7fab7b400700 1 -- 172.21.15.23:0/2494317851 shutdown_connections 2024-03-30T04:16:25.210 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.211+0000 7fab7b400700 1 -- 172.21.15.23:0/2494317851 wait complete. 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 Processor -- start 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 -- start start 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74106080 0x7fab741976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab7419ba90 0x7fab7419bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7419c510 con 0x7fab74100f60 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7419c680 con 0x7fab74106080 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab7b400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7419c840 con 0x7fab7419ba90 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab78a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab78a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56174/0 (socket says 172.21.15.23:56174) 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.212+0000 7fab78a00700 1 -- 172.21.15.23:0/274014725 learned_addr learned my addr 172.21.15.23:0/274014725 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab73e00700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74106080 0x7fab741976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab79400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab7419ba90 0x7fab7419bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 -- 172.21.15.23:0/274014725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab7419ba90 msgr2=0x7fab7419bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab7419ba90 0x7fab7419bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 -- 172.21.15.23:0/274014725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74106080 msgr2=0x7fab741976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74106080 0x7fab741976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 -- 172.21.15.23:0/274014725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab680095f0 con 0x7fab74100f60 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab78a00700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fab68000c00 tx=0x7fab68005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab71600700 1 -- 172.21.15.23:0/274014725 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6801d070 con 0x7fab74100f60 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab741a3d40 con 0x7fab74100f60 2024-03-30T04:16:25.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab71600700 1 -- 172.21.15.23:0/274014725 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab68022470 con 0x7fab74100f60 2024-03-30T04:16:25.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.213+0000 7fab71600700 1 -- 172.21.15.23:0/274014725 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68017650 con 0x7fab74100f60 2024-03-30T04:16:25.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.214+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab741a41c0 con 0x7fab74100f60 2024-03-30T04:16:25.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.217+0000 7fab71600700 1 -- 172.21.15.23:0/274014725 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fab680225e0 con 0x7fab74100f60 2024-03-30T04:16:25.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 Processor -- start 2024-03-30T04:16:25.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 -- start start 2024-03-30T04:16:25.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab71600700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 0x7fab5c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab71600700 1 -- 172.21.15.23:0/274014725 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab680ce3d0 con 0x7fab74100f60 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc90069b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab79400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 0x7fab5c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc90141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8f5f990 con 0x558cc90065d0 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8f5d350 con 0x558cc9006ef0 2024-03-30T04:16:25.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.218+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8f58c40 con 0x558cc9011df0 2024-03-30T04:16:25.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab78a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc90069b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35516/0 (socket says 172.21.15.23:35516) 2024-03-30T04:16:25.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 -- 172.21.15.23:0/398106257 learned_addr learned my addr 172.21.15.23:0/398106257 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab79400700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc90141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 -- 172.21.15.23:0/398106257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 msgr2=0x558cc90141e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc90141e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 -- 172.21.15.23:0/398106257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 msgr2=0x558cc90069b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc90069b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 -- 172.21.15.23:0/398106257 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cc8dda3b0 con 0x558cc9006ef0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab79400700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc90141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab78a00700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc90069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab79400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 0x7fab5c0a3750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0076f0 tx=0x7fab6c007ab0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.219+0000 7fab73e00700 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fab60009cf0 tx=0x7fab60011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cf0cc431624028a server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.220+0000 7fab65c00700 1 -- 172.21.15.23:0/398106257 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60011d80 con 0x558cc9006ef0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.220+0000 7fab65c00700 1 -- 172.21.15.23:0/398106257 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab60004510 con 0x558cc9006ef0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.220+0000 7fab65c00700 1 -- 172.21.15.23:0/398106257 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60005240 con 0x558cc9006ef0 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.220+0000 7fab8eff7b80 1 -- 172.21.15.23:0/398106257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 msgr2=0x558cc90118b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.220+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fab60009cf0 tx=0x7fab60011440 comp rx=0 tx=0).stop 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 -- 172.21.15.23:0/398106257 shutdown_connections 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc90141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc90069b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/398106257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9006ef0 0x558cc90118b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 -- 172.21.15.23:0/398106257 >> 172.21.15.23:0/398106257 conn(0x558cc8f666a0 msgr2=0x558cc9002a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 -- 172.21.15.23:0/398106257 shutdown_connections 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 -- 172.21.15.23:0/398106257 wait complete. 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 -- start start 2024-03-30T04:16:25.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.221+0000 7fab8eff7b80 1 Processor -- start 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc90116f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc900ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8ef4d80 con 0x558cc9011df0 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8ef4a20 con 0x558cc90065d0 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc8ddc6a0 con 0x558cc9006ef0 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab8eff7b80 10 client.0 _ll_register_callbacks cb 0x558cc8efe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab79400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab79400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56198/0 (socket says 172.21.15.23:56198) 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab78a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc90116f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab79400700 1 -- 172.21.15.23:0/3887626357 learned_addr learned my addr 172.21.15.23:0/3887626357 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.222+0000 7fab73e00700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc900ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 -- 172.21.15.23:0/3887626357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 msgr2=0x558cc900ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc900ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 -- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 msgr2=0x558cc90116f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc90116f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 -- 172.21.15.23:0/3887626357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab741a41c0 con 0x558cc9011df0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab79400700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fab6c01ccc0 tx=0x7fab6c004b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8425a7fd788b86 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab5a000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00e050 con 0x558cc9011df0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab6c01c3a0 con 0x558cc9011df0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558cc8f19380 con 0x558cc9011df0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6c019830 con 0x558cc9011df0 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab8eff7b80 10 client.19368 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.223+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7fab60009920 con 0x558cc9011df0 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558cc8f19380 con 0x558cc9011df0 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fab6c01ab00 con 0x558cc9011df0 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 10 client.19368 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 10 client.19368 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 20 client.19368 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc9025d00 0x558cc9028120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 4 client.19368 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558cc8f51700 con 0x558cc9025d00 2024-03-30T04:16:25.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.224+0000 7fab78a00700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc9025d00 0x558cc9028120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.225+0000 7fab78a00700 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc9025d00 0x558cc9028120 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b8831868eb6593ce server_cookie=b5e440cc61262861 in_seq=0 out_seq=0 2024-03-30T04:16:25.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.225+0000 7fab5a000700 10 client.19368 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.225+0000 7fab5a000700 1 -- 172.21.15.23:0/3887626357 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558cc8f51700 con 0x558cc9025d00 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.225+0000 7fab5a000700 10 client.19368 handle_command_reply: tid=0 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 client.19368 shutdown 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 msgr2=0x558cc900f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fab6c01ccc0 tx=0x7fab6c004b40 comp rx=0 tx=0).stop 2024-03-30T04:16:25.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 shutdown_connections 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9011df0 0x558cc900f000 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc90116f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc9025d00 0x558cc9028120 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/3887626357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc900ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab5a000700 0 client.19368 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 >> 172.21.15.23:0/3887626357 conn(0x558cc8f666a0 msgr2=0x558cc9008530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 shutdown_connections 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 1 -- 172.21.15.23:0/3887626357 wait complete. 2024-03-30T04:16:25.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.226+0000 7fab8eff7b80 20 client.19368 trim_cache size 0 max 16384 2024-03-30T04:16:25.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 Processor -- start 2024-03-30T04:16:25.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 -- start start 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc901ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc900d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc9003710 con 0x558cc9006ef0 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc901d3c0 con 0x558cc90065d0 2024-03-30T04:16:25.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc90063c0 con 0x558cc9011df0 2024-03-30T04:16:25.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.240+0000 7fab73e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56212/0 (socket says 172.21.15.23:56212) 2024-03-30T04:16:25.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 -- 172.21.15.23:0/2169056606 learned_addr learned my addr 172.21.15.23:0/2169056606 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab79400700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc900d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 -- 172.21.15.23:0/2169056606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 msgr2=0x558cc900d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc900d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab78a00700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc901ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 -- 172.21.15.23:0/2169056606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 msgr2=0x558cc901ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc901ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 -- 172.21.15.23:0/2169056606 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558cc8d86b60 con 0x558cc9006ef0 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab78a00700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc901ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-30T04:16:25.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab73e00700 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fab60009cf0 tx=0x7fab60005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48773e47f793a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab66600700 1 -- 172.21.15.23:0/2169056606 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60011c80 con 0x558cc9006ef0 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.241+0000 7fab66600700 1 -- 172.21.15.23:0/2169056606 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab60011de0 con 0x558cc9006ef0 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab66600700 1 -- 172.21.15.23:0/2169056606 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60003b40 con 0x558cc9006ef0 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2169056606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 msgr2=0x558cc900d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fab60009cf0 tx=0x7fab60005dd0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2169056606 shutdown_connections 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9011df0 0x558cc900d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc90065d0 0x558cc901ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2169056606 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc9006ef0 0x558cc900d3d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2169056606 >> 172.21.15.23:0/2169056606 conn(0x558cc8f666a0 msgr2=0x558cc9009df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2169056606 shutdown_connections 2024-03-30T04:16:25.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.242+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2169056606 wait complete. 2024-03-30T04:16:25.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 -- start start 2024-03-30T04:16:25.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 Processor -- start 2024-03-30T04:16:25.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc901af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 0x558cc901b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc901ba70 con 0x558cc90065d0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc901bbb0 con 0x558cc9011df0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cc901bd70 con 0x558cc9006ef0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.243+0000 7fab8eff7b80 10 client.0 _ll_register_callbacks cb 0x558cc8efe470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56220/0 (socket says 172.21.15.23:56220) 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 -- 172.21.15.23:0/2415241796 learned_addr learned my addr 172.21.15.23:0/2415241796 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab79400700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 0x558cc901b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab73e00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc901af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 -- 172.21.15.23:0/2415241796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 msgr2=0x558cc901af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc901af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 -- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 msgr2=0x558cc901b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 0x558cc901b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 -- 172.21.15.23:0/2415241796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6001d040 con 0x558cc90065d0 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab79400700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 0x558cc901b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab78a00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fab68097630 tx=0x7fab680a1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c91a5ba39e868688 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.244+0000 7fab5a000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6809b130 con 0x558cc90065d0 2024-03-30T04:16:25.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fab680cce40 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab60004510 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6809ef60 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 10 client.19374 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558cc8f1eea0 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab60004510 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fab6809b390 con 0x558cc90065d0 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 10 client.19374 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 10 client.19374 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 20 client.19374 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.245+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc907ad70 0x558cc907d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.246+0000 7fab8eff7b80 4 client.19374 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.246+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 --> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558cc8da7cf0 con 0x558cc907ad70 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.246+0000 7fab73e00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc907ad70 0x558cc907d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.246+0000 7fab5a000700 10 client.19374 ms_handle_connect on v2:172.21.15.154:6832/3615295039 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.246+0000 7fab73e00700 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc907ad70 0x558cc907d160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a78e8c74e116f03c server_cookie=1882b94cfa9c520d in_seq=0 out_seq=0 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab5a000700 1 -- 172.21.15.23:0/2415241796 <== mds.4 v2:172.21.15.154:6832/3615295039 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558cc8da7cf0 con 0x558cc907ad70 2024-03-30T04:16:25.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab5a000700 10 client.19374 handle_command_reply: tid=0 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 client.19374 shutdown 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 msgr2=0x558cc901cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fab68097630 tx=0x7fab680a1040 comp rx=0 tx=0).stop 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 shutdown_connections 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558cc9011df0 0x558cc901b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558cc90065d0 0x558cc901cb70 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.154:6832/3615295039,v1:172.21.15.154:6833/3615295039] conn(0x558cc907ad70 0x558cc907d160 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 --2- 172.21.15.23:0/2415241796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558cc9006ef0 0x558cc901af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 >> 172.21.15.23:0/2415241796 conn(0x558cc8f666a0 msgr2=0x558cc900d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab5a000700 0 client.19374 ms_handle_reset on v2:172.21.15.154:6832/3615295039 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 shutdown_connections 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 1 -- 172.21.15.23:0/2415241796 wait complete. 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.247+0000 7fab8eff7b80 20 client.19374 trim_cache size 0 max 16384 2024-03-30T04:16:25.253 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:16:25.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 msgr2=0x7fab5c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 0x7fab5c0a3750 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0076f0 tx=0x7fab6c007ab0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 msgr2=0x7fab741971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fab68000c00 tx=0x7fab68005740 comp rx=0 tx=0).stop 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 shutdown_connections 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab7419ba90 0x7fab7419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab74100f60 0x7fab741971b0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fab5c0a1290 0x7fab5c0a3750 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 --2- 172.21.15.23:0/274014725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fab74106080 0x7fab741976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 >> 172.21.15.23:0/274014725 conn(0x7fab740fc7c0 msgr2=0x7fab740fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 shutdown_connections 2024-03-30T04:16:25.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.248+0000 7fab7b400700 1 -- 172.21.15.23:0/274014725 wait complete. 2024-03-30T04:16:25.271 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-30T04:16:25.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 -- 172.21.15.23:0/1617150244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a04106080 msgr2=0x7f9a0410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 --2- 172.21.15.23:0/1617150244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a04106080 0x7f9a0410dc00 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b0a0 tx=0x7f99fc00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 -- 172.21.15.23:0/1617150244 shutdown_connections 2024-03-30T04:16:25.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 --2- 172.21.15.23:0/1617150244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a04106080 0x7f9a0410dc00 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 --2- 172.21.15.23:0/1617150244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 0x7f9a04101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 --2- 172.21.15.23:0/1617150244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04101910 0x7f9a04105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 -- 172.21.15.23:0/1617150244 >> 172.21.15.23:0/1617150244 conn(0x7f9a040fc7c0 msgr2=0x7f9a040febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 -- 172.21.15.23:0/1617150244 shutdown_connections 2024-03-30T04:16:25.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.474+0000 7f9a0b200700 1 -- 172.21.15.23:0/1617150244 wait complete. 2024-03-30T04:16:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.475+0000 7f9a0b200700 1 Processor -- start 2024-03-30T04:16:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.475+0000 7f9a0b200700 1 -- start start 2024-03-30T04:16:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 0x7f9a04199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04106080 0x7f9a04199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0419e8a0 con 0x7f9a0419de20 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0419ea10 con 0x7f9a04106080 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a0b200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0419ebd0 con 0x7f9a04100f60 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56238/0 (socket says 172.21.15.23:56238) 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 -- 172.21.15.23:0/3530862465 learned_addr learned my addr 172.21.15.23:0/3530862465 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a03e00700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 0x7f9a04199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a03400700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04106080 0x7f9a04199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 -- 172.21.15.23:0/3530862465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 msgr2=0x7f9a04199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 0x7f9a04199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 -- 172.21.15.23:0/3530862465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04106080 msgr2=0x7f9a04199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.476+0000 7f9a09200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04106080 0x7f9a04199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a09200700 1 -- 172.21.15.23:0/3530862465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc009d00 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a09200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b070 tx=0x7f99fc00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a00c00700 1 -- 172.21.15.23:0/3530862465 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00e040 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a00c00700 1 -- 172.21.15.23:0/3530862465 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99fc01d480 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a00c00700 1 -- 172.21.15.23:0/3530862465 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99fc0052a0 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a041a60d0 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.477+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a041a64a0 con 0x7f9a0419de20 2024-03-30T04:16:25.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.480+0000 7f9a00c00700 1 -- 172.21.15.23:0/3530862465 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f99fc01d5f0 con 0x7f9a0419de20 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 Processor -- start 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a00c00700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 0x7f99e80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 -- start start 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a00c00700 1 -- 172.21.15.23:0/3530862465 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f99fc0cde60 con 0x7f9a0419de20 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a03400700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 0x7f99e80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a078b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641955990 con 0x558641a07df0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641953350 con 0x5586419fc5d0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.482+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55864194ec40 con 0x5586419fcef0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a078b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56248/0 (socket says 172.21.15.23:56248) 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a09200700 1 -- 172.21.15.23:0/3046849554 learned_addr learned my addr 172.21.15.23:0/3046849554 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 0x7f99e80a37d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a0419aa90 tx=0x7f99f0009f80 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 -- 172.21.15.23:0/3046849554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 msgr2=0x5586419fc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419fc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 -- 172.21.15.23:0/3046849554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 msgr2=0x558641a0a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 -- 172.21.15.23:0/3046849554 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586417d03b0 con 0x5586419fcef0 2024-03-30T04:16:25.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03e00700 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419fc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a09200700 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.483+0000 7f9a03400700 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a078b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011d50 tx=0x7f99f001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fb4872f6b5dba96 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.484+0000 7f99f9600700 1 -- 172.21.15.23:0/3046849554 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f001ddb0 con 0x5586419fcef0 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.484+0000 7f99f9600700 1 -- 172.21.15.23:0/3046849554 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99f0004cf0 con 0x5586419fcef0 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.484+0000 7f99f9600700 1 -- 172.21.15.23:0/3046849554 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f000f620 con 0x5586419fcef0 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.484+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/3046849554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 msgr2=0x558641a078b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.484+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a078b0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011d50 tx=0x7f99f001c000 comp rx=0 tx=0).stop 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/3046849554 shutdown_connections 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419fc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/3046849554 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a078b0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/3046849554 >> 172.21.15.23:0/3046849554 conn(0x55864195c6a0 msgr2=0x5586419f8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/3046849554 shutdown_connections 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/3046849554 wait complete. 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 -- start start 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.485+0000 7f9a1ed4cb80 1 Processor -- start 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fcef0 0x558641a04b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558641a07df0 0x558641a052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586418ead80 con 0x5586419fc5d0 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586418eaa20 con 0x5586419fcef0 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586417d26a0 con 0x558641a07df0 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a1ed4cb80 10 client.0 _ll_register_callbacks cb 0x5586418f4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56250/0 (socket says 172.21.15.23:56250) 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 -- 172.21.15.23:0/2998598764 learned_addr learned my addr 172.21.15.23:0/2998598764 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.492 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03400700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fcef0 0x558641a04b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a09200700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558641a07df0 0x558641a052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 -- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558641a07df0 msgr2=0x558641a052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558641a07df0 0x558641a052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 -- 172.21.15.23:0/2998598764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fcef0 msgr2=0x558641a04b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.486+0000 7f9a03e00700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fcef0 0x558641a04b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f9a03e00700 1 -- 172.21.15.23:0/2998598764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a041a64a0 con 0x5586419fc5d0 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f9a03e00700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f99f40128a0 tx=0x7f99f4012bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1228d96ba7112fda server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f99ee000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:25.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4017070 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f99f4004020 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55864190f380 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99f40122e0 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f9a1ed4cb80 10 client.19386 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.487+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f99f0011ac0 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55864190f380 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f99f4003680 con 0x5586419fc5d0 2024-03-30T04:16:25.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 10 client.19386 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 10 client.19386 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 20 client.19386 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a17a20 0x558641a19e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 4 client.19386 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558641947700 con 0x558641a17a20 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.488+0000 7f9a03400700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a17a20 0x558641a19e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.489+0000 7f9a03400700 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a17a20 0x558641a19e40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f41f6787a3121036 server_cookie=a8908ed732e4093e in_seq=0 out_seq=0 2024-03-30T04:16:25.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.489+0000 7f99ee000700 10 client.19386 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f99ee000700 1 -- 172.21.15.23:0/2998598764 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558641947700 con 0x558641a17a20 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f99ee000700 10 client.19386 handle_command_reply: tid=0 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 client.19386 shutdown 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 msgr2=0x558641a07820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f99f40128a0 tx=0x7f99f4012bb0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 shutdown_connections 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a17a20 0x558641a19e40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558641a07df0 0x558641a052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fc5d0 0x558641a07820 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f99ee000700 0 client.19386 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/2998598764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fcef0 0x558641a04b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 >> 172.21.15.23:0/2998598764 conn(0x55864195c6a0 msgr2=0x5586419fe6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.490+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 shutdown_connections 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.491+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/2998598764 wait complete. 2024-03-30T04:16:25.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.491+0000 7f9a1ed4cb80 20 client.19386 trim_cache size 0 max 16384 2024-03-30T04:16:25.502 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.504+0000 7f9a1ed4cb80 1 Processor -- start 2024-03-30T04:16:25.503 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.504+0000 7f9a1ed4cb80 1 -- start start 2024-03-30T04:16:25.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641a060c0 con 0x558641a07df0 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641a0efe0 con 0x5586419fc5d0 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586419f8f20 con 0x5586419fcef0 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a09200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35600/0 (socket says 172.21.15.23:35600) 2024-03-30T04:16:25.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 -- 172.21.15.23:0/4067741748 learned_addr learned my addr 172.21.15.23:0/4067741748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03400700 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 -- 172.21.15.23:0/4067741748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 msgr2=0x558641a05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a05240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 -- 172.21.15.23:0/4067741748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 msgr2=0x558641a0eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.505+0000 7f9a03e00700 1 -- 172.21.15.23:0/4067741748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55864177cb60 con 0x5586419fc5d0 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.506+0000 7f9a03e00700 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f99f4003b60 tx=0x7f99f4003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2591bce26fc92acf server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.506+0000 7f99fa000700 1 -- 172.21.15.23:0/4067741748 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4017070 con 0x5586419fc5d0 2024-03-30T04:16:25.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.506+0000 7f99fa000700 1 -- 172.21.15.23:0/4067741748 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99f4018040 con 0x5586419fc5d0 2024-03-30T04:16:25.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.506+0000 7f99fa000700 1 -- 172.21.15.23:0/4067741748 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4004b90 con 0x5586419fc5d0 2024-03-30T04:16:25.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/4067741748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 msgr2=0x5586419ff5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f99f4003b60 tx=0x7f99f4003b90 comp rx=0 tx=0).stop 2024-03-30T04:16:25.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/4067741748 shutdown_connections 2024-03-30T04:16:25.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x558641a07df0 0x558641a0eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586419fc5d0 0x5586419ff5b0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/4067741748 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fcef0 0x558641a05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/4067741748 >> 172.21.15.23:0/4067741748 conn(0x55864195c6a0 msgr2=0x5586419f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/4067741748 shutdown_connections 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/4067741748 wait complete. 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 -- start start 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.507+0000 7f9a1ed4cb80 1 Processor -- start 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fc5d0 0x558641a64c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558641a07df0 0x558641a0ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641a0d160 con 0x5586419fcef0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641a0d2d0 con 0x558641a07df0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x558641a0d490 con 0x5586419fc5d0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a1ed4cb80 10 client.0 _ll_register_callbacks cb 0x5586418f4470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fc5d0 0x558641a64c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56278/0 (socket says 172.21.15.23:56278) 2024-03-30T04:16:25.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 -- 172.21.15.23:0/174604108 learned_addr learned my addr 172.21.15.23:0/174604108 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a09200700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558641a07df0 0x558641a0ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 -- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fc5d0 msgr2=0x558641a64c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fc5d0 0x558641a64c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 -- 172.21.15.23:0/174604108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558641a07df0 msgr2=0x558641a0ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558641a07df0 0x558641a0ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.508+0000 7f9a03400700 1 -- 172.21.15.23:0/174604108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f0011ac0 con 0x5586419fcef0 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f9a03400700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f99f0005850 tx=0x7f99f0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8de51c2f41dda4f server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f99ee000700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f002b040 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f99f0032450 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99f0004cf0 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99f001dab0 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f9a1ed4cb80 10 client.19392 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.509+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558641914ea0 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99f0004cf0 con 0x5586419fcef0 2024-03-30T04:16:25.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f99f0038020 con 0x5586419fcef0 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 10 client.19392 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 10 client.19392 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 20 client.19392 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a6c970 0x558641a6ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 4 client.19392 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 --> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55864179dcf0 con 0x558641a6c970 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a03e00700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a6c970 0x558641a6ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f99ee000700 10 client.19392 ms_handle_connect on v2:172.21.15.193:6834/2492108108 2024-03-30T04:16:25.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.510+0000 7f9a03e00700 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a6c970 0x558641a6ed90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc443112f0d64a82 server_cookie=806f891033631de9 in_seq=0 out_seq=0 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f99ee000700 1 -- 172.21.15.23:0/174604108 <== mds.3 v2:172.21.15.193:6834/2492108108 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55864179dcf0 con 0x558641a6c970 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f99ee000700 10 client.19392 handle_command_reply: tid=0 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 client.19392 shutdown 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 msgr2=0x558641a65170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f99f0005850 tx=0x7f99f0005d00 comp rx=0 tx=0).stop 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 shutdown_connections 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:6834/2492108108,v1:172.21.15.193:6835/2492108108] conn(0x558641a6c970 0x558641a6ed90 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.517 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x558641a07df0 0x558641a0ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586419fc5d0 0x558641a64c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f99ee000700 0 client.19392 ms_handle_reset on v2:172.21.15.193:6834/2492108108 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 --2- 172.21.15.23:0/174604108 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586419fcef0 0x558641a65170 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 >> 172.21.15.23:0/174604108 conn(0x55864195c6a0 msgr2=0x5586419ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 shutdown_connections 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 1 -- 172.21.15.23:0/174604108 wait complete. 2024-03-30T04:16:25.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.511+0000 7f9a1ed4cb80 20 client.19392 trim_cache size 0 max 16384 2024-03-30T04:16:25.519 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:16:25.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 msgr2=0x7f99e80a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 0x7f99e80a37d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9a0419aa90 tx=0x7f99f0009f80 comp rx=0 tx=0).stop 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 msgr2=0x7f9a0419e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00b070 tx=0x7f99fc00bda0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 shutdown_connections 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9a0419de20 0x7f9a0419e2a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9a04100f60 0x7f9a04199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f99e80a1310 0x7f99e80a37d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 --2- 172.21.15.23:0/3530862465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a04106080 0x7f9a04199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 >> 172.21.15.23:0/3530862465 conn(0x7f9a040fc7c0 msgr2=0x7f9a040fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 shutdown_connections 2024-03-30T04:16:25.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.512+0000 7f9a0b200700 1 -- 172.21.15.23:0/3530862465 wait complete. 2024-03-30T04:16:25.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:25 smithi023 ceph-mon[38254]: pgmap v455: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:25.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:25 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/4190731146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:25.528 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-30T04:16:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:25 smithi154 ceph-mon[79161]: pgmap v455: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:25.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:25 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/4190731146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 -- 172.21.15.23:0/1364005450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98100f60 msgr2=0x7f9b98101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 --2- 172.21.15.23:0/1364005450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98100f60 0x7f9b98101340 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009990 tx=0x7f9b8c009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 -- 172.21.15.23:0/1364005450 shutdown_connections 2024-03-30T04:16:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 --2- 172.21.15.23:0/1364005450 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98106080 0x7f9b9810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 --2- 172.21.15.23:0/1364005450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98100f60 0x7f9b98101340 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 --2- 172.21.15.23:0/1364005450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98101910 0x7f9b98105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 -- 172.21.15.23:0/1364005450 >> 172.21.15.23:0/1364005450 conn(0x7f9b980fc7c0 msgr2=0x7f9b980febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 -- 172.21.15.23:0/1364005450 shutdown_connections 2024-03-30T04:16:25.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.730+0000 7f9ba0800700 1 -- 172.21.15.23:0/1364005450 wait complete. 2024-03-30T04:16:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.731+0000 7f9ba0800700 1 Processor -- start 2024-03-30T04:16:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 -- start start 2024-03-30T04:16:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98100f60 0x7f9b980730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98073b30 0x7f9b981a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981a59f0 con 0x7f9b98106080 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981a5b60 con 0x7f9b98100f60 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9ba0800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b981a5d20 con 0x7f9b98073b30 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56292/0 (socket says 172.21.15.23:56292) 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9b9d400700 1 -- 172.21.15.23:0/502068861 learned_addr learned my addr 172.21.15.23:0/502068861 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9b9de00700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98100f60 0x7f9b980730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.732+0000 7f9b9e800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98073b30 0x7f9b981a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 -- 172.21.15.23:0/502068861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98073b30 msgr2=0x7f9b981a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98073b30 0x7f9b981a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 -- 172.21.15.23:0/502068861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98100f60 msgr2=0x7f9b980730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98100f60 0x7f9b980730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 -- 172.21.15.23:0/502068861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0095f0 con 0x7f9b98106080 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b9d400700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e970 tx=0x7f9b8800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b92a00700 1 -- 172.21.15.23:0/502068861 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88009790 con 0x7f9b98106080 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b92a00700 1 -- 172.21.15.23:0/502068861 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b8800c7e0 con 0x7f9b98106080 2024-03-30T04:16:25.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b981a5fa0 con 0x7f9b98106080 2024-03-30T04:16:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9b92a00700 1 -- 172.21.15.23:0/502068861 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880054e0 con 0x7f9b98106080 2024-03-30T04:16:25.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.733+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b981a6480 con 0x7f9b98106080 2024-03-30T04:16:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.737+0000 7f9b92a00700 1 -- 172.21.15.23:0/502068861 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9b8800c950 con 0x7f9b98106080 2024-03-30T04:16:25.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.738+0000 7f9b92a00700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 0x7f9b800a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.738+0000 7f9bb42a1b80 1 Processor -- start 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.738+0000 7f9b92a00700 1 -- 172.21.15.23:0/502068861 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9b88014070 con 0x7f9b98106080 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.738+0000 7f9bb42a1b80 1 -- start start 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.738+0000 7f9b9de00700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 0x7f9b800a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b7e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b8c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1ad7990 con 0x5622c1b7eef0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1ad5350 con 0x5622c1b89df0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1ad0c40 con 0x5622c1b7e5d0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b7e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56308/0 (socket says 172.21.15.23:56308) 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9d400700 1 -- 172.21.15.23:0/2052277138 learned_addr learned my addr 172.21.15.23:0/2052277138 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9de00700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b7e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48454/0 (socket says 172.21.15.23:48454) 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9e800700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b8c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.739+0000 7f9b9de00700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 0x7f9b800a3750 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006010 tx=0x7f9b8c00b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 -- 172.21.15.23:0/2052277138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 msgr2=0x5622c1b7e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b7e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 -- 172.21.15.23:0/2052277138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 msgr2=0x5622c1b8c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b8c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 -- 172.21.15.23:0/2052277138 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622c19523b0 con 0x5622c1b7eef0 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9e800700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b8c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b9d400700 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9b880a1f70 tx=0x7f9b8800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48f208d6f0b9e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b87000700 1 -- 172.21.15.23:0/2052277138 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880cd160 con 0x5622c1b7eef0 2024-03-30T04:16:25.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b87000700 1 -- 172.21.15.23:0/2052277138 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b8809de50 con 0x5622c1b7eef0 2024-03-30T04:16:25.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.740+0000 7f9b87000700 1 -- 172.21.15.23:0/2052277138 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8809a480 con 0x5622c1b7eef0 2024-03-30T04:16:25.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:25 smithi193 ceph-mon[79581]: pgmap v455: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 5.4 MiB/s wr, 180 op/s 2024-03-30T04:16:25.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:25 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/4190731146' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2052277138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 msgr2=0x5622c1b898b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9b880a1f70 tx=0x7f9b8800cda0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2052277138 shutdown_connections 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b8c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b7e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2052277138 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b898b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2052277138 >> 172.21.15.23:0/2052277138 conn(0x5622c1ade6a0 msgr2=0x5622c1b7aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2052277138 shutdown_connections 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2052277138 wait complete. 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.741+0000 7f9bb42a1b80 1 -- start start 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 Processor -- start 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b7e5d0 0x5622c1b89810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 0x5622c1b871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1a6cd80 con 0x5622c1b7eef0 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1a6ca20 con 0x5622c1b7e5d0 2024-03-30T04:16:25.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c19546a0 con 0x5622c1b89df0 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9bb42a1b80 10 client.0 _ll_register_callbacks cb 0x5622c1a76470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.742+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 0x5622c1b871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56314/0 (socket says 172.21.15.23:56314) 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9d400700 1 -- 172.21.15.23:0/3775084778 learned_addr learned my addr 172.21.15.23:0/3775084778 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b7e5d0 0x5622c1b89810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 -- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b7e5d0 msgr2=0x5622c1b89810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b7e5d0 0x5622c1b89810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 -- 172.21.15.23:0/3775084778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 msgr2=0x5622c1b86b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 -- 172.21.15.23:0/3775084778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b94011230 con 0x5622c1b89df0 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9d400700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b9e800700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 0x5622c1b871a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9b94007f00 tx=0x7f9b940116a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d858cc3b7a30ea1b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b7b400700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.743+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9400e040 con 0x5622c1b89df0 2024-03-30T04:16:25.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622c1a91380 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9b94004db0 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b94003a00 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9bb42a1b80 10 client.29344 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f9b981a6480 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5622c1a91380 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9b94016400 con 0x5622c1b89df0 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.744+0000 7f9bb42a1b80 10 client.29344 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9bb42a1b80 10 client.29344 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:16:25.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9bb42a1b80 20 client.29344 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1b99aa0 0x5622c1b9bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9bb42a1b80 4 client.29344 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622c1ac9700 con 0x5622c1b99aa0 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.745+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1b99aa0 0x5622c1b9bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.746+0000 7f9b7b400700 10 client.29344 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.746+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1b99aa0 0x5622c1b9bec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab72a2b1b86698dc server_cookie=ba6e6ef6e6b744d2 in_seq=0 out_seq=0 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.746+0000 7f9b7b400700 1 -- 172.21.15.23:0/3775084778 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5622c1ac9700 con 0x5622c1b99aa0 2024-03-30T04:16:25.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.746+0000 7f9b7b400700 10 client.29344 handle_command_reply: tid=0 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 client.29344 shutdown 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 msgr2=0x5622c1b871a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 0x5622c1b871a0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9b94007f00 tx=0x7f9b940116a0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 shutdown_connections 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1b99aa0 0x5622c1b9bec0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b89df0 0x5622c1b871a0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b7e5d0 0x5622c1b89810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9b7b400700 0 client.29344 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:16:25.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3775084778 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b86b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 >> 172.21.15.23:0/3775084778 conn(0x5622c1ade6a0 msgr2=0x5622c1b80690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 shutdown_connections 2024-03-30T04:16:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3775084778 wait complete. 2024-03-30T04:16:25.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.747+0000 7f9bb42a1b80 20 client.29344 trim_cache size 0 max 16384 2024-03-30T04:16:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 Processor -- start 2024-03-30T04:16:25.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 -- start start 2024-03-30T04:16:25.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b81540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1b868f0 con 0x5622c1b7eef0 2024-03-30T04:16:25.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1b86e80 con 0x5622c1b89df0 2024-03-30T04:16:25.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1b90f50 con 0x5622c1b7e5d0 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56330/0 (socket says 172.21.15.23:56330) 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9b9de00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b81540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.761+0000 7f9b9d400700 1 -- 172.21.15.23:0/2086998050 learned_addr learned my addr 172.21.15.23:0/2086998050 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9e800700 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 -- 172.21.15.23:0/2086998050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 msgr2=0x5622c1b735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 -- 172.21.15.23:0/2086998050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 msgr2=0x5622c1b90a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 -- 172.21.15.23:0/2086998050 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622c18feb60 con 0x5622c1b7e5d0 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9d400700 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b9de00700 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b81540 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00b690 tx=0x7f9b8c023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=170c0a7fd2dae24b server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b87a00700 1 -- 172.21.15.23:0/2086998050 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c035a90 con 0x5622c1b7e5d0 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b87a00700 1 -- 172.21.15.23:0/2086998050 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b8c01e040 con 0x5622c1b7e5d0 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.762+0000 7f9b87a00700 1 -- 172.21.15.23:0/2086998050 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0049e0 con 0x5622c1b7e5d0 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2086998050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 msgr2=0x5622c1b81540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b81540 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00b690 tx=0x7f9b8c023000 comp rx=0 tx=0).stop 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2086998050 shutdown_connections 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1b735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b81540 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/2086998050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b90a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2086998050 >> 172.21.15.23:0/2086998050 conn(0x5622c1ade6a0 msgr2=0x5622c1ba07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2086998050 shutdown_connections 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/2086998050 wait complete. 2024-03-30T04:16:25.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.763+0000 7f9bb42a1b80 1 -- start start 2024-03-30T04:16:25.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.764+0000 7f9bb42a1b80 1 Processor -- start 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.764+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b8e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.764+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.764+0000 7f9bb42a1b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.764+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1b8f2d0 con 0x5622c1b7eef0 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1becdd0 con 0x5622c1b89df0 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9bb42a1b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622c1becf40 con 0x5622c1b7e5d0 2024-03-30T04:16:25.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9bb42a1b80 10 client.0 _ll_register_callbacks cb 0x5622c1a76470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9e800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35664/0 (socket says 172.21.15.23:35664) 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56332/0 (socket says 172.21.15.23:56332) 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9e800700 1 -- 172.21.15.23:0/3080786160 learned_addr learned my addr 172.21.15.23:0/3080786160 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b8e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.771 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 -- 172.21.15.23:0/3080786160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 msgr2=0x5622c1b8e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b8e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 -- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 msgr2=0x5622c1bec890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 -- 172.21.15.23:0/3080786160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622c19523b0 con 0x5622c1b7eef0 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9e800700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b9d400700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f9b88098370 tx=0x7f9b8809a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b0f8a79c674b93 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.765+0000 7f9b7b400700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:25.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880b83e0 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9b880a24a0 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b8809de50 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b880cd160 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 10 client.19410 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f9b8c02c040 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b8809de50 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f9b880a2010 con 0x5622c1b7eef0 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 10 client.19410 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:25.773 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 10 client.19410 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.e 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 20 client.19410 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1bee440 0x5622c1bf0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 4 client.19410 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.766+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 --> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5622c191fcf0 con 0x5622c1bee440 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.767+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1bee440 0x5622c1bf0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.767+0000 7f9b9de00700 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1bee440 0x5622c1bf0860 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69e8c461b9c85d90 server_cookie=7c780f9feab52c2b in_seq=0 out_seq=0 2024-03-30T04:16:25.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.767+0000 7f9b7b400700 10 client.19410 ms_handle_connect on v2:172.21.15.154:6834/1153342637 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9b7b400700 1 -- 172.21.15.23:0/3080786160 <== mds.2 v2:172.21.15.154:6834/1153342637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5622c191fcf0 con 0x5622c1bee440 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9b7b400700 10 client.19410 handle_command_reply: tid=0 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 client.19410 shutdown 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 msgr2=0x5622c1b8ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f9b88098370 tx=0x7f9b8809a1a0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 shutdown_connections 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:6834/1153342637,v1:172.21.15.154:6835/1153342637] conn(0x5622c1bee440 0x5622c1bf0860 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5622c1b89df0 0x5622c1bec890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622c1b7e5d0 0x5622c1b8e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 --2- 172.21.15.23:0/3080786160 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5622c1b7eef0 0x5622c1b8ebc0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9b7b400700 0 client.19410 ms_handle_reset on v2:172.21.15.154:6834/1153342637 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 >> 172.21.15.23:0/3080786160 conn(0x5622c1ade6a0 msgr2=0x5622c1b735a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 shutdown_connections 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 1 -- 172.21.15.23:0/3080786160 wait complete. 2024-03-30T04:16:25.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.768+0000 7f9bb42a1b80 20 client.19410 trim_cache size 0 max 16384 2024-03-30T04:16:25.777 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 msgr2=0x7f9b800a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 0x7f9b800a3750 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c006010 tx=0x7f9b8c00b260 comp rx=0 tx=0).stop 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 msgr2=0x7f9b980735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e970 tx=0x7f9b8800ec80 comp rx=0 tx=0).stop 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 shutdown_connections 2024-03-30T04:16:25.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b98073b30 0x7f9b981a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9b800a1290 0x7f9b800a3750 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9b98100f60 0x7f9b980730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 --2- 172.21.15.23:0/502068861 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b98106080 0x7f9b980735f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 >> 172.21.15.23:0/502068861 conn(0x7f9b980fc7c0 msgr2=0x7f9b980fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 shutdown_connections 2024-03-30T04:16:25.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.769+0000 7f9ba0800700 1 -- 172.21.15.23:0/502068861 wait complete. 2024-03-30T04:16:25.785 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-30T04:16:25.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 -- 172.21.15.23:0/2270393688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073180 msgr2=0x7f6478073560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 --2- 172.21.15.23:0/2270393688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073180 0x7f6478073560 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f64680098f0 tx=0x7f6468009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:25.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 -- 172.21.15.23:0/2270393688 shutdown_connections 2024-03-30T04:16:25.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 --2- 172.21.15.23:0/2270393688 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f647810d5a0 0x7f647810f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 --2- 172.21.15.23:0/2270393688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073180 0x7f6478073560 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 --2- 172.21.15.23:0/2270393688 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6478073aa0 0x7f647810d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 -- 172.21.15.23:0/2270393688 >> 172.21.15.23:0/2270393688 conn(0x7f64780fc7f0 msgr2=0x7f64780fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:25.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 -- 172.21.15.23:0/2270393688 shutdown_connections 2024-03-30T04:16:25.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.988+0000 7f647e400700 1 -- 172.21.15.23:0/2270393688 wait complete. 2024-03-30T04:16:25.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.989+0000 7f647e400700 1 Processor -- start 2024-03-30T04:16:25.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.989+0000 7f647e400700 1 -- start start 2024-03-30T04:16:25.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6478073180 0x7f64781994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.990 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073aa0 0x7f64781999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819e3c0 con 0x7f647810d5a0 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819e530 con 0x7f6478073180 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f647e400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819e6f0 con 0x7f6478073aa0 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56342/0 (socket says 172.21.15.23:56342) 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 -- 172.21.15.23:0/3906597469 learned_addr learned my addr 172.21.15.23:0/3906597469 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:25.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6476a00700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073aa0 0x7f64781999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6478073180 0x7f64781994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 -- 172.21.15.23:0/3906597469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073aa0 msgr2=0x7f64781999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073aa0 0x7f64781999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 -- 172.21.15.23:0/3906597469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6478073180 msgr2=0x7f64781994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6478073180 0x7f64781994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.990+0000 7f6477e00700 1 -- 172.21.15.23:0/3906597469 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64640095f0 con 0x7f647810d5a0 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f6477e00700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f647000ef90 tx=0x7f647000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f646fe00700 1 -- 172.21.15.23:0/3906597469 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470018070 con 0x7f647810d5a0 2024-03-30T04:16:25.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6468009520 con 0x7f647810d5a0 2024-03-30T04:16:25.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f646fe00700 1 -- 172.21.15.23:0/3906597469 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f647000f040 con 0x7f647810d5a0 2024-03-30T04:16:25.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f646fe00700 1 -- 172.21.15.23:0/3906597469 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f647000cd60 con 0x7f647810d5a0 2024-03-30T04:16:25.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.991+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64781a6030 con 0x7f647810d5a0 2024-03-30T04:16:25.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.994+0000 7f646fe00700 1 -- 172.21.15.23:0/3906597469 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f6470003a00 con 0x7f647810d5a0 2024-03-30T04:16:25.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.995+0000 7f646fe00700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 0x7f645c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.995+0000 7f646fe00700 1 -- 172.21.15.23:0/3906597469 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f64700ccc00 con 0x7f647810d5a0 2024-03-30T04:16:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.995+0000 7f6491e88b80 1 Processor -- start 2024-03-30T04:16:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6477400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 0x7f645c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 -- start start 2024-03-30T04:16:25.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:25.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec349990 con 0x5586ec3f05d0 2024-03-30T04:16:25.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec347350 con 0x5586ec3f0ef0 2024-03-30T04:16:25.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6491e88b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec342c40 con 0x5586ec3fbdf0 2024-03-30T04:16:26.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6477e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6477400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6476a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3fb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6477e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48508/0 (socket says 172.21.15.23:48508) 2024-03-30T04:16:26.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.996+0000 7f6477e00700 1 -- 172.21.15.23:0/3471749708 learned_addr learned my addr 172.21.15.23:0/3471749708 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6477400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 0x7f645c0a3750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f64680098c0 tx=0x7f6468019040 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 -- 172.21.15.23:0/3471749708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 msgr2=0x5586ec3fe1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 -- 172.21.15.23:0/3471749708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 msgr2=0x5586ec3f09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3f09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 -- 172.21.15.23:0/3471749708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586ec1c43b0 con 0x5586ec3f0ef0 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6477e00700 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:26.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f6476a00700 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3fb8b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6464005fd0 tx=0x7f6464011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba700beeba818c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.997+0000 7f646cc00700 1 -- 172.21.15.23:0/3471749708 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6464004310 con 0x5586ec3f0ef0 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f646cc00700 1 -- 172.21.15.23:0/3471749708 <== mon.2 v2:172.21.15.154:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6464004470 con 0x5586ec3f0ef0 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f646cc00700 1 -- 172.21.15.23:0/3471749708 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64640199d0 con 0x5586ec3f0ef0 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 -- 172.21.15.23:0/3471749708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 msgr2=0x5586ec3fb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3fb8b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6464005fd0 tx=0x7f6464011720 comp rx=0 tx=0).stop 2024-03-30T04:16:26.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 -- 172.21.15.23:0/3471749708 shutdown_connections 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3f09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 --2- 172.21.15.23:0/3471749708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3fb8b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.998+0000 7f6491e88b80 1 -- 172.21.15.23:0/3471749708 >> 172.21.15.23:0/3471749708 conn(0x5586ec3506a0 msgr2=0x5586ec3eca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.999+0000 7f6491e88b80 1 -- 172.21.15.23:0/3471749708 shutdown_connections 2024-03-30T04:16:26.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.999+0000 7f6491e88b80 1 -- 172.21.15.23:0/3471749708 wait complete. 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.999+0000 7f6491e88b80 1 -- start start 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:25.999+0000 7f6491e88b80 1 Processor -- start 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec2ded80 con 0x5586ec3f05d0 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec2dea20 con 0x5586ec3f0ef0 2024-03-30T04:16:26.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec1c66a0 con 0x5586ec3fbdf0 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6491e88b80 10 client.0 _ll_register_callbacks cb 0x5586ec2e8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56354/0 (socket says 172.21.15.23:56354) 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 -- 172.21.15.23:0/1441887615 learned_addr learned my addr 172.21.15.23:0/1441887615 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6476a00700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477e00700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 -- 172.21.15.23:0/1441887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 msgr2=0x5586ec3f8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3f8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.000+0000 7f6477400700 1 -- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 msgr2=0x5586ec3f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6477400700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6477400700 1 -- 172.21.15.23:0/1441887615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6470084040 con 0x5586ec3f05d0 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6477e00700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6477400700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6468009230 tx=0x7f6468009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd0597b940453b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6461600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646802a050 con 0x5586ec3f05d0 2024-03-30T04:16:26.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f646800fe70 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5586ec303380 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f646800bba0 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.001+0000 7f6491e88b80 10 client.19422 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f6464009b80 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5586ec303380 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f6468037070 con 0x5586ec3f05d0 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 10 client.19422 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:26.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 10 client.19422 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 20 client.19422 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec40b8a0 0x5586ec40dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 4 client.19422 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.002+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5586ec33b700 con 0x5586ec40b8a0 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.003+0000 7f6476a00700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec40b8a0 0x5586ec40dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.003+0000 7f6476a00700 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec40b8a0 0x5586ec40dcc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc87491db98951d7 server_cookie=9c7bebb1a570a383 in_seq=0 out_seq=0 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.003+0000 7f6461600700 10 client.19422 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:16:26.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6461600700 1 -- 172.21.15.23:0/1441887615 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5586ec33b700 con 0x5586ec40b8a0 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6461600700 10 client.19422 handle_command_reply: tid=0 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 client.19422 shutdown 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 msgr2=0x5586ec3fb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6468009230 tx=0x7f6468009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 shutdown_connections 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec40b8a0 0x5586ec40dcc0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec3f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec3fb6f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 --2- 172.21.15.23:0/1441887615 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6461600700 0 client.19422 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 >> 172.21.15.23:0/1441887615 conn(0x5586ec3506a0 msgr2=0x5586ec3f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 shutdown_connections 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.004+0000 7f6491e88b80 1 -- 172.21.15.23:0/1441887615 wait complete. 2024-03-30T04:16:26.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.005+0000 7f6491e88b80 20 client.19422 trim_cache size 0 max 16384 2024-03-30T04:16:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 Processor -- start 2024-03-30T04:16:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 -- start start 2024-03-30T04:16:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3f05d0 0x5586ec40dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3ebf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec403050 con 0x5586ec3fbdf0 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec3f6fb0 con 0x5586ec3f0ef0 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.018+0000 7f6491e88b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec4080b0 con 0x5586ec3f05d0 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3f05d0 0x5586ec40dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56362/0 (socket says 172.21.15.23:56362) 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 -- 172.21.15.23:0/2840323109 learned_addr learned my addr 172.21.15.23:0/2840323109 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6476a00700 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3ebf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 -- 172.21.15.23:0/2840323109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3f05d0 msgr2=0x5586ec40dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3f05d0 0x5586ec40dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 -- 172.21.15.23:0/2840323109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 msgr2=0x5586ec3ebf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3ebf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 -- 172.21.15.23:0/2840323109 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5586ec170b60 con 0x5586ec3fbdf0 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6476a00700 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f6477e00700 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64700c9eb0 tx=0x7f647000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627fdb0d660ea609 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.019+0000 7f646d600700 1 -- 172.21.15.23:0/2840323109 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700992c0 con 0x5586ec3fbdf0 2024-03-30T04:16:26.020 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f646d600700 1 -- 172.21.15.23:0/2840323109 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f647009a060 con 0x5586ec3fbdf0 2024-03-30T04:16:26.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f646d600700 1 -- 172.21.15.23:0/2840323109 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470099e20 con 0x5586ec3fbdf0 2024-03-30T04:16:26.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 -- 172.21.15.23:0/2840323109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 msgr2=0x5586ec402a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f64700c9eb0 tx=0x7f647000cf70 comp rx=0 tx=0).stop 2024-03-30T04:16:26.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 -- 172.21.15.23:0/2840323109 shutdown_connections 2024-03-30T04:16:26.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3fbdf0 0x5586ec402a70 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3f05d0 0x5586ec40dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 --2- 172.21.15.23:0/2840323109 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec3ebf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 -- 172.21.15.23:0/2840323109 >> 172.21.15.23:0/2840323109 conn(0x5586ec3506a0 msgr2=0x5586ec3e55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 -- 172.21.15.23:0/2840323109 shutdown_connections 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.020+0000 7f6491e88b80 1 -- 172.21.15.23:0/2840323109 wait complete. 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 -- start start 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 Processor -- start 2024-03-30T04:16:26.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec400e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec4013c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec401900 con 0x5586ec3f05d0 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec401a40 con 0x5586ec3f0ef0 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5586ec401c00 con 0x5586ec3fbdf0 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.021+0000 7f6491e88b80 10 client.0 _ll_register_callbacks cb 0x5586ec2e8470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec4013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56378/0 (socket says 172.21.15.23:56378) 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 -- 172.21.15.23:0/4062012989 learned_addr learned my addr 172.21.15.23:0/4062012989 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6476a00700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec400e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 -- 172.21.15.23:0/4062012989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 msgr2=0x5586ec4013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec4013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 -- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 msgr2=0x5586ec400e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec400e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 -- 172.21.15.23:0/4062012989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64781a6030 con 0x5586ec3f05d0 2024-03-30T04:16:26.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6477400700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6468003d10 tx=0x7f6468005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30d5da178c969a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.022+0000 7f6461600700 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646802a040 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f646800bba0 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6464004470 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6468023410 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 10 client.19428 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5586ec308ea0 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6464004470 con 0x5586ec3f05d0 2024-03-30T04:16:26.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f6468037030 con 0x5586ec3f05d0 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 10 client.19428 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 10 client.19428 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 20 client.19428 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec460830 0x5586ec462c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 4 client.19428 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.023+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 --> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5586ec191cf0 con 0x5586ec460830 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6476a00700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec460830 0x5586ec462c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6476a00700 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec460830 0x5586ec462c20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e197cb01e2e782d server_cookie=ac64bc17e16164f6 in_seq=0 out_seq=0 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6461600700 10 client.19428 ms_handle_connect on v2:172.21.15.154:6836/3317012042 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6461600700 1 -- 172.21.15.23:0/4062012989 <== mds.1 v2:172.21.15.154:6836/3317012042 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5586ec191cf0 con 0x5586ec460830 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6461600700 10 client.19428 handle_command_reply: tid=0 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.024+0000 7f6491e88b80 1 client.19428 shutdown 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 msgr2=0x5586ec402760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f6468003d10 tx=0x7f6468005dc0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 shutdown_connections 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:6836/3317012042,v1:172.21.15.154:6837/3317012042] conn(0x5586ec460830 0x5586ec462c20 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5586ec3fbdf0 0x5586ec4013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5586ec3f05d0 0x5586ec402760 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 --2- 172.21.15.23:0/4062012989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5586ec3f0ef0 0x5586ec400e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 >> 172.21.15.23:0/4062012989 conn(0x5586ec3506a0 msgr2=0x5586ec3f3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6461600700 0 client.19428 ms_handle_reset on v2:172.21.15.154:6836/3317012042 2024-03-30T04:16:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 shutdown_connections 2024-03-30T04:16:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 1 -- 172.21.15.23:0/4062012989 wait complete. 2024-03-30T04:16:26.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f6491e88b80 20 client.19428 trim_cache size 0 max 16384 2024-03-30T04:16:26.033 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:16:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 msgr2=0x7f645c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 0x7f645c0a3750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f64680098c0 tx=0x7f6468019040 comp rx=0 tx=0).stop 2024-03-30T04:16:26.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 msgr2=0x7f647819dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.025+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f647000ef90 tx=0x7f647000c3a0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 shutdown_connections 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f647810d5a0 0x7f647819dd80 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f645c0a1290 0x7f645c0a3750 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6478073180 0x7f64781994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 --2- 172.21.15.23:0/3906597469 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6478073aa0 0x7f64781999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 >> 172.21.15.23:0/3906597469 conn(0x7f64780fc7f0 msgr2=0x7f64780fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 shutdown_connections 2024-03-30T04:16:26.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.026+0000 7f647e400700 1 -- 172.21.15.23:0/3906597469 wait complete. 2024-03-30T04:16:26.042 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-30T04:16:26.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 -- 172.21.15.23:0/2269093430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 msgr2=0x7fdd34101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 --2- 172.21.15.23:0/2269093430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd34101350 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdd240098f0 tx=0x7fdd24009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 -- 172.21.15.23:0/2269093430 shutdown_connections 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 --2- 172.21.15.23:0/2269093430 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34106090 0x7fdd3410dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 --2- 172.21.15.23:0/2269093430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd34101350 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 --2- 172.21.15.23:0/2269093430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34101920 0x7fdd34105960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 -- 172.21.15.23:0/2269093430 >> 172.21.15.23:0/2269093430 conn(0x7fdd340fc7f0 msgr2=0x7fdd340fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 -- 172.21.15.23:0/2269093430 shutdown_connections 2024-03-30T04:16:26.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.247+0000 7fdd39000700 1 -- 172.21.15.23:0/2269093430 wait complete. 2024-03-30T04:16:26.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.248+0000 7fdd39000700 1 Processor -- start 2024-03-30T04:16:26.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.248+0000 7fdd39000700 1 -- start start 2024-03-30T04:16:26.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34106090 0x7fdd340735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 0x7fdd341a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341a5a60 con 0x7fdd34073b30 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341a5ba0 con 0x7fdd34106090 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd39000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd341a5d60 con 0x7fdd34100f70 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd32800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 0x7fdd341a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48548/0 (socket says 172.21.15.23:48548) 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd31e00700 1 -- 172.21.15.23:0/3623467551 learned_addr learned my addr 172.21.15.23:0/3623467551 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.249+0000 7fdd31400700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34106090 0x7fdd340735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 -- 172.21.15.23:0/3623467551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34106090 msgr2=0x7fdd340735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34106090 0x7fdd340735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 -- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 msgr2=0x7fdd341a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 0x7fdd341a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 -- 172.21.15.23:0/3623467551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c009d00 con 0x7fdd34100f70 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd32800700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 0x7fdd341a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd31e00700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdd24005950 tx=0x7fdd2400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd2aa00700 1 -- 172.21.15.23:0/3623467551 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2401c070 con 0x7fdd34100f70 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd2aa00700 1 -- 172.21.15.23:0/3623467551 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd2400b9c0 con 0x7fdd34100f70 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd24009520 con 0x7fdd34100f70 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd2aa00700 1 -- 172.21.15.23:0/3623467551 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd240038e0 con 0x7fdd34100f70 2024-03-30T04:16:26.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.250+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd341a6210 con 0x7fdd34100f70 2024-03-30T04:16:26.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.255+0000 7fdd2aa00700 1 -- 172.21.15.23:0/3623467551 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fdd24020030 con 0x7fdd34100f70 2024-03-30T04:16:26.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 Processor -- start 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 -- start start 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd2aa00700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 0x7fdd180a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd2aa00700 1 -- 172.21.15.23:0/3623467551 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd240cdde0 con 0x7fdd34100f70 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd898990 con 0x55f1cd94adf0 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd896350 con 0x55f1cd93fef0 2024-03-30T04:16:26.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd891c40 con 0x55f1cd93f5d0 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.256+0000 7fdd32800700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 0x7fdd180a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd32800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:48564/0 (socket says 172.21.15.23:48564) 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 -- 172.21.15.23:0/2802050091 learned_addr learned my addr 172.21.15.23:0/2802050091 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31400700 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 -- 172.21.15.23:0/2802050091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 msgr2=0x55f1cd94a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 -- 172.21.15.23:0/2802050091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 msgr2=0x55f1cd94d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 -- 172.21.15.23:0/2802050091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1cd7133b0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd31e00700 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdd2409a8a0 tx=0x7fdd2409a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42757051ce6d76d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.257+0000 7fdd32800700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 0x7fdd180a3750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009fd0 tx=0x7fdd2c0093d0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd1f400700 1 -- 172.21.15.23:0/2802050091 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2409aba0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd1f400700 1 -- 172.21.15.23:0/2802050091 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd2409e9d0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd1f400700 1 -- 172.21.15.23:0/2802050091 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24005140 con 0x55f1cd93f5d0 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/2802050091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 msgr2=0x55f1cd93f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fdd2409a8a0 tx=0x7fdd2409a580 comp rx=0 tx=0).stop 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/2802050091 shutdown_connections 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd93f9b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.262 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/2802050091 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/2802050091 >> 172.21.15.23:0/2802050091 conn(0x55f1cd89f6a0 msgr2=0x55f1cd93ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/2802050091 shutdown_connections 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.258+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/2802050091 wait complete. 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.259+0000 7fdd4cb27b80 1 -- start start 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.259+0000 7fdd4cb27b80 1 Processor -- start 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93fef0 0x55f1cd947b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd948120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd82dd80 con 0x55f1cd94adf0 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd82da20 con 0x55f1cd93f5d0 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd7156a0 con 0x55f1cd93fef0 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd4cb27b80 10 client.0 _ll_register_callbacks cb 0x55f1cd837470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35740/0 (socket says 172.21.15.23:35740) 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 -- 172.21.15.23:0/1561203478 learned_addr learned my addr 172.21.15.23:0/1561203478 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd32800700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd948120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31400700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93fef0 0x55f1cd947b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 -- 172.21.15.23:0/1561203478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93fef0 msgr2=0x55f1cd947b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93fef0 0x55f1cd947b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 -- 172.21.15.23:0/1561203478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 msgr2=0x55f1cd948120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd948120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.260+0000 7fdd31e00700 1 -- 172.21.15.23:0/1561203478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd341a6210 con 0x55f1cd93f5d0 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdd240a2a60 tx=0x7fdd24005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe91be5c2a98757 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 10 client.0 ms_handle_connect on v2:172.21.15.154:3300/0 2024-03-30T04:16:26.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mon.2 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2409aba0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mon.2 v2:172.21.15.154:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd240a2f90 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1cd852380 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mon.2 v2:172.21.15.154:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd24005140 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 10 client.29396 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f1cd6bfb60 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mon.2 v2:172.21.15.154:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f1cd852380 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mon.2 v2:172.21.15.154:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdd2409ae90 con 0x55f1cd93f5d0 2024-03-30T04:16:26.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 10 client.29396 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 10 client.29396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 20 client.29396 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd95a130 0x55f1cd95c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 4 client.29396 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.261+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1cd88a700 con 0x55f1cd95a130 2024-03-30T04:16:26.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.262+0000 7fdd31400700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd95a130 0x55f1cd95c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.262+0000 7fdd31400700 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd95a130 0x55f1cd95c550 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb110f45cdfd87c server_cookie=508c9ca63ccf9334 in_seq=0 out_seq=0 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.262+0000 7fdd13e00700 10 client.29396 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.263+0000 7fdd13e00700 1 -- 172.21.15.23:0/1561203478 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55f1cd88a700 con 0x55f1cd95a130 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.263+0000 7fdd13e00700 10 client.29396 handle_command_reply: tid=0 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.263+0000 7fdd4cb27b80 1 client.29396 shutdown 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 msgr2=0x55f1cd94a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fdd240a2a60 tx=0x7fdd24005bb0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 shutdown_connections 2024-03-30T04:16:26.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd948120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94a790 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd95a130 0x55f1cd95c550 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1561203478 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93fef0 0x55f1cd947b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 >> 172.21.15.23:0/1561203478 conn(0x55f1cd89f6a0 msgr2=0x55f1cd9416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd13e00700 0 client.29396 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 shutdown_connections 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1561203478 wait complete. 2024-03-30T04:16:26.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.264+0000 7fdd4cb27b80 20 client.29396 trim_cache size 0 max 16384 2024-03-30T04:16:26.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 Processor -- start 2024-03-30T04:16:26.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 -- start start 2024-03-30T04:16:26.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd9517e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd94adf0 0x55f1cd93b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd948f00 con 0x55f1cd93f5d0 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd941140 con 0x55f1cd93fef0 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd940850 con 0x55f1cd94adf0 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.278+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56420/0 (socket says 172.21.15.23:56420) 2024-03-30T04:16:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 -- 172.21.15.23:0/1014824622 learned_addr learned my addr 172.21.15.23:0/1014824622 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd32800700 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd94adf0 0x55f1cd93b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31400700 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd9517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 -- 172.21.15.23:0/1014824622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd94adf0 msgr2=0x55f1cd93b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd94adf0 0x55f1cd93b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 -- 172.21.15.23:0/1014824622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 msgr2=0x55f1cd9517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd9517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 -- 172.21.15.23:0/1014824622 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1cd857ea0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd31e00700 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdd240970a0 tx=0x7fdd24009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e948ecb5c0390e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd1fe00700 1 -- 172.21.15.23:0/1014824622 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24021380 con 0x55f1cd93f5d0 2024-03-30T04:16:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd1fe00700 1 -- 172.21.15.23:0/1014824622 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd240a2f90 con 0x55f1cd93f5d0 2024-03-30T04:16:26.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.279+0000 7fdd1fe00700 1 -- 172.21.15.23:0/1014824622 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2409a320 con 0x55f1cd93f5d0 2024-03-30T04:16:26.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1014824622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 msgr2=0x55f1cd95c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fdd240970a0 tx=0x7fdd24009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:26.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1014824622 shutdown_connections 2024-03-30T04:16:26.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd94adf0 0x55f1cd93b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd93f5d0 0x55f1cd95c550 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/1014824622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd9517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1014824622 >> 172.21.15.23:0/1014824622 conn(0x55f1cd89f6a0 msgr2=0x55f1cd942e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1014824622 shutdown_connections 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/1014824622 wait complete. 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.280+0000 7fdd4cb27b80 1 -- start start 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 Processor -- start 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd9acf20 con 0x55f1cd94adf0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd9ad060 con 0x55f1cd93fef0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cd9ad1f0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd4cb27b80 10 client.0 _ll_register_callbacks cb 0x55f1cd837470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd31e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd32800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd31400700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35758/0 (socket says 172.21.15.23:35758) 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.281+0000 7fdd31400700 1 -- 172.21.15.23:0/4126839908 learned_addr learned my addr 172.21.15.23:0/4126839908 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 -- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 msgr2=0x55f1cd94f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 -- 172.21.15.23:0/4126839908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 msgr2=0x55f1cd94fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 -- 172.21.15.23:0/4126839908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1cd6bfb60 con 0x55f1cd93f5d0 2024-03-30T04:16:26.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31400700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd32800700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd31e00700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94f0a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdd240050f0 tx=0x7fdd240092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e06a20672d58003 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd13e00700 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24021380 con 0x55f1cd93f5d0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fdd2409aba0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd2409e9d0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.282+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd2409ae00 con 0x55f1cd93f5d0 2024-03-30T04:16:26.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 10 client.29386 fetch_fsmap learned FSMap version 31 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f1cd717cd0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdd2409e9d0 con 0x55f1cd93f5d0 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdd240a2b00 con 0x55f1cd93f5d0 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 10 client.29386 fetch_fsmap finished waiting for FSMap version 31 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 10 client.29386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 20 client.29386 populate_metadata read hostname 'smithi023' 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd9ae4e0 0x55f1cd9b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:26.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 4 client.29386 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 --> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f1cd6e0cf0 con 0x55f1cd9ae4e0 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.283+0000 7fdd31400700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd9ae4e0 0x55f1cd9b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd13e00700 10 client.29386 ms_handle_connect on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd31400700 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd9ae4e0 0x55f1cd9b08d0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bc7deb5cec50ef server_cookie=f966b5ac26ab9b53 in_seq=0 out_seq=0 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd13e00700 1 -- 172.21.15.23:0/4126839908 <== mds.0 v2:172.21.15.154:6838/2911311868 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f1cd6e0cf0 con 0x55f1cd9ae4e0 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd13e00700 10 client.29386 handle_command_reply: tid=0 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd4cb27b80 1 client.29386 shutdown 2024-03-30T04:16:26.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 msgr2=0x55f1cd94f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.284+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94f0a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fdd240050f0 tx=0x7fdd240092d0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 shutdown_connections 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f1cd94adf0 0x55f1cd94fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:6838/2911311868,v1:172.21.15.154:6839/2911311868] conn(0x55f1cd9ae4e0 0x55f1cd9b08d0 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cd93f5d0 0x55f1cd94f0a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 --2- 172.21.15.23:0/4126839908 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x55f1cd93fef0 0x55f1cd94f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd13e00700 0 client.29386 ms_handle_reset on v2:172.21.15.154:6838/2911311868 2024-03-30T04:16:26.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 >> 172.21.15.23:0/4126839908 conn(0x55f1cd89f6a0 msgr2=0x55f1cd93b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 shutdown_connections 2024-03-30T04:16:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 1 -- 172.21.15.23:0/4126839908 wait complete. 2024-03-30T04:16:26.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd4cb27b80 20 client.29386 trim_cache size 0 max 16384 2024-03-30T04:16:26.294 INFO:teuthology.orchestra.run.smithi023.stdout:[] 2024-03-30T04:16:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 msgr2=0x7fdd180a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 0x7fdd180a3750 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009fd0 tx=0x7fdd2c0093d0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 msgr2=0x7fdd340730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:26.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.285+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdd24005950 tx=0x7fdd2400fa00 comp rx=0 tx=0).stop 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 shutdown_connections 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd34073b30 0x7fdd341a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd34100f70 0x7fdd340730b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fdd180a1290 0x7fdd180a3750 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 --2- 172.21.15.23:0/3623467551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd34106090 0x7fdd340735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 >> 172.21.15.23:0/3623467551 conn(0x7fdd340fc7f0 msgr2=0x7fdd340fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 shutdown_connections 2024-03-30T04:16:26.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:26.286+0000 7fdd39000700 1 -- 172.21.15.23:0/3623467551 wait complete. 2024-03-30T04:16:27.303 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-30T04:16:27.303 INFO:tasks.fwd_scrub:done joining 2024-03-30T04:16:27.303 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-30T04:16:27.315 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:16:27.524 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.525+0000 7fe080c00700 1 -- 172.21.15.23:0/2808594832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c101810 msgr2=0x7fe07c105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.525+0000 7fe080c00700 1 --2- 172.21.15.23:0/2808594832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c101810 0x7fe07c105850 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe068009990 tx=0x7fe068009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 -- 172.21.15.23:0/2808594832 shutdown_connections 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 --2- 172.21.15.23:0/2808594832 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c105f80 0x7fe07c10db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 --2- 172.21.15.23:0/2808594832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c100e60 0x7fe07c101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 --2- 172.21.15.23:0/2808594832 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c101810 0x7fe07c105850 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 -- 172.21.15.23:0/2808594832 >> 172.21.15.23:0/2808594832 conn(0x7fe07c0fc700 msgr2=0x7fe07c0feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 -- 172.21.15.23:0/2808594832 shutdown_connections 2024-03-30T04:16:27.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.526+0000 7fe080c00700 1 -- 172.21.15.23:0/2808594832 wait complete. 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 Processor -- start 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 -- start start 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c100e60 0x7fe07c199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 0x7fe07c1998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c19e6d0 con 0x7fe07c105f80 2024-03-30T04:16:27.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c19e840 con 0x7fe07c19dc50 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.527+0000 7fe080c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c19ea00 con 0x7fe07c100e60 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 0x7fe07c1998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c100e60 0x7fe07c199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe07a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe07a800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:35770/0 (socket says 172.21.15.23:35770) 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe07a800700 1 -- 172.21.15.23:0/3515445065 learned_addr learned my addr 172.21.15.23:0/3515445065 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 -- 172.21.15.23:0/3515445065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c100e60 msgr2=0x7fe07c199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:27.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c100e60 0x7fe07c199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 -- 172.21.15.23:0/3515445065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 msgr2=0x7fe07c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 -- 172.21.15.23:0/3515445065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680095f0 con 0x7fe07c105f80 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe07a800700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.528+0000 7fe079400700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 0x7fe07c1998b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe0680098e0 tx=0x7fe068005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.529+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06801d070 con 0x7fe07c105f80 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.529+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe068022470 con 0x7fe07c105f80 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.529+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068017650 con 0x7fe07c105f80 2024-03-30T04:16:27.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.529+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c1a5f00 con 0x7fe07c105f80 2024-03-30T04:16:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.529+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c1a62c0 con 0x7fe07c105f80 2024-03-30T04:16:27.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.530+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c102330 con 0x7fe07c105f80 2024-03-30T04:16:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.532+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe068022850 con 0x7fe07c105f80 2024-03-30T04:16:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.533+0000 7fe072a00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 0x7fe0600a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.533+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe0680ce270 con 0x7fe07c105f80 2024-03-30T04:16:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.533+0000 7fe079e00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 0x7fe0600a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:27.532 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.534+0000 7fe079e00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 0x7fe0600a37d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009c60 tx=0x7fe06c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:27.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.537+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe0680975a0 con 0x7fe07c105f80 2024-03-30T04:16:27.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.790+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe07c101810 con 0x7fe07c105f80 2024-03-30T04:16:27.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.791+0000 7fe072a00700 1 -- 172.21.15.23:0/3515445065 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+25230 (secure 0 0 0) 0x7fe06809abc0 con 0x7fe07c105f80 2024-03-30T04:16:27.790 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:27.791 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:16:16.205603+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1767310006":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3829741445":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3110970888":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/354023371":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/4064094531":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6801/2266065890":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2266065890":"2024-03-31T04:01:56.045849+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]},{"pool":4,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 msgr2=0x7fe0600a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 0x7fe0600a37d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009c60 tx=0x7fe06c009260 comp rx=0 tx=0).stop 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 msgr2=0x7fe07c1998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 0x7fe07c1998b0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe0680098e0 tx=0x7fe068005740 comp rx=0 tx=0).stop 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 shutdown_connections 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe0600a1310 0x7fe0600a37d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe07c19dc50 0x7fe07c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe07c100e60 0x7fe07c199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 --2- 172.21.15.23:0/3515445065 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe07c105f80 0x7fe07c1998b0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:27.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 >> 172.21.15.23:0/3515445065 conn(0x7fe07c0fc700 msgr2=0x7fe07c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:27.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 shutdown_connections 2024-03-30T04:16:27.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:27.794+0000 7fe080c00700 1 -- 172.21.15.23:0/3515445065 wait complete. 2024-03-30T04:16:27.812 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:16:27.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:16:28.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.018+0000 7f285ca00700 1 -- 172.21.15.23:0/404171203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858100f60 msgr2=0x7f2858101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.018+0000 7f285ca00700 1 --2- 172.21.15.23:0/404171203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858100f60 0x7f2858101340 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2848009990 tx=0x7f2848009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 -- 172.21.15.23:0/404171203 shutdown_connections 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 --2- 172.21.15.23:0/404171203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2858106080 0x7f285810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 --2- 172.21.15.23:0/404171203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858100f60 0x7f2858101340 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 --2- 172.21.15.23:0/404171203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858101910 0x7f2858105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 -- 172.21.15.23:0/404171203 >> 172.21.15.23:0/404171203 conn(0x7f28580fc7c0 msgr2=0x7f28580febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:28.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 -- 172.21.15.23:0/404171203 shutdown_connections 2024-03-30T04:16:28.019 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.019+0000 7f285ca00700 1 -- 172.21.15.23:0/404171203 wait complete. 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 Processor -- start 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 -- start start 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858100f60 0x7f2858199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285819dd30 0x7f285819e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.021 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285819e7b0 con 0x7f2858106080 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285819e920 con 0x7f285819dd30 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f285ca00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285819eae0 con 0x7f2858100f60 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f2855400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f2855400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42466/0 (socket says 172.21.15.23:42466) 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.020+0000 7f2855400700 1 -- 172.21.15.23:0/2715109281 learned_addr learned my addr 172.21.15.23:0/2715109281 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855e00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858100f60 0x7f2858199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.022 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2856800700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285819dd30 0x7f285819e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 -- 172.21.15.23:0/2715109281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858100f60 msgr2=0x7f2858199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858100f60 0x7f2858199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 -- 172.21.15.23:0/2715109281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285819dd30 msgr2=0x7f285819e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285819dd30 0x7f285819e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 -- 172.21.15.23:0/2715109281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28480095f0 con 0x7f2858106080 2024-03-30T04:16:28.023 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f2855400700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f284400e970 tx=0x7f284400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844009790 con 0x7f2858106080 2024-03-30T04:16:28.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f284400c7e0 con 0x7f2858106080 2024-03-30T04:16:28.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28581a5fe0 con 0x7f2858106080 2024-03-30T04:16:28.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.021+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28440054e0 con 0x7f2858106080 2024-03-30T04:16:28.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.022+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28581a64c0 con 0x7f2858106080 2024-03-30T04:16:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.023+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2858102430 con 0x7f2858106080 2024-03-30T04:16:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.025+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f284400c950 con 0x7f2858106080 2024-03-30T04:16:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.026+0000 7f284ea00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 0x7f283c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.026+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2844014070 con 0x7f2858106080 2024-03-30T04:16:28.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.026+0000 7f2855e00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 0x7f283c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.027+0000 7f2855e00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 0x7f283c0a3750 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f284800b2e0 tx=0x7f2848005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.030+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2844096360 con 0x7f2858106080 2024-03-30T04:16:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.244+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2858101910 con 0x7f2858106080 2024-03-30T04:16:28.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.244+0000 7f284ea00700 1 -- 172.21.15.23:0/2715109281 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2844099980 con 0x7f2858106080 2024-03-30T04:16:28.243 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:16:28.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 msgr2=0x7f283c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 0x7f283c0a3750 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f284800b2e0 tx=0x7f2848005fd0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 msgr2=0x7f2858199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f284400e970 tx=0x7f284400ec80 comp rx=0 tx=0).stop 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 shutdown_connections 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f283c0a1290 0x7f283c0a3750 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f285819dd30 0x7f285819e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2858100f60 0x7f2858199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 --2- 172.21.15.23:0/2715109281 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2858106080 0x7f2858199990 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 >> 172.21.15.23:0/2715109281 conn(0x7f28580fc7c0 msgr2=0x7f28580fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 shutdown_connections 2024-03-30T04:16:28.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.247+0000 7f285ca00700 1 -- 172.21.15.23:0/2715109281 wait complete. 2024-03-30T04:16:28.265 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:16:28.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.468+0000 7f3ad1200700 1 -- 172.21.15.23:0/2307285866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 msgr2=0x7f3acc10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.468+0000 7f3ad1200700 1 --2- 172.21.15.23:0/2307285866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc10dc00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4009f90 tx=0x7f3ac400b330 comp rx=0 tx=0).stop 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 -- 172.21.15.23:0/2307285866 shutdown_connections 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 --2- 172.21.15.23:0/2307285866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc10dc00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 --2- 172.21.15.23:0/2307285866 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 --2- 172.21.15.23:0/2307285866 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc101910 0x7f3acc105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 -- 172.21.15.23:0/2307285866 >> 172.21.15.23:0/2307285866 conn(0x7f3acc0fc7c0 msgr2=0x7f3acc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 -- 172.21.15.23:0/2307285866 shutdown_connections 2024-03-30T04:16:28.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.469+0000 7f3ad1200700 1 -- 172.21.15.23:0/2307285866 wait complete. 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 Processor -- start 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 -- start start 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 0x7f3acc19e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc19e840 con 0x7f3acc19ddc0 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc19e9b0 con 0x7f3acc100f60 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.470+0000 7f3ad1200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc19eb70 con 0x7f3acc106080 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 0x7f3acc19e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3ac9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3ac9e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc1994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54156/0 (socket says 172.21.15.23:54156) 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3ac9e00700 1 -- 172.21.15.23:0/1329127113 learned_addr learned my addr 172.21.15.23:0/1329127113 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3ac9400700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 -- 172.21.15.23:0/1329127113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 msgr2=0x7f3acc199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 -- 172.21.15.23:0/1329127113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 msgr2=0x7f3acc1994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc1994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 -- 172.21.15.23:0/1329127113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8009520 con 0x7f3acc19ddc0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3aca800700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 0x7f3acc19e240 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400b6c0 tx=0x7f3ac4008a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.471+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400e040 con 0x7f3acc19ddc0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.472+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac4009c40 con 0x7f3acc19ddc0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.472+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ac4008ec0 con 0x7f3acc19ddc0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.472+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4005040 con 0x7f3acc19ddc0 2024-03-30T04:16:28.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.472+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3acc1a63b0 con 0x7f3acc19ddc0 2024-03-30T04:16:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.473+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3acc102430 con 0x7f3acc19ddc0 2024-03-30T04:16:28.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.475+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f3ac401d470 con 0x7f3acc19ddc0 2024-03-30T04:16:28.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.476+0000 7f3ac2a00700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 0x7f3ab00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.476+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3ac40ce420 con 0x7f3acc19ddc0 2024-03-30T04:16:28.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.476+0000 7f3ac9e00700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 0x7f3ab00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.477+0000 7f3ac9e00700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 0x7f3ab00a3750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005950 tx=0x7f3abc009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.478 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.480+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f3ac4097520 con 0x7f3acc19ddc0 2024-03-30T04:16:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:28 smithi193 ceph-mon[79581]: pgmap v456: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 5.4 MiB/s wr, 179 op/s 2024-03-30T04:16:28.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:28 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3515445065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:28 smithi023 ceph-mon[38254]: pgmap v456: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 5.4 MiB/s wr, 179 op/s 2024-03-30T04:16:28.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:28 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3515445065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:28.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:28 smithi154 ceph-mon[79161]: pgmap v456: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 5.4 MiB/s wr, 179 op/s 2024-03-30T04:16:28.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:28 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3515445065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:28.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.701+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3acc19a7e0 con 0x7f3acc19ddc0 2024-03-30T04:16:28.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.702+0000 7f3ac2a00700 1 -- 172.21.15.23:0/1329127113 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3ac400bb90 con 0x7f3acc19ddc0 2024-03-30T04:16:28.700 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:16:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 msgr2=0x7f3ab00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 0x7f3ab00a3750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3abc005950 tx=0x7f3abc009310 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 msgr2=0x7f3acc19e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 0x7f3acc19e240 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400b6c0 tx=0x7f3ac4008a10 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 shutdown_connections 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3acc19ddc0 0x7f3acc19e240 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f3ab00a1290 0x7f3ab00a3750 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3acc100f60 0x7f3acc1994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 --2- 172.21.15.23:0/1329127113 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3acc106080 0x7f3acc199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 >> 172.21.15.23:0/1329127113 conn(0x7f3acc0fc7c0 msgr2=0x7f3acc10d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 shutdown_connections 2024-03-30T04:16:28.705 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.704+0000 7f3ad1200700 1 -- 172.21.15.23:0/1329127113 wait complete. 2024-03-30T04:16:28.721 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 -- 172.21.15.23:0/2758527707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 msgr2=0x7ff50010dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 --2- 172.21.15.23:0/2758527707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff50010dc00 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800b0a0 tx=0x7ff4f800b3b0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 -- 172.21.15.23:0/2758527707 shutdown_connections 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 --2- 172.21.15.23:0/2758527707 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff50010dc00 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 --2- 172.21.15.23:0/2758527707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 0x7ff500101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 --2- 172.21.15.23:0/2758527707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff500101910 0x7ff500105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 -- 172.21.15.23:0/2758527707 >> 172.21.15.23:0/2758527707 conn(0x7ff5000fc7c0 msgr2=0x7ff5000febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:28.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 -- 172.21.15.23:0/2758527707 shutdown_connections 2024-03-30T04:16:28.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.926+0000 7ff506a00700 1 -- 172.21.15.23:0/2758527707 wait complete. 2024-03-30T04:16:28.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.927+0000 7ff506a00700 1 Processor -- start 2024-03-30T04:16:28.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.927+0000 7ff506a00700 1 -- start start 2024-03-30T04:16:28.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.927+0000 7ff506a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 0x7ff500199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff506a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff506a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 0x7ff50019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff506a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50019e8b0 con 0x7ff500106080 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff506a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50019ea20 con 0x7ff500100f60 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff506a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50019ebe0 con 0x7ff50019de30 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42498/0 (socket says 172.21.15.23:42498) 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 -- 172.21.15.23:0/3360479592 learned_addr learned my addr 172.21.15.23:0/3360479592 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:28.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ffe00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 0x7ff500199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff504a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 0x7ff50019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 -- 172.21.15.23:0/3360479592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 msgr2=0x7ff50019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 0x7ff50019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 -- 172.21.15.23:0/3360479592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 msgr2=0x7ff500199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 0x7ff500199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 -- 172.21.15.23:0/3360479592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4f8009d00 con 0x7ff500106080 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff504a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 0x7ff50019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:28.930 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.928+0000 7ff4ff400700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00e970 tx=0x7ff4ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.929+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec009790 con 0x7ff500106080 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.929+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4ec00eec0 con 0x7ff500106080 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.929+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec005590 con 0x7ff500106080 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.929+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5001a6140 con 0x7ff500106080 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.929+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5001a6620 con 0x7ff500106080 2024-03-30T04:16:28.931 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.930+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff500102430 con 0x7ff500106080 2024-03-30T04:16:28.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.932+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ff4ec00cbd0 con 0x7ff500106080 2024-03-30T04:16:28.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.933+0000 7ff4fcc00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 0x7ff4e40a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:28.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.933+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7ff4ec014070 con 0x7ff500106080 2024-03-30T04:16:28.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.933+0000 7ff4ffe00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 0x7ff4e40a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:28.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.934+0000 7ff4ffe00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 0x7ff4e40a37d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009c60 tx=0x7ff4f0009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:28.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:28.937+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ff4ec096280 con 0x7ff500106080 2024-03-30T04:16:29.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.158+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff500101910 con 0x7ff500106080 2024-03-30T04:16:29.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.159+0000 7ff4fcc00700 1 -- 172.21.15.23:0/3360479592 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff4ec004ef0 con 0x7ff500106080 2024-03-30T04:16:29.158 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:16:29.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 msgr2=0x7ff4e40a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 0x7ff4e40a37d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0009c60 tx=0x7ff4f0009260 comp rx=0 tx=0).stop 2024-03-30T04:16:29.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 msgr2=0x7ff500199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00e970 tx=0x7ff4ec00ec80 comp rx=0 tx=0).stop 2024-03-30T04:16:29.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 shutdown_connections 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff50019de30 0x7ff50019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ff4e40a1310 0x7ff4e40a37d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff500100f60 0x7ff500199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 --2- 172.21.15.23:0/3360479592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff500106080 0x7ff500199b80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.161+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 >> 172.21.15.23:0/3360479592 conn(0x7ff5000fc7c0 msgr2=0x7ff5000fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.162+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 shutdown_connections 2024-03-30T04:16:29.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.162+0000 7ff506a00700 1 -- 172.21.15.23:0/3360479592 wait complete. 2024-03-30T04:16:29.179 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 -- 172.21.15.23:0/2179715094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 msgr2=0x7f6a5410d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/2179715094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a5410d060 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6a400098f0 tx=0x7f6a40009c00 comp rx=0 tx=0).stop 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 -- 172.21.15.23:0/2179715094 shutdown_connections 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/2179715094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a5410d5a0 0x7f6a5410f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/2179715094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a54073180 0x7f6a54073560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/2179715094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a5410d060 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 -- 172.21.15.23:0/2179715094 >> 172.21.15.23:0/2179715094 conn(0x7f6a540fc7f0 msgr2=0x7f6a540fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:29.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 -- 172.21.15.23:0/2179715094 shutdown_connections 2024-03-30T04:16:29.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.387+0000 7f6a5bc00700 1 -- 172.21.15.23:0/2179715094 wait complete. 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.388+0000 7f6a5bc00700 1 Processor -- start 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.388+0000 7f6a5bc00700 1 -- start start 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 0x7f6a54199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a54199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5419e340 con 0x7f6a5410d5a0 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5419e4b0 con 0x7f6a54073180 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a5bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5419e670 con 0x7f6a54073aa0 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 0x7f6a54199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 0x7f6a54199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:54194/0 (socket says 172.21.15.23:54194) 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42506/0 (socket says 172.21.15.23:42506) 2024-03-30T04:16:29.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59200700 1 -- 172.21.15.23:0/1349545663 learned_addr learned my addr 172.21.15.23:0/1349545663 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a53e00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a54199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 -- 172.21.15.23:0/1349545663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 msgr2=0x7f6a54199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a54199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 -- 172.21.15.23:0/1349545663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 msgr2=0x7f6a54199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 0x7f6a54199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.389+0000 7f6a59c00700 1 -- 172.21.15.23:0/1349545663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a480095f0 con 0x7f6a5410d5a0 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a59c00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00ef90 tx=0x7f6a4c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c018070 con 0x7f6a5410d5a0 2024-03-30T04:16:29.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a40009520 con 0x7f6a5410d5a0 2024-03-30T04:16:29.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a4c00f040 con 0x7f6a5410d5a0 2024-03-30T04:16:29.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4c00cc30 con 0x7f6a5410d5a0 2024-03-30T04:16:29.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.390+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a541a5fb0 con 0x7f6a5410d5a0 2024-03-30T04:16:29.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.391+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a541087b0 con 0x7f6a5410d5a0 2024-03-30T04:16:29.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.393+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f6a4c003a00 con 0x7f6a5410d5a0 2024-03-30T04:16:29.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.394+0000 7f6a51600700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 0x7f6a3c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.394+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f6a4c0ccb70 con 0x7f6a5410d5a0 2024-03-30T04:16:29.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.395+0000 7f6a59200700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 0x7f6a3c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.395+0000 7f6a59200700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 0x7f6a3c0a3750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a48005d10 tx=0x7f6a48009310 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:29.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.398+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f6a4c095c70 con 0x7f6a5410d5a0 2024-03-30T04:16:29.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2715109281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1329127113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.493 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:29 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3360479592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2715109281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1329127113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:29 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3360479592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.614+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6a54067af0 con 0x7f6a5410d5a0 2024-03-30T04:16:29.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.614+0000 7f6a51600700 1 -- 172.21.15.23:0/1349545663 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6a4c099290 con 0x7f6a5410d5a0 2024-03-30T04:16:29.613 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:16:29.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 msgr2=0x7f6a3c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 0x7f6a3c0a3750 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6a48005d10 tx=0x7f6a48009310 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 msgr2=0x7f6a5419dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00ef90 tx=0x7f6a4c00c3a0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 shutdown_connections 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a5410d5a0 0x7f6a5419dd00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f6a3c0a1290 0x7f6a3c0a3750 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a54073180 0x7f6a54199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 --2- 172.21.15.23:0/1349545663 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6a54073aa0 0x7f6a54199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.616 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 >> 172.21.15.23:0/1349545663 conn(0x7f6a540fc7f0 msgr2=0x7f6a540fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:29.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 shutdown_connections 2024-03-30T04:16:29.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.617+0000 7f6a5bc00700 1 -- 172.21.15.23:0/1349545663 wait complete. 2024-03-30T04:16:29.635 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-30T04:16:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2715109281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1329127113' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:29 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3360479592' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:29.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 -- 172.21.15.23:0/421284305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 msgr2=0x7f09440735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 --2- 172.21.15.23:0/421284305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f09440735c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0938009990 tx=0x7f0938009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 -- 172.21.15.23:0/421284305 shutdown_connections 2024-03-30T04:16:29.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 --2- 172.21.15.23:0/421284305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f094410d430 0x7f094410f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 --2- 172.21.15.23:0/421284305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f09440735c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 --2- 172.21.15.23:0/421284305 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0944073b00 0x7f094410cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 -- 172.21.15.23:0/421284305 >> 172.21.15.23:0/421284305 conn(0x7f09440fc700 msgr2=0x7f09440feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 -- 172.21.15.23:0/421284305 shutdown_connections 2024-03-30T04:16:29.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.837+0000 7f094bc00700 1 -- 172.21.15.23:0/421284305 wait complete. 2024-03-30T04:16:29.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.838+0000 7f094bc00700 1 Processor -- start 2024-03-30T04:16:29.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.838+0000 7f094bc00700 1 -- start start 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0944073b00 0x7f0944199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 0x7f094419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419e260 con 0x7f09440731e0 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419e3d0 con 0x7f094410d430 2024-03-30T04:16:29.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f094bc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419e590 con 0x7f0944073b00 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42526/0 (socket says 172.21.15.23:42526) 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 -- 172.21.15.23:0/2608736837 learned_addr learned my addr 172.21.15.23:0/2608736837 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0943e00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0944073b00 0x7f0944199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949c00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 0x7f094419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 -- 172.21.15.23:0/2608736837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0944073b00 msgr2=0x7f0944199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0944073b00 0x7f0944199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 -- 172.21.15.23:0/2608736837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 msgr2=0x7f094419dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 0x7f094419dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949200700 1 -- 172.21.15.23:0/2608736837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09380095f0 con 0x7f09440731e0 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.839+0000 7f0949c00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 0x7f094419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f0949200700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0938000c00 tx=0x7f0938005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093801d070 con 0x7f09440731e0 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0938022470 con 0x7f09440731e0 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938017650 con 0x7f09440731e0 2024-03-30T04:16:29.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094419e810 con 0x7f09440731e0 2024-03-30T04:16:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.840+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09441a60b0 con 0x7f09440731e0 2024-03-30T04:16:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.841+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944108640 con 0x7f09440731e0 2024-03-30T04:16:29.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.843+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f09380225e0 con 0x7f09440731e0 2024-03-30T04:16:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.845+0000 7f0941600700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 0x7f092c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.845+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f09380ce320 con 0x7f09440731e0 2024-03-30T04:16:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.845+0000 7f0943e00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 0x7f092c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:29.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.846+0000 7f0943e00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 0x7f092c0a37d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f094419a890 tx=0x7f0930009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:29.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:29.849+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f093805d9f0 con 0x7f09440731e0 2024-03-30T04:16:30.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.126+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f094419a640 con 0x7f09440731e0 2024-03-30T04:16:30.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.137+0000 7f0941600700 1 -- 172.21.15.23:0/2608736837 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+733921 (secure 0 0 0) 0x7f09380973c0 con 0x7f09440731e0 2024-03-30T04:16:30.146 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 27, 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 15627, 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 250950136 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 250950136 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.148 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/250950136", 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2024-03-30T04:01:56.046075+0000", 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540138322906710015, 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2024-03-30T04:16:30.149 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 25627, 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540138322906710015, 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.150 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.153 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.154 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.155 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.156 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.157 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.160 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-30T04:16:30.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.168 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-30T04:16:30.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T04:16:30.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.177 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-30T04:16:30.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-30T04:16:30.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:30.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.188 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.192 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-30T04:16:30.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2024-03-30T04:16:30.198 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.202 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.206 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.211 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-30T04:16:30.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.219 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T04:16:30.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.224 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T04:16:30.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.229 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-30T04:16:30.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T04:16:30.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.234 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-30T04:16:30.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T04:16:30.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.242 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-30T04:16:30.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T04:16:30.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-30T04:16:30.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.255 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-30T04:16:30.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T04:16:30.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.263 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-30T04:16:30.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.270 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T04:16:30.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-30T04:16:30.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-30T04:16:30.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.283 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-30T04:16:30.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T04:16:30.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-30T04:16:30.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-30T04:16:30.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.295 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-30T04:16:30.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.300 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.301 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.305 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.306 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-30T04:16:30.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.315 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T04:16:30.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.324 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T04:16:30.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.327 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T04:16:30.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.332 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T04:16:30.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T04:16:30.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.342 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.344 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T04:16:30.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T04:16:30.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.357 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.362 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.367 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.369 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.371 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T04:16:30.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.373 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.374 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.375 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.376 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T04:16:30.377 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.378 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T04:16:30.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.385 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.386 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-30T04:16:30.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-30T04:16:30.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.395 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.399 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.401 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.402 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-30T04:16:30.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:30.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.416 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.417 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.418 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.422 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.425 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.429 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.432 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-30T04:16:30.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T04:16:30.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-30T04:16:30.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-30T04:16:30.444 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.446 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.447 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.450 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.454 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.455 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.456 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.458 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.465 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.466 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.470 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.471 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-30T04:16:30.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-30T04:16:30.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-30T04:16:30.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:30.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.483 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.485 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.486 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.489 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.490 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.491 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.492 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.493 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.494 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.495 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.496 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.497 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.498 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.499 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.501 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-30T04:16:30.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.503 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.506 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.510 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.513 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-30T04:16:30.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-30T04:16:30.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.516 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.517 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.518 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.521 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.522 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-30T04:16:30.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-30T04:16:30.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-30T04:16:30.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-30T04:16:30.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.533 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.535 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.537 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.539 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.542 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.548 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.551 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.553 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.557 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.560 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.563 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.566 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.567 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.573 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.574 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.579 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.582 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.583 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.584 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.591 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.592 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.596 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.597 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.601 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-30T04:16:30.603 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.606 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.607 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.608 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.609 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.612 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.613 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.614 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.615 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.616 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.619 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.622 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.623 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.625 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T04:16:30.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:30.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T04:16:30.638 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:30.643 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.645 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.646 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.650 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.651 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-30T04:16:30.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.654 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.655 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.656 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.659 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.660 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.663 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.666 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:30.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.675 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.676 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.680 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.682 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-30T04:16:30.683 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.685 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.686 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.687 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.689 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.690 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.693 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.694 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.695 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.696 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.697 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.698 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.699 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.700 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.701 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.702 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.703 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-30T04:16:30.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.707 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-30T04:16:30.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.715 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-30T04:16:30.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-30T04:16:30.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-30T04:16:30.722 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.724 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.727 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.731 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.732 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.734 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.736 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.740 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.743 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.745 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.749 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.751 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.753 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.754 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.756 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.757 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.762 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.764 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.765 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-30T04:16:30.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T04:16:30.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-30T04:16:30.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T04:16:30.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-30T04:16:30.777 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.780 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.781 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-30T04:16:30.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.786 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.787 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.789 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.790 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.793 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.797 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.801 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.806 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.807 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.810 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.812 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.814 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.821 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-30T04:16:30.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-30T04:16:30.824 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.829 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.830 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.831 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.834 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-30T04:16:30.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.842 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.843 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 25634, 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2024-03-30T04:16:30.844 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540138322906710015, 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:30.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T04:16:30.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.848 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.850 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.852 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-30T04:16:30.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.858 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.859 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-30T04:16:30.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-30T04:16:30.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.865 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.866 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.871 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-30T04:16:30.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T04:16:30.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-30T04:16:30.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:30.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.883 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.885 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.886 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.887 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.888 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.889 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.890 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.891 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.892 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.893 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:30.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T04:16:30.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:30.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-30T04:16:30.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T04:16:30.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.904 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.905 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.906 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-30T04:16:30.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T04:16:30.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-30T04:16:30.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.913 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.914 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.916 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-30T04:16:30.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.922 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.924 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.926 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-30T04:16:30.927 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T04:16:30.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-30T04:16:30.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.933 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T04:16:30.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-30T04:16:30.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-30T04:16:30.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-30T04:16:30.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.944 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.948 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:30.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-30T04:16:30.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:30.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.959 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:30.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:30.965 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.966 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.970 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-30T04:16:30.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-30T04:16:30.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:30.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-30T04:16:30.977 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-30T04:16:30.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T04:16:30.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-30T04:16:30.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T04:16:30.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-30T04:16:30.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.991 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.996 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:30.998 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:30.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.002 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.003 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.007 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-30T04:16:31.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T04:16:31.011 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.012 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.013 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T04:16:31.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.018 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-30T04:16:31.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T04:16:31.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T04:16:31.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T04:16:31.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.039 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T04:16:31.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.043 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.045 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T04:16:31.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T04:16:31.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T04:16:31.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T04:16:31.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T04:16:31.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T04:16:31.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T04:16:31.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T04:16:31.075 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.078 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.083 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-30T04:16:31.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.093 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.100 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:31.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:31.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.114 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:31.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.118 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-30T04:16:31.119 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.121 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.127 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-30T04:16:31.130 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.131 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-30T04:16:31.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.137 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.141 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.143 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.144 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.147 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-30T04:16:31.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-30T04:16:31.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.153 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.157 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-30T04:16:31.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.162 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.163 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.167 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.171 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-30T04:16:31.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-30T04:16:31.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-30T04:16:31.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-30T04:16:31.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-30T04:16:31.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.182 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.189 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.191 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.199 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.204 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.208 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-30T04:16:31.211 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-30T04:16:31.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.213 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.219 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-30T04:16:31.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-30T04:16:31.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T04:16:31.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.232 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.236 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.237 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.241 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.242 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.245 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.246 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.250 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.254 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.255 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:31.257 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.258 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.259 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-30T04:16:31.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-30T04:16:31.270 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.272 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.276 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.279 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.285 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.286 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.290 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.292 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.293 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.294 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.295 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.300 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-30T04:16:31.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.306 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.310 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-30T04:16:31.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.314 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-30T04:16:31.316 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.321 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.325 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.326 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-30T04:16:31.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.331 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.332 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-30T04:16:31.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-30T04:16:31.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-30T04:16:31.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:31.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.343 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.344 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.346 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.347 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-30T04:16:31.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.353 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.359 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.360 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.362 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:31.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.371 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.373 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.376 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.377 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.378 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.379 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.380 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.382 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.383 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.384 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.385 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.386 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.387 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.390 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.392 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.397 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.399 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.400 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-30T04:16:31.401 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.402 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.403 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.404 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-30T04:16:31.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.407 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.408 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.410 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.411 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.412 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.413 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.416 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.417 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.418 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.419 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-30T04:16:31.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T04:16:31.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.422 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.423 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.424 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.432 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.433 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.434 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.437 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.440 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.441 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.445 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.446 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.447 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.448 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.450 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.453 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.454 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.455 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.456 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.459 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-30T04:16:31.462 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.466 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-30T04:16:31.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:31.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-30T04:16:31.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-30T04:16:31.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.483 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.484 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.486 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.488 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.493 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.494 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.498 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.501 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.505 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.506 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.507 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.508 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.511 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.512 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.513 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.514 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.515 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.516 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-30T04:16:31.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.519 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.520 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.525 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.526 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.530 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-30T04:16:31.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.536 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.537 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.538 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "restful", 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_schedule" 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.539 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.540 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.541 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.543 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.547 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.548 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.550 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.552 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.553 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-30T04:16:31.556 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-30T04:16:31.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.558 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-30T04:16:31.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.563 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-30T04:16:31.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:31.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-30T04:16:31.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-30T04:16:31.571 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.572 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-30T04:16:31.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-30T04:16:31.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.578 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.582 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.583 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.584 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.585 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.586 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.587 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap" 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.593 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.594 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.595 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.596 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.597 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.601 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.602 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.603 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-30T04:16:31.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-30T04:16:31.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-30T04:16:31.610 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.612 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.613 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.614 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2024-03-30T04:16:31.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.617 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.618 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.619 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-30T04:16:31.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.623 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-30T04:16:31.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-30T04:16:31.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.628 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-30T04:16:31.629 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.630 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.632 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-30T04:16:31.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.635 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.636 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.637 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.638 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-30T04:16:31.639 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.641 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-30T04:16:31.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.648 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.649 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.650 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.651 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.654 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.656 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.660 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.661 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.663 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.664 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-30T04:16:31.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-30T04:16:31.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-30T04:16:31.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.677 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.678 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.679 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.680 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.683 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-30T04:16:31.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-30T04:16:31.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.688 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-30T04:16:31.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-30T04:16:31.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.696 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.698 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.699 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.700 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.701 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.705 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-30T04:16:31.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.709 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-30T04:16:31.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-30T04:16:31.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-30T04:16:31.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-30T04:16:31.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.729 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.731 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.732 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.733 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.734 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.735 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.736 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-30T04:16:31.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.742 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.744 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.745 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.746 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-30T04:16:31.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.750 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.752 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.753 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.754 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-30T04:16:31.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.761 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-30T04:16:31.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-30T04:16:31.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-30T04:16:31.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.769 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.770 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.772 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-30T04:16:31.773 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.776 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-30T04:16:31.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.779 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.780 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-30T04:16:31.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.782 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-30T04:16:31.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.787 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.789 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.790 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.791 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.793 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.794 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.795 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:31.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.802 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.803 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.804 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-30T04:16:31.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.808 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.810 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.813 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.814 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.816 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.817 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.818 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.819 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.820 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.821 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.822 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.826 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-30T04:16:31.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.831 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.832 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.835 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.837 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.839 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.841 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.843 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-30T04:16:31.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.845 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.847 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.848 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.850 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.852 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-30T04:16:31.853 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.856 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.857 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.859 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.860 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.861 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.862 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.863 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.866 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-30T04:16:31.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-30T04:16:31.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-30T04:16:31.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.877 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.878 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.881 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.885 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.886 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.887 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.888 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.890 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.893 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-30T04:16:31.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.895 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.897 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.898 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.899 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.900 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.902 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.903 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.905 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.906 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.909 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.913 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.914 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-30T04:16:31.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-30T04:16:31.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-30T04:16:31.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-30T04:16:31.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-30T04:16:31.925 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.926 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.927 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.930 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.931 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.932 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.934 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.936 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.940 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.943 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.945 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.946 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.947 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.949 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.952 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.953 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.956 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.957 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.960 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.965 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.966 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.970 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.971 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.974 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.976 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.977 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.980 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:31.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.985 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:31.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:31.989 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.992 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:31.996 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.997 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:31.998 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:31.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.001 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.005 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.007 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.008 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.010 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.011 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.015 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.016 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.020 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-30T04:16:32.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:32.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-30T04:16:32.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.031 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-30T04:16:32.034 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.037 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.039 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.043 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.047 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.056 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.064 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.066 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.068 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.069 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.072 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.077 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.081 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-30T04:16:32.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-30T04:16:32.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.086 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.088 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.092 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.096 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-30T04:16:32.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.101 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.106 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.114 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-30T04:16:32.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-30T04:16:32.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.124 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.127 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.130 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.131 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.133 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.136 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.137 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.138 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.142 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.145 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:32.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.150 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.151 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.154 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.159 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-30T04:16:32.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-30T04:16:32.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-30T04:16:32.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.167 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.170 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-30T04:16:32.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-30T04:16:32.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.179 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.180 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-30T04:16:32.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.192 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.194 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.198 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.199 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.201 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.202 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.203 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.204 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.207 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.210 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.214 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.215 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-30T04:16:32.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.225 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-30T04:16:32.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-30T04:16:32.230 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-30T04:16:32.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-30T04:16:32.233 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.234 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-30T04:16:32.235 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-30T04:16:32.236 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2024-03-30T04:16:32.237 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-30T04:16:32.238 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-30T04:16:32.239 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 89, 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2702400394 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.241 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3577657736 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.242 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3215394962 2024-03-30T04:16:32.243 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.244 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 417492168 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 623714799 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-30T04:16:32.246 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2563248446 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-30T04:16:32.247 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:32.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:30 smithi154 ceph-mon[79161]: pgmap v457: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 4.6 MiB/s wr, 160 op/s 2024-03-30T04:16:32.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:30 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1349545663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:32.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:30 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2608736837' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:32.249 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:30 smithi193 ceph-mon[79581]: pgmap v457: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 4.6 MiB/s wr, 160 op/s 2024-03-30T04:16:32.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:30 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1349545663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:32.250 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:30 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2608736837' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 msgr2=0x7f092c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:32.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 0x7f092c0a37d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f094419a890 tx=0x7f0930009260 comp rx=0 tx=0).stop 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 msgr2=0x7f0944199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0938000c00 tx=0x7f0938005740 comp rx=0 tx=0).stop 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 shutdown_connections 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f092c0a1310 0x7f092c0a37d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094410d430 0x7f094419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09440731e0 0x7f0944199340 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 --2- 172.21.15.23:0/2608736837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0944073b00 0x7f0944199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.144+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 >> 172.21.15.23:0/2608736837 conn(0x7f09440fc700 msgr2=0x7f09440fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.145+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 shutdown_connections 2024-03-30T04:16:32.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:30.145+0000 7f094bc00700 1 -- 172.21.15.23:0/2608736837 wait complete. 2024-03-30T04:16:32.259 DEBUG:tasks.check_counter:Getting stats from a 2024-03-30T04:16:32.259 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.a perf dump 2024-03-30T04:16:32.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:30 smithi023 ceph-mon[38254]: pgmap v457: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 4.6 MiB/s wr, 160 op/s 2024-03-30T04:16:32.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:30 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1349545663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:32.260 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:30 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2608736837' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-30T04:16:32.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:32 smithi193 ceph-mon[79581]: pgmap v458: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-03-30T04:16:32.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:32 smithi023 ceph-mon[38254]: pgmap v458: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-03-30T04:16:32.643 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:16:32.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:32 smithi154 ceph-mon[79161]: pgmap v458: 129 pgs: 129 active+clean; 775 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 4.5 MiB/s wr, 153 op/s 2024-03-30T04:16:33.480 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:33 smithi023 ceph-mon[38254]: pgmap v459: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 4.6 MiB/s wr, 159 op/s 2024-03-30T04:16:33.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:33 smithi193 ceph-mon[79581]: pgmap v459: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 4.6 MiB/s wr, 159 op/s 2024-03-30T04:16:33.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:33 smithi154 ceph-mon[79161]: pgmap v459: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 4.6 MiB/s wr, 159 op/s 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1946, 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 1946, 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 4778123, 2024-03-30T04:16:34.008 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 604411, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 6, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.196361147, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.065162003, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.054761242, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.054611003, 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:34.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1945, 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.101510778, 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000052190 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1214, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 1198, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 5705536, 2024-03-30T04:16:34.011 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 439995, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.161426716, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.050841291, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.076699842, 2024-03-30T04:16:34.012 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.037497706, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1197, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.062152549, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000051923 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.013 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1771728, 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 160960 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 41, 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 247, 2024-03-30T04:16:34.014 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 9936, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 523025, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 8, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.025903039, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.015204753, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.005425031, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.001328233, 2024-03-30T04:16:34.015 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 244, 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.033695237, 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000138095 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 510896 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.017 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4700, 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.129532375, 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000027560 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:34.018 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-30T04:16:34.019 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-30T04:16:34.020 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 958, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 958, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:34.021 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 958, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 7, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-30T04:16:34.022 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-30T04:16:34.023 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:34.024 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-30T04:16:34.025 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.026 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:34.027 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-30T04:16:34.028 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-30T04:16:34.029 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 3667, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-30T04:16:34.030 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 4, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 12100350, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 12095158, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.031 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 3667 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 154, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 885, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 731, 2024-03-30T04:16:34.032 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 31, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 31, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 0, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 958, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 958, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-30T04:16:34.033 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-30T04:16:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 51076, 2024-03-30T04:16:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 207156 2024-03-30T04:16:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-30T04:16:34.034 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-30T04:16:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-30T04:16:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-30T04:16:34.037 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.039 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.040 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:34.042 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.044 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-30T04:16:34.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.047 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.048 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.049 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.051 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-30T04:16:34.052 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.053 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.054 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.055 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 1, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 1, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 0, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 1, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-30T04:16:34.056 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 2416, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:34.057 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:34.058 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:34.059 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:34.060 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 202845, 2024-03-30T04:16:34.061 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 213, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:34.062 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 207, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 1094131, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 7815, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:34.063 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 2878, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-30T04:16:34.064 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 2878, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2878, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 1.973985947, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000685888 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-30T04:16:34.065 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2878, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2879 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 2878, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 2878, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1851, 2024-03-30T04:16:34.066 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 1025, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-30T04:16:34.067 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-30T04:16:34.068 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-30T04:16:34.069 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-30T04:16:34.070 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 107, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 23, 2024-03-30T04:16:34.071 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 6, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 8, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 2, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-30T04:16:34.072 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:34.073 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:34.074 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 3197, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 10186668, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 3197, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 3197, 2024-03-30T04:16:34.075 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 10186668, 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:34.076 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2878, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 7906051, 2024-03-30T04:16:34.077 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2878, 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 7906051, 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:34.079 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2878, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 2878, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2878, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2878, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.080 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:34.081 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.082 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x5631a01ea640": { 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:34.083 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:34.084 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.085 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:34.085 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:34.368 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-30T04:16:34.369 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-30T04:16:34.369 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-30T04:16:34.369 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:34.369 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:34.369 DEBUG:tasks.check_counter:Getting stats from d 2024-03-30T04:16:34.369 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.d perf dump 2024-03-30T04:16:34.739 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:16:35.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:35 smithi154 ceph-mon[79161]: pgmap v460: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:35.724 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:35 smithi023 ceph-mon[38254]: pgmap v460: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:35.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:35 smithi193 ceph-mon[79581]: pgmap v460: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:35.991 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-30T04:16:35.992 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.003575824, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000749328, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:35.993 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:35.994 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 211, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 189344, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-30T04:16:35.995 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.020819437, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.014061440, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.003199457, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 208, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.020407735, 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000098114 2024-03-30T04:16:35.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 189344 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.997 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 293, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 279, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1716896, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 158896, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.060294154, 2024-03-30T04:16:35.998 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.021388738, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.045131956, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.005425976, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 279, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.016169234, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000057954 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:35.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1716896, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 158896 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-30T04:16:36.000 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:36.001 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:36.002 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:36.003 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:36.004 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 150008, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 31, 2024-03-30T04:16:36.005 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-30T04:16:36.006 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:36.007 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-30T04:16:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-30T04:16:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:36.008 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 291, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1673138, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 291, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:36.009 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 291, 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1673138, 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:36.010 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:36.383 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-30T04:16:36.383 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-30T04:16:36.384 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-30T04:16:36.384 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-30T04:16:36.384 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-30T04:16:36.384 DEBUG:tasks.check_counter:Getting stats from g 2024-03-30T04:16:36.384 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.g perf dump 2024-03-30T04:16:36.759 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:16:37.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:37 smithi193 ceph-mon[79581]: pgmap v461: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:37.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:37 smithi023 ceph-mon[38254]: pgmap v461: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:37.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:37 smithi154 ceph-mon[79161]: pgmap v461: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 70 KiB/s wr, 16 op/s 2024-03-30T04:16:37.968 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:37.968 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 983, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 985, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 330233, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 305714, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 7, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.125164673, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.054418593, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.026519622, 2024-03-30T04:16:37.969 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.028729760, 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 983, 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.060772081, 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000061823 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:37.970 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2260, 2024-03-30T04:16:37.971 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2242, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2748115, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 765056, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.203739783, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.065986116, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.083168608, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.056275288, 2024-03-30T04:16:37.972 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2242, 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.097796663, 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000043620 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1775264, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 162592 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 5, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 213, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1668, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 515328, 2024-03-30T04:16:37.974 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 8, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.021489485, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.013769334, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.004115699, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000092023, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 210, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.022510155, 2024-03-30T04:16:37.975 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000107191 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:37.976 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 514384 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-30T04:16:37.977 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4877, 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.122981367, 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000025216 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-30T04:16:37.978 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-30T04:16:37.979 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:37.980 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 70, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 70, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:37.981 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 70, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 3, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-30T04:16:37.982 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-30T04:16:37.983 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T04:16:37.984 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:37.985 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-30T04:16:37.986 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-30T04:16:37.987 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:37.988 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-30T04:16:37.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 325, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-30T04:16:37.990 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 1, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 4674877, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 4674303, 2024-03-30T04:16:37.991 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 325 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-30T04:16:37.992 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 21, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 64, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 43, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 13, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 13, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 0, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 70, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 70, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-30T04:16:37.993 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 48640, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 207156 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:37.994 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.995 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.997 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-30T04:16:37.998 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:37.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.000 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.001 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.003 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-30T04:16:38.004 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.005 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.006 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.007 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:38.008 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.009 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.011 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.012 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 2, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 5, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 3, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 3, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-30T04:16:38.013 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 5471, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:38.014 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:38.015 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:38.016 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:38.017 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 198000, 2024-03-30T04:16:38.018 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 71, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:38.019 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 207, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 118892, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 744, 2024-03-30T04:16:38.020 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 2923, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-30T04:16:38.021 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 2923, 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2923, 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2.526982087, 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000864516 2024-03-30T04:16:38.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2923, 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2924 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.023 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 2923, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 2923, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1901, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 1020, 2024-03-30T04:16:38.024 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-30T04:16:38.025 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:38.026 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:38.027 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-30T04:16:38.028 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-30T04:16:38.029 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 107, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 23, 2024-03-30T04:16:38.030 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 6, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 8, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 2, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-30T04:16:38.031 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-30T04:16:38.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:38.033 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T04:16:38.034 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-30T04:16:38.034 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-30T04:16:38.034 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-30T04:16:38.034 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:38.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 3244, 2024-03-30T04:16:38.035 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 2768755, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 3244, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 3244, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2768755, 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:38.036 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:38.037 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2923, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 480578, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2923, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 480578, 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:38.038 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-30T04:16:38.039 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2923, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 2923, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2923, 2024-03-30T04:16:38.040 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2923, 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:38.041 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:38.042 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x55c1e5486640": { 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:38.044 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:38.045 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.046 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:38.046 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:38.346 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-30T04:16:38.346 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-30T04:16:38.346 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-30T04:16:38.346 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:38.346 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:38.347 DEBUG:tasks.check_counter:Getting stats from j 2024-03-30T04:16:38.347 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.j perf dump 2024-03-30T04:16:38.719 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.a/config 2024-03-30T04:16:39.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:39 smithi023 ceph-mon[38254]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 22 op/s 2024-03-30T04:16:39.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:39 smithi154 ceph-mon[79161]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 22 op/s 2024-03-30T04:16:39.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:39 smithi193 ceph-mon[79581]: pgmap v462: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 22 op/s 2024-03-30T04:16:39.907 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-30T04:16:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:39.907 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 291, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 279, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1714448, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 159632, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.068608525, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.022295516, 2024-03-30T04:16:39.908 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.055005136, 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.005769758, 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 278, 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.017738902, 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000063809 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:39.909 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1714448, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 159632 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 6, 2024-03-30T04:16:39.910 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 215, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 3312, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 190144, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.022250313, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.015554559, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.002862032, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T04:16:39.911 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 213, 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.018563938, 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000087154 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:39.912 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 190144 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-30T04:16:39.913 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.000588863, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000437084, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:39.914 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:39.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.916 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:39.917 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:39.918 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:39.919 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-30T04:16:39.920 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148994, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 31, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-30T04:16:39.921 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-30T04:16:39.922 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:39.923 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 293, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1673398, 2024-03-30T04:16:39.924 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 293, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 293, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1673398, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-30T04:16:39.925 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-30T04:16:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:39.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-30T04:16:39.926 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-30T04:16:40.321 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-30T04:16:40.322 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-30T04:16:40.322 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-30T04:16:40.322 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-30T04:16:40.322 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-30T04:16:40.322 DEBUG:tasks.check_counter:Getting stats from b 2024-03-30T04:16:40.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.b perf dump 2024-03-30T04:16:40.655 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T04:16:41.808 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-30T04:16:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 1540, 2024-03-30T04:16:41.808 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1691, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 2336465, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1002666, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 17, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.164751824, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.053022154, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.083912823, 2024-03-30T04:16:41.809 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.026863091, 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1690, 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.986041854, 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000583456 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1542, 2024-03-30T04:16:41.810 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000318134, 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000206 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1826944, 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 603920 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 315487, 2024-03-30T04:16:41.811 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 158426, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 152460036, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 25807138, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 12, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 14.313720352, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 6.147560162, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 11.037001079, 2024-03-30T04:16:41.812 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.079315547, 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 158428, 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3.927073840, 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000024787 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 473043, 2024-03-30T04:16:41.813 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.066440104, 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000140 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 513968, 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 279648 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:41.814 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 37705, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 37109, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 13096013, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 11598358, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 223, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 9, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 2.449574501, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.873927530, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 1.977260659, 2024-03-30T04:16:41.815 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.081628123, 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 37103, 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.503951879, 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000040534 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 72778, 2024-03-30T04:16:41.816 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.009634447, 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000132 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 12547056, 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 5796112 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-30T04:16:41.817 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 957, 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.348698003, 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000364365 2024-03-30T04:16:41.818 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 143, 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.006807597, 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000047605 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 34836, 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 587, 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 587, 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 4.451512456, 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.007583496 2024-03-30T04:16:41.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 34249, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 12, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-30T04:16:41.821 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 62, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 5, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 37, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 20, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 50, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 37, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 18, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 18, 2024-03-30T04:16:41.822 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 4, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 35318, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 763, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 34249, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 2, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 16, 2024-03-30T04:16:41.823 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 1, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 46, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 1, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 11, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 322, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:41.824 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 716, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 223963475, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 322, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 207, 2024-03-30T04:16:41.825 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 26, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 181, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 115, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 1, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 19, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 258, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:41.826 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 79, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 37, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 23, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 310 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-30T04:16:41.827 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 3, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 2, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 4, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 14, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 44, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 44, 2024-03-30T04:16:41.828 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 26, 2024-03-30T04:16:41.829 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 1, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 576, 2024-03-30T04:16:41.830 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 576, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 1, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 1, 2024-03-30T04:16:41.831 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 7580280, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 319, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.987097875, 2024-03-30T04:16:41.832 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003094350 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 63, 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 168, 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 105, 2024-03-30T04:16:41.833 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 49, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 52, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 3, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 71, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 160, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 89, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 7, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 125, 2024-03-30T04:16:41.834 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 118, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 61396, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 207156 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 35152, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 34836, 2024-03-30T04:16:41.835 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 45, 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 66, 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 83, 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.623803678, 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.007515706 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-30T04:16:41.836 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 181, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.038711086, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000213873 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.837 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 94, 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 3.060267058, 2024-03-30T04:16:41.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.032556032 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-30T04:16:41.839 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1, 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000190857, 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000190857 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.841 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 15, 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.002601740, 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000173449 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.842 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 18, 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.023157547, 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001286530 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 15, 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.138289711, 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.009219314 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-30T04:16:41.844 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 98, 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.136533093, 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001393194 2024-03-30T04:16:41.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 18, 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.190764660, 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.010598036 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.846 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.847 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9, 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.088055521, 2024-03-30T04:16:41.848 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.009783946 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-03-30T04:16:41.849 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.001662126, 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000831063 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.850 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.851 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9, 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.025800019, 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002866668 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-30T04:16:41.852 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 44, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.121675360, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002765349 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.853 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 7, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 5, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 228, 2024-03-30T04:16:41.855 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 114, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 890, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:41.856 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:41.857 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:41.858 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:41.859 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:41.860 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 302161, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 154, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 352, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 4, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:41.861 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-30T04:16:41.862 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 311998, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 2518, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-30T04:16:41.863 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 1439, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 3398181, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 37, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 1439, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1439, 2024-03-30T04:16:41.864 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 21.599441722, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.015010035 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1439, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1486 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 1439, 2024-03-30T04:16:41.865 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 390, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 1049, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 0, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 11, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 42, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 335, 2024-03-30T04:16:41.866 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 70, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 600, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:41.867 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 348, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 22, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-30T04:16:41.868 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-30T04:16:41.869 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-30T04:16:41.870 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 107, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 28, 2024-03-30T04:16:41.871 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 12, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 48, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 10 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-30T04:16:41.872 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 34913 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.873 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 44, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.874 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 354728, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 134197186, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 354728, 2024-03-30T04:16:41.875 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 354728, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 134197186, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.876 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:41.877 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 1439, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 3402977, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1439, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 3402977, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.878 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:41.879 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 1439, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 1439, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1439, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1439, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.880 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 44, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 4796, 2024-03-30T04:16:41.881 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 44, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 16, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 4796, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.882 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x5652f8d900a0": { 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 21198, 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:41.883 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 576, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 3385976, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 576, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 319, 2024-03-30T04:16:41.884 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 3364778, 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:41.885 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-30T04:16:42.168 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-03-30T04:16:42.168 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-30T04:16:42.168 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-30T04:16:42.168 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=15 2024-03-30T04:16:42.168 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=9 2024-03-30T04:16:42.168 DEBUG:tasks.check_counter:Getting stats from e 2024-03-30T04:16:42.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.e perf dump 2024-03-30T04:16:42.392 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T04:16:42.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:42 smithi154 ceph-mon[79161]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-30T04:16:42.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:42 smithi193 ceph-mon[79581]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-30T04:16:42.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:42 smithi023 ceph-mon[38254]: pgmap v463: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-30T04:16:43.456 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 649168, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1035016, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 214596416, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1134911315, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 20, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 4, 2024-03-30T04:16:43.457 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 40.804197189, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 18.541745409, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 25.372986241, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 1.529739423, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1035014, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 31.709304500, 2024-03-30T04:16:43.458 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000030636 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1394576, 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.209107000, 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000149 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 179270960, 2024-03-30T04:16:43.459 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 655840960 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 183795, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 292968, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 37870827, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 649616215, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 10, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-30T04:16:43.460 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 14.101385681, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 6.203607900, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 6.304378587, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 1.844606351, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 292968, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 9.803898084, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000033464 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.461 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 326449, 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.066186702, 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000202 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1812192, 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 189232 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.462 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 315693, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 581305, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 59137179, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 686842196, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 56, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 22.189034473, 2024-03-30T04:16:43.463 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 10.448920777, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 11.841252415, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 1.176127462, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 581302, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 15.821048464, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000027216 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:43.464 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 798013, 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.131125458, 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000164 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 518304 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-30T04:16:43.465 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 155803, 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 120.118396911, 2024-03-30T04:16:43.466 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000770963 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 91495, 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 12.532461249, 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000136974 2024-03-30T04:16:43.467 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 346325, 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 345596, 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 345596, 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 730.995333483, 2024-03-30T04:16:43.468 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002115173 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 729, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 209, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 14, 2024-03-30T04:16:43.469 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 17, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 3818, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 1337, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 2479, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 2, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 1909, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 176, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 647, 2024-03-30T04:16:43.470 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 647, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 16, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 464594, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 301081, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 728, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 21, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 3596, 2024-03-30T04:16:43.471 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 10, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 131, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 1, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 46, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 2622, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 616, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 211345520, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 2622, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 388, 2024-03-30T04:16:43.473 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 388, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 2234, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 7, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 34134, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 221804, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:43.474 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 101401, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 39712, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 19218, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 234899 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 6, 2024-03-30T04:16:43.475 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 0, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 22, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 31, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 49188, 2024-03-30T04:16:43.476 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 49188, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:43.477 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 12, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 31, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 225448, 2024-03-30T04:16:43.478 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 154500, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 154500, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 70948, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 539, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 355, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 227, 2024-03-30T04:16:43.479 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 227, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 129, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 1, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 708845943, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 1228484737, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 79047, 2024-03-30T04:16:43.480 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 203.111341675, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002569500 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 1655, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 50967, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 49312, 2024-03-30T04:16:43.481 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 159, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 268, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 109, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 3818, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 101880, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 98062, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 647, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 52407, 2024-03-30T04:16:43.482 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 51760, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 80164, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 198964 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 512170, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 593, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 346325, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 6828, 2024-03-30T04:16:43.483 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 1291, 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 49847, 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 93.820427184, 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001882167 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 832, 2024-03-30T04:16:43.484 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 18.168159890, 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.021836730 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.485 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 98771, 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 89.146294356, 2024-03-30T04:16:43.486 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000902555 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-30T04:16:43.487 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.488 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 101, 2024-03-30T04:16:43.490 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.721672573, 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.007145273 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.491 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9, 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.021048528, 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002338725 2024-03-30T04:16:43.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 134834, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 390.431816705, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002895648 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 10373, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 22.746180349, 2024-03-30T04:16:43.493 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002192825 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:43.494 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.005327841, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005327841 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.495 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 20, 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.797993793, 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.039899689 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.497 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1620, 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.988497949, 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001227467 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-30T04:16:43.498 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 49188, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 113.147914315, 2024-03-30T04:16:43.500 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002300315 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.501 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 1, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 2, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 1, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-30T04:16:43.502 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 44001, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 44001, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 2888, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:43.503 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:43.504 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:43.505 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:43.506 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:43.507 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 505075, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 1193, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 176, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 2, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:43.508 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:43.509 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 7403553, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 75069, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.510 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 184297, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 1234336569, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 33, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 184297, 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-30T04:16:43.511 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 184297, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 584.579877599, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003171944 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 184297, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 185001 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.512 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 184297, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 3934, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 180363, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 55, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 34, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 1312, 2024-03-30T04:16:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 80359, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 648, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 14, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 98936, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-30T04:16:43.514 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 2622, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 68, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-30T04:16:43.515 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-30T04:16:43.516 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-30T04:16:43.517 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 107, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 23, 2024-03-30T04:16:43.518 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 12, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 0, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 720, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 233 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-30T04:16:43.519 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 346982 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:43.520 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 49188, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:43.521 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 1148652, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 187168386, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 1148652, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1148652, 2024-03-30T04:16:43.522 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 187168386, 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:43.523 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 184297, 2024-03-30T04:16:43.524 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 1239704701, 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 184297, 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1239704701, 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:43.526 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 184297, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 184297, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 184297, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 184297, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 49188, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 5368132, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:43.528 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 49188, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 1312, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 5368132, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.529 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x55fe73a9e0a0": { 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 225448, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 1224290433, 2024-03-30T04:16:43.530 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 225448, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 79047, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 1224290433, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:43.531 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:43.532 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-30T04:16:43.604 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:43 smithi154 ceph-mon[79161]: pgmap v464: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-03-30T04:16:43.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:43 smithi193 ceph-mon[79581]: pgmap v464: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-03-30T04:16:43.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:43 smithi023 ceph-mon[38254]: pgmap v464: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 18 op/s 2024-03-30T04:16:43.818 INFO:tasks.check_counter:Daemon mds.e mds.exported=10 2024-03-30T04:16:43.818 INFO:tasks.check_counter:Daemon mds.e mds.imported=1 2024-03-30T04:16:43.818 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-30T04:16:43.818 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:43.818 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:43.818 DEBUG:tasks.check_counter:Getting stats from h 2024-03-30T04:16:43.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.h perf dump 2024-03-30T04:16:44.045 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T04:16:45.111 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-30T04:16:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 8897, 2024-03-30T04:16:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 9768, 2024-03-30T04:16:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 3048795, 2024-03-30T04:16:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 6522587, 2024-03-30T04:16:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 13, 2024-03-30T04:16:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-30T04:16:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.617424396, 2024-03-30T04:16:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.247791225, 2024-03-30T04:16:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.443217729, 2024-03-30T04:16:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.021532214, 2024-03-30T04:16:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 9766, 2024-03-30T04:16:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.420460469, 2024-03-30T04:16:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000043053 2024-03-30T04:16:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 16748, 2024-03-30T04:16:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.002431404, 2024-03-30T04:16:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000145 2024-03-30T04:16:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 2652816, 2024-03-30T04:16:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 3668688 2024-03-30T04:16:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 554, 2024-03-30T04:16:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 554, 2024-03-30T04:16:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 1855390, 2024-03-30T04:16:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 671494, 2024-03-30T04:16:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 60, 2024-03-30T04:16:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 6, 2024-03-30T04:16:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.075903987, 2024-03-30T04:16:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.017119060, 2024-03-30T04:16:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.060675659, 2024-03-30T04:16:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.009005036, 2024-03-30T04:16:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 553, 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.058823233, 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000106371 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 105, 2024-03-30T04:16:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000014713, 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000140 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1808176, 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 178160 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 219928, 2024-03-30T04:16:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 110900, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 126188877, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 18487704, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 20, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 10.147718436, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 4.355830867, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 7.819900146, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.005195114, 2024-03-30T04:16:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 110898, 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.349174786, 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000021183 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 330029, 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.046385464, 2024-03-30T04:16:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000140 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 595008 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-30T04:16:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1032, 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.467267394, 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000452778 2024-03-30T04:16:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 402, 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.024817870, 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000061735 2024-03-30T04:16:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 6495, 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 974, 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 974, 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 4.652235415, 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004776422 2024-03-30T04:16:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 5521, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 11, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-30T04:16:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 143, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 16, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 127, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 121, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 0, 2024-03-30T04:16:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 11, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 11, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 6, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 7858, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 1027, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 5521, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 7, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 36, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 3, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 43, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 38, 2024-03-30T04:16:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 616, 2024-03-30T04:16:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 211345520, 2024-03-30T04:16:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-30T04:16:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 38, 2024-03-30T04:16:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 38, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 28, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 820, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 700, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 425, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 116, 2024-03-30T04:16:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 923 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 3, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 8, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 6, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 11, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 3, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-30T04:16:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 177, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 177, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 1493, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 1493, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-30T04:16:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 2, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 2, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 6419850, 2024-03-30T04:16:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 480, 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.658154643, 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003454488 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 104, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 281, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 177, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 46, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 49, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 3, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 143, 2024-03-30T04:16:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 422, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 279, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 11, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 237, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 226, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 58896, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 198964 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 7416, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 6495, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 77, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 272, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 181, 2024-03-30T04:16:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.613497293, 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.014439211 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 6, 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.004007892, 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000667982 2024-03-30T04:16:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.147 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 296, 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.251790275, 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000850642 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 273, 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.227415057, 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000833022 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 37, 2024-03-30T04:16:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.600055448, 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.016217714 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 4, 2024-03-30T04:16:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.015469132, 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.003867283 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 177, 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.940000318, 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005310736 2024-03-30T04:16:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 3, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 1, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 820, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 410, 2024-03-30T04:16:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 837, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 311335, 2024-03-30T04:16:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 189, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 440, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 5, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 415605, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 3161, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 996, 2024-03-30T04:16:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 2276246, 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 27, 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 996, 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 996, 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 4.962872807, 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004982804 2024-03-30T04:16:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 996, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1037 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 996, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 63, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 933, 2024-03-30T04:16:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 0, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 10, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 25, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 505, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 28, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-30T04:16:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 366, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 38, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 20, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-30T04:16:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-30T04:16:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 107, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 23, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 17, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 5, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 34, 2024-03-30T04:16:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 11 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-30T04:16:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 6495 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-30T04:16:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 177, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 229375, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 110071929, 2024-03-30T04:16:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 229375, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 229375, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 110071929, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.185 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:45.186 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 996, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 2295539, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 996, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 2295539, 2024-03-30T04:16:45.187 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-30T04:16:45.188 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 996, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 996, 2024-03-30T04:16:45.189 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 996, 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 996, 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:45.190 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 177, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 19293, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 177, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:45.191 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 25, 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 19293, 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x5590a752a0a0": { 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 1722, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 1493, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 2225546, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 1493, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:45.193 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 480, 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 2223824, 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:45.194 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:45.195 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-30T04:16:45.494 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-30T04:16:45.494 INFO:tasks.check_counter:Daemon mds.h mds.imported=3 2024-03-30T04:16:45.494 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-30T04:16:45.494 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:45.495 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:45.497 DEBUG:tasks.check_counter:Getting stats from k 2024-03-30T04:16:45.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.k perf dump 2024-03-30T04:16:45.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:45 smithi154 ceph-mon[79161]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:45.721 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.b/config 2024-03-30T04:16:45.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:45 smithi193 ceph-mon[79581]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:45.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:45 smithi023 ceph-mon[38254]: pgmap v465: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:46.770 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-03-30T04:16:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 1339, 2024-03-30T04:16:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 1345, 2024-03-30T04:16:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 2070442, 2024-03-30T04:16:46.771 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 1200181, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 10, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 5, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.137061265, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.049150773, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.083627743, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.015123552, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1344, 2024-03-30T04:16:46.772 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.070980344, 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000052812 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 1612, 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000316345, 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000196 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.773 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 1790256, 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 180304 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 219103, 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 110242, 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 125956646, 2024-03-30T04:16:46.774 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 18634181, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 70, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 7, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 10.522540978, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 4.585069710, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 8.081711486, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.001174669, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 110239, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 2.056405033, 2024-03-30T04:16:46.775 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000018654 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 328972, 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.044105219, 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000134 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 159968, 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 645552 2024-03-30T04:16:46.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_messages": 108, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages": 286, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_bytes": 29990, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_bytes": 457992, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_created_connections": 14, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_active_connections": 8, 2024-03-30T04:16:46.777 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_total_time": 0.017944659, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_send_time": 0.007050272, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_recv_time": 0.006359217, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_running_fast_dispatch_time": 0.002806252, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 285, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.048153770, 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000168960 2024-03-30T04:16:46.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 13, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000002107, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000162 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_recv_encrypted_bytes": 4656, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "msgr_send_encrypted_bytes": 397568 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.779 INFO:teuthology.orchestra.run.smithi154.stdout: "cct": { 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "total_workers": 1, 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "unhealthy_workers": 0 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-MDSRank": { 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 294, 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.132016477, 2024-03-30T04:16:46.780 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000449035 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "queue_len": 0, 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "complete_latency": { 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 153, 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.007370021, 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000048170 2024-03-30T04:16:46.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "mds": { 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "request": 290, 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "reply": 288, 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "reply_latency": { 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 288, 2024-03-30T04:16:46.782 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.287837228, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.004471657 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_reply": 0, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "forward": 2, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_commit": 11, 2024-03-30T04:16:46.783 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_split": 0, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_merge": 0, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes": 51, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_top": 6, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_bottom": 45, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_pinned": 43, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_expired": 1, 2024-03-30T04:16:46.784 INFO:teuthology.orchestra.run.smithi154.stdout: "inodes_with_caps": 2, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "caps": 2, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "subtrees": 4, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse": 798, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_hit": 475, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_forward": 1, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_discover": 8, 2024-03-30T04:16:46.785 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "traverse_lock": 30, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "load_cent": 0, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "q": 0, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "exported": 0, 2024-03-30T04:16:46.786 INFO:teuthology.orchestra.run.smithi154.stdout: "exported_inodes": 0, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "imported": 1, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "imported_inodes": 3, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_backtrace_fetch": 5, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rfiles": 616, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rbytes": 211345520, 2024-03-30T04:16:46.787 INFO:teuthology.orchestra.run.smithi154.stdout: "root_rsnaps": 6, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_fetch": 5, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "scrub_file_inodes": 5, 2024-03-30T04:16:46.788 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_revoke": 1, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_grant": 191, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps": 64, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_caps_dirty": 35, 2024-03-30T04:16:46.789 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_cap_release": 30, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "process_request_cap_release": 301 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_cache": { 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update": 0, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_update_receipt": 11, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_try_discover": 8, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_send_discover": 14, 2024-03-30T04:16:46.790 INFO:teuthology.orchestra.run.smithi154.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_created": 43, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_enqueued": 43, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "strays_migrated": 0, 2024-03-30T04:16:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_started": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "recovery_completed": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_flush": 0, 2024-03-30T04:16:46.792 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "ireq_inodestats": 0 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_log": { 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "evadd": 329, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "evex": 0, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "evtrm": 0, 2024-03-30T04:16:46.793 INFO:teuthology.orchestra.run.smithi154.stdout: "ev": 329, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "evexg": 0, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "evexd": 0, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "segadd": 1, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "segex": 0, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "segtrm": 0, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "seg": 1, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "segexg": 0, 2024-03-30T04:16:46.794 INFO:teuthology.orchestra.run.smithi154.stdout: "segexd": 0, 2024-03-30T04:16:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "expos": 4194304, 2024-03-30T04:16:46.795 INFO:teuthology.orchestra.run.smithi154.stdout: "wrpos": 4682513, 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "rdpos": 0, 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "jlat": { 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 116, 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.656716356, 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005661347 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: "replayed": 0 2024-03-30T04:16:46.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_mem": { 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "ino": 40, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "ino+": 83, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "ino-": 43, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "dir": 42, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "dir+": 46, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "dir-": 4, 2024-03-30T04:16:46.797 INFO:teuthology.orchestra.run.smithi154.stdout: "dn": 51, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "dn+": 112, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "dn-": 61, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "cap": 2, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "cap+": 52, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "cap-": 50, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "rss": 53876, 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: "heap": 198964 2024-03-30T04:16:46.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_server": { 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_client_request": 583, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_request": 290, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_client_session": 33, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "handle_peer_request": 80, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "req_create_latency": { 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 45, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.576950540, 2024-03-30T04:16:46.799 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.012821123 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getattr_latency": { 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 2, 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.001220877, 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000610438 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.800 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "req_link_latency": { 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.801 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookup_latency": { 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 89, 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.138662814, 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.001558009 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.802 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupino_latency": { 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupname_latency": { 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.803 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:46.804 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "req_lssnap_latency": { 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mkdir_latency": { 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mknod_latency": { 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.806 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "req_mksnap_latency": { 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.807 INFO:teuthology.orchestra.run.smithi154.stdout: "req_open_latency": { 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "req_readdir_latency": { 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 79, 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.055277351, 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000699713 2024-03-30T04:16:46.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rename_latency": { 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 30, 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.403554442, 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.013451814 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.809 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.810 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmdir_latency": { 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.811 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setattr_latency": { 2024-03-30T04:16:46.812 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.813 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setlayout_latency": { 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.814 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "req_setxattr_latency": { 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "req_symlink_latency": { 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.815 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "req_unlink_latency": { 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 43, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.112171204, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.002608632 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:46.816 INFO:teuthology.orchestra.run.smithi154.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_sessions": { 2024-03-30T04:16:46.817 INFO:teuthology.orchestra.run.smithi154.stdout: "session_count": 2, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "session_add": 5, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "session_remove": 3, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_open": 2, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "sessions_stale": 0, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "total_load": 236, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "average_load": 118, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "avg_session_uptime": 895, 2024-03-30T04:16:46.818 INFO:teuthology.orchestra.run.smithi154.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "mempool": { 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:46.819 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:46.820 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:46.821 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:46.822 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_items": 0, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_bytes": 294303, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_anon_items": 152, 2024-03-30T04:16:46.823 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_bytes": 440, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "buffer_meta_items": 5, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_bytes": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_items": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:46.824 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_items": 207, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "pgmap_items": 0, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_bytes": 188161, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "mds_co_items": 1170, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:46.825 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_1_items": 0, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "unittest_2_items": 0 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "objecter": { 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "op_active": 0, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "op_laggy": 0, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send": 321, 2024-03-30T04:16:46.826 INFO:teuthology.orchestra.run.smithi154.stdout: "op_send_bytes": 511052, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "op_resend": 33, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "op_reply": 321, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "op_latency": { 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 321, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 1.726955719, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.005379924 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.827 INFO:teuthology.orchestra.run.smithi154.stdout: "op_inflight": 0, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "oplen_avg": { 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 321, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 365 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "op": 321, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "op_r": 23, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "op_w": 298, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "op_rmw": 0, 2024-03-30T04:16:46.828 INFO:teuthology.orchestra.run.smithi154.stdout: "op_pg": 0, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_stat": 0, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_create": 10, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_read": 18, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_write": 134, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writefull": 27, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_writesame": 0, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_append": 0, 2024-03-30T04:16:46.829 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_zero": 0, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_truncate": 0, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_delete": 98, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_mapext": 0, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_getxattr": 5, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_setxattr": 20, 2024-03-30T04:16:46.830 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_call": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_watch": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_notify": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "osdop_other": 0, 2024-03-30T04:16:46.831 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_active": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_send": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_resend": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "linger_ping": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_active": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_send": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "poolop_resend": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_active": 0, 2024-03-30T04:16:46.832 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_send": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "poolstat_resend": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_active": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_send": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "statfs_resend": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "command_active": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "command_send": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "command_resend": 0, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "map_epoch": 107, 2024-03-30T04:16:46.833 INFO:teuthology.orchestra.run.smithi154.stdout: "map_full": 0, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "map_inc": 23, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_sessions": 12, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_open": 18, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_session_close": 6, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "osd_laggy": 0, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_wr": 43, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_rd": 0, 2024-03-30T04:16:46.834 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_del": 10 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "oft": { 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_objs": 1, 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_updates": 0, 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "omap_total_removes": 0 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "osd-slow-ops": { 2024-03-30T04:16:46.835 INFO:teuthology.orchestra.run.smithi154.stdout: "slow_ops_count": 290 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "purge_queue": { 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_ops_high_water": 10, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing": 0, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executing_high_water": 5, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_executed": 43, 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:46.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 220546, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 108166650, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:46.837 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 220546, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 220546, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 108166650, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 104857600, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:46.839 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 321, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 515739, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 321, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 515739, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.840 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 1024, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 0, 2024-03-30T04:16:46.841 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 0, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 321, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 321, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 321, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 321, 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:46.842 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 0, 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:46.843 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 43, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 4687, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 43, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 18, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 4687, 2024-03-30T04:16:46.844 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "throttle-write_buf_throttle-0x564de5a900a0": { 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "val": 574, 2024-03-30T04:16:46.845 INFO:teuthology.orchestra.run.smithi154.stdout: "max": 3758096384, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "get_started": 0, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "get": 329, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "get_sum": 488209, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "get_or_fail_success": 329, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "take": 0, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "take_sum": 0, 2024-03-30T04:16:46.846 INFO:teuthology.orchestra.run.smithi154.stdout: "put": 116, 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "put_sum": 487635, 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "wait": { 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgcount": 0, 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "sum": 0.000000000, 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: "avgtime": 0.000000000 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-03-30T04:16:46.847 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-03-30T04:16:47.142 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-30T04:16:47.142 INFO:tasks.check_counter:Daemon mds.k mds.imported=1 2024-03-30T04:16:47.142 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=6 2024-03-30T04:16:47.142 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:47.143 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:47.143 DEBUG:tasks.check_counter:Getting stats from c 2024-03-30T04:16:47.143 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.c perf dump 2024-03-30T04:16:47.390 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T04:16:48.378 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T04:16:48.378 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:48.378 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2940, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2942, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3053263, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 914347, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.204707149, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.062603270, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.051728043, 2024-03-30T04:16:48.379 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.074063182, 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2940, 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.089726940, 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000030519 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.380 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 8, 2024-03-30T04:16:48.381 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 217, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 3868, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 519180, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.010737183, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.006537953, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002206951, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000081707, 2024-03-30T04:16:48.382 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 215, 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.016968961, 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000078925 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.383 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 518496 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 331, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 308, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1799980, 2024-03-30T04:16:48.384 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 164823, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 4, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.051257908, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.014465169, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.041429310, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.006198216, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 307, 2024-03-30T04:16:48.385 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.014053937, 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000045778 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.386 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1799712, 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 164512 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4955, 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.114225578, 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000023052 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-03-30T04:16:48.389 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:48.390 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 289, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 1, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 288, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:48.391 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 288, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 5, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-03-30T04:16:48.392 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-03-30T04:16:48.393 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T04:16:48.394 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:48.395 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-03-30T04:16:48.396 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:48.397 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:48.398 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:48.399 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 1490, 2024-03-30T04:16:48.400 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 2, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-30T04:16:48.401 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 6418128, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 6415769, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.402 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 1490 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 92, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 269, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 177, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 20, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 20, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 0, 2024-03-30T04:16:48.403 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 289, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 289, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 0, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 52808, 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 207156 2024-03-30T04:16:48.404 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-30T04:16:48.405 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.406 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.407 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:48.408 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.409 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.410 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-30T04:16:48.411 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.412 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.413 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.414 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.415 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.417 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-30T04:16:48.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.422 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 3, 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 1, 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 2, 2024-03-30T04:16:48.424 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 2919, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:48.425 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:48.426 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:48.427 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:48.428 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:48.429 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 184243, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 133, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-30T04:16:48.430 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 207, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:48.431 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 452362, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 3337, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-30T04:16:48.432 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 2943, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 2943, 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-30T04:16:48.433 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2943, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.384978427, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000470600 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2943, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2944 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.434 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 2943, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 2943, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 1907, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1034, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-03-30T04:16:48.435 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:48.436 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-30T04:16:48.437 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-30T04:16:48.438 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-30T04:16:48.439 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 107, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 23, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 4, 2024-03-30T04:16:48.440 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 5, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 1, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-30T04:16:48.441 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.442 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:48.443 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3275, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 4542535, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3275, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:48.444 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3275, 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4542535, 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:48.446 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2943, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2223829, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2943, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2223829, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.447 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:48.448 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2943, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2943, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2943, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2943, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.449 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:48.450 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.451 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x558fc4522500": { 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:48.452 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:48.453 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:48.454 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:48.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.454 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:48.454 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T04:16:48.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:48 smithi193 ceph-mon[79581]: pgmap v466: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:48.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:48 smithi023 ceph-mon[38254]: pgmap v466: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:48.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:48 smithi154 ceph-mon[79161]: pgmap v466: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 12 op/s 2024-03-30T04:16:48.749 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-30T04:16:48.749 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-30T04:16:48.749 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-30T04:16:48.749 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:48.749 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:48.749 DEBUG:tasks.check_counter:Getting stats from f 2024-03-30T04:16:48.750 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.f perf dump 2024-03-30T04:16:48.971 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T04:16:49.488 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:49 smithi193 ceph-mon[79581]: pgmap v467: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-03-30T04:16:49.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:49 smithi023 ceph-mon[38254]: pgmap v467: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-03-30T04:16:49.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:49 smithi154 ceph-mon[79161]: pgmap v467: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 18 op/s 2024-03-30T04:16:49.888 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T04:16:49.888 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:49.888 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 12732, 2024-03-30T04:16:49.888 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 16106, 2024-03-30T04:16:49.888 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4558121, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 13775793, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 17, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.982833544, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.400629859, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.710021300, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.034563731, 2024-03-30T04:16:49.889 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 16104, 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.722098983, 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000044839 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 24161, 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.003620674, 2024-03-30T04:16:49.890 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000149 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 4146624, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 8970176 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 222076, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 114438, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 126613736, 2024-03-30T04:16:49.891 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 23211247, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 19, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 7, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 8.922467163, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 3.339440217, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 7.848470544, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.026944480, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 114436, 2024-03-30T04:16:49.892 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.813867068, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000015850 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 333996, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.037550301, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000112 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-30T04:16:49.893 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 520384 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1753, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1741, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2182649, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 1046929, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 50, 2024-03-30T04:16:49.894 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.158335038, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.048925719, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.095200020, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.024118960, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1740, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.116602260, 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000067012 2024-03-30T04:16:49.895 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1592, 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000286282, 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000179 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1795520, 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 171728 2024-03-30T04:16:49.896 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2509, 2024-03-30T04:16:49.897 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.611585858, 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000642321 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1420, 2024-03-30T04:16:49.898 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.094887736, 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000066822 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 9795, 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 4275, 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4275, 2024-03-30T04:16:49.899 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 14.207963797, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003323500 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 5520, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 11, 2024-03-30T04:16:49.900 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 294, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 35, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 259, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 245, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 58, 2024-03-30T04:16:49.901 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 58, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 9, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 12662, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 3874, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 5520, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 9, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:49.902 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 26, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 5, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 74, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:49.903 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 84, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 616, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 211345520, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 84, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:49.904 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 10, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 10, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 74, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 1, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 168, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 2331, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:49.905 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 1163, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 641, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 253, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 2928 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-30T04:16:49.906 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 11, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 8, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 16, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 19, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 731, 2024-03-30T04:16:49.907 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 731, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:49.908 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 3681, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-30T04:16:49.909 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 3681, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 4, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 4, 2024-03-30T04:16:49.910 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 12139971, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 0, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1210, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 4.621524974, 2024-03-30T04:16:49.911 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003819442 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 0 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 175, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 906, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 731, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 55, 2024-03-30T04:16:49.912 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 59, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 4, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 294, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 1697, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 1403, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 58, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 854, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 796, 2024-03-30T04:16:49.913 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 62396, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 198964 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 12353, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 127, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 9795, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 81, 2024-03-30T04:16:49.914 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 429, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 743, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 4.456335809, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.005997760 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 215, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2.210075347, 2024-03-30T04:16:49.915 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.010279420 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.916 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1301, 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.600378213, 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000461474 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:49.917 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000140791, 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000140791 2024-03-30T04:16:49.918 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.919 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.920 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.921 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.922 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1162, 2024-03-30T04:16:49.923 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3.939767421, 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003390505 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 121, 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.556689012, 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004600735 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.924 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.925 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:49.926 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.927 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1, 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.013632794, 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.013632794 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.928 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-30T04:16:49.929 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.930 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 731, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2.430944410, 2024-03-30T04:16:49.931 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.003325505 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.932 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 1, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 0, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 1, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 3521, 2024-03-30T04:16:49.933 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 3521, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 887, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:49.934 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:49.935 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:49.936 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:49.937 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:49.938 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 307823, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 230, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 440, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 5, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:49.939 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 207, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 731370, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 6405, 2024-03-30T04:16:49.940 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 2892, 2024-03-30T04:16:49.941 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 8095729, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 33, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 2892, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2892, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 13.892781007, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.004803866 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-30T04:16:49.942 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2892, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 2936 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 2892, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 113, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 2779, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-30T04:16:49.943 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 0, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 10, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 29, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 1239, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 28, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-30T04:16:49.944 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 1475, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 84, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 20, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:49.945 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:49.946 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-30T04:16:49.947 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-30T04:16:49.948 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 107, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 23, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 12, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 14, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 2, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 38, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 0, 2024-03-30T04:16:49.949 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 13 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 1, 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.950 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 9924 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 10, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 5, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 731, 2024-03-30T04:16:49.951 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 236557, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 111506925, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:49.952 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 236557, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 236557, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 111506925, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.953 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:49.954 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2892, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 8175408, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2892, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 8175408, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.955 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:49.956 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 2892, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 2892, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 2892, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 2892, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.957 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:49.958 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 731, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 79679, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 731, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 29, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 79679, 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:49.959 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x560145f340a0": { 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 574, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:49.960 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3681, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 7945667, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3681, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 1210, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 7945093, 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:49.961 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:49.962 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:49.962 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:49.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.962 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:49.962 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T04:16:50.249 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-30T04:16:50.249 INFO:tasks.check_counter:Daemon mds.f mds.imported=5 2024-03-30T04:16:50.249 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-30T04:16:50.249 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:50.249 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:50.249 DEBUG:tasks.check_counter:Getting stats from i 2024-03-30T04:16:50.250 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.i perf dump 2024-03-30T04:16:50.472 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T04:16:51.409 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2180, 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2181, 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 4074784, 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 677679, 2024-03-30T04:16:51.410 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 6, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.156039854, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.045468224, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.043902644, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.050906717, 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:51.411 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2179, 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.062982808, 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000028904 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.412 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 6, 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 217, 2024-03-30T04:16:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 1976, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 520926, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 9, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 5, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.011018227, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.006229085, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.002811811, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.000103463, 2024-03-30T04:16:51.414 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 214, 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.019810587, 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000092572 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.415 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 519616 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1336, 2024-03-30T04:16:51.416 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1311, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 2049181, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 476416, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 3, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 2, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.125106211, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.040317767, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.053971767, 2024-03-30T04:16:51.417 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.033498999, 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1311, 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.046680052, 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000035606 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.418 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1802864, 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 165104 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-30T04:16:51.419 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 5293, 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.115604201, 2024-03-30T04:16:51.420 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000021840 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.421 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.422 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.422 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.422 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-30T04:16:51.422 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-03-30T04:16:51.423 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-03-30T04:16:51.423 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-30T04:16:51.423 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.423 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.424 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.424 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.424 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-30T04:16:51.424 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-03-30T04:16:51.424 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 90, 2024-03-30T04:16:51.425 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 0, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 90, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 90, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-30T04:16:51.426 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 4, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:51.427 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:51.428 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 716, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 223963475, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 6, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:51.429 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T04:16:51.430 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-03-30T04:16:51.431 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:51.432 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:51.433 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:51.434 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-30T04:16:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 577, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 1, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-30T04:16:51.436 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 4194304, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 7625171, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 7580280, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.437 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 577 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 42, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 132, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 90, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 15, 2024-03-30T04:16:51.438 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 26, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 11, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 90, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 136, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 46, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-03-30T04:16:51.439 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 47544, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 207156 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-03-30T04:16:51.440 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-30T04:16:51.441 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.442 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-30T04:16:51.443 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.444 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-30T04:16:51.445 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.446 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.447 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.448 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.449 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.449 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-30T04:16:51.449 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.450 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-30T04:16:51.451 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.452 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:51.453 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.454 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.455 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.456 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.457 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.458 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.459 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:51.460 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.461 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 2, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 7, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 5, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 4, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-03-30T04:16:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 8961, 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:51.464 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:51.465 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:51.466 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:51.467 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 190935, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 96, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-30T04:16:51.468 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 207, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:51.469 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 243165, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 1602, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:51.470 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 3184, 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 26, 2024-03-30T04:16:51.471 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 4, 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 3184, 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3184, 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 1.306979627, 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000410483 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.472 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 3184, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 3185 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 3184, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 3184, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-03-30T04:16:51.473 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2000, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 1181, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-03-30T04:16:51.474 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:51.475 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 2, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-30T04:16:51.476 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-30T04:16:51.477 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-30T04:16:51.478 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 107, 2024-03-30T04:16:51.479 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 23, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 5, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 7, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 2, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-03-30T04:16:51.480 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.481 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-30T04:16:51.482 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:51.483 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 3518, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 5789449, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 3518, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:51.484 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3518, 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 5789449, 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.485 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:51.486 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3184, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3430877, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3184, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3430877, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.487 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:51.488 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 3184, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 3184, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 3184, 2024-03-30T04:16:51.489 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 3184, 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.490 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:51.491 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.492 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x562bf5c42640": { 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:51.493 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:51.494 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:51.495 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T04:16:51.875 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-30T04:16:51.875 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-30T04:16:51.875 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-30T04:16:51.875 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:51.875 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:51.875 DEBUG:tasks.check_counter:Getting stats from l 2024-03-30T04:16:51.876 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:470100a4ccbcf7769052bf09a5413881ccf1b97f shell --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 -- ceph daemon mds.l perf dump 2024-03-30T04:16:52.097 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/mon.c/config 2024-03-30T04:16:52.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:52 smithi193 ceph-mon[79581]: pgmap v468: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-30T04:16:52.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:52 smithi023 ceph-mon[38254]: pgmap v468: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-30T04:16:52.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:52 smithi154 ceph-mon[79161]: pgmap v468: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-30T04:16:53.010 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-30T04:16:53.010 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-0": { 2024-03-30T04:16:53.010 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1241, 2024-03-30T04:16:53.010 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1243, 2024-03-30T04:16:53.010 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 406817085, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 386015, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 8, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.275232247, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.024001238, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.213220566, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.027967242, 2024-03-30T04:16:53.011 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1241, 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.037437566, 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000030167 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.012 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 0 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-1": { 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 1386, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 1601, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 512060696, 2024-03-30T04:16:53.013 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 950749, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 8, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 6, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.287732747, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.034288881, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.213001008, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.030095446, 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:53.014 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 1598, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.056624096, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000035434 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.015 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 520320 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "AsyncMessenger::Worker-2": { 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_messages": 2060, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages": 2042, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_bytes": 308484594, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_bytes": 705047, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_created_connections": 5, 2024-03-30T04:16:53.016 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_active_connections": 4, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_total_time": 0.350033910, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_send_time": 0.053144320, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_recv_time": 0.239946860, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_running_fast_dispatch_time": 0.053681472, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_messages_queue_lat": { 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 2042, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.078493642, 2024-03-30T04:16:53.017 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000038439 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_handle_ack_lat": { 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_recv_encrypted_bytes": 1777440, 2024-03-30T04:16:53.018 INFO:teuthology.orchestra.run.smithi193.stdout: "msgr_send_encrypted_bytes": 167328 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "cct": { 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "total_workers": 1, 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "unhealthy_workers": 0 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-MDSRank": { 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:53.019 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 7003, 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.214175921, 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000030583 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "finisher-PurgeQueue": { 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "queue_len": 0, 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "complete_latency": { 2024-03-30T04:16:53.020 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "mds": { 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "request": 0, 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "reply": 0, 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "reply_latency": { 2024-03-30T04:16:53.021 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_reply": 0, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "forward": 0, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_complete": 0, 2024-03-30T04:16:53.022 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_fetch_keys": 0, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_commit": 0, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_split": 0, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_merge": 0, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes": 51352, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_top": 23855, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_bottom": 27497, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pin_tail": 0, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_pinned": 17274, 2024-03-30T04:16:53.023 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_expired": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "inodes_with_caps": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "caps": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "subtrees": 13, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_hit": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_forward": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_discover": 0, 2024-03-30T04:16:53.024 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_dir_fetch": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_remote_ino": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "traverse_lock": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "load_cent": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "q": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "exported": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "exported_inodes": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "imported": 0, 2024-03-30T04:16:53.025 INFO:teuthology.orchestra.run.smithi193.stdout: "imported_inodes": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_dir_fetch": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_backtrace_fetch": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "openino_peer_discover": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rfiles": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rbytes": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "root_rsnaps": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_fetch": 0, 2024-03-30T04:16:53.026 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_set_tag": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_backtrace_repaired": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_inotable_repaired": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_inodes": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dir_base_inodes": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_dirfrag_rstats": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "scrub_file_inodes": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_inode_file_caps": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_revoke": 0, 2024-03-30T04:16:53.027 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_grant": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_trunc": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_caps_dirty": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_cap_release": 0, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "process_request_cap_release": 0 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_cache": { 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_update_receipt": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_try_discover": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_send_discover": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "dir_handle_discover": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_delayed": 0, 2024-03-30T04:16:53.029 INFO:teuthology.orchestra.run.smithi193.stdout: "num_strays_enqueuing": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_created": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_enqueued": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_reintegrated": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "strays_migrated": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_processing": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_enqueued": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "num_recovering_prioritized": 0, 2024-03-30T04:16:53.030 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_started": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "recovery_completed": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_enqueue_scrub": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_exportdir": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_flush": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragmentdir": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_fragstats": 0, 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: "ireq_inodestats": 0 2024-03-30T04:16:53.031 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_log": { 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "evadd": 0, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "evex": 0, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "evtrm": 0, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "ev": 225456, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "evexg": 0, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "evexd": 0, 2024-03-30T04:16:53.032 INFO:teuthology.orchestra.run.smithi193.stdout: "segadd": 0, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "segex": 0, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "segtrm": 0, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "seg": 129, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "segexg": 0, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "segexd": 0, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "expos": 708845943, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "wrpos": 1228581800, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "rdpos": 1228577471, 2024-03-30T04:16:53.033 INFO:teuthology.orchestra.run.smithi193.stdout: "jlat": { 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "replayed": 225456 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_mem": { 2024-03-30T04:16:53.034 INFO:teuthology.orchestra.run.smithi193.stdout: "ino": 1630, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "ino+": 50938, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "ino-": 49308, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dir": 149, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dir+": 258, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dir-": 109, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dn": 51352, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dn+": 51480, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "dn-": 128, 2024-03-30T04:16:53.035 INFO:teuthology.orchestra.run.smithi193.stdout: "cap": 0, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "cap+": 0, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "cap-": 0, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "rss": 101368, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "heap": 207156 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_server": { 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_client_request": 0, 2024-03-30T04:16:53.036 INFO:teuthology.orchestra.run.smithi193.stdout: "dispatch_server_request": 0, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_request": 0, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_client_session": 0, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "handle_peer_request": 0, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "req_create_latency": { 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.037 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getattr_latency": { 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getfilelock_latency": { 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.038 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "req_link_latency": { 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookup_latency": { 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.039 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookuphash_latency": { 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupino_latency": { 2024-03-30T04:16:53.040 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupname_latency": { 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.041 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupparent_latency": { 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lookupsnap_latency": { 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.042 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "req_lssnap_latency": { 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mkdir_latency": { 2024-03-30T04:16:53.043 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mknod_latency": { 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.044 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "req_mksnap_latency": { 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "req_open_latency": { 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.045 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "req_readdir_latency": { 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.046 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rename_latency": { 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "req_renamesnap_latency": { 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.047 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "req_snapdiff_latency": { 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmdir_latency": { 2024-03-30T04:16:53.048 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmsnap_latency": { 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.049 INFO:teuthology.orchestra.run.smithi193.stdout: "req_rmxattr_latency": { 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setattr_latency": { 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.050 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setdirlayout_latency": { 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setfilelock_latency": { 2024-03-30T04:16:53.051 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setlayout_latency": { 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.052 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "req_setxattr_latency": { 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "req_symlink_latency": { 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.053 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "req_unlink_latency": { 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_revoke_eviction": 0, 2024-03-30T04:16:53.054 INFO:teuthology.orchestra.run.smithi193.stdout: "cap_acquisition_throttle": 0, 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "req_getvxattr_latency": { 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_sessions": { 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "session_count": 1, 2024-03-30T04:16:53.055 INFO:teuthology.orchestra.run.smithi193.stdout: "session_add": 2, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "session_remove": 1, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_open": 1, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "sessions_stale": 0, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "total_load": 0, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "average_load": 0, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "avg_session_uptime": 2922, 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: "mdthresh_evicted": 0 2024-03-30T04:16:53.056 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "mempool": { 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_bytes": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bloom_filter_items": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_bytes": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_alloc_items": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_bytes": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_data_items": 0, 2024-03-30T04:16:53.057 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_onode_items": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_meta_items": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_bytes": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_other_items": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_cache_buffer_items": 0, 2024-03-30T04:16:53.058 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_bytes": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_extent_items": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_bytes": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_blob_items": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_shared_blob_items": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_inline_bl_items": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_bytes": 0, 2024-03-30T04:16:53.059 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_fsck_items": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_bytes": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_txc_items": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_deferred_items": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_bytes": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluestore_writing_items": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_bytes": 0, 2024-03-30T04:16:53.060 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_items": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_bytes": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_reader_items": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_bytes": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "bluefs_file_writer_items": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_bytes": 295497, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_anon_items": 1962, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_bytes": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "buffer_meta_items": 0, 2024-03-30T04:16:53.061 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_bytes": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_items": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_bytes": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_mapbl_items": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_bytes": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_pglog_items": 0, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_bytes": 9848, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_items": 207, 2024-03-30T04:16:53.062 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_bytes": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "osdmap_mapping_items": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_bytes": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "pgmap_items": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_bytes": 40455004, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "mds_co_items": 250629, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_bytes": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_1_items": 0, 2024-03-30T04:16:53.063 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_bytes": 0, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "unittest_2_items": 0 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "objecter": { 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "op_active": 0, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "op_laggy": 0, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send": 4354, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "op_send_bytes": 21, 2024-03-30T04:16:53.064 INFO:teuthology.orchestra.run.smithi193.stdout: "op_resend": 3, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "op_reply": 4354, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "op_latency": { 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4354, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 5.768909752, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.001324967 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "op_inflight": 0, 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "oplen_avg": { 2024-03-30T04:16:53.065 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 4354, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 4355 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "op": 4354, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "op_r": 4354, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "op_w": 0, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "op_rmw": 0, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "op_pg": 0, 2024-03-30T04:16:53.066 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_stat": 2336, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_create": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_read": 2016, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_write": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writefull": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_writesame": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_append": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_zero": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_truncate": 0, 2024-03-30T04:16:53.067 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_delete": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_mapext": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_sparse_read": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_clonerange": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_getxattr": 1, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_setxattr": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_cmpxattr": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_rmxattr": 0, 2024-03-30T04:16:53.068 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_resetxattrs": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_call": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_watch": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_notify": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_src_cmpxattr": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls": 0, 2024-03-30T04:16:53.069 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_pgls_filter": 0, 2024-03-30T04:16:53.070 INFO:teuthology.orchestra.run.smithi193.stdout: "osdop_other": 0, 2024-03-30T04:16:53.070 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_active": 0, 2024-03-30T04:16:53.070 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_send": 0, 2024-03-30T04:16:53.070 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_resend": 0, 2024-03-30T04:16:53.070 INFO:teuthology.orchestra.run.smithi193.stdout: "linger_ping": 0, 2024-03-30T04:16:53.072 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_active": 0, 2024-03-30T04:16:53.072 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_send": 0, 2024-03-30T04:16:53.072 INFO:teuthology.orchestra.run.smithi193.stdout: "poolop_resend": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_active": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_send": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "poolstat_resend": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_active": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_send": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "statfs_resend": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "command_active": 0, 2024-03-30T04:16:53.073 INFO:teuthology.orchestra.run.smithi193.stdout: "command_send": 0, 2024-03-30T04:16:53.074 INFO:teuthology.orchestra.run.smithi193.stdout: "command_resend": 0, 2024-03-30T04:16:53.074 INFO:teuthology.orchestra.run.smithi193.stdout: "map_epoch": 107, 2024-03-30T04:16:53.074 INFO:teuthology.orchestra.run.smithi193.stdout: "map_full": 0, 2024-03-30T04:16:53.074 INFO:teuthology.orchestra.run.smithi193.stdout: "map_inc": 23, 2024-03-30T04:16:53.074 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_sessions": 10, 2024-03-30T04:16:53.079 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_open": 10, 2024-03-30T04:16:53.080 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_session_close": 0, 2024-03-30T04:16:53.080 INFO:teuthology.orchestra.run.smithi193.stdout: "osd_laggy": 0, 2024-03-30T04:16:53.080 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_wr": 0, 2024-03-30T04:16:53.081 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_rd": 2, 2024-03-30T04:16:53.081 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_del": 0 2024-03-30T04:16:53.081 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.081 INFO:teuthology.orchestra.run.smithi193.stdout: "oft": { 2024-03-30T04:16:53.082 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_objs": 0, 2024-03-30T04:16:53.082 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_kv_pairs": 0, 2024-03-30T04:16:53.082 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_updates": 0, 2024-03-30T04:16:53.082 INFO:teuthology.orchestra.run.smithi193.stdout: "omap_total_removes": 0 2024-03-30T04:16:53.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.083 INFO:teuthology.orchestra.run.smithi193.stdout: "osd-slow-ops": { 2024-03-30T04:16:53.083 INFO:teuthology.orchestra.run.smithi193.stdout: "slow_ops_count": 0 2024-03-30T04:16:53.083 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.084 INFO:teuthology.orchestra.run.smithi193.stdout: "purge_queue": { 2024-03-30T04:16:53.084 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops": 0, 2024-03-30T04:16:53.084 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_ops_high_water": 0, 2024-03-30T04:16:53.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing": 0, 2024-03-30T04:16:53.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executing_high_water": 0, 2024-03-30T04:16:53.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_executed": 0, 2024-03-30T04:16:53.085 INFO:teuthology.orchestra.run.smithi193.stdout: "pq_item_in_journal": 0 2024-03-30T04:16:53.086 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.086 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-30T04:16:53.086 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:53.087 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:53.087 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:53.087 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 4683, 2024-03-30T04:16:53.087 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 1226921188, 2024-03-30T04:16:53.088 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:53.088 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 4683, 2024-03-30T04:16:53.088 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:53.088 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:53.089 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4683, 2024-03-30T04:16:53.089 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1226921188, 2024-03-30T04:16:53.089 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:53.090 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.090 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.090 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.090 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.091 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.091 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_bytes": { 2024-03-30T04:16:53.091 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:53.091 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 104857600, 2024-03-30T04:16:53.092 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:53.092 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:53.092 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:53.093 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:53.093 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:53.093 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 4354, 2024-03-30T04:16:53.093 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 1224393383, 2024-03-30T04:16:53.094 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4354, 2024-03-30T04:16:53.094 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 1224393383, 2024-03-30T04:16:53.094 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:53.094 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.095 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.095 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.095 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.095 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.096 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-objecter_ops": { 2024-03-30T04:16:53.096 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:53.096 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 1024, 2024-03-30T04:16:53.097 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:53.097 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:53.097 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:53.097 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:53.098 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:53.098 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 4354, 2024-03-30T04:16:53.098 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 4354, 2024-03-30T04:16:53.098 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 4354, 2024-03-30T04:16:53.099 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 4354, 2024-03-30T04:16:53.099 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:53.099 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.099 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.100 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.100 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.100 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.100 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle": { 2024-03-30T04:16:53.101 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:53.101 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:53.101 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:53.102 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:53.102 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:53.102 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:53.102 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:53.103 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:53.103 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:53.103 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:53.103 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:53.104 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:53.104 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.104 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.105 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.105 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.105 INFO:teuthology.orchestra.run.smithi193.stdout: }, 2024-03-30T04:16:53.105 INFO:teuthology.orchestra.run.smithi193.stdout: "throttle-write_buf_throttle-0x561aedfa0500": { 2024-03-30T04:16:53.106 INFO:teuthology.orchestra.run.smithi193.stdout: "val": 0, 2024-03-30T04:16:53.106 INFO:teuthology.orchestra.run.smithi193.stdout: "max": 3758096384, 2024-03-30T04:16:53.106 INFO:teuthology.orchestra.run.smithi193.stdout: "get_started": 0, 2024-03-30T04:16:53.106 INFO:teuthology.orchestra.run.smithi193.stdout: "get": 0, 2024-03-30T04:16:53.107 INFO:teuthology.orchestra.run.smithi193.stdout: "get_sum": 0, 2024-03-30T04:16:53.107 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_fail": 0, 2024-03-30T04:16:53.107 INFO:teuthology.orchestra.run.smithi193.stdout: "get_or_fail_success": 0, 2024-03-30T04:16:53.108 INFO:teuthology.orchestra.run.smithi193.stdout: "take": 0, 2024-03-30T04:16:53.108 INFO:teuthology.orchestra.run.smithi193.stdout: "take_sum": 0, 2024-03-30T04:16:53.108 INFO:teuthology.orchestra.run.smithi193.stdout: "put": 0, 2024-03-30T04:16:53.108 INFO:teuthology.orchestra.run.smithi193.stdout: "put_sum": 0, 2024-03-30T04:16:53.109 INFO:teuthology.orchestra.run.smithi193.stdout: "wait": { 2024-03-30T04:16:53.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgcount": 0, 2024-03-30T04:16:53.109 INFO:teuthology.orchestra.run.smithi193.stdout: "sum": 0.000000000, 2024-03-30T04:16:53.109 INFO:teuthology.orchestra.run.smithi193.stdout: "avgtime": 0.000000000 2024-03-30T04:16:53.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.110 INFO:teuthology.orchestra.run.smithi193.stdout: } 2024-03-30T04:16:53.110 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-30T04:16:53.366 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-30T04:16:53.366 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-30T04:16:53.366 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-30T04:16:53.366 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-30T04:16:53.366 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-30T04:16:53.367 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-30T04:16:53.377 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-30T04:16:53.378 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T04:16:53.378 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T04:16:53.427 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-30T04:16:53.427 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-30T04:16:53.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:53 smithi193 ceph-mon[79581]: pgmap v469: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 17 op/s 2024-03-30T04:16:53.513 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:16:53.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:53 smithi023 ceph-mon[38254]: pgmap v469: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 17 op/s 2024-03-30T04:16:53.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:53 smithi154 ceph-mon[79161]: pgmap v469: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 17 op/s 2024-03-30T04:16:53.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1305313776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0100e80 msgr2=0x7f9cc0101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:53.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1305313776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0100e80 0x7f9cc0101260 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0009990 tx=0x7f9cb0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1305313776 shutdown_connections 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1305313776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc0105fa0 0x7f9cc010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1305313776 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0100e80 0x7f9cc0101260 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1305313776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0101830 0x7f9cc0105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1305313776 >> 172.21.15.23:0/1305313776 conn(0x7f9cc00fc720 msgr2=0x7f9cc00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1305313776 shutdown_connections 2024-03-30T04:16:53.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.729+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1305313776 wait complete. 2024-03-30T04:16:53.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.730+0000 7f9cc4e00700 1 Processor -- start 2024-03-30T04:16:53.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 -- start start 2024-03-30T04:16:53.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0100e80 0x7f9cc0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:53.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 0x7f9cc019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc019c500 con 0x7f9cc0105fa0 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc019c670 con 0x7f9cc0100e80 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cc4e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc019c830 con 0x7f9cc019ba80 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cbd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cbe800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 0x7f9cc019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cbd400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42738/0 (socket says 172.21.15.23:42738) 2024-03-30T04:16:53.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cbd400700 1 -- 172.21.15.23:0/1651167515 learned_addr learned my addr 172.21.15.23:0/1651167515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.731+0000 7f9cbde00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0100e80 0x7f9cc0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 -- 172.21.15.23:0/1651167515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0100e80 msgr2=0x7f9cc0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0100e80 0x7f9cc0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 -- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 msgr2=0x7f9cc0197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 -- 172.21.15.23:0/1651167515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb00095f0 con 0x7f9cc019ba80 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbd400700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cbe800700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 0x7f9cc019bf00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800efc0 tx=0x7f9cb800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:53.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8018070 con 0x7f9cc019ba80 2024-03-30T04:16:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc01a3d30 con 0x7f9cc019ba80 2024-03-30T04:16:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cb800f040 con 0x7f9cc019ba80 2024-03-30T04:16:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb800cd40 con 0x7f9cc019ba80 2024-03-30T04:16:53.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.732+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc01a4210 con 0x7f9cc019ba80 2024-03-30T04:16:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.733+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc0109210 con 0x7f9cc019ba80 2024-03-30T04:16:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.735+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f9cb8003a00 con 0x7f9cc019ba80 2024-03-30T04:16:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.737+0000 7f9cb6a00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 0x7f9ca40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:53.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.737+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f9cb80ccd70 con 0x7f9cc019ba80 2024-03-30T04:16:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.737+0000 7f9cbde00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 0x7f9ca40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:53.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.737+0000 7f9cbde00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 0x7f9ca40a3750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006010 tx=0x7f9cb000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:53.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.739+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f9cb8095e10 con 0x7f9cc019ba80 2024-03-30T04:16:53.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.996+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cc0102350 con 0x7f9cc019ba80 2024-03-30T04:16:53.996 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:53.998+0000 7f9cb6a00700 1 -- 172.21.15.23:0/1651167515 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+25230 (secure 0 0 0) 0x7f9cb804e2a0 con 0x7f9cc019ba80 2024-03-30T04:16:53.997 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:53.997 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:16:16.205603+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1767310006":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3829741445":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3110970888":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/354023371":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/4064094531":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6801/2266065890":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2266065890":"2024-03-31T04:01:56.045849+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]},{"pool":4,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:16:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.000+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 msgr2=0x7f9ca40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.000+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 0x7f9ca40a3750 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9cb0006010 tx=0x7f9cb000b260 comp rx=0 tx=0).stop 2024-03-30T04:16:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.000+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 msgr2=0x7f9cc019bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:53.999 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.000+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 0x7f9cc019bf00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800efc0 tx=0x7f9cb800c3a0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 shutdown_connections 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cc019ba80 0x7f9cc019bf00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f9ca40a1290 0x7f9ca40a3750 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc0100e80 0x7f9cc0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 --2- 172.21.15.23:0/1651167515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cc0105fa0 0x7f9cc0197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 >> 172.21.15.23:0/1651167515 conn(0x7f9cc00fc720 msgr2=0x7f9cc00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 shutdown_connections 2024-03-30T04:16:54.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.001+0000 7f9cc4e00700 1 -- 172.21.15.23:0/1651167515 wait complete. 2024-03-30T04:16:54.019 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-30T03:56:12.826164+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-30T03:59:38.851515+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '79', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-30T03:59:39.896921+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-30T03:59:42.968880+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 25, 'snap_epoch': 106, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-30T04:16:54.019 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-30T04:16:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 -- 172.21.15.23:0/3173153117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228101900 msgr2=0x7f3228105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 --2- 172.21.15.23:0/3173153117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228101900 0x7f3228105940 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3214009990 tx=0x7f3214009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 -- 172.21.15.23:0/3173153117 shutdown_connections 2024-03-30T04:16:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 --2- 172.21.15.23:0/3173153117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3228106070 0x7f322810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 --2- 172.21.15.23:0/3173153117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228100f50 0x7f3228101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 --2- 172.21.15.23:0/3173153117 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228101900 0x7f3228105940 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 -- 172.21.15.23:0/3173153117 >> 172.21.15.23:0/3173153117 conn(0x7f32280fc7f0 msgr2=0x7f32280fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 -- 172.21.15.23:0/3173153117 shutdown_connections 2024-03-30T04:16:54.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.225+0000 7f3230000700 1 -- 172.21.15.23:0/3173153117 wait complete. 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.226+0000 7f3230000700 1 Processor -- start 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 -- start start 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228106070 0x7f3228199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 0x7f322819e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322819e7a0 con 0x7f3228100f50 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322819e910 con 0x7f322819dd20 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f3230000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322819ead0 con 0x7f3228106070 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f322d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f322d600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42750/0 (socket says 172.21.15.23:42750) 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f322d600700 1 -- 172.21.15.23:0/2792873609 learned_addr learned my addr 172.21.15.23:0/2792873609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f322cc00700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228106070 0x7f3228199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.227+0000 7f322e000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 0x7f322819e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 -- 172.21.15.23:0/2792873609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228106070 msgr2=0x7f3228199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228106070 0x7f3228199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 -- 172.21.15.23:0/2792873609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 msgr2=0x7f322819e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 0x7f322819e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 -- 172.21.15.23:0/2792873609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32140095f0 con 0x7f3228100f50 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322e000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 0x7f322819e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f322d600700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9a0 tx=0x7f321c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c009790 con 0x7f3228100f50 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f321c00cbd0 con 0x7f3228100f50 2024-03-30T04:16:54.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32281a5fd0 con 0x7f3228100f50 2024-03-30T04:16:54.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c018680 con 0x7f3228100f50 2024-03-30T04:16:54.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.228+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32281a6500 con 0x7f3228100f50 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.229+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3228102420 con 0x7f3228100f50 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.231+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f321c00cd40 con 0x7f3228100f50 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.232+0000 7f3226000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 0x7f32100a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.232+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f321c014070 con 0x7f3228100f50 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.233+0000 7f322cc00700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 0x7f32100a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.233+0000 7f322cc00700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 0x7f32100a37d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f322819aa30 tx=0x7f321400b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.237+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f321c0529f0 con 0x7f3228100f50 2024-03-30T04:16:54.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:54 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1651167515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:54.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.456+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3228101900 con 0x7f3228100f50 2024-03-30T04:16:54.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.457+0000 7f3226000700 1 -- 172.21.15.23:0/2792873609 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f321c096310 con 0x7f3228100f50 2024-03-30T04:16:54.455 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 msgr2=0x7f32100a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 0x7f32100a37d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f322819aa30 tx=0x7f321400b260 comp rx=0 tx=0).stop 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 msgr2=0x7f3228199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9a0 tx=0x7f321c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 shutdown_connections 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f32100a1310 0x7f32100a37d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322819dd20 0x7f322819e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3228100f50 0x7f3228199530 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 --2- 172.21.15.23:0/2792873609 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3228106070 0x7f3228199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 >> 172.21.15.23:0/2792873609 conn(0x7f32280fc7f0 msgr2=0x7f32280feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 shutdown_connections 2024-03-30T04:16:54.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.459+0000 7f3230000700 1 -- 172.21.15.23:0/2792873609 wait complete. 2024-03-30T04:16:54.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-30T04:16:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:54 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1651167515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 -- 172.21.15.23:0/4222346704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe794101910 msgr2=0x7fe794105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 --2- 172.21.15.23:0/4222346704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe794101910 0x7fe794105950 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe780009990 tx=0x7fe780009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 -- 172.21.15.23:0/4222346704 shutdown_connections 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 --2- 172.21.15.23:0/4222346704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794106080 0x7fe79410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 --2- 172.21.15.23:0/4222346704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794100f60 0x7fe794101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 --2- 172.21.15.23:0/4222346704 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe794101910 0x7fe794105950 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 -- 172.21.15.23:0/4222346704 >> 172.21.15.23:0/4222346704 conn(0x7fe7940fc7c0 msgr2=0x7fe7940febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 -- 172.21.15.23:0/4222346704 shutdown_connections 2024-03-30T04:16:54.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.682+0000 7fe79c200700 1 -- 172.21.15.23:0/4222346704 wait complete. 2024-03-30T04:16:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 Processor -- start 2024-03-30T04:16:54.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 -- start start 2024-03-30T04:16:54.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:54 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1651167515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:54.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 0x7fe7941994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 0x7fe7941999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe79419dd90 0x7fe79419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79419e810 con 0x7fe79419dd90 2024-03-30T04:16:54.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79419e980 con 0x7fe794106080 2024-03-30T04:16:54.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.683+0000 7fe79c200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79419eb40 con 0x7fe794100f60 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe798e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 0x7fe7941999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 0x7fe7941994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe798e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 0x7fe7941999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47934/0 (socket says 172.21.15.23:47934) 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe79a200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe79419dd90 0x7fe79419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe798e00700 1 -- 172.21.15.23:0/1396323970 learned_addr learned my addr 172.21.15.23:0/1396323970 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 -- 172.21.15.23:0/1396323970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 msgr2=0x7fe7941999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 0x7fe7941999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 -- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe79419dd90 msgr2=0x7fe79419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe79419dd90 0x7fe79419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.684+0000 7fe799800700 1 -- 172.21.15.23:0/1396323970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800095f0 con 0x7fe794100f60 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe799800700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 0x7fe7941994b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe78800e9f0 tx=0x7fe78800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78800c9c0 con 0x7fe794100f60 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7941a60a0 con 0x7fe794100f60 2024-03-30T04:16:54.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe788010430 con 0x7fe794100f60 2024-03-30T04:16:54.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe788018680 con 0x7fe794100f60 2024-03-30T04:16:54.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.685+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7941a65d0 con 0x7fe794100f60 2024-03-30T04:16:54.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.686+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe794102430 con 0x7fe794100f60 2024-03-30T04:16:54.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.688+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe788010860 con 0x7fe794100f60 2024-03-30T04:16:54.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.689+0000 7fe792000700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 0x7fe77c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:54.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.689+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7fe788014070 con 0x7fe794100f60 2024-03-30T04:16:54.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.690+0000 7fe798e00700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 0x7fe77c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:54.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.690+0000 7fe798e00700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 0x7fe77c0a3750 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe79419aa00 tx=0x7fe780005fd0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:54.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.691+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe788096410 con 0x7fe794100f60 2024-03-30T04:16:54.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.909+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe79419a7b0 con 0x7fe794100f60 2024-03-30T04:16:54.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.910+0000 7fe792000700 1 -- 172.21.15.23:0/1396323970 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe788099a30 con 0x7fe794100f60 2024-03-30T04:16:54.908 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:16:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.912+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 msgr2=0x7fe77c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 0x7fe77c0a3750 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe79419aa00 tx=0x7fe780005fd0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 msgr2=0x7fe7941994b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 0x7fe7941994b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fe78800e9f0 tx=0x7fe78800ed00 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 shutdown_connections 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe79419dd90 0x7fe79419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe794100f60 0x7fe7941994b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe77c0a1290 0x7fe77c0a3750 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 --2- 172.21.15.23:0/1396323970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe794106080 0x7fe7941999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 >> 172.21.15.23:0/1396323970 conn(0x7fe7940fc7c0 msgr2=0x7fe7940feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:54.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 shutdown_connections 2024-03-30T04:16:54.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:54.913+0000 7fe79c200700 1 -- 172.21.15.23:0/1396323970 wait complete. 2024-03-30T04:16:54.930 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 -- 172.21.15.23:0/3977531414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc101910 msgr2=0x7f2edc105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/3977531414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc101910 0x7f2edc105950 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009990 tx=0x7f2ec8009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 -- 172.21.15.23:0/3977531414 shutdown_connections 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/3977531414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc106080 0x7f2edc10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/3977531414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc100f60 0x7f2edc101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/3977531414 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc101910 0x7f2edc105950 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 -- 172.21.15.23:0/3977531414 >> 172.21.15.23:0/3977531414 conn(0x7f2edc0fc7c0 msgr2=0x7f2edc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:55.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 -- 172.21.15.23:0/3977531414 shutdown_connections 2024-03-30T04:16:55.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.135+0000 7f2ee1c00700 1 -- 172.21.15.23:0/3977531414 wait complete. 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 Processor -- start 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 -- start start 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc100f60 0x7f2edc0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc073b30 0x7f2edc1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1a5a60 con 0x7f2edc106080 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1a5bd0 con 0x7f2edc073b30 2024-03-30T04:16:55.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.136+0000 7f2ee1c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1a5d90 con 0x7f2edc100f60 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42778/0 (socket says 172.21.15.23:42778) 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 -- 172.21.15.23:0/177605847 learned_addr learned my addr 172.21.15.23:0/177605847 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2edb400700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc073b30 0x7f2edc1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2edaa00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc100f60 0x7f2edc0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 -- 172.21.15.23:0/177605847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc100f60 msgr2=0x7f2edc0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc100f60 0x7f2edc0730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 -- 172.21.15.23:0/177605847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc073b30 msgr2=0x7f2edc1a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc073b30 0x7f2edc1a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 -- 172.21.15.23:0/177605847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80095f0 con 0x7f2edc106080 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.137+0000 7f2eda000700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80059d0 tx=0x7f2ec8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.138+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec801d070 con 0x7f2edc106080 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.138+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2ec8022470 con 0x7f2edc106080 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.138+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8017650 con 0x7f2edc106080 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.138+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc1a6010 con 0x7f2edc106080 2024-03-30T04:16:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.138+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc1a63e0 con 0x7f2edc106080 2024-03-30T04:16:55.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.139+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc109350 con 0x7f2edc106080 2024-03-30T04:16:55.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.141+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f2ec80225e0 con 0x7f2edc106080 2024-03-30T04:16:55.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.143+0000 7f2ed3400700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 0x7f2ec00a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.143+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f2ec80cea30 con 0x7f2edc106080 2024-03-30T04:16:55.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.143+0000 7f2edaa00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 0x7f2ec00a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.144+0000 7f2edaa00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 0x7f2ec00a37d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009c60 tx=0x7f2ecc009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:55.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.146+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f2ec8097ad0 con 0x7f2edc106080 2024-03-30T04:16:55.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.370+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2edc067aa0 con 0x7f2edc106080 2024-03-30T04:16:55.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.370+0000 7f2ed3400700 1 -- 172.21.15.23:0/177605847 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ec809b0f0 con 0x7f2edc106080 2024-03-30T04:16:55.369 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-30T04:16:55.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.372+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 msgr2=0x7f2ec00a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.372+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 0x7f2ec00a37d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009c60 tx=0x7f2ecc009260 comp rx=0 tx=0).stop 2024-03-30T04:16:55.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.372+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 msgr2=0x7f2edc0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.372+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80059d0 tx=0x7f2ec8004730 comp rx=0 tx=0).stop 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 shutdown_connections 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f2ec00a1310 0x7f2ec00a37d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2edc073b30 0x7f2edc1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2edc100f60 0x7f2edc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 --2- 172.21.15.23:0/177605847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2edc106080 0x7f2edc0735f0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 >> 172.21.15.23:0/177605847 conn(0x7f2edc0fc7c0 msgr2=0x7f2edc0feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:55.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 shutdown_connections 2024-03-30T04:16:55.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.373+0000 7f2ee1c00700 1 -- 172.21.15.23:0/177605847 wait complete. 2024-03-30T04:16:55.390 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-30T04:16:55.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:55 smithi023 ceph-mon[38254]: pgmap v470: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2792873609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:55 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1396323970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 -- 172.21.15.23:0/778771666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 msgr2=0x7f31a4101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 --2- 172.21.15.23:0/778771666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4101340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 -- 172.21.15.23:0/778771666 shutdown_connections 2024-03-30T04:16:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 --2- 172.21.15.23:0/778771666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 --2- 172.21.15.23:0/778771666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4101340 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 --2- 172.21.15.23:0/778771666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a4101910 0x7f31a4105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 -- 172.21.15.23:0/778771666 >> 172.21.15.23:0/778771666 conn(0x7f31a40fc7c0 msgr2=0x7f31a40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:55.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 -- 172.21.15.23:0/778771666 shutdown_connections 2024-03-30T04:16:55.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.595+0000 7f31aae00700 1 -- 172.21.15.23:0/778771666 wait complete. 2024-03-30T04:16:55.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 Processor -- start 2024-03-30T04:16:55.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 -- start start 2024-03-30T04:16:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a4199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 0x7f31a419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a419e750 con 0x7f31a4106080 2024-03-30T04:16:55.599 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a419e8c0 con 0x7f31a419dcd0 2024-03-30T04:16:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.597+0000 7f31aae00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a419ea80 con 0x7f31a4100f60 2024-03-30T04:16:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a4199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 0x7f31a419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.600 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a8e00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 0x7f31a419e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.23:47970/0 (socket says 172.21.15.23:47970) 2024-03-30T04:16:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a8e00700 1 -- 172.21.15.23:0/110803164 learned_addr learned my addr 172.21.15.23:0/110803164 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 -- 172.21.15.23:0/110803164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 msgr2=0x7f31a4199440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4199440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.601 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 -- 172.21.15.23:0/110803164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 msgr2=0x7f31a419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 0x7f31a419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 -- 172.21.15.23:0/110803164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31940095f0 con 0x7f31a4106080 2024-03-30T04:16:55.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.598+0000 7f31a3400700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a4199980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f319000e9c0 tx=0x7f319000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:55.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.599+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f319000c9c0 con 0x7f31a4106080 2024-03-30T04:16:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.599+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3190010430 con 0x7f31a4106080 2024-03-30T04:16:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.599+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3190018680 con 0x7f31a4106080 2024-03-30T04:16:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.599+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a41a5fe0 con 0x7f31a4106080 2024-03-30T04:16:55.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.599+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a41a6460 con 0x7f31a4106080 2024-03-30T04:16:55.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.600+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4102430 con 0x7f31a4106080 2024-03-30T04:16:55.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.602+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f319000cb20 con 0x7f31a4106080 2024-03-30T04:16:55.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.603+0000 7f31a0c00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 0x7f31880a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:55.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.603+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f3190014070 con 0x7f31a4106080 2024-03-30T04:16:55.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.603+0000 7f31a3e00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 0x7f31880a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:55.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.604+0000 7f31a3e00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 0x7f31880a37d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f319400b2e0 tx=0x7f3194005960 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:55.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.607+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f31900529f0 con 0x7f31a4106080 2024-03-30T04:16:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:55 smithi154 ceph-mon[79161]: pgmap v470: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:55.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:55 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2792873609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:55 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1396323970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:55 smithi193 ceph-mon[79581]: pgmap v470: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2792873609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:55 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1396323970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-30T04:16:55.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.822+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f31a4101910 con 0x7f31a4106080 2024-03-30T04:16:55.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.822+0000 7f31a0c00700 1 -- 172.21.15.23:0/110803164 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v107) v1 ==== 103+0+11 (secure 0 0 0) 0x7f31900961a0 con 0x7f31a4106080 2024-03-30T04:16:55.821 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-30T04:16:55.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.824+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 msgr2=0x7f31880a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.824+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 0x7f31880a37d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f319400b2e0 tx=0x7f3194005960 comp rx=0 tx=0).stop 2024-03-30T04:16:55.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.824+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 msgr2=0x7f31a4199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.824+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a4199980 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f319000e9c0 tx=0x7f319000ecd0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 shutdown_connections 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f31880a1310 0x7f31880a37d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31a419dcd0 0x7f31a419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a4100f60 0x7f31a4199440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 --2- 172.21.15.23:0/110803164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f31a4106080 0x7f31a4199980 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:55.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 >> 172.21.15.23:0/110803164 conn(0x7f31a40fc7c0 msgr2=0x7f31a40fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 shutdown_connections 2024-03-30T04:16:55.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:55.825+0000 7f31aae00700 1 -- 172.21.15.23:0/110803164 wait complete. 2024-03-30T04:16:55.842 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-30T04:16:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 -- 172.21.15.23:0/2012307276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 msgr2=0x7f9638101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 --2- 172.21.15.23:0/2012307276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f9638101340 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f9628009990 tx=0x7f9628009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 -- 172.21.15.23:0/2012307276 shutdown_connections 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 --2- 172.21.15.23:0/2012307276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638106080 0x7f963810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 --2- 172.21.15.23:0/2012307276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f9638101340 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 --2- 172.21.15.23:0/2012307276 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638101910 0x7f9638105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 -- 172.21.15.23:0/2012307276 >> 172.21.15.23:0/2012307276 conn(0x7f96380fc7c0 msgr2=0x7f96380febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 -- 172.21.15.23:0/2012307276 shutdown_connections 2024-03-30T04:16:56.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.044+0000 7f963d800700 1 -- 172.21.15.23:0/2012307276 wait complete. 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.045+0000 7f963d800700 1 Processor -- start 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.045+0000 7f963d800700 1 -- start start 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638106080 0x7f96380735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638073b30 0x7f96381a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96381a5a30 con 0x7f9638100f60 2024-03-30T04:16:56.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96381a5b70 con 0x7f9638106080 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f963d800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96381a5d30 con 0x7f9638073b30 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42830/0 (socket says 172.21.15.23:42830) 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 -- 172.21.15.23:0/651352895 learned_addr learned my addr 172.21.15.23:0/651352895 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9637400700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638073b30 0x7f96381a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636000700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638106080 0x7f96380735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 -- 172.21.15.23:0/651352895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638073b30 msgr2=0x7f96381a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638073b30 0x7f96381a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 -- 172.21.15.23:0/651352895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638106080 msgr2=0x7f96380735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638106080 0x7f96380735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.046+0000 7f9636a00700 1 -- 172.21.15.23:0/651352895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280095f0 con 0x7f9638100f60 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f9636a00700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9628004730 tx=0x7f9628004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962801d070 con 0x7f9638100f60 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9628022470 con 0x7f9638100f60 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628017650 con 0x7f9638100f60 2024-03-30T04:16:56.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96381a5fb0 con 0x7f9638100f60 2024-03-30T04:16:56.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.047+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96381a6320 con 0x7f9638100f60 2024-03-30T04:16:56.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.048+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96381092f0 con 0x7f9638100f60 2024-03-30T04:16:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.050+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f96280225e0 con 0x7f9638100f60 2024-03-30T04:16:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.051+0000 7f962f400700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 0x7f961c0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.052+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f96280ce230 con 0x7f9638100f60 2024-03-30T04:16:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.052+0000 7f9636000700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 0x7f961c0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:56.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.052+0000 7f9636000700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 0x7f961c0a3750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9638074790 tx=0x7f9624009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:56.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.055+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f962805d9f0 con 0x7f9638100f60 2024-03-30T04:16:56.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.335+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9638102430 con 0x7f9638100f60 2024-03-30T04:16:56.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.335+0000 7f962f400700 1 -- 172.21.15.23:0/651352895 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v107) v1 ==== 81+0+999 (secure 0 0 0) 0x7f96280972d0 con 0x7f9638100f60 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1767310006 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2151815679 2024-03-31T03:53:51.749812+0000 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/604121731 2024-03-31T03:54:16.197829+0000 2024-03-30T04:16:56.334 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3829741445 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/783324330 2024-03-31T03:53:51.749812+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2471664699 2024-03-31T03:53:51.749812+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2173844842 2024-03-31T03:53:51.749812+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/443135190 2024-03-31T03:54:16.197829+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1161676542 2024-03-31T03:54:16.197829+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3110970888 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1219945839 2024-03-31T03:54:16.197829+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/354023371 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.335 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4064094531 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.336 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/2266065890 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.336 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/2266065890 2024-03-31T04:01:56.045849+0000 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 msgr2=0x7f961c0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 0x7f961c0a3750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9638074790 tx=0x7f9624009260 comp rx=0 tx=0).stop 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 msgr2=0x7f96380730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f9628004730 tx=0x7f9628004760 comp rx=0 tx=0).stop 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 shutdown_connections 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9638073b30 0x7f96381a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9638100f60 0x7f96380730b0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f961c0a1290 0x7f961c0a3750 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 --2- 172.21.15.23:0/651352895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9638106080 0x7f96380735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:56.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 >> 172.21.15.23:0/651352895 conn(0x7f96380fc7c0 msgr2=0x7f96380fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:56.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 shutdown_connections 2024-03-30T04:16:56.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:56.338+0000 7f963d800700 1 -- 172.21.15.23:0/651352895 wait complete. 2024-03-30T04:16:56.338 INFO:teuthology.orchestra.run.smithi023.stderr:listed 17 entries 2024-03-30T04:16:56.355 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-30T04:16:56.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:16:56.355 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-30T04:16:56.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:56 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/177605847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:56 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/110803164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/177605847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:56 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/110803164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:56 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/177605847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:56 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/110803164' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-30T04:16:56.769 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:16:56.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:16:56.769 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-30T04:16:56.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:16:56.837 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-30T04:16:56.878 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-30T04:16:56.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:16:56.880 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-30T04:16:56.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-30T04:16:56.988 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-30T04:16:57.111 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-30T04:16:57.122 INFO:tasks.cephadm:Teardown begin 2024-03-30T04:16:57.123 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-30T04:16:57.123 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:16:57.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.330+0000 7f8e5b600700 1 -- 172.21.15.23:0/2131942653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541026d0 msgr2=0x7f8e5410abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.330+0000 7f8e5b600700 1 --2- 172.21.15.23:0/2131942653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541026d0 0x7f8e5410abb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009990 tx=0x7f8e40009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 -- 172.21.15.23:0/2131942653 shutdown_connections 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 --2- 172.21.15.23:0/2131942653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e5410b1d0 0x7f8e5410d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 --2- 172.21.15.23:0/2131942653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e54101db0 0x7f8e54102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 --2- 172.21.15.23:0/2131942653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e541026d0 0x7f8e5410abb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 -- 172.21.15.23:0/2131942653 >> 172.21.15.23:0/2131942653 conn(0x7f8e540fb5f0 msgr2=0x7f8e540fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:57.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 -- 172.21.15.23:0/2131942653 shutdown_connections 2024-03-30T04:16:57.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.331+0000 7f8e5b600700 1 -- 172.21.15.23:0/2131942653 wait complete. 2024-03-30T04:16:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 Processor -- start 2024-03-30T04:16:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 -- start start 2024-03-30T04:16:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e541026d0 0x7f8e540fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e5410b1d0 0x7f8e54100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e541a5530 con 0x7f8e54101db0 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e541a56a0 con 0x7f8e5410b1d0 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e5b600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e541a5860 con 0x7f8e541026d0 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e58c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.332+0000 7f8e58c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:42846/0 (socket says 172.21.15.23:42846) 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 -- 172.21.15.23:0/3528153314 learned_addr learned my addr 172.21.15.23:0/3528153314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:57.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e53e00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e541026d0 0x7f8e540fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e59600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e5410b1d0 0x7f8e54100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 -- 172.21.15.23:0/3528153314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e541026d0 msgr2=0x7f8e540fff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e541026d0 0x7f8e540fff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 -- 172.21.15.23:0/3528153314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e5410b1d0 msgr2=0x7f8e54100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e5410b1d0 0x7f8e54100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 -- 172.21.15.23:0/3528153314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e400095f0 con 0x7f8e54101db0 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e58c00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9a0 tx=0x7f8e4800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48009790 con 0x7f8e54101db0 2024-03-30T04:16:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.333+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8e48010430 con 0x7f8e54101db0 2024-03-30T04:16:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.334+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e541a5b40 con 0x7f8e54101db0 2024-03-30T04:16:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.334+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48018680 con 0x7f8e54101db0 2024-03-30T04:16:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.334+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e541a60a0 con 0x7f8e54101db0 2024-03-30T04:16:57.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.335+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e541062a0 con 0x7f8e54101db0 2024-03-30T04:16:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.337+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f8e480105a0 con 0x7f8e54101db0 2024-03-30T04:16:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.338+0000 7f8e51600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 0x7f8e3c0a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.338+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f8e48014070 con 0x7f8e54101db0 2024-03-30T04:16:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.338+0000 7f8e53e00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 0x7f8e3c0a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.339+0000 7f8e53e00700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 0x7f8e3c0a37d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8e54101130 tx=0x7f8e4000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.342+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f8e480529f0 con 0x7f8e54101db0 2024-03-30T04:16:57.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:57 smithi023 ceph-mon[38254]: pgmap v471: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:57.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:57 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/651352895' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:16:57.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.608+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e540686d0 con 0x7f8e54101db0 2024-03-30T04:16:57.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.609+0000 7f8e51600700 1 -- 172.21.15.23:0/3528153314 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10931 (secure 0 0 0) 0x7f8e48096270 con 0x7f8e54101db0 2024-03-30T04:16:57.607 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:57.608 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:15:25.335802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:16:57.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 msgr2=0x7f8e3c0a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 0x7f8e3c0a37d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8e54101130 tx=0x7f8e4000b260 comp rx=0 tx=0).stop 2024-03-30T04:16:57.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 msgr2=0x7f8e540ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e9a0 tx=0x7f8e4800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 shutdown_connections 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f8e3c0a1310 0x7f8e3c0a37d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e5410b1d0 0x7f8e54100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8e54101db0 0x7f8e540ffa00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 --2- 172.21.15.23:0/3528153314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e541026d0 0x7f8e540fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 >> 172.21.15.23:0/3528153314 conn(0x7f8e540fb5f0 msgr2=0x7f8e540fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 shutdown_connections 2024-03-30T04:16:57.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.612+0000 7f8e5b600700 1 -- 172.21.15.23:0/3528153314 wait complete. 2024-03-30T04:16:57.613 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-30T04:16:57.630 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-30T04:16:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:57 smithi154 ceph-mon[79161]: pgmap v471: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:57.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:57 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/651352895' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:16:57.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:57 smithi193 ceph-mon[79581]: pgmap v471: 129 pgs: 129 active+clean; 772 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 11 op/s 2024-03-30T04:16:57.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:57 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/651352895' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-30T04:16:57.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 -- 172.21.15.23:0/295189321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f70101db0 msgr2=0x7f4f70102190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 --2- 172.21.15.23:0/295189321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f70101db0 0x7f4f70102190 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4f64009990 tx=0x7f4f64009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 -- 172.21.15.23:0/295189321 shutdown_connections 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 --2- 172.21.15.23:0/295189321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f7010b1d0 0x7f4f7010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 --2- 172.21.15.23:0/295189321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f70101db0 0x7f4f70102190 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 --2- 172.21.15.23:0/295189321 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f701026d0 0x7f4f7010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 -- 172.21.15.23:0/295189321 >> 172.21.15.23:0/295189321 conn(0x7f4f700fb5f0 msgr2=0x7f4f700fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:57.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.830+0000 7f4f77c00700 1 -- 172.21.15.23:0/295189321 shutdown_connections 2024-03-30T04:16:57.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.831+0000 7f4f77c00700 1 -- 172.21.15.23:0/295189321 wait complete. 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.831+0000 7f4f77c00700 1 Processor -- start 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 -- start start 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 0x7f4f700ffa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 0x7f4f70100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701a54c0 con 0x7f4f7010b1d0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701a5630 con 0x7f4f70101db0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f77c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f701a57f0 con 0x7f4f701026d0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f6fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f75c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 0x7f4f70100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f6fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:41100/0 (socket says 172.21.15.23:41100) 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.832+0000 7f4f6fe00700 1 -- 172.21.15.23:0/1055547161 learned_addr learned my addr 172.21.15.23:0/1055547161 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f75200700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 0x7f4f700ffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 -- 172.21.15.23:0/1055547161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 msgr2=0x7f4f700ffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 0x7f4f700ffa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 -- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 msgr2=0x7f4f70100480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 0x7f4f70100480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 -- 172.21.15.23:0/1055547161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640095f0 con 0x7f4f701026d0 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f75200700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 0x7f4f700ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f75c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 0x7f4f70100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6fe00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00e9f0 tx=0x7f4f5c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c00c9c0 con 0x7f4f701026d0 2024-03-30T04:16:57.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f701a5ad0 con 0x7f4f701026d0 2024-03-30T04:16:57.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4f5c010430 con 0x7f4f701026d0 2024-03-30T04:16:57.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.833+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c018680 con 0x7f4f701026d0 2024-03-30T04:16:57.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.834+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f701a5fe0 con 0x7f4f701026d0 2024-03-30T04:16:57.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.835+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f70106260 con 0x7f4f701026d0 2024-03-30T04:16:57.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.837+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f4f5c010860 con 0x7f4f701026d0 2024-03-30T04:16:57.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.838+0000 7f4f6d600700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 0x7f4f580a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:57.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.838+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4f5c014070 con 0x7f4f701026d0 2024-03-30T04:16:57.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.839+0000 7f4f75200700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 0x7f4f580a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:57.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.839+0000 7f4f75200700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 0x7f4f580a3750 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f6400b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:57.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:57.840+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4f5c0966f0 con 0x7f4f701026d0 2024-03-30T04:16:58.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.111+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f700686d0 con 0x7f4f701026d0 2024-03-30T04:16:58.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.112+0000 7f4f6d600700 1 -- 172.21.15.23:0/1055547161 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+10931 (secure 0 0 0) 0x7f4f5c099d10 con 0x7f4f701026d0 2024-03-30T04:16:58.110 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:58.111 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14484,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1931657906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1931657906},{"type":"v1","addr":"172.21.15.23:6835","nonce":1931657906}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14496,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/180927964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":180927964},{"type":"v1","addr":"172.21.15.23:6837","nonce":180927964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-30T03:59:40.432862+0000","modified":"2024-03-30T04:15:25.335802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24533,"mds_2":24509,"mds_3":24508,"mds_4":24476},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14544":{"gid":14544,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/667186519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":667186519},{"type":"v1","addr":"172.21.15.23:6839","nonce":667186519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6841/2954889723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":2954889723},{"type":"v1","addr":"172.21.15.23:6841","nonce":2954889723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24457":{"gid":24457,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/2731954066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":2731954066},{"type":"v1","addr":"172.21.15.193:6833","nonce":2731954066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24476":{"gid":24476,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.154:6833/3615295039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6832","nonce":3615295039},{"type":"v1","addr":"172.21.15.154:6833","nonce":3615295039}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":14,"addr":"172.21.15.193:6835/2492108108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2492108108},{"type":"v1","addr":"172.21.15.193:6835","nonce":2492108108}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24509":{"gid":24509,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":14,"addr":"172.21.15.154:6835/1153342637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6834","nonce":1153342637},{"type":"v1","addr":"172.21.15.154:6835","nonce":1153342637}]},"join_fscid":-1,"export_targets":[0,1,3,4],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24514":{"gid":24514,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6837/1829293538","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1829293538},{"type":"v1","addr":"172.21.15.193:6837","nonce":1829293538}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/606583058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":606583058},{"type":"v1","addr":"172.21.15.193:6839","nonce":606583058}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24533":{"gid":24533,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.154:6837/3317012042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6836","nonce":3317012042},{"type":"v1","addr":"172.21.15.154:6837","nonce":3317012042}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.154:6839/2911311868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6838","nonce":2911311868},{"type":"v1","addr":"172.21.15.154:6839","nonce":2911311868}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 msgr2=0x7f4f580a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 0x7f4f580a3750 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4f64006010 tx=0x7f4f6400b260 comp rx=0 tx=0).stop 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 msgr2=0x7f4f700fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c00e9f0 tx=0x7f4f5c00ed00 comp rx=0 tx=0).stop 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 shutdown_connections 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4f7010b1d0 0x7f4f70100480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4f580a1290 0x7f4f580a3750 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4f70101db0 0x7f4f700ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 --2- 172.21.15.23:0/1055547161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4f701026d0 0x7f4f700fff40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 >> 172.21.15.23:0/1055547161 conn(0x7f4f700fb5f0 msgr2=0x7f4f700fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.114+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 shutdown_connections 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.115+0000 7f4f77c00700 1 -- 172.21.15.23:0/1055547161 wait complete. 2024-03-30T04:16:58.114 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 31 2024-03-30T04:16:58.133 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T04:16:58.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.337+0000 7f4a89000700 1 -- 172.21.15.23:0/1468222769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 msgr2=0x7f4a84101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.337+0000 7f4a89000700 1 --2- 172.21.15.23:0/1468222769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84101240 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009990 tx=0x7f4a74009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.337+0000 7f4a89000700 1 -- 172.21.15.23:0/1468222769 shutdown_connections 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.337+0000 7f4a89000700 1 --2- 172.21.15.23:0/1468222769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a84105f80 0x7f4a8410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.337+0000 7f4a89000700 1 --2- 172.21.15.23:0/1468222769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84101240 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.338+0000 7f4a89000700 1 --2- 172.21.15.23:0/1468222769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84101810 0x7f4a84105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.338+0000 7f4a89000700 1 -- 172.21.15.23:0/1468222769 >> 172.21.15.23:0/1468222769 conn(0x7f4a840fc700 msgr2=0x7f4a840feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.338+0000 7f4a89000700 1 -- 172.21.15.23:0/1468222769 shutdown_connections 2024-03-30T04:16:58.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.338+0000 7f4a89000700 1 -- 172.21.15.23:0/1468222769 wait complete. 2024-03-30T04:16:58.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 Processor -- start 2024-03-30T04:16:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 -- start start 2024-03-30T04:16:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 0x7f4a84199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a8419dc20 0x7f4a8419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419e6a0 con 0x7f4a84100e60 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419e810 con 0x7f4a8419dc20 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a89000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8419e9d0 con 0x7f4a84105f80 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a81e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a81e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57326/0 (socket says 172.21.15.23:57326) 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a81e00700 1 -- 172.21.15.23:0/1436072835 learned_addr learned my addr 172.21.15.23:0/1436072835 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.339+0000 7f4a81400700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 0x7f4a84199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a82800700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a8419dc20 0x7f4a8419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 -- 172.21.15.23:0/1436072835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 msgr2=0x7f4a84199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 0x7f4a84199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 -- 172.21.15.23:0/1436072835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a8419dc20 msgr2=0x7f4a8419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a8419dc20 0x7f4a8419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 -- 172.21.15.23:0/1436072835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a740095f0 con 0x7f4a84100e60 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81400700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 0x7f4a84199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a81e00700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4a74005770 tx=0x7f4a740057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.340+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7401d070 con 0x7f4a84100e60 2024-03-30T04:16:58.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.341+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4a74022470 con 0x7f4a84100e60 2024-03-30T04:16:58.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.341+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a841a5ed0 con 0x7f4a84100e60 2024-03-30T04:16:58.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.341+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a74017650 con 0x7f4a84100e60 2024-03-30T04:16:58.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.341+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a841a62f0 con 0x7f4a84100e60 2024-03-30T04:16:58.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.342+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a84102330 con 0x7f4a84100e60 2024-03-30T04:16:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.344+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7f4a7400fb00 con 0x7f4a84100e60 2024-03-30T04:16:58.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.346+0000 7f4a7aa00700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 0x7f4a680a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.346+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7f4a740ce480 con 0x7f4a84100e60 2024-03-30T04:16:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.346+0000 7f4a81400700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 0x7f4a680a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.347+0000 7f4a81400700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 0x7f4a680a37d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4a8419a890 tx=0x7f4a70009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:58.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.350+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7f4a740976a0 con 0x7f4a84100e60 2024-03-30T04:16:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:58 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3528153314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:58 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1055547161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.607+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a8419a6e0 con 0x7f4a84100e60 2024-03-30T04:16:58.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.608+0000 7f4a7aa00700 1 -- 172.21.15.23:0/1436072835 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+25230 (secure 0 0 0) 0x7f4a74027090 con 0x7f4a84100e60 2024-03-30T04:16:58.607 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-30T04:16:58.607 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":107,"fsid":"eac7e298-ee48-11ee-b647-cb9ed24678a4","created":"2024-03-30T03:53:14.698271+0000","modified":"2024-03-30T04:16:16.205603+0000","last_up_change":"2024-03-30T03:59:02.614339+0000","last_in_change":"2024-03-30T03:58:45.416303+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-30T03:56:12.826164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-30T03:59:38.851515+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-30T03:59:39.896921+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-30T03:59:42.968880+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":25,"snap_epoch":106,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"60b581f8-6bf1-4e71-8ad6-974da9b9db58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6803","nonce":2819496083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6805","nonce":2819496083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6809","nonce":2819496083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":2819496083},{"type":"v1","addr":"172.21.15.23:6807","nonce":2819496083}]},"public_addr":"172.21.15.23:6803/2819496083","cluster_addr":"172.21.15.23:6805/2819496083","heartbeat_back_addr":"172.21.15.23:6809/2819496083","heartbeat_front_addr":"172.21.15.23:6807/2819496083","state":["exists","up"]},{"osd":1,"uuid":"4e38bd8a-9f96-4585-ad8f-a364a12543df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6801","nonce":1863895073}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6803","nonce":1863895073}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6807","nonce":1863895073}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1863895073},{"type":"v1","addr":"172.21.15.154:6805","nonce":1863895073}]},"public_addr":"172.21.15.154:6801/1863895073","cluster_addr":"172.21.15.154:6803/1863895073","heartbeat_back_addr":"172.21.15.154:6807/1863895073","heartbeat_front_addr":"172.21.15.154:6805/1863895073","state":["exists","up"]},{"osd":2,"uuid":"e2244e3c-4dd3-4c06-a796-07187e9a77a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6801","nonce":304660967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6803","nonce":304660967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6807","nonce":304660967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":304660967},{"type":"v1","addr":"172.21.15.193:6805","nonce":304660967}]},"public_addr":"172.21.15.193:6801/304660967","cluster_addr":"172.21.15.193:6803/304660967","heartbeat_back_addr":"172.21.15.193:6807/304660967","heartbeat_front_addr":"172.21.15.193:6805/304660967","state":["exists","up"]},{"osd":3,"uuid":"f3b063ec-ce2c-4142-a8c8-5bca5f8252bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6811","nonce":1020396955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6813","nonce":1020396955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6817","nonce":1020396955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":1020396955},{"type":"v1","addr":"172.21.15.23:6815","nonce":1020396955}]},"public_addr":"172.21.15.23:6811/1020396955","cluster_addr":"172.21.15.23:6813/1020396955","heartbeat_back_addr":"172.21.15.23:6817/1020396955","heartbeat_front_addr":"172.21.15.23:6815/1020396955","state":["exists","up"]},{"osd":4,"uuid":"c90bc547-0492-4582-be42-0b7cdd2f2fb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6809","nonce":2815560551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6811","nonce":2815560551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6815","nonce":2815560551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":2815560551},{"type":"v1","addr":"172.21.15.154:6813","nonce":2815560551}]},"public_addr":"172.21.15.154:6809/2815560551","cluster_addr":"172.21.15.154:6811/2815560551","heartbeat_back_addr":"172.21.15.154:6815/2815560551","heartbeat_front_addr":"172.21.15.154:6813/2815560551","state":["exists","up"]},{"osd":5,"uuid":"93e66232-688d-4fc5-a935-1c4e147cd6d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6809","nonce":1684695578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6811","nonce":1684695578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6815","nonce":1684695578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1684695578},{"type":"v1","addr":"172.21.15.193:6813","nonce":1684695578}]},"public_addr":"172.21.15.193:6809/1684695578","cluster_addr":"172.21.15.193:6811/1684695578","heartbeat_back_addr":"172.21.15.193:6815/1684695578","heartbeat_front_addr":"172.21.15.193:6813/1684695578","state":["exists","up"]},{"osd":6,"uuid":"5c236d02-e504-41f1-be6d-acc36f957d54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6819","nonce":4279410626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6821","nonce":4279410626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6825","nonce":4279410626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":4279410626},{"type":"v1","addr":"172.21.15.23:6823","nonce":4279410626}]},"public_addr":"172.21.15.23:6819/4279410626","cluster_addr":"172.21.15.23:6821/4279410626","heartbeat_back_addr":"172.21.15.23:6825/4279410626","heartbeat_front_addr":"172.21.15.23:6823/4279410626","state":["exists","up"]},{"osd":7,"uuid":"dca94a7d-3a01-47d2-aac6-b633e302111d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6816","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6817","nonce":4282776176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6818","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6819","nonce":4282776176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6822","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6823","nonce":4282776176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6820","nonce":4282776176},{"type":"v1","addr":"172.21.15.154:6821","nonce":4282776176}]},"public_addr":"172.21.15.154:6817/4282776176","cluster_addr":"172.21.15.154:6819/4282776176","heartbeat_back_addr":"172.21.15.154:6823/4282776176","heartbeat_front_addr":"172.21.15.154:6821/4282776176","state":["exists","up"]},{"osd":8,"uuid":"81abadd3-48ab-4d42-abd2-f687f5122279","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6817","nonce":54947619}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6819","nonce":54947619}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6823","nonce":54947619}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":54947619},{"type":"v1","addr":"172.21.15.193:6821","nonce":54947619}]},"public_addr":"172.21.15.193:6817/54947619","cluster_addr":"172.21.15.193:6819/54947619","heartbeat_back_addr":"172.21.15.193:6823/54947619","heartbeat_front_addr":"172.21.15.193:6821/54947619","state":["exists","up"]},{"osd":9,"uuid":"89bd59bf-4452-4a10-b466-f00fab24b496","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":63,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6827","nonce":2836554643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6829","nonce":2836554643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6833","nonce":2836554643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":2836554643},{"type":"v1","addr":"172.21.15.23:6831","nonce":2836554643}]},"public_addr":"172.21.15.23:6827/2836554643","cluster_addr":"172.21.15.23:6829/2836554643","heartbeat_back_addr":"172.21.15.23:6833/2836554643","heartbeat_front_addr":"172.21.15.23:6831/2836554643","state":["exists","up"]},{"osd":10,"uuid":"0803d869-d43c-46f0-9725-be71809628b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6824","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6825","nonce":1797593825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6826","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6827","nonce":1797593825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6830","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6831","nonce":1797593825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6828","nonce":1797593825},{"type":"v1","addr":"172.21.15.154:6829","nonce":1797593825}]},"public_addr":"172.21.15.154:6825/1797593825","cluster_addr":"172.21.15.154:6827/1797593825","heartbeat_back_addr":"172.21.15.154:6831/1797593825","heartbeat_front_addr":"172.21.15.154:6829/1797593825","state":["exists","up"]},{"osd":11,"uuid":"7746428c-6c3a-4801-922d-7dd9000dfe7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6825","nonce":2276508968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6827","nonce":2276508968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6831","nonce":2276508968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":2276508968},{"type":"v1","addr":"172.21.15.193:6829","nonce":2276508968}]},"public_addr":"172.21.15.193:6825/2276508968","cluster_addr":"172.21.15.193:6827/2276508968","heartbeat_back_addr":"172.21.15.193:6831/2276508968","heartbeat_front_addr":"172.21.15.193:6829/2276508968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:34.471058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:55:52.277802+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:09.874482+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:30.685856+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:56:48.294160+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:06.100628+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:27.260903+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:57:45.581514+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:03.126119+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:24.722353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:42.558014+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T03:58:59.958427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.8","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.6","mappings":[{"from":3,"to":6}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":7},{"from":5,"to":8}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":10}]},{"pgid":"4.1b","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:6801/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6800/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1767310006":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2151815679":"2024-03-31T03:53:51.749812+0000","172.21.15.23:6801/604121731":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3829741445":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/783324330":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2471664699":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/2173844842":"2024-03-31T03:53:51.749812+0000","172.21.15.23:0/443135190":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/1161676542":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/3110970888":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/1219945839":"2024-03-31T03:54:16.197829+0000","172.21.15.23:0/354023371":"2024-03-31T04:01:56.045849+0000","172.21.15.23:0/4064094531":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6801/2266065890":"2024-03-31T04:01:56.045849+0000","172.21.15.23:6800/2266065890":"2024-03-31T04:01:56.045849+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]},{"pool":4,"snaps":[{"begin":12,"length":1},{"begin":25,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T04:16:58.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 msgr2=0x7f4a680a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 0x7f4a680a37d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4a8419a890 tx=0x7f4a70009260 comp rx=0 tx=0).stop 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 msgr2=0x7f4a84199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4a74005770 tx=0x7f4a740057a0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 shutdown_connections 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7f4a680a1310 0x7f4a680a37d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a8419dc20 0x7f4a8419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4a84100e60 0x7f4a84199340 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 --2- 172.21.15.23:0/1436072835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4a84105f80 0x7f4a84199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 >> 172.21.15.23:0/1436072835 conn(0x7f4a840fc700 msgr2=0x7f4a840fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:58.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 shutdown_connections 2024-03-30T04:16:58.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.611+0000 7f4a89000700 1 -- 172.21.15.23:0/1436072835 wait complete. 2024-03-30T04:16:58.629 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-30T04:16:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:58 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3528153314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:58 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1055547161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:58 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3528153314' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:58 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1055547161' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-30T04:16:58.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 -- 172.21.15.23:0/3990428249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4101910 msgr2=0x7feed4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 --2- 172.21.15.23:0/3990428249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4101910 0x7feed4105950 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7feec0009990 tx=0x7feec0009ca0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 -- 172.21.15.23:0/3990428249 shutdown_connections 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 --2- 172.21.15.23:0/3990428249 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed4106080 0x7feed410dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 --2- 172.21.15.23:0/3990428249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed4101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 --2- 172.21.15.23:0/3990428249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4101910 0x7feed4105950 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 -- 172.21.15.23:0/3990428249 >> 172.21.15.23:0/3990428249 conn(0x7feed40fc7c0 msgr2=0x7feed40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 -- 172.21.15.23:0/3990428249 shutdown_connections 2024-03-30T04:16:58.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.831+0000 7feedc200700 1 -- 172.21.15.23:0/3990428249 wait complete. 2024-03-30T04:16:58.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.832+0000 7feedc200700 1 Processor -- start 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 -- start start 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 0x7feed4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 0x7feed419e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419e750 con 0x7feed419dcd0 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419e8c0 con 0x7feed4106080 2024-03-30T04:16:58.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feedc200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419ea80 con 0x7feed4100f60 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feeda200700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 0x7feed419e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feed9800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feed9800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:38964/0 (socket says 172.21.15.23:38964) 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feed9800700 1 -- 172.21.15.23:0/961487378 learned_addr learned my addr 172.21.15.23:0/961487378 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.833+0000 7feed8e00700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 0x7feed4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 -- 172.21.15.23:0/961487378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 msgr2=0x7feed4199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 0x7feed4199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 -- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 msgr2=0x7feed419e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:58.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 0x7feed419e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 -- 172.21.15.23:0/961487378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec00095f0 con 0x7feed4100f60 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed8e00700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 0x7feed4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feeda200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 0x7feed419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed9800700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feec800e9a0 tx=0x7feec800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8009790 con 0x7feed4100f60 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feec800cbd0 con 0x7feed4100f60 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed41a5fe0 con 0x7feed4100f60 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.834+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec8018680 con 0x7feed4100f60 2024-03-30T04:16:58.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.835+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed41a6460 con 0x7feed4100f60 2024-03-30T04:16:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.835+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4102430 con 0x7feed4100f60 2024-03-30T04:16:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.838+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7feec800cd40 con 0x7feed4100f60 2024-03-30T04:16:58.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.839+0000 7feed2000700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 0x7feebc0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.839+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9512+0+0 (secure 0 0 0) 0x7feec8014070 con 0x7feed4100f60 2024-03-30T04:16:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.839+0000 7feed8e00700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 0x7feebc0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:58.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.840+0000 7feed8e00700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 0x7feebc0a3750 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feec000b2e0 tx=0x7feec0005fb0 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:58.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:58.841+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7feec8096680 con 0x7feed4100f60 2024-03-30T04:16:59.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.106+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feed4067af0 con 0x7feed4100f60 2024-03-30T04:16:59.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.123+0000 7feed2000700 1 -- 172.21.15.23:0/961487378 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7feec8099ca0 con 0x7feed4100f60 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 msgr2=0x7feebc0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 0x7feebc0a3750 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7feec000b2e0 tx=0x7feec0005fb0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 msgr2=0x7feed41993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7feec800e9a0 tx=0x7feec800ecb0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 shutdown_connections 2024-03-30T04:16:59.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feed419dcd0 0x7feed419e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feed4100f60 0x7feed41993f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7feebc0a1290 0x7feebc0a3750 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 --2- 172.21.15.23:0/961487378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feed4106080 0x7feed4199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 >> 172.21.15.23:0/961487378 conn(0x7feed40fc7c0 msgr2=0x7feed40feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.126+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 shutdown_connections 2024-03-30T04:16:59.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.127+0000 7feedc200700 1 -- 172.21.15.23:0/961487378 wait complete. 2024-03-30T04:16:59.127 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-30T04:16:59.154 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-30T04:16:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 -- 172.21.15.23:0/81085432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184105ed0 msgr2=0x7fe18410da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 --2- 172.21.15.23:0/81085432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184105ed0 0x7fe18410da50 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00b0a0 tx=0x7fe17c00b3b0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 -- 172.21.15.23:0/81085432 shutdown_connections 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 --2- 172.21.15.23:0/81085432 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184105ed0 0x7fe18410da50 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 --2- 172.21.15.23:0/81085432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe184100db0 0x7fe184101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 --2- 172.21.15.23:0/81085432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184101760 0x7fe1841057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 -- 172.21.15.23:0/81085432 >> 172.21.15.23:0/81085432 conn(0x7fe1840fc670 msgr2=0x7fe1840fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 -- 172.21.15.23:0/81085432 shutdown_connections 2024-03-30T04:16:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.369+0000 7fe189a00700 1 -- 172.21.15.23:0/81085432 wait complete. 2024-03-30T04:16:59.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.370+0000 7fe189a00700 1 Processor -- start 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 -- start start 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 0x7fe184197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 0x7fe18419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18419c480 con 0x7fe184100db0 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18419c5f0 con 0x7fe184105ed0 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18419c7b0 con 0x7fe18419ba00 2024-03-30T04:16:59.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe183e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe183e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57396/0 (socket says 172.21.15.23:57396) 2024-03-30T04:16:59.373 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe183e00700 1 -- 172.21.15.23:0/2734476681 learned_addr learned my addr 172.21.15.23:0/2734476681 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe183400700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 0x7fe184197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.371+0000 7fe189000700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 0x7fe18419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 -- 172.21.15.23:0/2734476681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 msgr2=0x7fe18419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 0x7fe18419be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 -- 172.21.15.23:0/2734476681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 msgr2=0x7fe184197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:16:59.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 0x7fe184197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 -- 172.21.15.23:0/2734476681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c009d00 con 0x7fe184100db0 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe189000700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 0x7fe18419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183400700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 0x7fe184197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe183e00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe17400e9a0 tx=0x7fe17400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174009790 con 0x7fe184100db0 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1841a3d10 con 0x7fe184100db0 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe174010430 con 0x7fe184100db0 2024-03-30T04:16:59.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.372+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe174018680 con 0x7fe184100db0 2024-03-30T04:16:59.376 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.373+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1841a41e0 con 0x7fe184100db0 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.374+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1841091a0 con 0x7fe184100db0 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.376+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fe1740105a0 con 0x7fe184100db0 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.377+0000 7fe180c00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 0x7fe1680a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.377+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9854+0+0 (secure 0 0 0) 0x7fe174014070 con 0x7fe184100db0 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.377+0000 7fe183400700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 0x7fe1680a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:16:59.377 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.378+0000 7fe183400700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 0x7fe1680a3750 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe184198670 tx=0x7fe170009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:16:59.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.379+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fe1740964d0 con 0x7fe184100db0 2024-03-30T04:16:59.434 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.434 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.435 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: starting mds.e at 2024-03-30T04:16:59.435 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.435 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.435 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mds[116288]: -- 172.21.15.154:0/3053284540 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ae448c1a0 con 0x559ae4462800 2024-03-30T04:16:59.435 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: starting mds.b at 2024-03-30T04:16:59.436 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.436 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.436 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mds[115943]: -- 172.21.15.154:0/1194563842 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5621e84e41a0 con 0x5621e84bb000 2024-03-30T04:16:59.436 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: starting mds.h at 2024-03-30T04:16:59.436 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.437 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.437 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: starting mds.k at 2024-03-30T04:16:59.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: pgmap v472: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 159 KiB/s wr, 17 op/s 2024-03-30T04:16:59.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/1436072835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:59.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/961487378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: osdmap e108: 12 total, 12 up, 12 in 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T04:16:59.438 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:16:59 smithi154 ceph-mon[79161]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-30T04:16:59.492 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.493 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.493 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mds[116261]: -- 172.21.15.193:0/1053561521 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55965f3f21a0 con 0x55965f3c9000 2024-03-30T04:16:59.493 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: starting mds.c at 2024-03-30T04:16:59.493 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.494 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.494 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mds[115919]: -- 172.21.15.193:0/3735322776 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56523b2c81a0 con 0x56523b29f000 2024-03-30T04:16:59.494 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: starting mds.f at 2024-03-30T04:16:59.494 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.494 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: starting mds.l at 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: pgmap v472: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 159 KiB/s wr, 17 op/s 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/1436072835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/961487378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.495 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: osdmap e108: 12 total, 12 up, 12 in 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mon[79581]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.496 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-mds[116602]: -- 172.21.15.193:0/2004066351 <== mon.2 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe2d61c1a0 con 0x55fe2d5f2c00 2024-03-30T04:16:59.497 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:16:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: starting mds.i at 2024-03-30T04:16:59.521 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.522 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.522 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mds[88311]: -- 172.21.15.23:0/2949201559 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558bfb3de1a0 con 0x558bfb3b7000 2024-03-30T04:16:59.522 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: starting mds.g at 2024-03-30T04:16:59.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a[38231]: 2024-03-30T04:16:59.108+0000 7fe57ac00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: pgmap v472: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 159 KiB/s wr, 17 op/s 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/1436072835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/961487378' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-30T04:16:59.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: osdmap e108: 12 total, 12 up, 12 in 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mon[38254]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: ignoring --setuser ceph since I am not root 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: ignoring --setgroup ceph since I am not root 2024-03-30T04:16:59.524 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-mds[88697]: -- 172.21.15.23:0/3553996277 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565583c21a0 con 0x55655839ac00 2024-03-30T04:16:59.525 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:16:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: starting mds.a at 2024-03-30T04:16:59.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:16:59.650+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe184102280 con 0x7fe184100db0 2024-03-30T04:17:00.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.123+0000 7fe180c00700 1 -- 172.21.15.23:0/2734476681 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe174099af0 con 0x7fe184100db0 2024-03-30T04:17:00.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.127+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 msgr2=0x7fe1680a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:00.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.127+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 0x7fe1680a3750 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe184198670 tx=0x7fe170009260 comp rx=0 tx=0).stop 2024-03-30T04:17:00.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 msgr2=0x7fe184197120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:00.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe17400e9a0 tx=0x7fe17400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 shutdown_connections 2024-03-30T04:17:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe18419ba00 0x7fe18419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe184100db0 0x7fe184197120 secure :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fe17400e9a0 tx=0x7fe17400ecb0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fe1680a1290 0x7fe1680a3750 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 --2- 172.21.15.23:0/2734476681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe184105ed0 0x7fe184197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.128+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 >> 172.21.15.23:0/2734476681 conn(0x7fe1840fc670 msgr2=0x7fe1840fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:00.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.129+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 shutdown_connections 2024-03-30T04:17:00.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.129+0000 7fe189a00700 1 -- 172.21.15.23:0/2734476681 wait complete. 2024-03-30T04:17:00.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-30T04:17:00.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 -- 172.21.15.23:0/866954576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28101920 msgr2=0x7fcf28105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:00.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 --2- 172.21.15.23:0/866954576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28101920 0x7fcf28105960 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009990 tx=0x7fcf14009ca0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 -- 172.21.15.23:0/866954576 shutdown_connections 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 --2- 172.21.15.23:0/866954576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf2810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 --2- 172.21.15.23:0/866954576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 0x7fcf28101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 --2- 172.21.15.23:0/866954576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28101920 0x7fcf28105960 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 -- 172.21.15.23:0/866954576 >> 172.21.15.23:0/866954576 conn(0x7fcf280fc7f0 msgr2=0x7fcf280fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.358+0000 7fcf2f600700 1 -- 172.21.15.23:0/866954576 shutdown_connections 2024-03-30T04:17:00.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.359+0000 7fcf2f600700 1 -- 172.21.15.23:0/866954576 wait complete. 2024-03-30T04:17:00.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.359+0000 7fcf2f600700 1 Processor -- start 2024-03-30T04:17:00.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 -- start start 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 0x7fcf280730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28073b30 0x7fcf281a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf281a5970 con 0x7fcf28106090 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf281a5ae0 con 0x7fcf28073b30 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2f600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf281a5ca0 con 0x7fcf28100f70 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf27e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:00.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf27e00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57414/0 (socket says 172.21.15.23:57414) 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf27e00700 1 -- 172.21.15.23:0/2610755526 learned_addr learned my addr 172.21.15.23:0/2610755526 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2d600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28073b30 0x7fcf281a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.360+0000 7fcf2cc00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 0x7fcf280730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 -- 172.21.15.23:0/2610755526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 msgr2=0x7fcf280730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 0x7fcf280730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 -- 172.21.15.23:0/2610755526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28073b30 msgr2=0x7fcf281a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28073b30 0x7fcf281a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:00.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 -- 172.21.15.23:0/2610755526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140095f0 con 0x7fcf28106090 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf27e00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf14000c00 tx=0x7fcf14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1401d070 con 0x7fcf28106090 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf14021410 con 0x7fcf28106090 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf281a5f20 con 0x7fcf28106090 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14017650 con 0x7fcf28106090 2024-03-30T04:17:00.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.361+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf281a6350 con 0x7fcf28106090 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.363+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf28109360 con 0x7fcf28106090 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.364+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fcf14021580 con 0x7fcf28106090 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.366+0000 7fcf25600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 0x7fcf100a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.366+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 9854+0+0 (secure 0 0 0) 0x7fcf140ce2d0 con 0x7fcf28106090 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.366+0000 7fcf2cc00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 0x7fcf100a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:00.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.366+0000 7fcf2cc00700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 0x7fcf100a37d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009c60 tx=0x7fcf1c009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:00.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.369+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fcf14097280 con 0x7fcf28106090 2024-03-30T04:17:00.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: Cluster is now healthy 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6832/1102732252,v1:172.21.15.193:6834/1102732252] up:boot 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6834/2790266646,v1:172.21.15.154:6835/2790266646] up:boot 2024-03-30T04:17:00.522 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6836/1474773197,v1:172.21.15.154:6838/1474773197] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6837/4281786760,v1:172.21.15.193:6839/4281786760] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6837/2589298609,v1:172.21.15.154:6839/2589298609] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6833/3727113983,v1:172.21.15.193:6835/3727113983] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.154:6832/715107449,v1:172.21.15.154:6833/715107449] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.193:6836/629245101,v1:172.21.15.193:6838/629245101] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6838/1970726382,v1:172.21.15.23:6839/1970726382] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: mds.? [v2:172.21.15.23:6840/643499884,v1:172.21.15.23:6841/643499884] up:boot 2024-03-30T04:17:00.523 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: fsmap 12 up:standby 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:17:00.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:17:00.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:17:00.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:17:00.525 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:00 smithi023 ceph-mon[38254]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:17:00.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:00.592+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcf28074590 con 0x7fcf28106090 2024-03-30T04:17:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T04:17:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T04:17:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T04:17:00.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: Cluster is now healthy 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6832/1102732252,v1:172.21.15.193:6834/1102732252] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6834/2790266646,v1:172.21.15.154:6835/2790266646] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6836/1474773197,v1:172.21.15.154:6838/1474773197] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6837/4281786760,v1:172.21.15.193:6839/4281786760] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6837/2589298609,v1:172.21.15.154:6839/2589298609] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6833/3727113983,v1:172.21.15.193:6835/3727113983] up:boot 2024-03-30T04:17:00.682 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.154:6832/715107449,v1:172.21.15.154:6833/715107449] up:boot 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.193:6836/629245101,v1:172.21.15.193:6838/629245101] up:boot 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6838/1970726382,v1:172.21.15.23:6839/1970726382] up:boot 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: mds.? [v2:172.21.15.23:6840/643499884,v1:172.21.15.23:6841/643499884] up:boot 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: fsmap 12 up:standby 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:17:00.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:17:00.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:00 smithi154 ceph-mon[79161]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:17:00.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: Cluster is now healthy 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6832/1102732252,v1:172.21.15.193:6834/1102732252] up:boot 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6834/2790266646,v1:172.21.15.154:6835/2790266646] up:boot 2024-03-30T04:17:00.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6836/1474773197,v1:172.21.15.154:6838/1474773197] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6837/4281786760,v1:172.21.15.193:6839/4281786760] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6837/2589298609,v1:172.21.15.154:6839/2589298609] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6833/3727113983,v1:172.21.15.193:6835/3727113983] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.154:6832/715107449,v1:172.21.15.154:6833/715107449] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.193:6836/629245101,v1:172.21.15.193:6838/629245101] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6838/1970726382,v1:172.21.15.23:6839/1970726382] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: mds.? [v2:172.21.15.23:6840/643499884,v1:172.21.15.23:6841/643499884] up:boot 2024-03-30T04:17:00.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2734476681' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: fsmap 12 up:standby 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-30T04:17:00.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-30T04:17:00.746 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-30T04:17:00.746 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:00 smithi193 ceph-mon[79581]: from='mgr.15627 172.21.15.23:0/3313983844' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-30T04:17:01.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.274+0000 7fcf25600700 1 -- 172.21.15.23:0/2610755526 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcf1409a8a0 con 0x7fcf28106090 2024-03-30T04:17:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.278+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 msgr2=0x7fcf100a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.278+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 0x7fcf100a37d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009c60 tx=0x7fcf1c009260 comp rx=0 tx=0).stop 2024-03-30T04:17:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.278+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 msgr2=0x7fcf280735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:01.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.278+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fcf14000c00 tx=0x7fcf14004730 comp rx=0 tx=0).stop 2024-03-30T04:17:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 shutdown_connections 2024-03-30T04:17:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fcf100a1310 0x7fcf100a37d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf28073b30 0x7fcf281a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcf28100f70 0x7fcf280730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 --2- 172.21.15.23:0/2610755526 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcf28106090 0x7fcf280735f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 >> 172.21.15.23:0/2610755526 conn(0x7fcf280fc7f0 msgr2=0x7fcf280fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 shutdown_connections 2024-03-30T04:17:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.279+0000 7fcf2f600700 1 -- 172.21.15.23:0/2610755526 wait complete. 2024-03-30T04:17:01.279 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2024-03-30T04:17:01.300 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-30T04:17:01.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.507+0000 7ffb92c00700 1 -- 172.21.15.23:0/818463578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c106080 msgr2=0x7ffb8c10dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.507+0000 7ffb92c00700 1 --2- 172.21.15.23:0/818463578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c106080 0x7ffb8c10dc00 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7ffb8400b0a0 tx=0x7ffb8400b3b0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 -- 172.21.15.23:0/818463578 shutdown_connections 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 --2- 172.21.15.23:0/818463578 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c106080 0x7ffb8c10dc00 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 --2- 172.21.15.23:0/818463578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 --2- 172.21.15.23:0/818463578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c101910 0x7ffb8c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 -- 172.21.15.23:0/818463578 >> 172.21.15.23:0/818463578 conn(0x7ffb8c0fc7c0 msgr2=0x7ffb8c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:01.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 -- 172.21.15.23:0/818463578 shutdown_connections 2024-03-30T04:17:01.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.508+0000 7ffb92c00700 1 -- 172.21.15.23:0/818463578 wait complete. 2024-03-30T04:17:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.509+0000 7ffb92c00700 1 Processor -- start 2024-03-30T04:17:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.509+0000 7ffb92c00700 1 -- start start 2024-03-30T04:17:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c106080 0x7ffb8c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 0x7ffb8c19c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c6a0 con 0x7ffb8c19bc20 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c810 con 0x7ffb8c106080 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb92c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c19c9d0 con 0x7ffb8c100f60 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb8be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb8be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.23:39056/0 (socket says 172.21.15.23:39056) 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb8be00700 1 -- 172.21.15.23:0/103893256 learned_addr learned my addr 172.21.15.23:0/103893256 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb90c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 0x7ffb8c19c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.510+0000 7ffb8b400700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c106080 0x7ffb8c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 -- 172.21.15.23:0/103893256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c106080 msgr2=0x7ffb8c197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c106080 0x7ffb8c197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 -- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 msgr2=0x7ffb8c19c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 0x7ffb8c19c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 -- 172.21.15.23:0/103893256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84009d00 con 0x7ffb8c100f60 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb90c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 0x7ffb8c19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb8be00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9a0 tx=0x7ffb7c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c009790 con 0x7ffb8c100f60 2024-03-30T04:17:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb7c010430 con 0x7ffb8c100f60 2024-03-30T04:17:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c1a3f30 con 0x7ffb8c100f60 2024-03-30T04:17:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c018680 con 0x7ffb8c100f60 2024-03-30T04:17:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.511+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c1a43b0 con 0x7ffb8c100f60 2024-03-30T04:17:01.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.512+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c109350 con 0x7ffb8c100f60 2024-03-30T04:17:01.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.514+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7ffb7c0105a0 con 0x7ffb8c100f60 2024-03-30T04:17:01.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.515+0000 7ffb88c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 0x7ffb700a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:01.514 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.516+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9358+0+0 (secure 0 0 0) 0x7ffb7c014070 con 0x7ffb8c100f60 2024-03-30T04:17:01.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.516+0000 7ffb8b400700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 0x7ffb700a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:01.515 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.516+0000 7ffb8b400700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 0x7ffb700a3750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c198890 tx=0x7ffb78009260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:01.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.520+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7ffb7c0529f0 con 0x7ffb8c100f60 2024-03-30T04:17:01.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:01 smithi023 ceph-mon[38254]: pgmap v474: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 175 KiB/s wr, 14 op/s 2024-03-30T04:17:01.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:01 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:01 smithi154 ceph-mon[79161]: pgmap v474: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 175 KiB/s wr, 14 op/s 2024-03-30T04:17:01.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:01 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:01.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:01 smithi193 ceph-mon[79581]: pgmap v474: 129 pgs: 129 active+clean; 774 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 175 KiB/s wr, 14 op/s 2024-03-30T04:17:01.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:01 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:01.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:01.780+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb8c102430 con 0x7ffb8c100f60 2024-03-30T04:17:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.294+0000 7ffb88c00700 1 -- 172.21.15.23:0/103893256 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) v1 ==== 169+0+0 (secure 0 0 0) 0x7ffb7c0962f0 con 0x7ffb8c100f60 2024-03-30T04:17:02.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.298+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 msgr2=0x7ffb700a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.298+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 0x7ffb700a3750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c198890 tx=0x7ffb78009260 comp rx=0 tx=0).stop 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.298+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 msgr2=0x7ffb8c197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.298+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00e9a0 tx=0x7ffb7c00ecb0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 shutdown_connections 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb8c19bc20 0x7ffb8c19c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb8c100f60 0x7ffb8c197340 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7ffb700a1290 0x7ffb700a3750 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 --2- 172.21.15.23:0/103893256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb8c106080 0x7ffb8c197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 >> 172.21.15.23:0/103893256 conn(0x7ffb8c0fc7c0 msgr2=0x7ffb8c0feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 shutdown_connections 2024-03-30T04:17:02.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.299+0000 7ffb92c00700 1 -- 172.21.15.23:0/103893256 wait complete. 2024-03-30T04:17:02.301 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2024-03-30T04:17:02.323 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-30T04:17:02.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.530+0000 7fd443a00700 1 -- 172.21.15.23:0/528764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c100f60 msgr2=0x7fd43c101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.530+0000 7fd443a00700 1 --2- 172.21.15.23:0/528764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c100f60 0x7fd43c101340 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd430009990 tx=0x7fd430009ca0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 -- 172.21.15.23:0/528764848 shutdown_connections 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 --2- 172.21.15.23:0/528764848 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c106080 0x7fd43c10dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 --2- 172.21.15.23:0/528764848 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c100f60 0x7fd43c101340 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 --2- 172.21.15.23:0/528764848 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c101910 0x7fd43c105950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 -- 172.21.15.23:0/528764848 >> 172.21.15.23:0/528764848 conn(0x7fd43c0fc7c0 msgr2=0x7fd43c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:02.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 -- 172.21.15.23:0/528764848 shutdown_connections 2024-03-30T04:17:02.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.531+0000 7fd443a00700 1 -- 172.21.15.23:0/528764848 wait complete. 2024-03-30T04:17:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 Processor -- start 2024-03-30T04:17:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 -- start start 2024-03-30T04:17:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c100f60 0x7fd43c074c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:02.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 0x7fd43c073aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1a9de0 con 0x7fd43c106080 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1a9f50 con 0x7fd43c100f60 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd443a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c1aa110 con 0x7fd43c073640 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.532+0000 7fd43be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57448/0 (socket says 172.21.15.23:57448) 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 -- 172.21.15.23:0/3923961859 learned_addr learned my addr 172.21.15.23:0/3923961859 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd441000700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c100f60 0x7fd43c074c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:02.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd441a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 0x7fd43c073aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 -- 172.21.15.23:0/3923961859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 msgr2=0x7fd43c073aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 0x7fd43c073aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 -- 172.21.15.23:0/3923961859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c100f60 msgr2=0x7fd43c074c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c100f60 0x7fd43c074c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 -- 172.21.15.23:0/3923961859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4300095f0 con 0x7fd43c106080 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd441a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 0x7fd43c073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-30T04:17:02.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd43be00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd42800e970 tx=0x7fd42800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:02.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd428009790 con 0x7fd43c106080 2024-03-30T04:17:02.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c1aa3f0 con 0x7fd43c106080 2024-03-30T04:17:02.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd42800eec0 con 0x7fd43c106080 2024-03-30T04:17:02.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.533+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd428005590 con 0x7fd43c106080 2024-03-30T04:17:02.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.534+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c1aa900 con 0x7fd43c106080 2024-03-30T04:17:02.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.535+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c1092a0 con 0x7fd43c106080 2024-03-30T04:17:02.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.537+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 27) v1 ==== 135996+0+0 (secure 0 0 0) 0x7fd42800cbd0 con 0x7fd43c106080 2024-03-30T04:17:02.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.538+0000 7fd439600700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 0x7fd4240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-30T04:17:02.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.538+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 8725+0+0 (secure 0 0 0) 0x7fd428014070 con 0x7fd43c106080 2024-03-30T04:17:02.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.538+0000 7fd441000700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 0x7fd4240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-30T04:17:02.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.539+0000 7fd441000700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 0x7fd4240a3750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd430006010 tx=0x7fd43000b260 comp rx=0 tx=0).ready entity=mgr.15627 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-30T04:17:02.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.542+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182738 (secure 0 0 0) 0x7fd4280963d0 con 0x7fd43c106080 2024-03-30T04:17:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:02 smithi154 ceph-mon[79161]: osdmap e109: 12 total, 12 up, 12 in 2024-03-30T04:17:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:02 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/103893256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:02.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:02 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:02.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:02.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:02 smithi193 ceph-mon[79581]: osdmap e109: 12 total, 12 up, 12 in 2024-03-30T04:17:02.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:02 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/103893256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:02.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:02 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:02.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:02.766+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd43c074540 con 0x7fd43c106080 2024-03-30T04:17:02.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/2610755526' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:02.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:02 smithi023 ceph-mon[38254]: osdmap e109: 12 total, 12 up, 12 in 2024-03-30T04:17:02.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:02 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/103893256' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:02.771 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:02 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:03.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.296+0000 7fd439600700 1 -- 172.21.15.23:0/3923961859 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v111) v1 ==== 178+0+0 (secure 0 0 0) 0x7fd428004ef0 con 0x7fd43c106080 2024-03-30T04:17:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 msgr2=0x7fd4240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 0x7fd4240a3750 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd430006010 tx=0x7fd43000b260 comp rx=0 tx=0).stop 2024-03-30T04:17:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 msgr2=0x7fd43c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-30T04:17:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd42800e970 tx=0x7fd42800ec80 comp rx=0 tx=0).stop 2024-03-30T04:17:03.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 shutdown_connections 2024-03-30T04:17:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd43c073640 0x7fd43c073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:6800/250950136,v1:172.21.15.23:6801/250950136] conn(0x7fd4240a1290 0x7fd4240a3750 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43c100f60 0x7fd43c074c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 --2- 172.21.15.23:0/3923961859 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd43c106080 0x7fd43c0730b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-30T04:17:03.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.301+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 >> 172.21.15.23:0/3923961859 conn(0x7fd43c0fc7c0 msgr2=0x7fd43c0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-30T04:17:03.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.303+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 shutdown_connections 2024-03-30T04:17:03.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-30T04:17:03.303+0000 7fd443a00700 1 -- 172.21.15.23:0/3923961859 wait complete. 2024-03-30T04:17:03.302 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' removed 2024-03-30T04:17:03.331 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:17:03.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:17:03.513 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:17:03.543 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:03 smithi193 ceph-mon[79581]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:03.544 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:03 smithi193 ceph-mon[79581]: osdmap e110: 12 total, 12 up, 12 in 2024-03-30T04:17:03.544 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:03 smithi193 ceph-mon[79581]: pgmap v477: 33 pgs: 33 active+clean; 261 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:17:03.544 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:03 smithi193 ceph-mon[79581]: from='client.? 172.21.15.23:0/3923961859' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:03.548 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-30T04:17:03.549 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T04:17:03.590 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T04:17:03.608 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-30T04:17:03.625 INFO:tasks.cephadm:Stopping all daemons... 2024-03-30T04:17:03.626 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-30T04:17:03.626 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a 2024-03-30T04:17:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:03 smithi154 ceph-mon[79161]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:03 smithi154 ceph-mon[79161]: osdmap e110: 12 total, 12 up, 12 in 2024-03-30T04:17:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:03 smithi154 ceph-mon[79161]: pgmap v477: 33 pgs: 33 active+clean; 261 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:17:03.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:03 smithi154 ceph-mon[79161]: from='client.? 172.21.15.23:0/3923961859' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:03.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-mon[38254]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-30T04:17:03.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-mon[38254]: osdmap e110: 12 total, 12 up, 12 in 2024-03-30T04:17:03.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-mon[38254]: pgmap v477: 33 pgs: 33 active+clean; 261 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-30T04:17:03.688 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-mon[38254]: from='client.? 172.21.15.23:0/3923961859' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-30T04:17:03.992 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 systemd[1]: Stopping Ceph mon.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:04.271 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a[38231]: 2024-03-30T04:17:03.993+0000 7fe581a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a[38231]: 2024-03-30T04:17:03.993+0000 7fe581a00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-30T04:17:04.272 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 30 04:17:04 smithi023 bash[127511]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-a 2024-03-30T04:17:04.623 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.a.service' 2024-03-30T04:17:04.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:04.724 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-30T04:17:04.724 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-30T04:17:04.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.b 2024-03-30T04:17:05.070 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:04 smithi154 systemd[1]: Stopping Ceph mon.b for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:05.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-b[79138]: 2024-03-30T04:17:05.071+0000 7fd7cce0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:05.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-b[79138]: 2024-03-30T04:17:05.071+0000 7fd7cce0c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-30T04:17:05.802 INFO:journalctl@ceph.mon.b.smithi154.stdout:Mar 30 04:17:05 smithi154 bash[122578]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-b 2024-03-30T04:17:05.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.b.service' 2024-03-30T04:17:05.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:05.877 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-30T04:17:05.877 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-30T04:17:05.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.c 2024-03-30T04:17:06.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:05 smithi193 systemd[1]: Stopping Ceph mon.c for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:06.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-c[79558]: 2024-03-30T04:17:06.113+0000 7f6557f77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:06.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-c[79558]: 2024-03-30T04:17:06.113+0000 7f6557f77700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-30T04:17:06.587 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 30 04:17:06 smithi193 bash[123832]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mon-c 2024-03-30T04:17:06.594 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mon.c.service' 2024-03-30T04:17:06.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:06.645 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-30T04:17:06.645 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-30T04:17:06.645 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x 2024-03-30T04:17:07.016 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:17:06 smithi023 systemd[1]: Stopping Ceph mgr.x for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:07.271 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:17:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:17:07.017+0000 7fe1fbc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:07.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:17:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x[38504]: 2024-03-30T04:17:07.017+0000 7fe1fbc00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-30T04:17:07.628 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 30 04:17:07 smithi023 bash[127650]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-x 2024-03-30T04:17:07.657 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.x.service' 2024-03-30T04:17:07.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:07.742 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-30T04:17:07.742 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-30T04:17:07.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.y 2024-03-30T04:17:08.106 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:17:07 smithi154 systemd[1]: Stopping Ceph mgr.y for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:08.433 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:17:08 smithi154 bash[122718]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-y 2024-03-30T04:17:08.434 INFO:journalctl@ceph.mgr.y.smithi154.stdout:Mar 30 04:17:08 smithi154 systemd[1]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-30T04:17:08.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.y.service' 2024-03-30T04:17:08.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:08.509 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-30T04:17:08.510 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-30T04:17:08.510 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.z 2024-03-30T04:17:08.850 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:17:08 smithi193 systemd[1]: Stopping Ceph mgr.z for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:09.191 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:17:08 smithi193 bash[123974]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mgr-z 2024-03-30T04:17:09.191 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 30 04:17:08 smithi193 systemd[1]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-30T04:17:09.197 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mgr.z.service' 2024-03-30T04:17:09.235 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:09.235 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-30T04:17:09.235 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-30T04:17:09.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.0 2024-03-30T04:17:09.607 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 04:17:09 smithi023 systemd[1]: Stopping Ceph osd.0 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:10.021 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 04:17:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T04:17:09.608+0000 7fbda2200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:10.021 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 04:17:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T04:17:09.608+0000 7fbda2200700 -1 osd.0 111 *** Got signal Terminated *** 2024-03-30T04:17:10.022 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 04:17:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0[57208]: 2024-03-30T04:17:09.608+0000 7fbda2200700 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:15.771 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 30 04:17:15 smithi023 bash[127792]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-0 2024-03-30T04:17:17.039 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.0.service' 2024-03-30T04:17:17.125 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:17.125 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-30T04:17:17.126 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-30T04:17:17.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.1 2024-03-30T04:17:17.431 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 04:17:17 smithi154 systemd[1]: Stopping Ceph osd.1 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:17.431 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 04:17:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T04:17:17.369+0000 7f08643f3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:17.432 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 04:17:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T04:17:17.369+0000 7f08643f3700 -1 osd.1 111 *** Got signal Terminated *** 2024-03-30T04:17:17.432 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 04:17:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1[84148]: 2024-03-30T04:17:17.369+0000 7f08643f3700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:23.431 INFO:journalctl@ceph.osd.1.smithi154.stdout:Mar 30 04:17:23 smithi154 bash[122864]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-1 2024-03-30T04:17:24.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.1.service' 2024-03-30T04:17:24.404 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:24.405 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-30T04:17:24.405 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-30T04:17:24.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.2 2024-03-30T04:17:24.742 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 04:17:24 smithi193 systemd[1]: Stopping Ceph osd.2 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:24.743 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 04:17:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T04:17:24.644+0000 7f0d10c94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:24.743 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 04:17:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T04:17:24.644+0000 7f0d10c94700 -1 osd.2 111 *** Got signal Terminated *** 2024-03-30T04:17:24.743 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 04:17:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2[85377]: 2024-03-30T04:17:24.644+0000 7f0d10c94700 -1 osd.2 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:30.492 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 30 04:17:30 smithi193 bash[124121]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-2 2024-03-30T04:17:31.313 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.2.service' 2024-03-30T04:17:31.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:31.353 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-30T04:17:31.354 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-30T04:17:31.354 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.3 2024-03-30T04:17:31.746 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 04:17:31 smithi023 systemd[1]: Stopping Ceph osd.3 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:32.021 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 04:17:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T04:17:31.746+0000 7fcdd5600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:32.022 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 04:17:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T04:17:31.746+0000 7fcdd5600700 -1 osd.3 111 *** Got signal Terminated *** 2024-03-30T04:17:32.022 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 04:17:31 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3[65102]: 2024-03-30T04:17:31.746+0000 7fcdd5600700 -1 osd.3 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:36.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:36 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:36.467+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:37.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:37 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:37.450+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:37.771 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 30 04:17:37 smithi023 bash[128077]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-3 2024-03-30T04:17:38.364 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:38 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:38.108+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:38.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:38 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:38.425+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:39.230 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.3.service' 2024-03-30T04:17:39.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:39.317 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-30T04:17:39.317 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-30T04:17:39.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.4 2024-03-30T04:17:39.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:39 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:39.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:39.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 04:17:39 smithi154 systemd[1]: Stopping Ceph osd.4 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:39.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 04:17:39 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T04:17:39.558+0000 7fa374a53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:39.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 04:17:39 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T04:17:39.558+0000 7fa374a53700 -1 osd.4 111 *** Got signal Terminated *** 2024-03-30T04:17:39.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 04:17:39 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4[92665]: 2024-03-30T04:17:39.558+0000 7fa374a53700 -1 osd.4 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:39.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:39 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:39.460+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:40.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:40 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:40.146+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:40.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:40 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:40.464+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:41.453 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:41 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:41.193+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:41.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:41 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:41.159+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:41.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:41 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:41.455+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:42.482 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:42 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:42.185+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:42.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:42 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:42.121+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:42.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:42 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:42.483+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:43.447 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:43 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:43.137+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:43.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:43 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:43.162+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:43.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:43 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:43.448+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:43.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:43 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:43.591+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:44.466 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:44 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:44.164+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:44.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:44 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:44.156+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:44.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:44 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:44.156+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:44.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:44 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:44.467+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:44.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:44 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:44.629+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:45.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:44 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:44.687+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:45.438 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:45 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:45.161+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:45.439 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:45 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:45.161+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:45.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:45.158+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:45.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:45.158+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:45.681 INFO:journalctl@ceph.osd.4.smithi154.stdout:Mar 30 04:17:45 smithi154 bash[123120]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-4 2024-03-30T04:17:45.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:45 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:45.439+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:45.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:45 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:45.600+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:46.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:45 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:45.665+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:46.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:45 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:45.769+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:46.492 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:46.439+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:46.493 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:46.197+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:46.493 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:46.197+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:46.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:46 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:46.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:46.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:46 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:46.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:46.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.4.service' 2024-03-30T04:17:46.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:46.735 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-30T04:17:46.736 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-30T04:17:46.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.5 2024-03-30T04:17:46.980 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 systemd[1]: Stopping Ceph osd.5 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:46.980 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:46.643+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:47.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:46 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:46.642+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:47.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:46 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:46.801+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:47.242 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:46.981+0000 7ffa75bcc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:47.243 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:46.981+0000 7ffa75bcc700 -1 osd.5 111 *** Got signal Terminated *** 2024-03-30T04:17:47.243 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:46 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:46.981+0000 7ffa75bcc700 -1 osd.5 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:47.243 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:47.182+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:47.243 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:47.182+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:47.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:47.139+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:47.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:47.139+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:47.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:47.625+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:47.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:47 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:47.390+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:47.768 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:47.514+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:48.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:47.659+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:48.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:47 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:47.769+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:48.492 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:48.185+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:48.492 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:48.185+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:48.493 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:48.427+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:48.493 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:48.427+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:48.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:48.134+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:48.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:48.134+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:48.780 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:48.512+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:48.992 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:48 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:48.646+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:49.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:48 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:48.623+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:49.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:48 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:48.782+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:49.391 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:49.139+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:49.391 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:49.139+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:49.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:49.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:49.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:49.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:49.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:49.615+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:49.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:49.392+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:49.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:49 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:49.392+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:49.817 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:49.530+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:50.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:49 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:49.599+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:50.183 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:49 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:49.818+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:50.437 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:50.185+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:50.437 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:50.185+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:50.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:50.142+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:50.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:50.142+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:50.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:50.586+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:50.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:50.437+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:50.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:50 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:50.437+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:50.931 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:50.564+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:50.931 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:50.793+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:50.931 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:50 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:50.793+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:51.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:50 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:50.575+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:51.484 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:51.182+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:14.321495+0000 front 2024-03-30T04:17:14.321356+0000 (oldest deadline 2024-03-30T04:17:40.221269+0000) 2024-03-30T04:17:51.484 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5[93443]: 2024-03-30T04:17:51.182+0000 7ffa6e1ba700 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.223419+0000 front 2024-03-30T04:17:21.223266+0000 (oldest deadline 2024-03-30T04:17:44.623991+0000) 2024-03-30T04:17:51.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:51.170+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:51.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:51.170+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:51.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:51.540+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:51.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:51.485+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:51.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:51 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:51.485+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:51.830 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:51.581+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:52.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:51.529+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:52.022 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:51 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:51.529+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:52.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:51.831+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:52.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:51 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:51.831+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:52.511 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:52.157+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:52.511 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:52.157+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:52.743 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 30 04:17:52 smithi193 bash[124375]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-5 2024-03-30T04:17:52.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:52.569+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:52.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:52.569+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:52.744 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:52.447+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:52.744 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:52.447+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:52.744 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:52 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:52.447+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:52.771 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:52.512+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:52.771 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:52 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:52.512+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:52.847 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:52.557+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:52.848 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:52.557+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:53.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:52.849+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:53.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:52.849+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:53.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:53.155+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:53.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:53.155+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:53.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:53.578+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:53.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:53.578+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:53.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:53.448+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:53.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:53.448+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:53.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:53 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:53.448+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:53.791 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.5.service' 2024-03-30T04:17:53.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:17:53.833 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-30T04:17:53.833 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-30T04:17:53.834 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.6 2024-03-30T04:17:53.897 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:53.583+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:53.898 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:53.583+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:53.898 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:53.558+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:53.898 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:53 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:53.558+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:54.150 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:53 smithi023 systemd[1]: Stopping Ceph osd.6 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:17:54.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:53.899+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:54.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:53 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:53.899+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:54.523 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:54.201+0000 7fb002e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:17:54.523 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:54.201+0000 7fb002e00700 -1 osd.6 111 *** Got signal Terminated *** 2024-03-30T04:17:54.523 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:54.201+0000 7fb002e00700 -1 osd.6 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:17:54.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:54.150+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:54.524 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:54.150+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:54.524 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:54.150+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:54.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:54 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:54.542+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:54.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:54 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:54.542+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:54.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:54 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:54.443+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:54.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:54 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:54.443+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:54.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:54 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:54.443+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:54.895 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:54 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:54.605+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:54.896 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:54 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:54.605+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:55.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:54.555+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:55.022 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:54 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:54.555+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:55.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:54 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:54.897+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:55.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:54 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:54.897+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:55.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:55.181+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:55.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:55.181+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:55.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:55.181+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:55.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:55 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:55.570+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:55.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:55 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:55.570+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:55.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:55 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:55.395+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:55.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:55 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:55.395+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:55.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:55 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:55.395+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:55.894 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:55 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:55.559+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:55.894 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:55 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:55.559+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:56.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:55.574+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:56.022 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:55 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:55.574+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:56.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:55 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:55.895+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:56.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:55 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:55.895+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:56.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:56.160+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:56.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:56.160+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:56.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:56.160+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:56.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:56 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:56.601+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:56.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:56 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:56.601+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:56.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:56 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:56.369+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:56.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:56 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:56.369+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:56.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:56 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:56.369+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:56.887 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:56.526+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:56.887 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:56.526+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:57.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:56.590+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:57.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:56 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:56.590+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:57.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:56.888+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:57.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:56.888+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:57.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:57 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:57.118+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:57.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:57 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:57.118+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:57.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:57 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:57.118+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:57.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:57 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:57.572+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:57.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:57 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:57.572+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:57.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:57 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:57.361+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:57.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:57 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:57.361+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:57.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:57 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:57.361+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:57.850 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:57 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:57.521+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:57.850 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:57 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:57.521+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:58.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:57 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:57.603+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:58.022 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:57 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:57.603+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:58.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:57 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:57.851+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:58.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:57 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:57.851+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:58.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:57 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:57.851+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:17:58.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:58 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:58.158+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:58.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:58 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:58.158+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:58.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:58 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:58.158+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:58.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:58 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:58.524+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:58.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:58 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:58.524+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:58.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:58 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:58.324+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:58.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:58 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:58.324+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:58.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:58 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:58.324+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:58.803 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:58 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:58.504+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:58.803 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:58 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:58.504+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:59.021 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:58 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:58.580+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.613527+0000 front 2024-03-30T04:17:21.613565+0000 (oldest deadline 2024-03-30T04:17:43.913127+0000) 2024-03-30T04:17:59.022 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:58 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6[73008]: 2024-03-30T04:17:58.580+0000 7faffe600700 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.414599+0000 front 2024-03-30T04:17:27.414609+0000 (oldest deadline 2024-03-30T04:17:51.514370+0000) 2024-03-30T04:17:59.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:58 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:58.805+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:17:59.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:58 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:58.805+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:17:59.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:58 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:58.805+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:17:59.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:59.139+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:17:59.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:59.139+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:17:59.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:17:59 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:17:59.139+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:17:59.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:59.541+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:17:59.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:17:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:17:59.541+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:17:59.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:59.297+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:17:59.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:59.297+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:17:59.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:17:59 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:17:59.297+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:17:59.838 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:59.489+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:17:59.838 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:59.489+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:17:59.838 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:17:59.489+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:00.129 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 30 04:17:59 smithi023 bash[128344]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-6 2024-03-30T04:18:00.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:59.839+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:00.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:59.839+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:00.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:17:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:17:59.839+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:00.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:00.129+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:00.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:00.129+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:00.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:00.129+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:00.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:00 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:00.129+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:00.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:00 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:00.557+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:00.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:00 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:00.557+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:00.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:00 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:00.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:00.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:00 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:00.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:00.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:00 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:00.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:00.845 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:00.447+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:00.845 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:00.447+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:00.845 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:00.447+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:01.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:00.846+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:01.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:00.846+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:01.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:00 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:00.846+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:01.425 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:01 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:01.102+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:01.425 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:01 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:01.102+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:01.425 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:01 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:01.102+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:01.426 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:01 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:01.102+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:01.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.6.service' 2024-03-30T04:18:01.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:01.560 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-30T04:18:01.560 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-30T04:18:01.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.7 2024-03-30T04:18:01.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:01.304+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:01.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:01.304+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:01.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:01.304+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:01.804 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:01.804 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:01.804 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:01.804 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 systemd[1]: Stopping Ceph osd.7 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:01.992 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:01.586+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:01.993 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:01.586+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:01.993 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:01 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:01.586+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:02.181 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.805+0000 7f1c33216700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:02.181 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.805+0000 7f1c33216700 -1 osd.7 111 *** Got signal Terminated *** 2024-03-30T04:18:02.181 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:01.805+0000 7f1c33216700 -1 osd.7 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:18:02.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:01.819+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:02.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:01.819+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:02.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:01 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:01.819+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:02.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:02 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:02.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:02.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:02 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:02.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:02.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:02 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:02.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:02.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:02 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:02.116+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:02.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:02.541+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:02.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:02.541+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:02.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:02.541+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:02.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:02.337+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:02.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:02.337+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:02.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:02 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:02.337+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:02.789 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:02.456+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:02.789 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:02.456+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:02.790 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:02.456+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:03.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:02.791+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:03.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:02.791+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:03.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:02 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:02.791+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:03.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:03.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:03.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:03.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:03.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:03.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:03.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:03 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:03.161+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:03.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:03.346+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:03.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:03.346+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:03.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:03.346+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:03.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:03.514+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:03.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:03.514+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:03.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:03 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:03.514+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:03.771 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:03.444+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:03.771 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:03.444+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:03.772 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:03.444+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:04.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:03.773+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:04.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:03.773+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:04.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:03 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:03.773+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:04.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:04 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:04.171+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:04.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:04 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:04.171+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:04.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:04 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:04.171+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:04.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:04 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:04.171+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:04.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:04.360+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:04.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:04.360+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:04.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:04.360+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:04.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:04.465+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:04.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:04.465+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:04.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:04 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:04.465+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:04.786 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:04.459+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:04.786 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:04.459+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:04.786 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:04.459+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:05.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:04.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:05.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:04.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:05.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:04.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:05.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:05 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:05.206+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:05.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:05 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:05.206+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:05.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:05 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:05.206+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:05.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:05 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:05.206+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:05.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:05.400+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:05.400+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:05.400+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:05.468+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:05.468+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:05.468+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:05.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:05 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:05.468+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:05.782 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:05.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:05.782 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:05.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:05.783 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:05.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:05.783 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:05.468+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:39.451994+0000 front 2024-03-30T04:17:39.451968+0000 (oldest deadline 2024-03-30T04:18:05.352101+0000) 2024-03-30T04:18:06.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:05.784+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:06.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:05.784+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:06.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:05 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:05.784+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:06.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:06 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:06.214+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:06.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:06 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:06.214+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:06.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:06 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:06.214+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:06.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:06 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:06.214+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:06.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:06 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:06.214+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:06.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:06.412+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:06.412+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:06.412+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:06.412+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:06.499+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:06.499+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:06.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:06.499+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:06.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:06 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:06.499+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:06.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:06.460+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:20.749004+0000 front 2024-03-30T04:17:20.749021+0000 (oldest deadline 2024-03-30T04:17:47.149823+0000) 2024-03-30T04:18:06.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:06.460+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.150483+0000 front 2024-03-30T04:17:27.150424+0000 (oldest deadline 2024-03-30T04:17:52.450390+0000) 2024-03-30T04:18:06.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:06.460+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:35.951815+0000 front 2024-03-30T04:17:35.951720+0000 (oldest deadline 2024-03-30T04:17:59.451670+0000) 2024-03-30T04:18:06.752 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7[100631]: 2024-03-30T04:18:06.460+0000 7f1c2f80c700 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:39.451994+0000 front 2024-03-30T04:17:39.451968+0000 (oldest deadline 2024-03-30T04:18:05.352101+0000) 2024-03-30T04:18:07.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:06.753+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:07.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:06.753+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:07.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:06.753+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:07.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:06 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:06.753+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:07.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:07.211+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:07.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:07.211+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:07.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:07.211+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:07.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:07.211+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:07.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:07 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:07.211+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:07.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:07.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:07.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:07.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:07.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:07.527+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:07.527+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:07.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:07.527+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:07.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:07.527+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:07.786 INFO:journalctl@ceph.osd.7.smithi154.stdout:Mar 30 04:18:07 smithi154 bash[123380]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-7 2024-03-30T04:18:08.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:07 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:07.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:08.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:07 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:07.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:08.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:07 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:07.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:08.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:07 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:07.787+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:08.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:08.217+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:08.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:08.217+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:08.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:08.217+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:08.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:08.217+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:08.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:08 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:08.217+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:08.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.7.service' 2024-03-30T04:18:08.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:08.739 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-30T04:18:08.739 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-30T04:18:08.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.8 2024-03-30T04:18:08.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:08.396+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:08.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:08.396+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:08.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:08.396+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:08.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:08.396+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:08.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.547+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:08.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.547+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:08.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.547+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:08.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.547+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:09.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:08 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:08.837+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:09.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:08 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:08.837+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:09.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:08 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:08.837+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:09.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:08 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:08.837+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:09.242 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 systemd[1]: Stopping Ceph osd.8 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:09.243 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.969+0000 7f7343f53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:09.243 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.969+0000 7f7343f53700 -1 osd.8 111 *** Got signal Terminated *** 2024-03-30T04:18:09.243 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:08 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:08.969+0000 7f7343f53700 -1 osd.8 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:18:09.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:09.252+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:09.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:09.252+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:09.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:09.252+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:09.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:09.252+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:09.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:09 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:09.253+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:09.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:09.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:09.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:09.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:09.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:09.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:09.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:09.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:09.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:09.508+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:09.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:09.508+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:09.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:09.508+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:09.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:09 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:09.508+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:10.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:09 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:09.788+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:10.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:09 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:09.788+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:10.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:09 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:09.788+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:10.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:09 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:09.788+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:10.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:10.222+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:10.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:10.222+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:10.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:10.222+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:10.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:10.222+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:10.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:10 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:10.222+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:10.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:10.356+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:10.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:10.356+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:10.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:10.356+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:10.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:10.356+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:10.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:10.489+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:10.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:10.489+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:10.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:10.489+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:10.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:10 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:10.489+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:11.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:10 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:10.772+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:11.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:10 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:10.772+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:11.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:10 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:10.772+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:11.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:10 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:10.772+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:11.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:11.247+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:11.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:11.247+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:11.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:11.247+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:11.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:11.247+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:11.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:11 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:11.247+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:11.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:11.317+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:11.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:11.317+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:11.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:11.317+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:11.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:11.317+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:11.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:11.488+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:11.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:11.488+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:11.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:11.488+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:11.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:11.488+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:12.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:11 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:11.815+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:12.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:11 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:11.815+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:12.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:11 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:11.815+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:12.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:11 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:11.815+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:12.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:12.257+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:12.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:12.257+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:12.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:12.257+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:12.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:12.257+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:12.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:12 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:12.257+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:12.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:12.358+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:12.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:12.358+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:12.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:12.358+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:12.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:12.358+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:12.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:12.530+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:12.743 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:12.530+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:12.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:12.530+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:12.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:12 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:12.530+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:13.183 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:12.862+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:13.184 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:12.862+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:13.184 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:12.862+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:13.184 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:12.862+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:13.184 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:12 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:12.862+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:13.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:13.251+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:13.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:13.251+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:13.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:13.251+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:13.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:13.251+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:13.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:13 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:13.251+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:13.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:13.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:13.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:13.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:13.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:13.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:13.744 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:13.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:13.744 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:13.560+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:17.467955+0000 front 2024-03-30T04:17:17.467965+0000 (oldest deadline 2024-03-30T04:17:43.268389+0000) 2024-03-30T04:18:13.745 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:13.560+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:27.770250+0000 front 2024-03-30T04:17:27.770327+0000 (oldest deadline 2024-03-30T04:17:52.470428+0000) 2024-03-30T04:18:13.745 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:13.560+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:36.472323+0000 front 2024-03-30T04:17:36.472283+0000 (oldest deadline 2024-03-30T04:18:01.172162+0000) 2024-03-30T04:18:13.745 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:13 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8[101433]: 2024-03-30T04:18:13.560+0000 7f733c541700 -1 osd.8 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.673067+0000 front 2024-03-30T04:17:41.673195+0000 (oldest deadline 2024-03-30T04:18:05.173029+0000) 2024-03-30T04:18:14.180 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:13.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:14.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:13.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:14.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:13.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:14.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:13.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:14.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:13 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:13.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:14.742 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 30 04:18:14 smithi193 bash[124642]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-8 2024-03-30T04:18:14.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:14.383+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:14.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:14.383+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:14.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:14.383+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:14.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:14 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:14.383+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:14.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:14.274+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:14.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:14.274+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:14.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:14.274+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:14.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:14.274+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:14.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:14 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:14.274+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:15.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:14.916+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:15.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:14.916+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:15.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:14.916+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:15.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:14.916+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:15.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:14 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:14.916+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:15.655 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:15.411+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:15.656 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:15.411+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:15.656 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:15.411+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:15.656 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:15.411+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:15.663 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.8.service' 2024-03-30T04:18:15.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:15.704 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-30T04:18:15.704 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-30T04:18:15.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.9 2024-03-30T04:18:15.770 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:15.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:15.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:15.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:15.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:15.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:15.313+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:16.021 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:15 smithi023 systemd[1]: Stopping Ceph osd.9 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:16.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:15.904+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:16.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:15.904+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:16.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:15.904+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:16.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:15.904+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:16.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:15 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:15.904+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:16.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.074+0000 7f3be1e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.074+0000 7f3be1e00700 -1 osd.9 111 *** Got signal Terminated *** 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.074+0000 7f3be1e00700 -1 osd.9 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:16.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:16.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:16 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:16.271+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:16.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:16.378+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:16.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:16.378+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:16.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:16.378+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:16.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:16.378+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:16.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:16 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:16.378+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:17.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:16.917+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:17.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:16.917+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:17.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:16.917+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:17.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:16.917+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:17.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:16 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:16.917+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:17.521 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:17.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:17.522 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:17.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:17.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:17.523 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:17 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:17.260+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:17.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:17 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:17.333+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:17.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:17 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:17.333+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:17.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:17 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:17.333+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:17.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:17 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:17.333+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:17.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:17 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:17.333+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:18.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:17.945+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:18.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:17.945+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:18.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:17.945+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:18.432 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:17.945+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:18.432 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:17 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:17.945+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:18.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:18 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:18.306+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:18.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:18 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:18.306+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:18.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:18 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:18.306+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:18.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:18 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:18.306+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:18.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:18 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:18.306+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:18.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:18.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:18.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:18.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:18.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:18.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:18 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:18.284+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:19.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:18.996+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:19.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:18.996+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:19.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:18.996+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:19.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:18.996+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:19.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:18 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:18.996+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:19.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:19.339+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:19.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:19.339+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:19.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:19.339+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:19.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:19.339+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:19.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:19.339+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:19.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:19.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:19.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:19.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:19.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:19.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:19 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:19.309+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:20.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:19.961+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:20.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:19.961+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:20.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:19.961+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:20.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:19.961+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:20.432 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:19 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:19.961+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:20.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:20 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:20.334+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:20.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:20 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:20.334+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:20.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:20 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:20.334+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:20.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:20 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:20.334+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:20.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:20 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:20.334+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:20.771 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:13.562263+0000 front 2024-03-30T04:17:13.562234+0000 (oldest deadline 2024-03-30T04:17:37.662487+0000) 2024-03-30T04:18:20.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:18.163737+0000 front 2024-03-30T04:17:18.163685+0000 (oldest deadline 2024-03-30T04:17:43.364535+0000) 2024-03-30T04:18:20.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:29.265825+0000 front 2024-03-30T04:17:29.265776+0000 (oldest deadline 2024-03-30T04:17:53.866588+0000) 2024-03-30T04:18:20.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.967810+0000 front 2024-03-30T04:17:34.967976+0000 (oldest deadline 2024-03-30T04:17:59.668261+0000) 2024-03-30T04:18:20.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.471254+0000 front 2024-03-30T04:17:43.471302+0000 (oldest deadline 2024-03-30T04:18:06.172562+0000) 2024-03-30T04:18:20.772 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:20 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9[80955]: 2024-03-30T04:18:20.314+0000 7f3bd8600700 -1 osd.9 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.374342+0000 front 2024-03-30T04:17:51.374285+0000 (oldest deadline 2024-03-30T04:18:14.873891+0000) 2024-03-30T04:18:21.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:20 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:20.947+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:21.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:20 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:20.947+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:21.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:20 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:20.947+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:21.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:20 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:20.947+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:21.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:20 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:20.947+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:21.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:21.288+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:21.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:21.288+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:21.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:21.288+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:21.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:21.288+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:21.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:21 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:21.288+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:22.022 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 30 04:18:21 smithi023 bash[128601]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-9 2024-03-30T04:18:22.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:21 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:21.909+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:22.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:21 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:21.909+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:22.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:21 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:21.909+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:22.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:21 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:21.909+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:22.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:21 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:21.909+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:22.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:22.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:22.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:22.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:22.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:22.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:22 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:22.255+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:23.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:22 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:22.887+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:23.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:22 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:22.887+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:23.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:22 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:22.887+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:23.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:22 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:22.887+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:23.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:22 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:22.887+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:23.213 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.9.service' 2024-03-30T04:18:23.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:23.301 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-30T04:18:23.301 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-30T04:18:23.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.10 2024-03-30T04:18:23.681 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 systemd[1]: Stopping Ceph osd.10 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:23.681 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.541+0000 7f08b3126700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:23.681 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.541+0000 7f08b3126700 -1 osd.10 111 *** Got signal Terminated *** 2024-03-30T04:18:23.681 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.541+0000 7f08b3126700 -1 osd.10 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:18:23.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:23.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:23.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:23.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:23.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:23.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:23 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:23.265+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:24.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.850+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:24.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.850+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:24.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.850+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:24.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.850+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:24.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:23 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:23.850+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:24.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:24.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:24.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:24.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:24.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:24.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:24 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:24.289+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:25.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:24 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:24.900+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:58.306224+0000 front 2024-03-30T04:17:58.306176+0000 (oldest deadline 2024-03-30T04:18:24.205944+0000) 2024-03-30T04:18:25.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:25.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:25.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:25.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:25.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:25.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:25 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:25.314+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:26.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:26.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:26.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:26.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:26.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:26.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:25 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:25.884+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:58.306224+0000 front 2024-03-30T04:17:58.306176+0000 (oldest deadline 2024-03-30T04:18:24.205944+0000) 2024-03-30T04:18:26.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:26.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:26.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:26.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:26.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:26.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:26 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:26.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:27.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:27.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:27.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:27.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:27.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:27.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:26 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:26.873+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:58.306224+0000 front 2024-03-30T04:17:58.306176+0000 (oldest deadline 2024-03-30T04:18:24.205944+0000) 2024-03-30T04:18:27.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:27.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:27.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:27.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:27.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:27.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:27 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:27.380+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:28.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:21.997773+0000 front 2024-03-30T04:17:21.997794+0000 (oldest deadline 2024-03-30T04:17:44.798482+0000) 2024-03-30T04:18:28.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.200311+0000 front 2024-03-30T04:17:28.200370+0000 (oldest deadline 2024-03-30T04:17:50.500388+0000) 2024-03-30T04:18:28.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-30T04:17:34.601618+0000 front 2024-03-30T04:17:34.601460+0000 (oldest deadline 2024-03-30T04:17:57.501373+0000) 2024-03-30T04:18:28.181 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:43.703812+0000 front 2024-03-30T04:17:43.703939+0000 (oldest deadline 2024-03-30T04:18:06.603974+0000) 2024-03-30T04:18:28.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:47.705236+0000 front 2024-03-30T04:17:47.705190+0000 (oldest deadline 2024-03-30T04:18:12.405026+0000) 2024-03-30T04:18:28.182 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:27 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10[109222]: 2024-03-30T04:18:27.841+0000 7f08ab714700 -1 osd.10 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:58.306224+0000 front 2024-03-30T04:17:58.306176+0000 (oldest deadline 2024-03-30T04:18:24.205944+0000) 2024-03-30T04:18:28.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:28.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:28 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:28.355+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:29.431 INFO:journalctl@ceph.osd.10.smithi154.stdout:Mar 30 04:18:29 smithi154 bash[123636]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-10 2024-03-30T04:18:29.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:29.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:29.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:29.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:29.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:29.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:29.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:29 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:29.375+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:30.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.10.service' 2024-03-30T04:18:30.401 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:30.402 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-30T04:18:30.402 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-30T04:18:30.402 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.11 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:30.648 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.393+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:30.649 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 systemd[1]: Stopping Ceph osd.11 for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:30.649 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.648+0000 7f2bcec62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:30.649 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.648+0000 7f2bcec62700 -1 osd.11 111 *** Got signal Terminated *** 2024-03-30T04:18:30.992 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:30 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:30.648+0000 7f2bcec62700 -1 osd.11 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-30T04:18:31.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:31.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:31 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:31.344+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:32.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:32.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:32 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:32.349+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:33.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:33.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:33 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:33.374+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:34.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:34.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:34 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:34.335+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:35.742 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-30T04:17:11.686308+0000 front 2024-03-30T04:17:11.686170+0000 (oldest deadline 2024-03-30T04:17:36.386075+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6804 osd.1 since back 2024-03-30T04:17:22.287380+0000 front 2024-03-30T04:17:22.287398+0000 (oldest deadline 2024-03-30T04:17:47.587303+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-30T04:17:28.588910+0000 front 2024-03-30T04:17:28.588945+0000 (oldest deadline 2024-03-30T04:17:51.489116+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6812 osd.4 since back 2024-03-30T04:17:41.991250+0000 front 2024-03-30T04:17:41.991130+0000 (oldest deadline 2024-03-30T04:18:06.091294+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-30T04:17:51.392387+0000 front 2024-03-30T04:17:51.392453+0000 (oldest deadline 2024-03-30T04:18:15.492535+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-30T04:17:55.493072+0000 front 2024-03-30T04:17:55.493145+0000 (oldest deadline 2024-03-30T04:18:21.393045+0000) 2024-03-30T04:18:35.743 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:35 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11[110005]: 2024-03-30T04:18:35.343+0000 7f2bc7250700 -1 osd.11 111 heartbeat_check: no reply from 172.21.15.154:6820 osd.7 since back 2024-03-30T04:18:03.094078+0000 front 2024-03-30T04:18:03.094132+0000 (oldest deadline 2024-03-30T04:18:27.794081+0000) 2024-03-30T04:18:36.492 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 30 04:18:36 smithi193 bash[124899]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-osd-11 2024-03-30T04:18:37.394 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@osd.11.service' 2024-03-30T04:18:37.434 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:37.434 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-30T04:18:37.435 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-30T04:18:37.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.a 2024-03-30T04:18:37.772 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:18:37 smithi023 systemd[1]: Stopping Ceph mds.a for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:38.271 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:18:37 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: 2024-03-30T04:18:37.804+0000 7f368b600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:38.272 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:18:37 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a[88674]: 2024-03-30T04:18:37.804+0000 7f368b600700 -1 mds.a *** got signal Terminated *** 2024-03-30T04:18:39.760 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 30 04:18:39 smithi023 bash[128868]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-a 2024-03-30T04:18:39.788 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.a.service' 2024-03-30T04:18:39.872 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:39.872 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-30T04:18:39.872 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-30T04:18:39.872 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.d 2024-03-30T04:18:40.219 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 04:18:39 smithi023 systemd[1]: Stopping Ceph mds.d for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:40.521 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 04:18:40 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-d[87720]: 2024-03-30T04:18:40.220+0000 7f500a400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:40.522 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 04:18:40 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-d[87720]: 2024-03-30T04:18:40.220+0000 7f500a400700 -1 mds.d *** got signal Terminated *** 2024-03-30T04:18:42.271 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 30 04:18:42 smithi023 bash[129006]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-d 2024-03-30T04:18:42.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.d.service' 2024-03-30T04:18:42.455 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:42.455 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-30T04:18:42.455 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-30T04:18:42.456 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.g 2024-03-30T04:18:42.771 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:18:42 smithi023 systemd[1]: Stopping Ceph mds.g for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:43.271 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:18:42 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: 2024-03-30T04:18:42.803+0000 7fa2ee600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:43.271 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:18:42 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g[88288]: 2024-03-30T04:18:42.803+0000 7fa2ee600700 -1 mds.g *** got signal Terminated *** 2024-03-30T04:18:47.271 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 30 04:18:46 smithi023 bash[129144]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-g 2024-03-30T04:18:47.388 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.g.service' 2024-03-30T04:18:47.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:47.468 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-30T04:18:47.469 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-30T04:18:47.469 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.j 2024-03-30T04:18:47.771 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 04:18:47 smithi023 systemd[1]: Stopping Ceph mds.j for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:48.271 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 04:18:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-j[87333]: 2024-03-30T04:18:47.852+0000 7f51c4800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:48.271 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 04:18:47 smithi023 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-j[87333]: 2024-03-30T04:18:47.852+0000 7f51c4800700 -1 mds.j *** got signal Terminated *** 2024-03-30T04:18:51.771 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 30 04:18:51 smithi023 bash[129284]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-j 2024-03-30T04:18:51.971 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.j.service' 2024-03-30T04:18:52.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:52.052 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-30T04:18:52.052 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-30T04:18:52.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.b 2024-03-30T04:18:52.431 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:18:52 smithi154 systemd[1]: Stopping Ceph mds.b for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:52.431 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:18:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:18:52.312+0000 7f66bca58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:52.431 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:18:52 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b[116265]: 2024-03-30T04:18:52.312+0000 7f66bca58700 -1 mds.b *** got signal Terminated *** 2024-03-30T04:18:55.681 INFO:journalctl@ceph.mds.b.smithi154.stdout:Mar 30 04:18:55 smithi154 bash[123901]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-b 2024-03-30T04:18:55.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.b.service' 2024-03-30T04:18:55.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:55.739 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-30T04:18:55.740 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-30T04:18:55.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.e 2024-03-30T04:18:56.019 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:18:55 smithi154 systemd[1]: Stopping Ceph mds.e for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:18:56.430 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:18:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: 2024-03-30T04:18:56.020+0000 7f26f370b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:18:56.431 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:18:56 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e[115579]: 2024-03-30T04:18:56.020+0000 7f26f370b700 -1 mds.e *** got signal Terminated *** 2024-03-30T04:18:59.670 INFO:journalctl@ceph.mds.e.smithi154.stdout:Mar 30 04:18:59 smithi154 bash[124038]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-e 2024-03-30T04:18:59.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.e.service' 2024-03-30T04:18:59.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:18:59.712 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-30T04:18:59.713 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-30T04:18:59.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.h 2024-03-30T04:18:59.931 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:18:59 smithi154 systemd[1]: Stopping Ceph mds.h for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:00.431 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:18:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: 2024-03-30T04:18:59.978+0000 7f876729b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:00.431 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:18:59 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h[115920]: 2024-03-30T04:18:59.978+0000 7f876729b700 -1 mds.h *** got signal Terminated *** 2024-03-30T04:19:03.681 INFO:journalctl@ceph.mds.h.smithi154.stdout:Mar 30 04:19:03 smithi154 bash[124179]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-h 2024-03-30T04:19:03.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.h.service' 2024-03-30T04:19:03.762 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:03.762 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-30T04:19:03.762 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-30T04:19:03.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.k 2024-03-30T04:19:04.017 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:19:03 smithi154 systemd[1]: Stopping Ceph mds.k for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:04.430 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:19:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: 2024-03-30T04:19:04.017+0000 7f1b83a1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:04.431 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:19:04 smithi154 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k[115237]: 2024-03-30T04:19:04.017+0000 7f1b83a1e700 -1 mds.k *** got signal Terminated *** 2024-03-30T04:19:07.680 INFO:journalctl@ceph.mds.k.smithi154.stdout:Mar 30 04:19:07 smithi154 bash[124317]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-k 2024-03-30T04:19:07.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.k.service' 2024-03-30T04:19:07.740 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:07.740 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-30T04:19:07.740 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-30T04:19:07.740 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.c 2024-03-30T04:19:08.242 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:19:07 smithi193 systemd[1]: Stopping Ceph mds.c for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:08.243 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:19:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: 2024-03-30T04:19:07.976+0000 7fdda645e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:08.243 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:19:07 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c[116238]: 2024-03-30T04:19:07.976+0000 7fdda645e700 -1 mds.c *** got signal Terminated *** 2024-03-30T04:19:11.656 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 30 04:19:11 smithi193 bash[125165]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-c 2024-03-30T04:19:11.662 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.c.service' 2024-03-30T04:19:11.699 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:11.700 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-30T04:19:11.700 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-30T04:19:11.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.f 2024-03-30T04:19:11.958 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:19:11 smithi193 systemd[1]: Stopping Ceph mds.f for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:12.242 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:19:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: 2024-03-30T04:19:11.958+0000 7fe72ecc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:12.243 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:19:11 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f[115896]: 2024-03-30T04:19:11.958+0000 7fe72ecc4700 -1 mds.f *** got signal Terminated *** 2024-03-30T04:19:15.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 30 04:19:15 smithi193 bash[125305]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-f 2024-03-30T04:19:15.608 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.f.service' 2024-03-30T04:19:15.643 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:15.643 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-30T04:19:15.644 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-30T04:19:15.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.i 2024-03-30T04:19:15.898 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:19:15 smithi193 systemd[1]: Stopping Ceph mds.i for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:16.242 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:19:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: 2024-03-30T04:19:15.899+0000 7f46dbb0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:16.243 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:19:15 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i[116579]: 2024-03-30T04:19:15.899+0000 7f46dbb0e700 -1 mds.i *** got signal Terminated *** 2024-03-30T04:19:19.683 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 30 04:19:19 smithi193 bash[125445]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-i 2024-03-30T04:19:19.688 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.i.service' 2024-03-30T04:19:19.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:19.723 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-30T04:19:19.723 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-30T04:19:19.723 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.l 2024-03-30T04:19:19.989 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:19:19 smithi193 systemd[1]: Stopping Ceph mds.l for eac7e298-ee48-11ee-b647-cb9ed24678a4... 2024-03-30T04:19:20.242 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:19:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: 2024-03-30T04:19:19.991+0000 7fcf34a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-30T04:19:20.242 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:19:19 smithi193 ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l[115554]: 2024-03-30T04:19:19.991+0000 7fcf34a00700 -1 mds.l *** got signal Terminated *** 2024-03-30T04:19:23.686 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 30 04:19:23 smithi193 bash[125586]: ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4-mds-l 2024-03-30T04:19:23.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac7e298-ee48-11ee-b647-cb9ed24678a4@mds.l.service' 2024-03-30T04:19:23.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T04:19:23.727 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-30T04:19:23.727 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T04:19:23.992 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:19:27.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T04:19:27.208 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:19:28.681 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force --keep-logs 2024-03-30T04:19:28.864 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:19:30.244 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:30.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:30.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-30T04:19:30.391 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-30T04:19:30.393 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi023/crash 2024-03-30T04:19:30.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T04:19:30.478 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T04:19:30.479 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-30T04:19:30.480 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi154/crash 2024-03-30T04:19:30.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T04:19:30.537 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T04:19:30.557 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-03-30T04:19:30.560 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi193/crash 2024-03-30T04:19:30.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash -- . 2024-03-30T04:19:30.610 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-03-30T04:19:30.611 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-03-30T04:19:30.612 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-30T04:19:30.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-30T04:19:30.696 INFO:teuthology.orchestra.run.smithi023.stderr:grep: /var/log/ceph/eac7e298-ee48-11ee-b647-cb9ed24678a4/ceph.log: No such file or directory 2024-03-30T04:19:30.701 INFO:tasks.cephadm:Compressing logs... 2024-03-30T04:19:30.702 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:19:30.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:19:30.709 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:19:30.736 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T04:19:30.767 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T04:19:30.786 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-30T04:27:00.524 INFO:tasks.cephadm:Archiving logs... 2024-03-30T04:27:00.525 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi023/log 2024-03-30T04:27:00.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T04:27:16.069 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi154/log 2024-03-30T04:27:16.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T04:28:09.801 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi193/log 2024-03-30T04:28:09.803 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T04:28:27.416 INFO:tasks.cephadm:Removing cluster... 2024-03-30T04:28:27.417 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force 2024-03-30T04:28:27.684 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:28:28.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force 2024-03-30T04:28:28.891 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:28:29.483 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac7e298-ee48-11ee-b647-cb9ed24678a4 --force 2024-03-30T04:28:29.666 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: eac7e298-ee48-11ee-b647-cb9ed24678a4 2024-03-30T04:28:30.045 INFO:tasks.cephadm:Removing cephadm ... 2024-03-30T04:28:30.046 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T04:28:30.093 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T04:28:30.127 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-30T04:28:30.145 INFO:tasks.cephadm:Teardown complete 2024-03-30T04:28:30.146 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T04:28:30.158 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T04:28:30.158 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T04:28:30.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T04:28:30.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T04:28:30.305 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T04:28:30.317 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T04:28:30.317 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T04:28:30.322 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T04:28:30.324 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T04:28:30.339 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2024-03-30T04:28:30.340 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1406us[+1406us] +/- 90ms 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1741us[-1723us] +/- 55ms 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1387us[+1387us] +/- 88ms 2024-03-30T04:28:30.344 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:28:30.358 INFO:teuthology.orchestra.run.smithi023.stderr:bash: ntpq: command not found 2024-03-30T04:28:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T04:28:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-30T04:28:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +1419us[+1443us] +/- 90ms 2024-03-30T04:28:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -1712us[-1687us] +/- 55ms 2024-03-30T04:28:30.366 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 +1430us[+1454us] +/- 88ms 2024-03-30T04:28:30.367 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:28:30.398 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-30T04:28:30.398 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-03-30T04:28:30.398 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +1402us[+1381us] +/- 90ms 2024-03-30T04:28:30.399 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -1834us[-1855us] +/- 55ms 2024-03-30T04:28:30.399 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +1418us[+1418us] +/- 88ms 2024-03-30T04:28:30.399 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-30T04:28:30.399 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T04:28:30.410 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T04:28:30.411 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T04:28:30.422 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T04:28:30.565 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T04:28:30.742 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-30T04:28:30.812 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T04:28:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:30.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T04:28:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:30.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-30T04:28:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:30.947 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T04:28:30.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T04:28:30.968 INFO:teuthology.task.internal:Duration was 3107.701412 seconds 2024-03-30T04:28:30.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T04:28:30.979 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T04:28:30.980 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T04:28:30.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T04:28:30.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T04:28:31.034 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T04:28:31.068 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T04:28:31.093 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-30T04:28:31.307 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T04:28:31.307 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-30T04:28:31.308 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T04:28:31.363 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-03-30T04:28:31.363 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T04:28:31.403 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-30T04:28:31.403 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T04:28:31.425 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T04:28:31.425 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:28:31.429 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:28:31.445 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T04:28:31.561 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T04:28:31.561 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T04:28:31.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T04:28:31.567 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T04:28:31.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T04:28:31.783 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T04:28:31.783 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T04:28:31.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T04:28:31.908 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T04:28:31.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T04:28:31.948 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:31.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:31.954 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:31.989 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-03-30T04:28:32.002 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-30T04:28:32.013 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-30T04:28:32.048 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:32.088 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:32.104 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T04:28:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T04:28:32.121 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T04:28:32.133 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T04:28:32.135 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi023 2024-03-30T04:28:32.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T04:28:32.217 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi154 2024-03-30T04:28:32.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T04:28:32.258 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435/remote/smithi193 2024-03-30T04:28:32.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T04:28:32.301 INFO:teuthology.task.internal:Removing archive directory... 2024-03-30T04:28:32.301 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T04:28:32.306 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T04:28:32.308 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-30T04:28:32.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T04:28:32.364 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T04:28:32.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T04:28:32.377 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T04:28:32.378 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T04:28:32.388 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T04:28:32.390 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T04:28:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 04:28 /home/ubuntu/cephtest 2024-03-30T04:28:32.406 INFO:teuthology.orchestra.run.smithi193.stdout: 262157 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 04:28 /home/ubuntu/cephtest 2024-03-30T04:28:32.425 INFO:teuthology.orchestra.run.smithi023.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 04:28 /home/ubuntu/cephtest 2024-03-30T04:28:32.428 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T04:28:32.440 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T04:28:32.465 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T04:28:32.491 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-30T04:28:32.492 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T04:28:32.514 DEBUG:teuthology.nuke:shortname: smithi154 2024-03-30T04:28:32.514 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T04:28:32.519 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.845700', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLp9uxysJxnBuYZiO+DnHycmfUebGiDiHI5DiVTfPUaHsatE07i0IWHz4qRqrXOR9S1dbVVWs5V2a9AMGb6ZZwM='} 2024-03-30T04:28:32.537 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-30T04:28:32.538 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T04:28:32.541 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.846856', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChxT9Qy5LIUldjLYonVCLcq3DDsp8y4977m4nZE51QPNBRejvuoHOOw61rNqkGQUZAnDpZ+ruC3pQ9o0+Wb7eg='} 2024-03-30T04:28:32.561 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-29_18:05:24-fs-wip-rishabh-testing-20240327.051042-reef-testing-default-smithi/7629435', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 03:24:00.844227', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKF3oidbcYiEISL3SZW28ovg1Xn7gOPBmKsLkOW9aGGueB2vGeKAal9lIMX3o+rKbbDxDit/fb3xYx0fe8jFfSw='} 2024-03-30T04:28:32.574 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-30T04:28:32.575 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T04:28:32.600 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-30T04:28:32.600 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:32.616 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-30T04:28:32.616 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T04:28:32.630 INFO:teuthology.orchestra.console.smithi154:Power off 2024-03-30T04:28:32.630 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T04:28:32.639 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-30T04:28:32.640 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:32.648 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-30T04:28:32.653 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-03-30T04:28:32.653 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:32.664 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T04:28:32.677 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-30T04:28:36.649 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:36.665 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:36.678 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:36.760 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-30T04:28:36.774 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T04:28:36.786 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-30T04:28:40.765 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:40.776 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:40.787 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:40.876 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-30T04:28:40.885 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-30T04:28:40.897 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-03-30T04:28:44.878 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:44.887 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:44.898 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T04:28:44.989 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-30T04:28:44.997 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-30T04:28:45.008 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-03-30T04:28:45.091 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-30T04:28:45.098 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-30T04:28:45.109 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-03-30T04:28:45.216 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} duration: 3107.7014124393463 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi023 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=470100a4ccbcf7769052bf09a5413881ccf1b97f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=99130478e84d41748b55014704480009 status: fail success: false 2024-03-30T04:28:45.217 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T04:28:45.299 INFO:teuthology.run:FAIL